1 Introduction↩︎

Human-in-the-loop Reinforcement Learning (HITL-RL) is a growing approach for training and finetuning autonomous agents [1]. This practice is a subdomain that sits within RL, a branch of machine learning in which agents learn to take actions by maximizing a reward function, conditioned on the state of their environment [2]. Existing HITL-RL techniques include learning from demonstration [[3]][4], preference learning [5], and intervention learning [6].

Many of these approaches share the common trait of requiring explicit feedback from human users, generally through speech, physical demonstration, or interface-based preference expression. This approach can introduce a few problems. For example, sustained attention and providing expert feedback may lead to cognitive overload for the human teacher [7]. Second, these systems may introduce bias by assuming that a user’s implicit reactions follow a general distribution of the training set. They assumme users move, speak, or interact with an agent in a similar manner to others [8]. These methods can exclude people with motor or speech impairments from finetuning agent behavior.

Implicit HITL-RL relaxes the first problem by utilizing passive feedback from users to convey knowledge to the agent. Unlike explicit feedback, where users attempt to shape the policy through direct instruction, implicit systems utilize indirect human reactions to guide agent behavior through gesture, facial expression, or voice inflection [9] [10]. However, these methods leave the second problem unresolved, as they generally require users to be able to move and speak.

Our work targets this issue directly by using brain signals as the main feedback channel directing the agent. Sometimes referred to as intrinsic feedback, this feedback signal is implied from human biosignals. Emerging work in this domain has introduced encephalography-based (EEG) feedback as a binary signal for training agents via RL [11]. Instead, we use functional near-infrared spectroscopy (fNIRS), which captures a slower hemodynamic signal with greater spatial resolution, and can be more robust to certain artifacts that plague EEG [12]. We propose using fNIRS and EEG in tandem, as opposed to in replacement, as both signals offer complementary features that can be useful for future multimodal, hybrid systems.

We use fNIRS data from a prerecorded data, and use an offline approach to guide a simulated robot’s policy via reinforcement learning. We map fNIRS feature vectors to a label describing agent optimality, and use this neural classification to guide the robot’s training loop, examining effects on learning.

Our key contributions are:

• We propose, develop, and test a framework for integrating offline neural data into an RL loop, examining multiple methods of algorithm enhancement.

• We demonstrate this framework using purely offline fNIRS data and show an improvement in success rate and total episode returns, indicating that the signal is useful without the hardware and calibration overhead of real-time BCI systems, making this approach more deployable in settings where a real-time pipeline is impractical.

• We conduct an ablation study isolating the individual and combined contributions of each augmentation mechanism, finding that augmenting the transition’s Q-value and Priority perform best, but augmenting Reward has no effect.

We further examine the effect of classifier noise on learning outcomes with a small ablation study. This study allows us to characterize how much noise the framework can tolerate before its benefits become nonsignificant.

2 Related Work↩︎

HITL-RL is a growing approach using human feedback to guide agent training [1]. Our approach leverages passive fNIRS signals as feedback, letting users interact with an agent without speaking or moving.
Human-in-the-Loop RL: Early HITL-RL approaches used binary evaluative feedback to guide learning. [13] introduced a framework letting non-expert users guide agent training via evaluative feedback (e.g., good, bad). [14] extends these ideas to deep RL, integrating feedback into various parts of the algorithm to shape policy. We adopt some of these ideas, injecting the neural signal into the RL loop by augmenting the reward, Q-value, and priority of an MDP tuple at a given timestep.
Implicit Human-in-the-Loop RL: Implicit HITL-RL shares this framework, but feedback takes the form of subliminal expressions not directed at the agent. [9] used facial expressions and gestures to update an agent’s policy in real-time, decoding expressions into learning task statistics before applying them to the RL loop. [10] similarly uses decoded emotional states ("happy," "sad") as positive or negative reward.
EEG for Intrinsic HITL-RL: Recent work has explored implicit brain signals, particularly EEG-measured error-related potentials (ErrPs), as HITL-RL feedback [15]–[18]. ErrPs are binary signals detected via EEG when a person perceives an error, evaluated in game domains like Maze and MuJoCo’s Ant and Cheetah [15], [17], and in simulated robot domains [16], [18], [19]. Most approaches use this binary signal to augment or replace a sparse reward function. We extend this approach through using the fNIRS hemodynamic response as our neural signal, and augmenting Q-value and trajectory prioritization on top of reward, learning from a limited, offline dataset.

3 Theoretical Framework↩︎

3.1 The Reinforcement Learning Problem↩︎

A reinforcement learning agent learns a task in OpenAI Gymnasium’s Fetch Pick-and-Place domain (v4), a robotic manipulation environment structured as a Markov Decision Process (MDP), described by the tuple \((\mathcal{S}, \mathcal{A}, \mathcal{P}, \mathcal{R}, \gamma)\). \(\mathcal{S}\) is the set of observable states, \(\mathcal{A}\) the set of actions, and \(\mathcal{P}\) is the transition function that determines the next state given the state and action at time \(t\): \(P(s_{t+1}|s_t,a_t)\). \(R\) denotes the reward function, and \(\gamma\), the discount factor, modulates the weight of future rewards.

3.2 The Performance Model↩︎

Previous work has mapped human brain signals to various granularities of agent performance [20], [21]. We adopt this approach to test whether neural data decoded into a value that reflects action optimality can improve a robot’s policy by modulating the RL algorithm. We denote the Multilayer Perceptron (MLP) performance model as \(\mathcal{M}_g(f_t) = \hat{y}_t\), where \(f_t\) is the neural feature vector at time \(t\), \(g\) the granularity of the optimality label, and \(\hat{y}_t\) the predicted class. We denote the confidence of the binary or ternary classifier as \(c_{t} = P(y_t = \hat{y}_t \mid f_t)\): the probability that predicted class \(\hat{y}_t\) matches true class \(y_t\), given \(f_t\). For regressors mapping neural features to a continuous label representing action-selection error, we denote the parallel quantity as \(c_{t} = 1-\hat{e}_t\), the estimated probability that the agent chose a near-optimal action given \(f_t\).

4 Methods↩︎

Our framework, NEURO-LOOP, is comprised of two main components: (1) the neural classification problem and (2) the RL loop. [21] addresses the neural classification problem, presenting three performance model variants mapping passive fNIRS feature vectors to agent performance labels. This section covers the second component which uses the performance model’s output to guide the RL algorithm.
Goal: The robot’s goal is to find, pick up, and place a black cube in a randomized location on the table in front of it.
DDPG-HER with Prioritization: We train the robot with Deep Deterministic Policy Gradient with Hindsight Experience Replay (DDPG-HER) [22]. We further implement [23]-like prioritization, where batched experiences are given priority instead of being uniformly sampled.
Finetuning Checkpoints: We inject the prerecorded MDP tuples at success rate checkpoints determined at evaluation time. The agent learns from scratch, and once it passes this checkpoint, it learns from the dataset until exhausted, then resumes online learning until the total episode count is reached. Our goal is to determine whether this framework is better suited for early, mid or late-stage training.
Interaction Tasks: We record and utilize passive fNIRS data from a participant’s prefrontal cortex during one of two interaction tasks.
Passive: The participant only observes the agent. State-action pairs are chosen by a near-optimal policy that is forced to execute a mix of optimal and suboptimal actions (roughly 50% each), so every MDP tuple carries a known performance label.
Active: The participant teleoperates the robot, so the state-action pairs aligned with fNIRS data are those the participant selected via controller. Since participants were not expert demonstrators, active MDP tuples are commonly suboptimal. Pooled experiments combine active and passive data.
Model Granularity: Training a performance model requires choosing the granularity of the performance label, \(g \in \{b, d, e\}\). Binary models predict Optimal or Not Optimal, \(g_b \in \{0, 1\}\). Ternary models predict Optimal, Suboptimal, or Not Optimal, \(g_d \in \{0, 1, 2\}\). Continuous models are regressors that predict the error between the agent’s chosen action and actions chosen by near-optimal policies, \(g_e \in \mathbb{R}\).
Tuning the Performance Model Output: We train a performance model for every combination of interaction task and model granularity, using all available participant data in a multi-subject training approach. We then tune the neural classification to align with the signal it augments. The regressor output is recalibrated as: \(n_t = (x_t + c_{t} \cdot \mu_X[0])\cdot \beta\) and the classifier output: \(n_t = (x_t + c_{t} \cdot \mu_X[\hat{y_t}])\cdot \beta\), where \(\hat{y}\) is the raw model output, \(x_t\) the parameter to be augmented (Reward, Priority, Q-value), and \(\mu_X\) is an array that describes the expected value of some parameter for each optimality class. From here, \(n_t\) is what is applied to the RL loop.
Augmentation Techniques: Conditions build on DDPG-HER with Prioritization, differing only in how \(n_t\) modulates RL.
Baseline: Data is injected at the finetuning checkpoint, but the neural augmentation step is withheld.
Prioritization: \(n_t\) modulates trajectory priority. TD-error \(\delta\) is the fallback priority for all non-Prioritization conditions.
Reward Augmentation: \(n_t\) modulates the domain’s reward.
Q-Augmentation: \(n_t\) modulates the state-action Q-value.
All: Combines Prioritization, Reward, and Q-Augmentation.
Model Noise: We inject synthetic noise into the performance model’s output to test its effect on learning. With probability \(\tau\), classifier outputs are flipped, or regressor outputs are shifted by a random value between 0 and 1.

5 Experiments and Results↩︎

Dataset: We evaluate NEURO-LOOP on a publicly available dataset that pairs fNIRS recordings with the behavior of an autonomous agent. We focus on the Robot domain, where participants either observe or teleoperate a robot performing a fetch-and-place task. For each timestep, the dataset provides an MDP tuple consisting of the agent’s state, action, and reward, along with the participant’s brain signal at that moment. Each MDP tuple is additionally labeled with three measures of how optimal the robot’s selected action was.

Participants: We analyzed brain data from 21 participants (14 female, 11 male), ages 19–27. Each participant completed one or both of two possible interaction tasks with at least 13 participants per condition.
Metrics: We tracked the agent’s total episode return and success rate at each evaluation cycle. Evaluations were run every 10,000 training steps at 25 episodes per evaluation.

5.1 Augmentation Methods↩︎

Q-Augmentation and All consistently outperformed Baseline and all other conditions (Fig. 2). We believe this is because Q-Augmentation most directly influences the policy.
Prioritization ranked third, showing a modest but consistent improvement over Baseline across most interaction tasks. This may be because Baseline is already prioritizing trajectories with TD-error, so Prioritization is competing with an already well-established ranking scheme.
Reward Augmentation showed no significant improvement, performing similarly to Baseline. These results are consistent with that of [18] finding that applying neural feedback to the agent’s reward pipeline performed similarly to agents learning from a dense reward function. All reported differences use 95% confidence intervals, shown as error bars.

5.2 Model Granularity↩︎

Passive Experiments: Continuous outperformed binary and ternary in Q-augmentation; binary and continuous performed similarly elsewhere. Ternary underperformed overall but still outpaced Baseline in all conditions except Reward Augmentation.
Active Experiments: Binary and ternary outperformed continuous, which only marginally outpaced Baseline in All.
Pooled Experiments: Binary performed best while continuous and ternary struggled. However, ternary All and continuous Prioritization and Q-Augmentation still modestly outpaced Baseline. We theorize ternary underperformed due to noisier output from the performance model: F1 scores were  0.47–0.50 for three-class models versus 0.67–0.72 for binary. Consistent with this analysis, setting model noise to 0.5 made Passive and Pooled binary performance converge toward ternary.

Continuous granularity performed worst in Active, possibly because demonstration data accumulates error, making an error-based signal harder to parse and learn from. Ternary, despite generally underperforming, performed similarly to binary in Active. Across granularities, Prioritization surprisingly showed little variance in performance. This could possibly be because, as a sort of ranking system, separating trajectories into "good" or "bad" is sufficient, with magnitude adding little more. It would be interesting to test whether this pattern changes with alterations to the algorithm’s memory buffer.

5.3 Interaction Task↩︎

As expected, Passive interaction tasks led to the fastest learning. This is likely because Passive tasks maintain a roughly 50/50 split between optimal and suboptimal trajectories, so it saw greater optimal performance earlier on. Active tasks are noisier since participants directly teleoperate the robot, meaning many trajectories are suboptimal, and many fail to reach the goal at all. Pooled data, which combines both interaction tasks, trains faster than Active alone but slower than Passive alone. This pattern is consistent with Pooled experiments using a mixture of the two.

5.4 Ablation↩︎

Finetune checkpoint ablations show that early injection yields the best performance across all conditions. A threshold of 0.2 still produces a measurable benefit, but beyond that point, the neural signal no longer improves learning.

We ran a brief ablation to test whether performance gains stem from the neural signal itself, rather than effects of recalibrating the performance model’s output, or some other artifact. In conditions with significant gains (e.g., Q-Aug), adding more noise to the performance model further degraded performance. For conditions without significant gains (e.g., Reward Aug), added noise had little to no effect (Fig. 3). This pattern supports the interpretation that performance improvements are driven by the neural signal instead of by artifacts in the framework.

6 Conclusion and Future Work↩︎

In this work, we show that fNIRS signals can be used to update a robot learning algorithm and improve training efficiency. We introduce NEURO-LOOP, a framework for evaluating different augmentation methods, comparing model granularity, and testing robustness to noise and finetuning checkpoints. Our results demonstrate that this pipeline is effective using offline data, offering a practical alternative for settings where real-time BCI setups are impractical. We believe that NEURO-LOOP may provide a useful starting point for future work extending this approach to real-time, physically embodied human-robot interaction settings.

There are several limitations to this research that we believe may inspire future work. First, agent performance labels are a function of the agent’s optimality, and they are not guaranteed to reflect the participant’s true judgment. We assume that the participant’s judgment of the agent aligns with that of a near-optimal policy, however we believe that exploring labeling schemes to capture user preference more directly will benefit these fNIRS-based systems.

Second, our models avoid the credit assignment problem by using 4-second neural feature windows. Future work should explore a more standardized approach to handling the temporal delay between the hemodynamic response captured by fNIRS and the interaction with the agent at that time.

Finally, further work should compare this method against other human-in-the-loop (HITL) approaches. Understanding how policy learning differs across HITL paradigms may clarify what the neural signal is actually contributing beyond what other feedback channels already provide. We also believe that extending these methods to an online setting and to a physical robot are natural next steps.

Acknowledgments↩︎

We thank our colleagues at Tufts University, the MuLIP Lab, and the HCI lab for their feedback and support at each step of this project.

Code & Datasets↩︎

Code: https://github.com/your-profile/OfflineNeuroloop/tree/iros2026
Dataset: https://github.com/your-profile/fNIRS2RL

References↩︎