1 Introduction↩︎

Vision-language-action (VLA) models have recently shown strong capability across diverse robotic tasks, including single-arm manipulation [1]–[12], bimanual manipulation [13]–[19] and dexterous manipulation [20]–[23]. Among them, diffusion-based VLAs [12], [14], [17] become popular due to their strong performance. However, when deployed in the real world, these pretrained models often encounter distribution shifts in scenes, objects, viewpoints, and contact dynamics. They can significantly degrade performance, especially in long-horizon, contact-rich, or high-precision manipulation tasks. While reinforcement learning (RL) provides a natural way to adapt VLA policies through task-specific interactions [24]–[35], on-robot RL remains costly: collecting real-world experience is expensive, failures consume substantial resources, and adaptation is constrained by tight wall-clock time budgets. This motivates the need for efficient VLA adaptation methods under limited on-robot interaction and wall-clock time budget.

One efficient way to avoid the high computational costs of adapting diffusion-based generative policies is noise-space finetuning [25], [33], [34], [36]. Specifically, we freeze the pretrained diffusion-based VLA as a denoising generator and steer a small noise actor to generate the noise. This confines adaptation to a small learnable module, reducing optimization cost while keeping the pretrained VLA frozen. As a result, the decoded actions naturally remain anchored to the pretrained policy prior, thus benefit from the priors in pretrained VLAs. However, since the VLA can degenerate under out-of-distribution (OOD) deployment conditions, policy improvement depends heavily on autonomous exploration, which can be inefficient under sparse rewards or large deployment shifts.

A natural way to mitigate this exploration bottleneck is to incorporate human guidance. While existing approaches for demonstrations [37], corrective interventions [38]–[40], and human-in-the-loop learning [24], [34], [41], [42] provide effective ways to refine robot behavior, they typically operate within the action space. Such action-space updates are often heavier and less stable compared with noise-space finetuning, since they require optimizing the action-generation pathway of a large generative policy. However, incorporating human corrective actions into noise-space is non-trivial. The mapping from noise to action is induced by a complex, learned transport process. Consequently, recovering a noise variable from a human-corrected action requires an implicit reverse Euler update whose velocity depends on the unknown preimage[43], [44]. Alternatively, backpropagating action-level supervision through the entire decoding chain is another option but it is computationally prohibitive and prone to unstable optimization signals.

To address this challenge, we propose UniSteer, a Unified Noise Steering framework that integrates human corrective guidance into noise-space online RL through approximate action-to-noise inversion. Although this inversion is not solved directly in closed form, we show that a tractable approximation yields low inversion error and provides reliable supervision in practice. Building on this inversion, we introduce a noise-space finetuning framework that unifies RL and human guidance through a shared noise-space interface, allowing the use of human corrections to supervise the noise actor that optimized via RL in the meanwhile. We validate this framework through extensive real-world robotic adaptation experiments, showing that it outperforms strong baselines with reduced time-to-performance and fewer human interventions. Our contributions are threefold:

1. We introduce UniSteer, the first unified noise-space steering framework that incorporates human guidance into noise-space online RL for efficient real-world adaptation.

2. We propose an approximate action-to-noise inversion method for flow-matching policies, enabling consistent noise supervision under a frozen generator.

3. We demonstrate that our method achieves reduced time-to-performance with fewer human interventions than strong baselines across diverse real-world robotic manipulation tasks.

2 Related work↩︎

2.1 Vision-Language-Action Models↩︎

Vision-Language-Action (VLA) models have become a central paradigm for robot learning [1]–[19]. Early methods formulate action prediction as autoregressive token generation to integrate robot control into language-model-style sequence modeling [1]–[4], [10], [13], [18]. Other methods attach continuous action heads or diffusion-style decoders to vision-language models for better representation of high-dimensional continuous control [5], [17], [45], [46]. More recently, many VLA policies adopt flow-matching action heads, showing strong generative capability and promising performance in real-world robotic manipulation [8], [12], [14]–[16], [19]. These policies generate action chunks by transporting initial noise variables to continuous actions through a learned state-conditioned velocity field. Our work focuses on this class of flow-matching VLA policies due to their competitive performance and leverages the initial noise variable as a natural interface for policy steering.

2.2 Adaptation of Flow-Matching VLA Policies↩︎

Reinforcement learning provides a natural approach for downstream VLA adaptation, but directly applying it to flow-matching policies is challenging. These policies generate actions through multi-step decoding, making end-to-end optimization costly and unstable. Moreover, their deterministic generation process makes the action log probabilities or likelihood ratios required by many RL algorithms difficult to obtain. Recent methods address this challenge by reformulating policy-gradient objectives for flow policies, such as RLinf [30] and FPO [47]. Other works avoid difficult online RL updates in the real world and instead improve policies through demonstrations, human interventions, or value-weighted behavior cloning, including HG-DAgger [38], AWR-style methods [48]–[50], and RECAP-style value-weighted imitation [26]. Another line of work keeps the pretrained generative policy frozen and adapts only a lightweight module, such as a residual correction network[51], [52] or a noise-space policy [25], [33], [34], [36]. Compared with the above two lines of work, this paradigm offers a more lightweight and stable adaptation strategy by freezing the pretrained VLA and updating only a small module. Our method follows this lightweight adaptation paradigm, but differs from prior noise-space methods that mainly rely on autonomous exploration: we translate action-space human corrections into noise-space supervision targets, allowing human interventions to directly guide the compact noise actor while keeping the flow-matching VLA frozen.

3 Problem Formulation↩︎

3.1 Flow matching policies.↩︎

We consider VLA policies parameterized as conditional flow matching models. Given a state \(s\), the policy maps an initial noise variable \(z_0 \sim \mathcal{N}(0,I)\in \mathbb{R}^d\) to an action chunk \(a \in \mathbb{R}^d\) by integrating a state-conditioned velocity field \(v_\theta(z_t,t,s)\), where \(\theta\) is the parameter of the network: \[\frac{d z_t}{dt} = v_\theta(z_t,t,s).\] At each flow step \(k\), the network predicts a velocity update conditioned on the current noise, the flow time, and the state. Starting from \(z_0\) at \(t=0\) and integrating forward to \(t=1\) produces the terminal sample deterministically \[a = G_\theta(s, z_0) := z_1 = z_0 + \int_{0}^{1} v_\theta(z_t,t,s)\,dt,\] where \(a\) denotes the predicted action chunk. In practice, this continuous-time process is implemented using a K-step Euler discretization with step size \(\Delta t\).

3.2 Noise-space policy optimization.↩︎

We formulate robotic policy learning as a Markov Decision Process (MDP) \(\mathcal{M}=(\mathcal{S},\mathcal{A},P,r,\gamma)\), where \(\mathcal{S}\) is the state space, \(\mathcal{A}\subseteq\mathbb{R}^d\) is the continuous action space, \(P(s' \mid s,a)\) is the transition dynamics, \(r(s,a)\) is the reward function, and \(\gamma \in [0,1)\) is the discount factor. In our setting, the action is in fact the noise variable \(z\) because we freeze the decoding process \(G_\theta(s,z)\). Specially, a lightweight policy \(\psi_\phi(z \mid s)\) is trained to select the initial noise variable conditioned on the current state, and the frozen decoder maps it to an action chunk: \[z \sim \psi_\phi(\cdot \mid s), \qquad a = G_\theta(s,z).\] We further learn a noise-space critic \(Q_\omega(s,z)\), which estimates the expected discounted return of the trajectory. \(Q_\omega(s,z)\) is trained with the standard TD loss: \[\mathcal{L}_{Q_\omega} = \left\| Q_\omega(s,z) - \big(r + \gamma \bar Q_\omega(s',z')\big) \right\|^2,\]

The RL objective is therefore to optimize the policy with respect to this value function, while the pretrained flow decoder remains fixed. \[\mathcal{L}_{\mathrm{RL}} = - Q_\omega(s,z), \qquad z \sim \psi_\phi(s).\]

4 Method↩︎

4.1 Action-to-Noise Inversion↩︎

Given the current state \(s\) and a human-provided corrective action chunk \(a^h\), our goal is to recover a noise target \(\hat{z}\) such that, the frozen flow-matching policy can decode it into an action chunk close to \(a^h\). Ideally, the desired noise target is the inverse image \(G_\theta(s,\cdot)^{-1}(a^h)\). However, since this noise-to-action mapping is realized through a multi-step nonlinear flow decoder, directly inverting it is difficult. We therefore exploit its stepwise Euler structure and invert the decoder one step at a time along the frozen decoding trajectory.

Remark. This proposition establishes the controllability of the noise-space actor. Intuitively, the global Lipschitz continuity of the velocity field prevents constant mapping or squashed mapping (such as \(G_\theta(s, z_0) = \tanh(z_0)\)).

For the frozen flow matching policy, forward decoding proceeds through \(K\) iterative Euler updates. At flow step \(k\), the inverse of a single update can be written as a fixed-point equation: \[x = y - \Delta t\, v_\theta(x,t_k,s) = g_y(x).\] where \(x\) and \(y\) denote the input and output of that decoder step, respectively, and \(v_\theta\) is the frozen velocity model.

Consequently, the inversion of the Euler step exists uniquely and can be recovered by fixed-point iteration. Starting from the corrective action chunk as the terminal state, we recover the noise states recursively as \[\hat{z}_k = \mathrm{Inv}_k(\hat{z}_{k-1}, s), \qquad k=1,\dots,K, \qquad \hat{z}_0 = a^h\] \[\mathrm{Inv}_k(\hat{z}_{k-1}, s) \;:=\; \operatorname*{fix}\Big(z \mapsto \hat{z}_{k-1} - \Delta t\, v_\theta(z,t_k,s)\Big).\] \(\mathrm{Inv}_k\) is defined as the fixed point of the inverse Euler map. In practice, we approximate this fixed point with \(M\) steps of iteration: \[z_k^{(m+1)} = \hat{z}_{k-1} - \Delta t\, v_\theta\!\left(z_k^{(m)}, t_k, s\right), \quad m=0,\dots,M-1, \qquad z_k^{(0)}=\hat{z}_{k-1}\] After all \(K\) backward steps, we obtain the recovered initial noise \(\hat{z} := \hat{z}_K,\) which is used as the noise supervision target.

4.2 Unified Noise Steering Framework↩︎

With the inversion procedure, human corrective actions can be translated into noise-space supervision targets under the frozen flow matching policy. During online interaction, the noise actor \(\psi_\phi\) selects an initial noise variable conditioned on the current state, and the frozen flow matching policy decodes it into an action chunk: \[z_t \sim \psi_\phi(\cdot \mid s_t), \qquad a_t = G_\theta(s_t, z_t).\] During autonomous interaction, we execute the decoded action chunk \(a_t\) and store the resulting transition \((s_t, z_t, r_t, s_{t+1})\) in the RL buffer \(\mathcal{B}_{\mathrm{RL}}\). When a human takes over and provides a corrective action chunk \(a_t^h\), we apply the inversion procedure in Sec4.1 to recover the corresponding noise target \(\hat{z}_t^h\) and execute the corrective action. The corrected transition \((s_t, \hat{z}_t^h, r_t, s_{t+1})\) is stored in the RL buffer \(\mathcal{B}_{\mathrm{RL}}\) for value learning and in the demo buffer \(\mathcal{B}_{\mathrm{demo}}\) for correction supervision. This design allows human interventions to improve both critic learning and actor optimization under a unified noise-space representation.

For policy optimization, we first update the policy by sampling batches from demo buffer \(\mathcal{B}_{\mathrm{demo}}\) by \[\mathcal{L}_{\mathrm{demo}} = ||\psi_{\phi}(s) - \hat{z}_h||^2_2.\] Then we sample from \(\mathcal{B}_{\mathrm{RL}}\) and conduct optimization using \(\mathcal{L}_{\mathrm{RL}}\). This design allows RL and human guidance to update the same noise actor through a shared noise interface. The RL branch improves the policy from reward feedback, while the correction branch injects action-space human guidance after translating it into noise-space targets through inversion. Algorithm 5 summarizes the overall training procedure.

5 Experiment↩︎

5.1 Experimental Setup↩︎

5.1.0.1 Tasks and Platform

We evaluate our method on four real-world manipulation tasks: pick up spoon, stack blocks, insert square, and fold towel, as shown in Figure 3. These tasks cover a broad range of manipulation skills, including pick-and-place, precise contact-rich operations, and deformable-object manipulation. Together, they provide a compact benchmark for evaluating both broad spatial generalization and fine-grained manipulation refinement under distribution shift. All experiments are conducted on an AgileX Piper robot with a master–slave teleoperation setup for human takeover. For all tasks, the observations consist of two RGB images from a side camera and a wrist camera, together with the robot’s 6D end-effector pose and gripper state. The action space is defined as the end-effector target pose with gripper openness, and the robot is controlled at 30 Hz. We initialize the policy from a pre-trained \(\pi_0\) architecture checkpoint with a flow-matching action head and warm it up using 30 task demonstrations per task. All training runs are conducted on a single NVIDIA A100 GPU.

5.1.0.2 Baselines

To fairly evaluate the effectiveness of our method, we compare against two representative baselines: DSRL[25], DAgger[37]. DSRL serves as the primary noise-space RL baseline, as it performs policy improvement under the same frozen generative policy backbone but without human intervention, thereby isolating the benefit of incorporating corrective guidance. DAgger represents a human-in-the-loop imitation learning approach, where corrective actions are aggregated and used for supervised policy updates in action space. Together, these baselines cover the relevant comparison axes for our setting: noise-space RL without human assistance, action-space imitation learning with human corrections. We initially considered HIL-SERL[41] as an additional human-in-the-loop online RL baseline. However, consistent with observations in RLToken [52], we found that HIL-SERL is poorly matched to our high-frequency control setting with sparse terminal rewards. In preliminary experiments, HIL-SERL was ineffective and unstable, and therefore did not provide a reliable comparison for our setting.

5.1.0.3 Evaluation Metrics

We use real-world task success rate as the metric for evaluating each method’s performance. For each task, the final policy is evaluated over 20 real-world trials, and the success rate is computed as the percentage of successful trials. For Pick up Spoon, Stack Blocks, and Insert Square, we randomly select 10 distinct initial object positions and evaluate each position twice. Among these positions, 20% are out-of-distribution cases not covered by the initial demonstration data, allowing us to evaluate both in-distribution adaptation and spatial generalization to unseen object placements. For Fold Towel, since the task involves deformable-object manipulation and does not admit the same discrete position-based evaluation protocol, we directly run the policy for 20 consecutive trials and report the resulting success rate.

5.2 Main Results↩︎

2.8pt 6pt

Table ¿tbl:tab:main95results? reports the real-world adaptation performance of UniSteer and all baselines across four manipulation tasks. UniSteer consistently achieves the highest success rate on all tasks, improving the average success rate from 20% for the initial policy to 90% after adaptation. Compared with DSRL, which performs reinforcement learning only in the noise space without human guidance, UniSteer improves the average success rate by 35 percentage points. Compared with DAgger, which uses human corrections for supervised action-space imitation, UniSteer improves the average success rate by 30 percentage points. These results show that translating human corrections into noise-space supervision substantially improves the efficiency of noise-space policy optimization.

The improvement is consistent across tasks with different manipulation characteristics. On Pick up Spoon and Stack Blocks, UniSteer achieves 90% and 95% overall success rates, respectively, indicating strong adaptation performance on pick-and-place and spatial rearrangement tasks. On Insert Square, UniSteer reaches 100% success, showing that the proposed noise-space steering is particularly effective for precision-sensitive contact-rich manipulation. Fold Towel remains the most challenging task due to deformable-object dynamics and higher execution variability, but UniSteer still improves the success rate to 75%, outperforming both DSRL and DAgger by a large margin.

The ID/OOD split further shows that UniSteer improves not only in-distribution adaptation but also spatial generalization to unseen object placements. For all three position-based tasks, the initial policy completely fails on OOD positions, obtaining 0% OOD success. After adaptation, UniSteer achieves 100% OOD success on Pick up Spoon, Stack Blocks, and Insert Square. In contrast, DSRL remains weak on OOD positions, achieving 0%, 0%, and 25% success on the three tasks, respectively. DAgger improves OOD performance on some tasks but is less consistent, achieving 75%, 100%, and 25%. These results suggest that human-guided noise-space supervision helps the policy adapt beyond the demonstrated state distribution, while still preserving strong in-distribution performance.

Overall, these results show that UniSteer improves both sample-efficient policy adaptation and final task performance under limited real-world interaction budgets, while also enhancing spatial generalization to unseen initial object positions.

5.3 Analysis of Adaptation Efficiency↩︎

We further analyze the trajectory composition collected during each adaptation round and present the results in Figure 4 (a). Compared with DSRL, which relies entirely on autonomous model rollouts, UniSteer uses fewer model-only trajectories while achieving substantially higher final success rates. This suggests that the gain of UniSteer does not come from collecting more autonomous experience, but from using human corrective signals to guide exploration more effectively. In contrast, DAgger relies exclusively on human-collected trajectories, requiring 8 pure human trajectories per round for every task. UniSteer requires far fewer pure human trajectories, averaging only 0.98 per round across tasks, while additionally leveraging a small number of mixed trajectories where the model acts autonomously before human takeover.

This trajectory composition highlights the key efficiency advantage of UniSteer. Rather than treating human interventions only as action-space demonstrations, UniSteer converts them into noise-space supervision targets and uses them jointly with reinforcement learning. As a result, even a small number of human and mixed trajectories can steer the noise actor toward more useful exploration, leading to better adaptation performance than both purely autonomous noise-space RL and action-space imitation from human corrections.

Figure 4 (b) shows the average online adaptation curve across the four real-world tasks. UniSteer improves more rapidly than both DSRL and DAgger throughout training, indicating better time-to-performance under the same real-world interaction budget. This suggests that human corrective guidance, when translated into noise-space supervision, accelerates policy improvement rather than merely improving the final converged performance.

The different learning trends also reflect the limitations of the two baselines. DSRL relies only on autonomous exploration in noise space, which makes improvement slow under sparse rewards and real-world interaction constraints. This issue is especially pronounced when the initial success rate is low: successful trajectories are rarely collected, so the policy receives limited useful reward signals and exploration becomes highly inefficient. DAgger benefits from human corrections early in training, but action-space imitation alone does not consistently translate corrections into robust policy improvement. In contrast, UniSteer uses human interventions to steer the noise actor toward more promising noise regions while continuing to optimize with reinforcement learning, leading to faster and more stable adaptation.

5.4 Analysis of Action-to-Noise Supervision Strategies↩︎

3.5pt

We compare fixed-point inversion with two alternative strategies for using action-space supervision. Optimization-based inversion treats the noise as an optimizable variable and searches for a target noise that reconstructs the human corrective action through the frozen decoder. Direct action supervision does not construct an explicit noise target; instead, it feeds the actor-predicted noise into the frozen decoder and updates the actor through action reconstruction loss.

Table 1 shows that fixed-point inversion achieves the best trade-off among efficiency, inversion accuracy, and downstream success. Compared with optimization-based inversion, it recovers more reliable noise targets with much lower computational cost. Compared with direct action supervision, it avoids expensive backpropagation through the frozen decoder during actor updates and reduces the risk of shifting the learned noise distribution away from regions where the decoder is well calibrated. These results support our design choice of using fixed-point inversion to construct explicit, model-consistent noise targets. This provides an efficient and stable supervision signal for learning from human corrective actions in noise space.

5.5 Analysis of Fixed-Point Inversion Iterations↩︎

6pt

We analyze the effect of the number of fixed-point iterations \(M\) on inversion quality and computation cost. For each recovered noise target, we decode it through the frozen flow-matching decoder and measure the reconstructed action-chunk MSE against the original human correction. Table 2 shows that fixed-point inversion already achieves low reconstruction error with a small number of iterations, and that increasing \(M\) yields quickly diminishing returns while steadily increasing computation. This indicates that the fixed-point iteration converges rapidly in the human-correction regime and produces model-consistent noise targets efficiently. We use \(M=16\) as a practical trade-off, since it achieves nearly saturated reconstruction accuracy while remaining efficient for online adaptation.

5.6 Analysis of Training Schedule↩︎

3.2pt

Table 3 ablates the effect of different training-stage schedules. The SFT-then-RL schedule used in UniSteer achieves the best overall performance, suggesting that the two stages play complementary roles. The supervised stage first pulls the noise actor toward target noise regions inferred from human corrections, providing a more effective starting point for exploration. The subsequent RL stage then further improves the policy through task-level rewards, leading to both more efficient exploration and stronger final performance.

When the order is reversed, the supervised update after RL can overwrite or distort the reward-optimized noise distribution learned during reinforcement learning, reducing the benefit of prior exploration. Using only SFT suffers from limitations similar to imitation-based methods: it relies solely on corrective data, can overfit to the collected correction distribution, and may forget previously learned behaviors, leading to weaker generalization. Using only RL avoids supervised overwriting, but exploration remains inefficient when the initial success rate is low and useful reward signals are sparse. Overall, these results show that supervised noise-space steering is most effective when used to guide exploration before RL, while RL is needed to consolidate and improve the policy under task-level rewards.

6 Conclusion↩︎

We presented UniSteer, the first unified noise-space finetuning framework that integrates human corrective guidance into online RL for flow-matching VLA adaptation. The key idea is to bridge the mismatch between action-space human corrections and noise-space policy optimization through approximate action-to-noise inversion, allowing both human guidance and reward-driven learning to operate through a shared noise interface while keeping the generative decoder frozen. This formulation preserves the efficiency and stability of lightweight noise-space adaptation, while addressing its reliance on inefficient autonomous exploration. More broadly, we show that human interventions can serve not only as action-space demonstrations, but also as structured noise supervision for steering frozen generative robot policies, offering a practical path toward more efficient and scalable real-world VLA adaptation.

7 Theoretical Analysis and Proofs↩︎

7.1 Invertibility of the Continuous Flow Decoder↩︎

7.1.0.1 Setup and statement.

For a fixed state \(s\), consider the continuous-time flow decoder \[\frac{d z_t}{dt}=v_\theta(z_t,t,s), \qquad t\in[0,1],\] and denote the induced noise-to-action map by \(G_\theta(s,z_0)=z_1\). We show that this map is bijective under standard Lipschitz regularity.

7.1.0.2 Proposition A.1.

Assume that \(v_\theta(\cdot,t,s)\) is Lipschitz continuous for all \(t\in[0,1]\). Then, for any fixed \(s\), \(G_\theta(s,\cdot)\) is bijective.

7.1.0.3 Proof.

The Lipschitz assumption guarantees existence and uniqueness of the ODE solution for every initial condition. For surjectivity, fix any target action \(a\in\mathbb{R}^d\) and solve the reverse-time ODE initialized at \(a\): \[\begin{gather} \frac{d y_\tau}{d\tau}=-v_\theta(y_\tau,1-\tau,s), \qquad y_0=a, \\ z_0:=y_1,\qquad z_t:=y_{1-t}. \end{gather}\] Then \[\frac{d z_t}{dt} = -\frac{d y_\tau}{d\tau}\Big|_{\tau=1-t} = v_\theta(y_{1-t},t,s) = v_\theta(z_t,t,s), \qquad z_1=y_0=a.\] Thus \(z_t\) is a valid forward trajectory of the original ODE and \(G_\theta(s,z_0)=a\), proving surjectivity. For injectivity, if \(G_\theta(s,z_0)=G_\theta(s,z_0')=a\), then the reverse-time ODE from the same terminal value \(a\) has a unique solution, so the recovered initial values must coincide: \(z_0=z_0'\). Therefore, \(G_\theta(s,\cdot)\) is bijective. \(\square\)

7.2 One-Step Fixed-Point Inversion↩︎

7.2.0.1 Setup.

We analyze the inverse of one Euler step of the frozen flow decoder. The forward step, inverse equation, and associated fixed-point map are \[\begin{gather} y=x+\Delta t\,v_\theta(x,t_k,s), \\ x=y-\Delta t\,v_\theta(x,t_k,s), \\ g_y(x):=y-\Delta t\,v_\theta(x,t_k,s). \end{gather}\]

7.2.0.2 Proposition A.2.

Assume that \(v_\theta(\cdot,t_k,s)\) is \(L\)-Lipschitz and \(\Delta t L<1\). Then \(g_y\) is a contraction, admits a unique fixed point \(x^\star\), and the iteration \[x^{(m+1)}=g_y(x^{(m)})\] converges to \(x^\star\) from any initialization.

7.2.0.3 Proof.

For any \(x_1,x_2\), the Lipschitz constant of \(g_y\) is bounded by \[\begin{align} \|g_y(x_1)-g_y(x_2)\| &= \Delta t \|v_\theta(x_1,t_k,s)-v_\theta(x_2,t_k,s)\| \\ &\le \Delta t L\|x_1-x_2\|. \end{align}\] Since \(\Delta t L<1\), \(g_y\) is a contraction. By the Banach fixed-point theorem, it admits a unique fixed point \(x^\star\), and the Picard iteration converges to \(x^\star\) from any initialization. \(\square\)

7.3 Finite-Step Approximation Error↩︎

7.3.0.1 Setup and statement.

In practice, the exact fixed point is approximated by a finite number of fixed-point iterations. Let \(x^\star\) be the unique fixed point of \(g_y\), and let \(\rho:=\Delta t L<1\). Then the \(M\)-step approximation satisfies \[\|x^{(M)}-x^\star\| \le \rho^M \|x^{(0)}-x^\star\|.\]

7.3.0.2 Proof.

The contraction property gives \[\begin{align} \|x^{(m+1)}-x^\star\| &= \|g_y(x^{(m)})-g_y(x^\star)\| \\ &\le \rho \|x^{(m)}-x^\star\|. \end{align}\] Recursively applying this inequality yields \[\|x^{(M)}-x^\star\| \le \rho^M \|x^{(0)}-x^\star\|.\] An alternative a posteriori bound is \[\|x^{(M)}-x^\star\| \le \frac{\rho^M}{1-\rho}\|x^{(1)}-x^{(0)}\|.\] Therefore, the finite-step inversion error decays geometrically with the number of iterations. \(\square\)

7.4 Error Accumulation in Recursive Inversion↩︎

7.4.0.1 Setup.

We analyze the cumulative error induced by applying approximate inversion over \(K\) Euler steps. Let \(F_k^{-1}\) denote the exact inverse of the \(k\)-th Euler step and \(\mathrm{Inv}_k^{(M)}\) denote its \(M\)-step fixed-point approximation. Starting from the human corrective action \(a^h\), the exact and approximate recursive inversions are \[\begin{align} &z_0^\star=a^h, &&z_k^\star=F_k^{-1}(z_{k-1}^\star,s), &&k=1,\dots,K, \\ &\hat{z}_0=a^h, &&\hat{z}_k=\mathrm{Inv}_k^{(M)}(\hat{z}_{k-1},s), &&k=1,\dots,K. \end{align}\]

7.4.0.2 Assumptions.

For each step \(k\), assume that the exact inverse is Lipschitz and the finite-step approximation error is geometrically bounded: \[\begin{align} \|F_k^{-1}(y_1,s)-F_k^{-1}(y_2,s)\| &\le c_k\|y_1-y_2\|, \\ \|\mathrm{Inv}_k^{(M)}(y,s)-F_k^{-1}(y,s)\| &\le C_k\rho_k^M, \qquad 0<\rho_k<1. \end{align}\]

7.4.0.3 Proposition A.3.

Under the above assumptions, the recursive inversion error satisfies \[\|\hat{z}_K-z_K^\star\| \le \sum_{k=1}^{K} \left( C_k\rho_k^M \prod_{\ell=k+1}^{K} c_\ell \right).\] In particular, for any \(\delta>0\), there exists \(M_0\) such that for all \(M\ge M_0\), \[\|\hat{z}_K-z_K^\star\|<\delta.\]

7.4.0.4 Proof.

Define the stepwise error \(e_k:=\|\hat{z}_k-z_k^\star\|\). Since both recursions start from the same terminal action, \(e_0=0\). For each step \(k\ge1\), we have \[\begin{align} e_k &= \|\mathrm{Inv}_k^{(M)}(\hat{z}_{k-1},s) - F_k^{-1}(z_{k-1}^\star,s)\| \\ &\le \|\mathrm{Inv}_k^{(M)}(\hat{z}_{k-1},s) - F_k^{-1}(\hat{z}_{k-1},s)\| \\ &\quad+ \|F_k^{-1}(\hat{z}_{k-1},s) - F_k^{-1}(z_{k-1}^\star,s)\| \\ &\le C_k\rho_k^M+c_k e_{k-1}. \end{align}\] Here the first inequality adds and subtracts \(F_k^{-1}(\hat{z}_{k-1},s)\), and the second inequality uses the approximation bound and Lipschitz continuity. Unrolling the recursion from \(e_0=0\) gives \[e_K \le \sum_{k=1}^{K} \left( C_k\rho_k^M \prod_{\ell=k+1}^{K} c_\ell \right).\] Since \(\rho_k\in(0,1)\) for every fixed \(k\), each term converges to zero as \(M\to\infty\). The right-hand side is a finite sum, so it also converges to zero. Therefore, for any \(\delta>0\), there exists \(M_0\) such that \(M\ge M_0\) implies \(\|\hat{z}_K-z_K^\star\|<\delta\). \(\square\)

8 Limitations↩︎

UniSteer still has several limitations. First, its action-to-noise inversion depends on the frozen flow-matching decoder, and the recovered noise targets may be slightly shifted from the initial distribution of the noise actor. Second, our experiments are conducted on four tasks with one robot platform, and broader evaluation across more robots, tasks, and longer-horizon settings is needed.

9 Algorithm of UniSteer Framework↩︎

10 Hyperparameters↩︎

5pt

11 Case Study↩︎

Figure 6 provides a qualitative comparison of early exploration behavior on the Pick up Spoon task. During early adaptation, DSRL relies entirely on autonomous noise-space exploration. As shown in Figure 6 (a), the robot repeatedly approaches the spoon from above but fails to reach a graspable pose. This leads to almost no successful trajectories, making reward-driven improvement inefficient under sparse task rewards.

In contrast, UniSteer uses human corrective actions to provide noise-space supervision through action-to-noise inversion. As shown in Figure 6 (b), after receiving human-guided noise targets, the robot explores lower motions around the spoon and is able to grasp it successfully. This qualitative behavior illustrates the key advantage of UniSteer: human corrections steer the noise actor toward more promising exploration regions, enabling the policy to collect successful trajectories earlier and improving the efficiency of subsequent reinforcement learning.

Figure 7 illustrates a typical forgetting behavior of DAgger during online adaptation. As shown in Figure 7 (a), after the second adaptation round, DAgger learns to grasp the upright spoon configuration. However, when evaluated on the tilted spoon configuration that was learned in the previous round, the policy fails to preserve the earlier behavior, as shown in Figure 7 (b). This suggests that directly aggregating corrective actions in action space can introduce interference between different correction distributions, causing newly learned behaviors to overwrite earlier ones. In contrast, UniSteer translates human corrections into noise-space supervision and combines them with reward-driven optimization, which helps reduce such forgetting while improving exploration.

12 Real-world experimental setup↩︎

References↩︎

---

1. For each fixed state \(s\), assume that \(t \mapsto v_\theta(z,t,s)\) is continuous for every \(z \in \mathbb{R}^d\), and that there exists a constant \(L_s > 0\) such that \(\|v_\theta(z,t,s)-v_\theta(z',t,s)\| \le L_s \|z-z'\|, \forall z,z' \in \mathbb{R}^d,\;\forall t \in [0,1].\)↩︎