1 Introduction↩︎

Designing a dense reward function for a long-horizon manipulation task is notoriously hard. Hand-engineered shaping signals are brittle and rarely transfer across embodiments or tasks; reward learning from human demonstrations or preferences is expensive and difficult to scale. A recent line of work circumvents both options by using a frozen pretrained vision-language model (VLM) as a zero-shot reward model [1]–[3]. The agent receives, for each rendered frame, a similarity score between the frame’s image embedding and a short natural-language description of the desired outcome.

1.0.0.1 The single-prompt failure mode.

While appealing, a single global prompt that describes the final goal state is poorly suited to long-horizon control with randomized initial conditions. Empirically the language reward is near-flat for most of the trajectory: the gripper spends many steps far from any frame configuration that the prompt describes, so the embedding similarity changes only marginally as it moves. At the same time, any one camera is hostage to gripper- and cube-induced occlusion: a perspective that conveniently shows the cube one moment can be entirely blocked by the gripper the next. Both effects waste signal.

1.0.0.2 Contributions.

We propose Reinforced Micro-task Learning (RMTL), a pipeline that decomposes a manipulation task into a small set of language-described micro-tasks and trains an agent to switch between them, with VLM rewards aggregated over multiple camera views and a reverse curriculum on initial conditions (1). Concretely, we contribute the following:

1. We show that decomposing the VLM reward into three short, stage-specific language prompts (approach, align, grasp) turns a near-flat single-prompt signal into a piecewise-monotone signal that PPO can climb, on the same environment and same VLM (§3.3, 4.2; 5).

2. We show that a small learned hierarchical manager, BC-warmstarted from a rule-based selector and refined by REINFORCE over a frozen PPO worker, matches or exceeds the rule while remaining stable (§3.6, 4.2; 7).

3. We show that multi-view aggregation of per-camera VLM scores reduces per-view occlusion noise and yields a smoother training signal than any single view, with no gradient-shape penalty (§3.2; [fig:multiview_snap,fig:multiview_story]).

4. We show that a reverse curriculum is the necessary glue: without it the language reward is too uninformative at the random-init endpoint to bootstrap PPO (§3.4, 4.2; 6).

The remainder of the paper introduces related work (§2), develops the method (§3), presents experiments (§4), and discusses limitations and future work (§5, 6).

2 Related work↩︎

2.0.0.1 RL for robotic manipulation.

Reinforcement learning in simulation has produced standard benchmarks for contact-rich manipulation. The Fetch family of mobile-manipulator tasks introduced by [4] highlights the recurring difficulty of reward formulation: in pick-and-place, agents typically rely on dense hand-designed geometric rewards or on sparse binary success signals. Sparse rewards demand excessive exploration, while dense geometric shaping is sensitive to local minima and assumes access to privileged simulator state. Our work removes this privileged state from the reward path and instead derives a dense reward purely from rendered visual observations.

2.0.0.2 Pixel-based and visual RL.

Moving from state to pixel observations introduces a sample-efficiency gap that has been partially closed by representation learning. RAD [5] shows that diverse image augmentations alone let pixel-based RL match state-based methods, and CURL [6] achieves the same with contrastive self-supervision. These methods improve the encoder of the policy but still rely on environment-provided task rewards. We are orthogonal: the environment provides no intrinsic task reward and the visual observation is the reward, computed by an external frozen VLM.

2.0.0.3 VLMs as reward models.

A growing body of work uses pretrained vision-language models to score RL trajectories. [1] introduce VLM-RMs, a family of zero-shot rewards that contrast a target prompt against a baseline using CLIP [7]; we build directly on this projected scorer and extend it to per-micro-task prompts and multi-view aggregation. [2] use a video-text encoder for one-shot imitation. [3] treat the VLM as a code generator for hand-shaped reward functions. Other lines pretrain representations whose embeddings double as rewards: VIP [8] derives a value-implicit embedding from human videos, LIV [9] adds language-image rewards from action-free video, and [10] train an internet-scale contrastive reward model for language-conditioned tasks in Minecraft. Across all of these the reward is conditioned on a single, global language prompt describing the final goal, which is the regime in which we identify the near-flat-reward failure mode and which our micro-task decomposition is designed to address.

2.0.0.4 Hierarchical RL and task decomposition.

Hierarchical methods that decouple high-level skill selection from low-level control are a long-standing line of research; the options framework of [11] formalises the multi-level control structure, and option-critic [12] provides an end-to-end gradient. For continuous control, HIRO [13] successfully applies a goal-conditioned two-level hierarchy. A long-standing open question in this line is how to discover sub-goals automatically. Our RMTL manager grounds sub-goals directly in language: a small (\(\approx 2\)k-parameter), discrete-action manager selects a micro-task prompt and is trained with REINFORCE on segment returns. Stage 1 supplies the manager with a rule-based teacher; Stage 2 replaces that rule with the learned manager itself, turning heuristic phase selection into a fully learned policy.

2.0.0.5 Language-conditioned policies vs.language as reward.

Beyond reward generation, language has been used directly to condition control policies and ground robotic affordances. RT-1 [14] tokenises language instructions and camera images and outputs motor commands; SayCan [15] grounds the semantic knowledge of LLMs in pretrained robotic skills so that high-level planners can sequence them. These methods rely heavily on demonstration datasets and imitation learning. Our setting is purely RL: language is used only as a supervisory signal for the reward, and policies are trained from scratch in simulation without expert demonstrations.

2.0.0.6 Curriculum and reverse curriculum learning.

Reverse curricula [16] schedule the distribution of initial conditions from easy to hard. We adopt a scalar form: a single difficulty knob \(d \in [0, 1]\) interpolates the gripper between an "above-cube" trivial start and a fully randomized workspace init, and is gated by per-stage success thresholds.

2.0.0.7 Imitation-warmstarted policies.

Behaviour cloning followed by RL is a common recipe [17]; we use it for the manager (not the worker), with rule-based teacher labels available essentially for free.

3 Method↩︎

We organize the method around the data flow shown in 2: at each step the environment renders a small bank of camera views, a frozen VLM scores each view against the prompt of the currently active micro-task, the per-view scores are mean-aggregated into a scalar reward, and a PPO worker uses that reward to update its policy. The active micro-task is selected either by a fixed distance-based rule (Stage 1) or by a learned manager (Stage 2). A reverse curriculum schedules the distribution of initial gripper configurations.

3.1 Preliminaries and problem formulation↩︎

We formulate robotic manipulation as an episodic finite-horizon reinforcement-learning problem. The interaction between the robot and the environment is modelled as a Markov decision process \(\mathcal{M} = (\mathcal{S}, \mathcal{A}, P, \rho_0, H, \gamma)\), with state space \(\mathcal{S}\), continuous action space \(\mathcal{A}\), transition dynamics \(P(s_{t+1}\mid s_t, a_t)\), initial-state distribution \(\rho_0\), horizon \(H\), and discount factor \(\gamma\). At each time step the agent observes \(o_t = O(s_t)\), samples \(a_t \sim \pi_\theta(\cdot \mid o_t)\) and receives a scalar reward \(r_t\). The policy is trained to maximize the expected discounted return: \[J(\theta) = \mathbb{E}_{s_0 \sim \rho_0,\, a_t \sim \pi_\theta} \left[\sum_{t=0}^{H-1} \gamma^t r_t\right]. \label{eq:rl-objective}\tag{1}\] In manipulation, specifying \(r_t\) is the main bottleneck. Hand-designed dense rewards are brittle and require privileged simulator states, while sparse success rewards make exploration unfeasible in long-horizon tasks where terminal states are rarely reached by an untrained policy.

We instead consider rewards derived from a frozen pretrained vision-language model. Let \(I_t = \mathcal{R}(s_t)\) denote a rendered image of the environment state (or a tuple of such images, one per camera view), and let \(F_\psi(I_t, p)\) be the scalar compatibility score assigned by a VLM with fixed parameters \(\psi\) to image \(I_t\) and text prompt \(p\). The policy parameters \(\theta\) are optimized; \(\psi\) is never updated. In our implementation \(F_\psi\) is instantiated as a CLIP-style projected-embedding reward (§3.3), but the rest of the method only assumes access to a scalar image-text alignment score.

A direct approach is to use a single global prompt \(p_G\) describing the final task outcome, \[r_t^G = F_\psi(I_t,\, p_G). \label{eq:single-global-reward}\tag{2}\] For example, \(p_G\) may describe a robot gripper lifting a cube. This formulation is simple, but it assumes that similarity to the final goal prompt provides a meaningful dense measure of progress throughout the trajectory. In long-horizon manipulation that assumption is violated: the robot must pass through qualitatively different stages (reaching, aligning, grasping, lifting), and early states—necessary for eventual success—may have weak semantic similarity to the final goal prompt. The single-prompt reward then becomes nearly flat over large portions of the trajectory, leading to poor credit assignment and weak exploration guidance, especially under randomized initial conditions where most early interactions otherwise carry no learning signal.

The problem we address is therefore: given a manipulation MDP, a frozen VLM, and natural-language task descriptions, define a reward that is visually grounded, language-conditioned, and sensitive to intermediate task progress across the full trajectory.

3.2 Aggregate multi-view semantic scoring↩︎

Manipulation scenes are often partially occluded or ambiguous from a single viewpoint: the gripper may hide the cube, a close-up may omit global pose context, and a wide view may sacrifice fine contact detail. We therefore compute the VLM score independently for each of \(V{=}6\) fixed viewpoints and take the arithmetic mean as the scalar reward fed to PPO. This aggregate score is a simple way to trade off complementary visual evidence without fusing pixels beforehand.

3 makes the disagreement concrete on one frame: all six cameras see the same instantaneous state, yet their active-micro-task scores differ substantially. The mean smooths these per-view fluctuations while preserving the coarse semantic trend toward the prompt. Over full episodes, two independent rollouts in 8 show the same pattern: the multi-view mean tracks a smoother trajectory than any single camera.

3.3 Phased VLM reward↩︎

We score images using the projection-style scorer of [1] adapted to per-micro-task targets. Let \(E\) denote the (fixed) image encoder and \(T\) the text encoder of a VLM. We encode two groups of prompts once per training run:

• a baseline prompt \(b\) (a frame with no task progress, e.g."a robot gripper and a red cube on a table"),

• a set of \(K\) micro-task target prompts \(\{\tau_i\}_{i=1}^K\) (one per micro-task, e.g."a robot gripper pinching a red cube between two fingers").

Their \(\ell_2\)-normalised text embeddings are denoted \(e_b = T(b)\) and \(e_{\tau_i} = T(\tau_i)\). For micro-task \(i\), the direction \(d_i = e_{\tau_i} - e_b\) identifies the semantic axis from "no progress" to "micro-task \(i\) satisfied"; the projection \[\begin{align} P_\alpha(d_i) &= \alpha \, \frac{d_i d_i^{\top}}{\lVert d_i \rVert_2^2} + (1 - \alpha) \, I, \label{eq:projection} \end{align}\tag{3}\] trades off between rewarding only motion along that axis (\(\alpha \to 1\)) and unrestricted cosine similarity to \(e_{\tau_i}\) (\(\alpha \to 0\)). For each camera view \(v\) we compute a per-view image embedding \(x_v = E(I_v)\) and score \[\begin{align} r_v^{(i)} &= 1 - \tfrac{1}{2}\, \bigl\lVert (x_v - e_{\tau_i})\, P_\alpha(d_i) \bigr\rVert_2^2, \label{eq:per-view-reward} \end{align}\tag{4}\] and aggregate over views by the mean \[\begin{align} r_t^{(i)} &= \frac{1}{V} \sum_{v=1}^{V} r_v^{(i)}. \label{eq:multiview-mean} \end{align}\tag{5}\] We use PE-Core-bigG-14-448 [18] as the VLM, with \(\alpha = 0.9\); the analysis that motivates this choice is given in 9. The scorer in [1] also admits an optional negative-prompt term that subtracts cosine similarity to a distractor prompt; we did not find it necessary in the final configuration.

3.4 Reverse curriculum↩︎

Training directly under the hardest randomised resets is brittle: the gripper can start far enough from meaningful scene structure that the VLM provides almost no similarity gradient, so exploration never connects to success. We therefore adopt a reverse curriculum that gradually widens the distribution of initial gripper (and finger) poses from an “easy” region (near the object, structured approach) toward the full randomised workspace draw used at convergence. The schedule is organised into five difficulty levels; each level exposes strictly more challenging initial conditions than the previous one. Progression is governed by online success statistics so that the agent is only exposed to the next level once it reliably solves the current one. Formally, this is the same reverse-curriculum principle as [16], specialised to randomised grippers in manipulation. Full numeric defaults (stage ranges, gating statistics, minimum dwell time) are listed in 7.

3.5 Stage 1: PPO worker with rule-based micro-task switching↩︎

We instantiate RMTL with \(K = 3\) micro-tasks aligned with the phases of pick-and-place: approach, align, and grasp, each associated with one short target prompt (full prompt text in 7). At every step a deterministic rule selects the active micro-task; the phase-aware scorer indexes the corresponding \(\tau_{i_t}\) in Eq. 4 , and the worker receives the VLM reward of the currently active micro-task only — never a sum or average across the three. The worker is trained with PPO [19], [20]; full hyperparameters are listed in 1.

3.6 Stage 2: learned hierarchical manager↩︎

Stage 2 replaces the rule-based selector of §3.5 with a learned manager \(\pi_M(\phi \mid s)\): a small MLP with a \(K\)-way softmax over the same three micro-tasks, shared across all parallel environments. At every \(H = 3\) environment steps the manager takes one decision \(\phi \in \{0, 1, 2\}\), which is broadcast to the worker as a phase-onehot input and to the reward pipeline as the index for \(\tau_{\phi}\) in Eq. 4 . The manager is BC-warmstarted from rule-based labels and then refined by REINFORCE on segment returns; the worker is loaded from a Stage-1 PPO checkpoint and its actor is frozen for the first \(50{,}000\) environment steps to stabilise the manager’s REINFORCE target during the most non-stationary phase of training, then thawed at a reduced learning rate. The full procedure (BC warmstart, REINFORCE objective, KL anchor, \(\varepsilon\)-greedy exploration, freeze schedule, and the Stage-2 data-flow diagram) is given in 7, with hyperparameters in 2 and the end-to-end algorithm in 9. Joint manager-and-worker training from scratch is left to future work (§6).

4 Experiments↩︎

4.1 Setup↩︎

4.1.0.1 Environment.

We instantiate RMTL on the FetchPickAndPlace-v4 environment from Gymnasium-Robotics [4], a 7-DoF Fetch arm tasked with picking up a cube and holding it stably with the gripper closed; physics is simulated by MuJoCo [21]. The environment exposes a 25-dimensional state observation \(s_t\) comprising gripper pose, cube pose, finger opening, and goal location. Actions are 4-dimensional: gripper \(\Delta xyz\) plus finger opening. Episode success is detected geometrically once the gripper has stably grasped and lifted the cube above the table (full criterion in 7). Episodes are at most 40 steps long. Object, goal, and gripper positions are randomized within workspace bounds at every reset; the gripper’s initial spatial spread is then modulated by the reverse curriculum (§3.4).

4.1.0.2 Visual domain.

Pretrained VLMs are trained primarily on natural images and read raw MuJoCo renderings poorly: low-texture surfaces, ambiguous object colors and visual artefacts make image-text alignment unreliable. To narrow this domain gap we apply a high-fidelity rendering setup: the manipulated cube is rendered in a saturated red, the table is given a wooden appearance, the floor and background are set to neutral grey tones, and lighting and material parameters are tuned for contrast and object separation (4). These changes do not alter the task dynamics or success condition; they only make the rendered scene more semantically legible to the frozen VLM. The rendering setup is identical across all methods we compare.

4.1.0.3 Compute.

We train every variant on a single NVIDIA A100 GPU with 16 parallel CPU environments per learner. Stage-1 runs typically converge near 600k environment steps; Stage-2 manager runs are trained for an additional 200k environment steps starting from a loaded Stage-1 checkpoint.

The complete training setup for the Stage-1 PPO worker is summarised in 1; Stage-2 hyperparameters specific to the learned hierarchical manager are listed in 2. Hyperparameters not repeated in 2 (environment, multi-view rendering, VLM scorer, reward shaping, RC, PPO worker) are inherited unchanged from 1.

4.2 Results↩︎

We report all curves as eval success rate against environment steps.

4.2.0.1 Micro-task vs.single-prompt baseline.

5 compares RMTL (three per-micro-task prompts + multi-view) against two strict counterparts that share the exact same RL hyperparameters and only disable the contribution under test: single global prompt + multi-view and single global prompt + single view. Decomposing the global prompt into three micro-task prompts is the dominant gap: the single-prompt variants fail to converge on the randomised protocol, whereas the per-micro-task variant passes the rule-based PPO baseline.

4.2.0.2 Reverse-curriculum progression.

6 shows env steps spent in each RC level and the rolling per-level success rate during Stage-1 training. The pattern that the agent dwells the longest at the hardest levels indicates the randomised-init regime is the binding difficulty rather than any intermediate one.

4.2.0.3 Behavioural comparison: rule vs.learned manager.

7 compares four policies on Fetch under random initial conditions. The two hierarchical columns share the same worker checkpoint and differ only in the phase signal — a distance-based rule vs.the learned manager — which isolates the effect of the manager from any change in the worker. The full RMTL stack with the learned manager attains \(\sim\) \(98\%\) success, up from \(94\%\) for the rule-based selector and \(89\%\) for the non-hierarchical PPO worker, while a single-prompt PPO baseline at the same compute fails entirely (\(0\%\)). The learned manager agrees with the rule on \(73\%\) of its decisions, yet still outperforms it: the rule is therefore not optimal but a useful inductive bias, and using it as a BC teacher (7) lets the manager reach an end policy that clears the rule’s ceiling at a fraction of the training cost required to learn phase selection from scratch.

4.2.0.4 Multi-view vs.single-view properties.

8 traces the multi-view mean reward against a single-view baseline (wide_view) over two independent eval rollouts. In both episodes the multi-view mean is smoother than any single view and preserves information about gripper configuration that a hand-engineered dense distance reward does not.

5 Discussion and limitations↩︎

5.0.0.1 Benefits of RMTL.

Three components account for most of the improvement over a single-prompt VLM-RL baseline. (i) The micro-task decomposition converts a near-flat single-prompt VLM signal into a piecewise-monotone signal that PPO can climb (5). (ii) Multi-view aggregation reduces per-view occlusion noise and yields a smoother training signal than any single view (8). (iii) The reverse curriculum bridges the easy-init and random-init regimes where the language reward alone is insufficient to bootstrap exploration. The behavioural ablation in 7 further isolates the contribution of the learned manager: replacing the distance-based rule with a learned policy lifts random-init grasp-detection success from \(84\%\) to \(\sim\) \(98\%\).

5.0.0.2 Limitations.

The reward signal inherits the perceptual capabilities and limitations of the underlying VLM: scenes whose lighting, shading, or close-up texture fall outside the VLM’s training distribution can yield noisy similarity scores, and current VLMs exhibit limited fine-grained spatial reasoning that can make per-step reward ambiguous in contact-rich phases. The grasp prompt also naturally saturates once the cube is firmly held, reducing gradient signal near the goal. We address the visual gap with a high-fidelity rendering setup (4) and the saturation issue with light reward post-processing (7); improving robustness to varied visual conditions and stronger VLM spatial reasoning remain open challenges.

6 Future work↩︎

The most promising next step is to remove the last manual element in RMTL by pairing it with an LLM-based micro-task disassembler: given a high-level natural-language task description, an LLM would automatically propose the stage-aligned prompt set and a candidate phase-selection rule, replacing today’s three hand-written prompts and making the pipeline applicable to new tasks without manual prompt engineering. We also plan to expand the evaluation to additional manipulation tasks.

7 Implementation details↩︎

7.0.0.1 Algorithm.

The RMTL algorithm is given below.

7.0.0.2 Stage-2 data flow.

10 sketches the full Stage-2 wiring. ManagerNet drives both the worker (via a phase-onehot input) and the phase-aware scorer (via the active prompt index), while the distance-based rule classifier is retained only as an offline BC label source and as the KL-anchor target during REINFORCE.

7.0.0.3 BC warmstart.

Before any RL update, we run a behaviour-cloning phase: the rule-based phase classifier is rolled out for 250 episodes on the loaded PPO 560k checkpoint, producing \((s, i_{\text{rule}})\) pairs. The manager is then fit to these labels with cross-entropy for 3500 updates at batch size 64.

7.0.0.4 REINFORCE on segment returns.

After warmstart, the manager is trained at the end of every PPO rollout with a standard REINFORCE objective on segments of length \(H\): \[\begin{align} \mathcal{L}_M &= -\, \mathbb{E}_{\text{seg}}\!\left[ \log \pi_M(\phi \mid s_{\text{seg}}) \cdot A_{\text{seg}} \right] \;-\; c_H \mathcal{H}\!\bigl[\pi_M\bigr] \;+\; w_{\text{KL}} \,\mathrm{KL}\!\bigl(\pi_{\text{rule}} \,\|\, \pi_M\bigr), \label{eq:manager-loss} \end{align}\tag{6}\] where \(A_{\text{seg}}\) is the rollout-internal \(z\)-score of the segment return \(\sum_{t \in \text{seg}} r_t\), \(\mathcal{H}[\cdot]\) is the distributional entropy with bonus \(c_H = 0.01\), and the third term is a KL anchor to the rule-based policy with weight \(w_{\text{KL}} = 0.30\). Action selection uses \(\varepsilon\)-greedy exploration with \(\varepsilon = 0.20\). The optimizer is Adam at learning rate \(3 \times 10^{-4}\).

7.0.0.5 Frozen worker.

Throughout this paper, Stage 2 starts from a specific Stage-1 checkpoint (PPO 560k env steps) loaded into the worker. For the first \(50{,}000\) environment steps the worker actor’s parameters are frozen (\(\texttt{requires\_grad} = \mathrm{False}\) on features + mlp_extractor.policy_net + action_net + log_std), keeping only the critic trainable. After 50k steps a callback unfreezes the actor and reduces the worker learning rate to \(1 \times 10^{-4}\). The BC labels above are also generated against this same 560k checkpoint for distributional consistency. Freezing the worker stabilises the manager’s REINFORCE target during the most non-stationary phase of training, which we found important for avoiding mode collapse on the manager’s softmax.

7.0.0.6 Phase prompts.

The three Stage-1 micro-task target prompts and the shared baseline prompt are written once and reused across runs. They are:

• baseline \(b\): “a robot gripper and a red cube on a table”;

• approach target \(\tau_{\text{approach}}\): “a two-finger robot gripper with open jaws centered above a red cube”;

• align target \(\tau_{\text{align}}\): “a robot gripper pinching a red cube between two fingers”;

• grasp target \(\tau_{\text{grasp}}\): “a robot gripper lifting a red cube between its fingers”.

The rule-based selector (Stage 1) chooses the active micro-task deterministically from a simple geometric feature; the learned manager (Stage 2) replaces this rule with a small policy over the same three prompts.

7.0.0.7 Camera bank.

The six camera viewpoints used by all multi-view configurations are defined as MuJoCo free-camera settings (azimuth, elevation, distance, lookat). Their parameters are: wide_view \((132, -14, 1.5)\), front \((180, -20, 1.1)\), front_side \((45, -30, 1.2)\), isometric \((225, -35, 1.1)\), side \((90, -15, 1.2)\), close_up \((135, -25, 0.8)\), all looking at approximately \((1.3, 0.75, 0.42)\). Realistic-visuals shading is on.

7.0.0.8 Termination bonus.

An episode terminates with success once the gripper is within \(0.025\) m of the cube and the cube has been lifted at least \(0.02\) m above the table for two consecutive steps; on success we add a one-shot reward of \(+1.0\) to the final step.

7.0.0.9 Reverse-curriculum thresholds.

The 5 RC stages have \((d, [d_{\min}, d_{\max}], \theta_{\text{succ}}, n_{\min})\) values \((0,\,[0,0.08],\,0.5,\,50)\), \((0.15,\,[0.08, 0.25],\,0.5,\,80)\), \((0.4,\,[0.25, 0.50],\,0.45,\,120)\), \((0.65,\,[0.50, 0.75],\,0.4,\,150)\), and \((1.0,\,[0.75, 1.0],\,-,\,0)\) (the final stage is ungated).

7.0.0.10 Manager hyperparameters.

Manager horizon \(H = 3\), hidden sizes \([128, 128]\), learning rate \(3 \times 10^{-4}\), entropy coefficient \(0.01\), KL-to-rule weight \(0.30\), \(\varepsilon\)-greedy \(0.20\), BC pretrain updates \(3500\), BC batch size \(64\), BC pretrain episodes \(250\), BC pretrain checkpoint PPO 560k, frozen-worker until 50k env steps, unfreeze learning rate \(1 \times 10^{-4}\).

8 Hyperparameters↩︎

8.0.0.1 Stage 1 hyperparameters.

Stage 1 training hyperparameters are given in the table below.

8.0.0.2 Stage 2 hyperparameters.

Stage 2 training hyperparameters are given in the table below.

9 Projection-mix sweep↩︎

This appendix expands on the choice of the projection-mix scalar \(\alpha = 0.9\) used in Eq. 3 .

9.0.0.1 Interpreting \(\alpha\).

The scalar \(\alpha \in [0,1]\) interpolates between unprojected cosine similarity (\(\alpha{=}0\)) and projection strictly along the target–baseline axis (\(\alpha{=}1\)). As \(\alpha\) increases, the raw per-step VLM reward typically spans a wider numerical range for the same underlying trajectory: the temporal profile—which micro-task is active and how similarity evolves—remains largely unchanged, but PPO sees larger step-to-step differences and therefore a stronger gradient signal. 11 summarises this on one episode: all \(\alpha\) curves follow the same phase structure, but higher \(\alpha\) stretches the vertical extent.

12 aggregates the same episodes over two scalar state coordinates. In gripper–cube distance (left), reward climbs monotonically as the gripper approaches the cube; the family of \(\alpha\) curves overlaps, so \(\alpha\) does not flip which physical variable the reward tracks. In object lift height (right), the on-table (\(h{\approx}0\)) vs.lifted (\(h{>}0\)) transition marks the switch into the terminal grasp micro-task; after that the prompt is saturated, so variation along height is dominated by visual noise rather than an informative gradient. Together, these plots justify \(\alpha{=}0.9\) as a high-amplitude setting that preserves the phase structure, and they motivate keying the rule-based selector on distance rather than lift height (§3.5).

References↩︎

---

1. Corresponding author.↩︎