1 Introduction↩︎

Long-horizon manipulation maps one stable task goal to a changing sequence of low-level actions. For example, the task put both moka pots on the stove includes approaching the first pot, grasping it, transporting it, placing it, and repeating the sequence for the second pot. We began with a practical intuition: if the task is divided into subtasks, the language can describe the current action more precisely, and the action distribution within each segment should become narrower. With less variation in each segment, a vision-language-action (VLA) policy might be easier to train.

Our results did not follow this intuition uniformly. In direct fine-tuning, both the stage-text and ordinal-state variants were below the Original mean. Adding a segment signal did not automatically improve the policy, and in some cases it reduced rollout success. That mismatch brings us back to a more basic question than whether segmentation is useful: how should subtask information be represented and introduced during action-policy training?

One possible explanation is that stage conditioning changes how examples share representations across phases. The Original task instruction keeps one language representation for all stages, while observations and robot state distinguish the current behavior. TaskCtx instead appends the active stage phrase to the task prompt, supplying a different language condition for each stage. Most vision-language parameters are frozen, while the top four language-model layers remain trainable. TaskCtx changes the language condition seen by both these upper layers and the action module using limited task data.

We study the interface question on LIBERO-10 with GR00T N1.6. A progress module estimates completion of the active stage and controls when the system advances. The resulting stage pointer is exposed to the action policy either as active-stage text or as an ordinal stage index in state. We compare these designs under two arrangements: direct fine-tuning from the pretrained model and continuation from a full-task baseline. The comparisons use the same stage annotations, controller, GR00T backbone, fine-tuning scope, and paired action-policy runs.

We make two contributions. First, we formulate subtask-conditioned VLA fine-tuning as an interface problem between a stage-tracking module and an action policy, comparing current-stage text with an ordinal stage-state signal under shared annotations and control logic. Second, we evaluate the two interfaces under direct fine-tuning and continuation from a full-task baseline. Explicit stage information does not exceed the Original mean under direct training, while Ordinal Stage-State achieves the highest continuation mean and exceeds both alternatives in all three paired runs.

2 Related Work↩︎

Recent VLA systems have made imitation-based robot learning increasingly capable. RT-2 connects web-scale vision-language knowledge to robotic control, while OpenVLA, Octo, RoboFlamingo, and SmolVLA explore open or efficient adaptation across manipulation settings [1]–[5]. Continuous-action systems such as \(\pi_0\) and GR00T N1 further support direct action generation with flow- or diffusion-style heads [6], [7]. However, stronger models do not remove a practical bottleneck: collecting diverse robot demonstrations remains expensive. Adding descriptions to trajectories that have already been collected can be substantially cheaper than gathering new physical interactions, and modern vision-language models can help propose detailed action segments for review. This creates an appealing possibility: extract more supervision from fixed robot data by describing what the robot is doing at each stage.

Long-horizon systems already suggest that intermediate language can organize behavior. SayCan connects high-level reasoning to executable skills, RT-H inserts language motions between task instructions and low-level actions, and DeMiAn enriches demonstrations with segment-level descriptions [8]–[10]. These works make detailed subtask annotation technically plausible, but they do not make its benefit automatic. Once a trajectory has been divided into stages, the action policy still needs an interface through which that information can affect learning. Our question comes from the practical use of annotation rather than annotation generation itself: does an active-stage label help a fine-tuned VLA, and should it enter through language or state?

Progress-aware systems provide a natural way to keep track of the active stage. ProgVLA and ProgressVLA estimate task progress, SeqVLA uses completion to trigger transitions, and CycleVLA and See-Plan-Rewind use progress or milestones for monitoring and recovery [11]–[15]. We use completion in this internal role: a progress model advances a shared stage pointer, but its completion scalar is not given to the action policy. The experiment focuses on a concrete module-interface choice under the same stage tracker and annotations: expose the active stage as semantic text, or as a normalized ordinal value in robot state. We study this choice by fine-tuning GR00T N1.6 with its visual encoder and most language-model layers frozen [16].

3 Method↩︎

Let \(o_t\) be the visual observation, \(s_t\) the robot state, and \(l_{\mathrm{task}}\) the original task instruction. The action module predicts a low-level chunk \(a_{t:t+H}\). We insert an ordered action-stage representation between \(l_{\mathrm{task}}\) and this chunk. Figure 1 shows how the progress and action models form a closed loop with the environment, with the estimated stage used by the next action request.

3.1 VLA Backbone and Fine-Tuning Scope↩︎

We use the pretrained NVIDIA GR00T N1.6-3B model [7], [16]. Its VLM encodes camera images and task language. State and action projectors connect robot inputs to a 32-layer Diffusion Transformer (DiT), which predicts relative action chunks with flow matching. We freeze the visual encoder and all but the top four language-model layers. We fine-tune those four language layers together with the action-side projectors, vision-language layer normalization, and DiT on LIBERO.

3.2 Action Segmentation↩︎

For each LIBERO task, Qwen2.5-VL [17] receives the original task information and instructions for action segmentation. It produces the ordered stage phrases and demonstration frame ranges used in this study. We use one fixed set of stage phrases and demonstration frame boundaries for all methods. The annotation procedure is not an experimental variable; we compare how the resulting stage information is exposed to the action module. These frame ranges label demonstrations and are not provided as an oracle during rollout. Adjacent low-level motions are grouped into semantic stages, so one stage can include approach, grasp, transport, and placement motions when they serve the same subgoal. The resulting sequence \(C=(c_1,\ldots,c_K)\) lies between the complete task instruction and low-level action chunks.

3pt

Table 1 shows three stages from the fixed annotation used throughout the study. Frame boundaries vary across demonstrations.

3.3 Progress Module↩︎

Frame boundaries \([b_k,e_k]\) assign each training frame to stage \(k\). The local target is \[p_t=\operatorname{clip}\left(\frac{t-b_k}{e_k-b_k},0,1\right).\] The progress model \(g_\phi\) is initialized from the pretrained GR00T checkpoint and receives \(o_t\), \(s_t\), \(l_{\mathrm{task}}\), and the active phrase \(c_k\). During progress training, the visual encoder and all but the top four language-model layers are frozen. The top four language layers, action-side projector, diffusion/DiT module, and action-head VLLN remain trainable, while a linear scalar head followed by a sigmoid predicts progress. We optimize Smooth L1 loss, disable the action loss, and use zero action tokens during progress-only training.

At rollout, the system maintains a stage pointer \(k_t\) and queries \[\hat{p}_t=g_\phi(o_t,s_t,[l_{\mathrm{task}};c_{k_t}]).\] The pointer advances when \(\hat{p}_t>0.8\) after at least 15 stage frames. The pointer is exposed to the action policy either as the active stage label or as its normalized ordinal index. The completion estimate only advances the pointer and is not provided to the action policy.

3.4 Action-Module Interfaces↩︎

Figure 2 summarizes the two alternative mappings from the shared stage pointer to the action-model inputs.

Original. The comparison receives no explicit stage input: \[a_{t:t+H}=\pi_A(o_t,s_t,l_{\mathrm{task}}).\]

Semantic Stage Interface (TaskCtx). The active phrase is sent through language: \[a_{t:t+H}=\pi_B(o_t,s_t,[l_{\mathrm{task}};c_{k_t}]).\]

Ordinal Stage-State Interface. The original instruction is preserved, and the active-stage index is appended to state: \[a_{t:t+H}=\pi_C(o_t,[s_t;z_t],l_{\mathrm{task}}).\] The progress module controls the one-based pointer \(k_t\in\{1,\ldots,K_\tau\}\) for task \(\tau\), which the processor maps to \(z_t\). Let \(K_{\max}=\max_\tau K_\tau\); for LIBERO-10, \(K_{\max}=6\). GR00T applies min-max state normalization: \[z_t=2\frac{k_t-1}{K_{\max}-1}-1.\] Thus, the action model receives the clipped value \(z_t\in[-1,1]\), not \(k_t\) and not a task-relative value normalized by \(K_\tau\). Appending \(z_t\) expands the active dataset and processor state schema from eight to nine dimensions. The GR00T state encoder already contains the corresponding ninth input column, so no model tensor is resized. All projector weights, including the nonzero ninth column, are restored from the same source checkpoint within each comparison. The original eight-dimensional LIBERO interface does not activate the ninth slot during baseline fine-tuning.

3pt

3.5 Training Arrangements↩︎

We evaluate two training arrangements. Direct fine-tuning introduces each interface while adapting the pretrained GR00T policy to the downstream tasks. Baseline-first continuation first adapts the policy with the complete task instruction and then introduces each interface from the resulting full-task checkpoint. Within each arrangement and paired run, Original, TaskCtx, and Ordinal Stage-State share the same source initialization and training budget.

3.6 Closed-Loop Deployment↩︎

The progress and action models run as a pipeline (Fig. 1). While the environment executes the current 8-step action chunk, the progress module estimates active-stage completion. Its output updates the stage pointer and conditions the next action-chunk request. Progress inference runs in parallel and completes before the next request.

4 Experiments↩︎

4.1 Experiment Setup↩︎

Benchmark and data. We evaluate on LIBERO-10 [18]. One action model is trained on all ten tasks using the first 30 demonstrations per task. All experiments use the LIBERO/GR00T eight-dimensional state and seven-dimensional action formats described above. Ordinal Stage-State appends one normalized scalar to state.

Models. We train the original-language baseline and two stage-conditioned direct variants from pretrained GR00T and evaluate them at 5k steps. For each training seed, continuation variants start from the same corresponding full-task baseline-5k checkpoint and continue for 1k steps. A single fixed progress model is trained for 10k steps and shared across all action-policy seeds and both arrangements. Its completion estimate controls the stage pointer but is not sent to the action policy.

Compute and rollout. Training and rollout use a server with four NVIDIA A800 GPUs. Each method is evaluated on all 10 tasks with 50 requested episodes per task and \(n_{\mathrm{envs}}=10\). We use rollout seed 7001, action and progress strides of 8, threshold 0.8, and at least 15 stage frames before advancement. The actual episode count is read from one CSV row per completed episode. Each arrangement is repeated in three paired action-policy runs with fixed random seeds 101, 202, and 303.

4.2 Direct Fine-Tuning↩︎

3.2pt

Table 3 answers whether explicit stage information helps when introduced at the start of downstream adaptation. Neither stage-conditioned interface exceeds the Original mean. Ordinal Stage-State is 4.12 percentage points above TaskCtx and exceeds it in two of the three paired runs, but remains 3.09 points below Original in mean. TaskCtx is 7.21 points below Original in mean, although it is higher in Run 3.

4.3 Baseline-First Continuation↩︎

3.2pt

Table 4 asks which interface has the highest mean after full-task baseline fine-tuning. The ordering is Ordinal Stage-State \(>\) TaskCtx \(>\) Original. Ordinal Stage-State exceeds TaskCtx and Original on all three paired runs, improving their means by 3.75 and 4.69 percentage points, respectively. TaskCtx is 0.93 points above Original in mean and exceeds it in one run.

4.4 Result Scope↩︎

The reported standard deviations are across three independent paired action-policy runs, with the rollout seed fixed. A single 10k-step progress model is shared across all action-policy runs and both arrangements. The two absolute success-rate ranges are not combined because the arrangements use different initialization histories and training trajectories. Each arrangement is compared internally. Ordinal Stage-State has a higher mean than TaskCtx in both arrangements, but neither stage-conditioned interface exceeds Original under direct training.

5 Discussion↩︎

The direct results complicate our initial intuition that a more precise stage description should make action learning easier. Knowing the active stage can reduce action ambiguity, but neither stage-conditioned interface exceeds Original under direct fine-tuning. The interface itself becomes part of the learning problem.

5.1 Interface Representation and Data Sharing↩︎

The higher mean of Ordinal Stage-State relative to TaskCtx can be read through the existing GR00T computation path. In the action DiT, the robot state token and noised action tokens form the hidden sequence that produces cross-attention queries, while visual and language tokens provide keys and values.

TaskCtx modifies the prompt before vision-language encoding. Each stage produces a different language context, and the action queries must retrieve stage semantics through cross-attention. Because the visual encoder and most language layers are frozen, adaptation to these prompts mainly occurs in the top four language layers and the action module.

Ordinal Stage-State keeps the full-task prompt unchanged. The normalized stage index passes through the state projector and becomes part of the state-action sequence before the first DiT block. It participates in self-attention from the beginning of action computation and changes the queries used in later cross-attention layers.

These paths also suggest a difference in how examples share representations. TaskCtx partitions one task’s demonstrations across several stage-conditioned language contexts, whereas Ordinal Stage-State retains a shared full-task language context and separates stages with a low-dimensional action-side variable. More specific conditioning may reduce ambiguity within a stage, but it also leaves fewer examples under each condition. This variance trade-off offers one account of why Ordinal Stage-State has a higher mean than TaskCtx in both arrangements while both remain below Original under direct fine-tuning.

5.2 Introducing Stage Information after Full-Task Fine-Tuning↩︎

The continuation results add a second observation: Ordinal Stage-State exceeds both alternatives in all three paired runs after the shared full-task baseline. During direct fine-tuning, the policy must adapt to LIBERO and learn the new stage interface at the same time. Baseline-first continuation instead introduces stage information after the policy has learned a basic mapping from task language, observations, and robot state to actions.

For Ordinal Stage-State, the full-task prompt learned during baseline fine-tuning remains unchanged. The continuation phase introduces only the ordinal state variable, which may act as a compact adjustment to the existing action representation. TaskCtx augments the baseline prompt with stage-dependent context, requiring the upper language layers and action module to adapt the existing language-action relationship to several new contexts. This difference in compatibility with the baseline policy offers one explanation for the continuation ordering. It also shifts the practical question from whether stage annotations are available to how the stage-tracking module communicates with the action policy and when that interface is introduced.

5.3 Limitations↩︎

The study is limited to LIBERO-10, GR00T N1.6-3B, the current parameter-freezing setup, and the evaluated training budgets. Direct and continuation use different initialization histories and update budgets. TaskCtx and Ordinal Stage-State provide different forms of stage information, so the comparison does not isolate the input channel alone. The stage annotations, progress model, and rollout seed are fixed across action-policy runs. Three paired action-policy runs provide only a limited view of variation across training seeds.

6 Conclusion↩︎

We study how active subtask information should be exposed to a VLA action policy. Under direct fine-tuning, neither stage text nor ordinal stage state exceeds the Original mean, showing that explicit stage annotations do not automatically simplify policy learning. After continuation from a full-task baseline, Ordinal Stage-State achieves the highest mean and exceeds both Original and TaskCtx in all three paired runs. These results suggest that a low-dimensional stage-state interface can be more effective than modifying the language input processed by a mostly frozen vision-language backbone, particularly when adapting an existing full-task policy. Whether this pattern carries to matched training schedules, other benchmarks, and real robots remains open.

References↩︎