1 Introduction↩︎

Long-horizon household tasks require robots to maintain semantic state across many visually grounded actions. Making microwave popcorn, for example, means navigating to the kitchen, opening the microwave, inserting the bag, closing the door, and activating the appliance. No single language-conditioned action solves such a task; it requires composing many skills while preserving the conditions that make the next skill executable.

A natural architecture is to let an agent orchestrate vision-language-action (VLA) skills. In this view, VLA policies act as general-purpose, language-conditioned visuomotor tools, while the agent layer handles task decomposition, state tracking, verification, recovery, and evidence collection. Compared with hand-engineered task and motion planners or symbolic tool APIs[1], [2], this boundary promises broader coverage: new objects and appliances need not each come with a new symbolic controller. The challenge is that learned skills expose a less explicit interface. They may complete the local behavior requested by the agent without producing a state that is usable by the next skill.

We call this the semantic handoff problem (Sec. 2). A skill can satisfy its own postcondition yet leave the robot, objects, or camera views in a state from which the following skill cannot start. For example, a navigation skill before a grasp should not be considered complete merely because the target object is visible somewhere in the scene; the object should be close enough and positioned plausibly for grasping. Similarly, opening a door may satisfy an open-door postcondition while moving the next target out of view, creating a downstream grounding failure. Thus, long-horizon execution requires judging not only whether the current skill appears complete, but whether the resulting state is ready for the next skill.

We study this problem in BEHAVIOR-1K [3] using a compact \(\pi_{0.5}\)-based VLA skill library trained from cleaned skill demonstrations and an agent execution harness that performs bounded plan-act-verify-replan control. The harness represents each skill call with typed arguments, a step budget, and a handoff-aware postcondition; executes the corresponding VLA checkpoint; verifies progress from head and wrist observations with a multi-view VLM verifier; and records replayable traces of skill attempts, verifier decisions, replans, and final task predicates. Our goal is not to claim state-of-the-art BEHAVIOR-1K task success. Instead, we ask what such an agent harness can reveal when long-horizon rollouts fail.

This diagnostic framing separates isolated skill competence from compositional robustness. We evaluate the same skill checkpoints both from clean skill-boundary snapshots and from chained terminal states produced by previous skills. Several skills succeed frequently from curated snapshots under human-reviewed verification, yet composed rollouts still stall from the messy states produced during execution. The resulting traces turn near-zero task success into actionable evidence: failures can be attributed to next-skill readiness, target grounding, or control execution from chained states, pointing to the data and verification capabilities that VLA skill libraries must improve.

This paper makes three contributions:

• We formulate long-horizon agent-orchestrated VLA execution as semantic skill composition, where each skill call has typed arguments, a budget, an expected postcondition, and an implicit next-skill readiness requirement; we identify semantic handoff failures as cases where a skill appears complete but does not leave the state ready for the next skill.

• We instantiate this formulation as a BEHAVIOR-1K semantic execution harness: \(\pi_{0.5}\)-based skill checkpoints trained from cleaned skill segments are invoked through a bounded plan-act-verify-replan loop with multi-view VLM verification, handoff-aware postconditions, recovery, and replayable trace artifacts.

• We provide trace-backed diagnostics showing that snapshot-state skill competence does not by itself carry over to chained initial conditions. On a representative BEHAVIOR-1K rollout round, the harness attributes failed skill attempts to next-skill readiness, target grounding, and control execution from chained states; a controlled arm-reach verifier ablation surfaces additional readiness failures and triggers more re-navigation attempts.

2 Semantic Handoff Problem↩︎

Long-horizon execution chains language-conditioned skills, but the boundary between two skills is under-specified. Each skill is trained and benchmarked to reach its own postcondition \(\phi_t\) from a curated skill-boundary state; in composition, however, the next skill \(s_{t+1}\) must start from whatever state the previous skill \(s_t\) actually leaves behind. A skill can therefore satisfy \(\phi_t\) and still strand the robot, objects, or camera views in a state from which \(s_{t+1}\) cannot execute—a semantic handoff failure. The agent must judge not merely whether the current state is a plausible completion of \(s_t\), but whether it is a valid starting state for \(s_{t+1}\).

A standard verifier checks only \(\phi_t\), which is insufficient at a boundary. For instance, a move_to(can) step followed by pick_up_from(can) should not advance merely because the can is visible at a distance; it should advance only when the can is also plausibly reachable by the gripper. The same checkpoint is also exercised on different inputs in the two settings: a single-skill benchmark evaluates move_to from a restored demonstration snapshot, whereas an end-to-end rollout invokes it after the robot has navigated, opened doors, and shifted viewpoints, so the next skill no longer starts from the curated skill-boundary distribution. This competence drop at skill boundaries is a distribution shift in the imitation-learning sense [4], but distinct from classic within-rollout compounding error: each skill’s legitimate terminal state is already out of distribution for the next skill—a shift between two composed policies rather than within a single one.

In the current system we approximate this handoff requirement directly in the skill postconditions rather than through a separate predicate: the natural-language postcondition authored for each move_to step requires the target to be shown close-up and at arm-reach distance, not merely visible. This is a deliberately limited instantiation; a principled, next-skill-indexed readiness predicate \(\rho(s_t,s_{t+1})\) verified jointly with the postcondition, \(\phi_t \wedge \rho\), is a design direction we develop in Sec. 5.3.

3 Semantic Execution Harness↩︎

Our harness separates the agent layer from the embodiment layer. The agent layer selects skill calls, maintains task state, invokes verification, and triggers recovery. The embodiment layer owns simulation, VLA inference, and low-level action execution. Figure 1 summarizes the system.

3.1 Semantic Skill Contracts↩︎

We represent a skill call as \[s_t = (\text{name}, \text{args}, \text{prompt}, B_s, K_s, \phi_t),\] where \(\text{name}\) selects a skill type, \(\text{args}\) bind task objects, \(\text{prompt}\) is the language instruction passed to the VLA policy, \(B_s\) is the maximum skill budget, \(K_s\) is the verifier interval, and \(\phi_t\) is the expected postcondition. The current skill library includes move_to, pick_up_from, place_on, place_on_next_to, place_in, press, turn_on_switch, open_door, and close_door. We use these names throughout; trace artifacts label move_to and pick_up_from as navigate_to and grasp. press and turn_on_switch are distinct skills that share one checkpoint group: a button/switch press versus a toggle-on action.

The VLA executor is trained as part of the system rather than assumed as an external oracle: the skills are \(\pi_{0.5}\) vision-language-action policies [5], a PaliGemma vision-language backbone with a flow-matching action expert over three RGB views from the head and two wrists, plus proprioceptive state. We clean BEHAVIOR-1K teleoperation demonstrations [3] into skill segments and train in two stages: a pooled all-skill mid-training, then skill-group specialization, with each skill type routed to its group’s checkpoint at deployment (Table 5). We use skill-specific budgets \(B_s\) because VLA policies lack a reliable stop signal and BEHAVIOR-1K skill durations vary widely (Table 4).

3.2 Step-Level Plan-Act-Verify-Replan↩︎

For each skill, the harness executes a VLA action chunk for \(K_s\) environment steps, collects head and wrist observations, and queries a VLM verifier over the head, left-wrist, and right-wrist views together with the current skill’s postcondition. The skill advances as soon as a verification returns a positive verdict above a confidence threshold of \(0.6\); an uncertain verdict never terminates a skill, a fail-safe against flaky calls. Otherwise execution continues until the skill budget is exhausted or the agent replans. Because the agent, not the VLA, owns termination, every rollout produces a step-resolved verifier trajectory rather than a single end-of-skill verdict.

4 Diagnostic Evaluation↩︎

4.1 Setup↩︎

We evaluate in BEHAVIOR-1K with two complementary protocols: an isolated single-skill benchmark and a composed end-to-end rollout. The single-skill benchmark restores the simulator snapshot at a cleaned demonstration segment’s start frame from the validation split and runs the corresponding skill-group checkpoint in isolation, scoring success with the multi-view VLM verifier whose decisions are confirmed by human review. These snapshots are drawn from the same task instances we evaluate end-to-end, so the isolated-vs-composed comparison holds the scene and instance fixed and varies only the skill’s initial-state distribution—a clean snapshot versus the chained terminal state of the previous skill. The end-to-end protocol composes the same checkpoints through the agent loop under live VLA execution, each skill starting from the terminal state of the previous one; we run official BEHAVIOR-1K public test instances per task, taken from the 2025 challenge test_instances.csv, one instance per round across three rounds. Execution is step-level plan-act-verify-replan: the agent verifies every \(K=200\) simulator steps with gemini-2.5-flash multi-view verification and advances a skill only when its postcondition holds, including an arm-reach handoff criterion on navigation, terminating on a per-skill step budget and a global episode budget of twice the human-demonstration length. Every rollout writes a structured trace of skill dispatches, verifier decisions, replans, final task-predicate checks, and media, which we parse into per-skill attempts; failed attempts are classified from structured verifier reasons as diagnostic labels, not human ground truth. The progress score aggregates all three rounds (\(n{=}3\)); the per-attempt failure attribution and verifier-call diagnostics are reported on one representative round.

4.2 Several Skills Are Competent From Curated Snapshot States↩︎

Table 1 summarizes the isolated skill benchmark. Several skills achieve strong judge success from curated skill-boundary states: place_on reaches 6/6, open_door reaches 7/7, pick_up_from reaches 28/29, place_in reaches 11/13, and move_to reaches 27/35. Notably, the placement skills—which dominate the end-to-end bottlenecks below—are themselves competent from snapshot states once trained to convergence, as Table 5 shows, so their chained failures cannot be attributed to weak placement policies in isolation. The skill library is therefore not uniformly weak in isolation.

8pt

4.3 Competence Does Not Survive Composition↩︎

The single-skill and end-to-end protocols evaluate different semantic states. In the single-skill benchmark, a grasp skill starts with the object, robot pose, and camera view close to a successful demonstration boundary. In end-to-end execution, skill \(s_t\) starts from the actual terminal state produced by \(s_{t-1}\): shifted gripper poses, displaced objects, off-angle camera views, or ambiguous object instances. The result is a skill-to-task gap: VLA skills that are competent from snapshot states become unreliable when called from chained states. Crucially, both protocols route the same skill-group checkpoints (Table 1): although they use different success criteria—isolated skill success is VLM-judged with human review, composed task success is the BDDL predicate—the checkpoint that scores highly in isolation is exactly the policy that stalls in composition. The gap is not visible in isolated snapshot evaluation alone: because the same checkpoints are used in both protocols, the traces suggest that chained boundary states expose a robustness failure that is not explained by isolated skill success rates. We hypothesize that simply scaling the same snapshot-only training distribution would not fully close this gap; such scaling adds density on the curated boundary distribution, where the skills are already competent, rather than coverage of the chained states where the observed failures concentrate. Targeted chained-state data and readiness-aware verification are therefore likely necessary.

4.4 Attributing Handoff Failures to Actionable Causes↩︎

Table 2 reports a quantitative per-task signal over three instances of each task: a progress score equal to the fraction of the task’s reference skill sequence, the BEHAVIOR-1K teacher-hint plan, that a rollout completes, scored by hand from the rollout video. Because task-level predicate success is near zero, this graded score is what separates a run that stalls at its first navigation from one that clears most of its plan: mean progress across the thirty rollouts is 19.5%, ranging from roughly half the reference plan on the radio and popcorn tasks down to zero on Easter eggs, and the score localizes where competence collapses—contact-rich manipulation and late multi-object steps.

6pt

Beyond this aggregate progress score, the harness attributes every failed attempt to a specific cause. The three trace-derived causes—target grounding or scene search, control execution, and next-skill readiness (Table 3)—are all VLA-side; each points at a concrete skill and chained-state distribution to improve and maps to a targeted remedy. Crucially, all three are transfer failures: the same checkpoints are competent in isolation (Table 1), so the breakdown cannot be diagnosed from isolated skill success alone. This is the payoff of the agent layer: it turns near-zero task success into a prioritized list of what the VLA must learn next. Notably, the binding failure at most handoffs is not premature advance through a weak state but the policy failing to reach a ready state from its chained start, exhausting the skill budget. Tightening the navigation verifier to require an arm-reach rather than a merely same-region handoff is a controlled ablation that changes only the verifier criterion, holding tasks, instances, and checkpoints fixed. This sharpens the readiness signal: it surfaces 12 additional readiness failures and 25 more re-navigation attempts and recovers the radio task, while leaving control/commit the dominant cause (Table 6). A blinded human audit of the verifier (Appendix, Table 7) confirms its failure decisions: of the verifier-flagged failures the annotators could adjudicate, 20 of 21 were judged real (over-strict \(0.05\)), so a reported failure is a real failure rather than a verifier artifact. Because the failure categories remain verifier-derived and the audited set is small, we report the category counts as preliminary indicators rather than calibrated rates. Table 3 aggregates the failed-attempt categories; per-task attempt, verifier-call, and termination diagnostics appear in the appendix, drawn from a representative ten-rollout round, one instance per task, that issues 196 verifier calls in total.

5pt

5 Discussion↩︎

5.1 Implications↩︎

The results support reading the agent layer as a diagnostic instrument for long-horizon robotics, and should be read as preliminary diagnostics. The harness executes, verifies, and records each rollout, turning a budget-limited run into a step-resolved trace of where execution stalls. These traces reveal concrete blocked states where the remaining issue is most often the chained-state distribution the VLA was never trained on—a skill-boundary distribution shift—rather than a uniformly weak policy; target grounding is the other recurring factor. Further ablations that rerun the same task suite while varying handoff checks, verifier cadence, or recovery strength would measure whether the bottleneck moves; we leave these to future work.

5.2 Limitations↩︎

The current end-to-end evidence covers a small number of completed pilot traces, so bottleneck counts should not be read as population-level estimates. The verifier is VLM-based; a blinded two-annotator audit (Table 7) confirms its failure decisions (20 of 21 verifier-flagged failures judged real, over-strict \(0.05\)), but it validates only the binary failure decision—the failure-category labels remain verifier-derived and may carry bias for object placement, grasp visibility, and articulated-object state, and the audited set is small. The task scripts use some under-specified object names such as door or storage box; robust instance-level binding remains open. Finally, recovery is handled by the agent replanning within the episode budget—in practice it usually re-navigates to re-establish the handoff before retrying a failed skill. A no-recovery versus replanning ablation, runnable in our harness, is designed to separate VLA skill brittleness from recovery limitations; an oracle skill-boundary reset remains a future upper bound because our current live execution setup does not expose arbitrary skill-start state restoration.

5.3 Future Work: Next-Skill Readiness Verification↩︎

The handoff failures diagnosed above motivate a principled treatment of next-skill readiness (Sec. 2). Rather than hand-authoring an arm-reach clause into each move_to postcondition, the verifier could check \[\phi_t \wedge \rho(s_t, s_{t+1}),\] where \(\rho(s_t, s_{t+1})\) is a next-skill readiness predicate generated from the type and arguments of \(s_{t+1}\): \[\rho(s_t, s_{t+1}) = R_{\text{name}(s_{t+1})}(\text{args}(s_{t+1}), o_t),\] \(o_t\) is the current multi-view visual evidence and \(R\) is a finite template library indexed by the next skill type. For example, a move_to before pick_up_from would pass only when the target is plausibly graspable, whereas one before open_door would instead require the handle within reach (Table 9). Evaluating this design—in particular a postcondition-only versus handoff-aware ablation that isolates the effect of \(\rho\)—requires a verifier conditioned on the next skill and is left to future work.

6 Related Work↩︎

Robot policy learning. Language-conditioned manipulation has progressed from modular imitation-learning systems such as CLIPort and PerAct [6], [7] and prompt-conditioned agents such as VIMA [8] to large robot policies and VLA models including Gato, RT-1, RT-2, Open X-Embodiment, Octo, OpenVLA, \(\pi_0\), and diffusion policies [5], [9]–[15]. Recent work further strengthens the VLA substrate through world models and learned memory [16], [17]. We build a compact \(\pi_{0.5}\)-based VLA skill library from cleaned BEHAVIOR-1K skill data, but focus on the execution problem around that library: how trained skills should be composed, verified, recovered, and diagnosed under chained initial conditions.

Task and motion planning and programmatic policies. Task and motion planning combines symbolic task structure with continuous motion, grasp, placement, and collision constraints [18], [19]; code-generating approaches such as Code as Policies synthesize executable robot programs over perception and control APIs [1]. MAESTRO and TIGeR similarly enrich VLM agents with active perception, geometry, planning, control, or exact geometric-computation tools [20], [21]. H-WM guides task and motion planning with a hierarchical world model that curbs error accumulation across long-horizon VLA execution [22]. These systems show the value of stable tools with inspectable semantics. Our harness treats trained VLA skills as agent-callable tools and supplies a missing runtime interface: typed skill contracts, handoff-sensitive postcondition checks, and trace-backed failure attribution.

Language-conditioned agents and semantic grounding. SayCan, Inner Monologue, and ReAct-style systems use language models to select, reason over, or call robot tools [23]–[25]. Recent agentic robot systems add structured workflows, multi-agent decomposition, temporal verification, unified data-collection/deployment loops, and stronger embodied VLM backbones [26]–[29]. Semantic grounding work also uses language or vision-language models to build spatial constraints, value maps, or reusable atomic skills [30], [31]. We target execution-time grounding at skill handoffs: a step should advance not when the target is merely visible but when the state is ready for the next skill—a readiness notion we fold into the skill postcondition and check online rather than as an offline label.

7 Conclusion↩︎

We presented a semantic execution harness for agent-orchestrated VLA long-horizon skill composition. The harness represents skills through typed semantic contracts, executes VLA policies under bounded step-level verification, and surfaces semantic handoff failures at skill boundaries. Preliminary BEHAVIOR-1K diagnostics show a gap between snapshot skill competence and chained-state robustness. This gap is best interpreted as a roadmap for VLA improvement: the agent layer supplies task control, verification, recovery, and evidence, while trace-backed semantic diagnostics identify the VLA recovery states and grounding capabilities needed for reliable long-horizon robots.

Appendix↩︎

Training and Routing Details↩︎

The skills are \(\pi_{0.5}\) vision-language-action policies [5]: a PaliGemma 2B vision-language backbone coupled with a 300M-parameter flow-matching action expert that consumes three \(224\times224\) RGB views from an egocentric head camera and two wrist cameras together with proprioceptive state and predicts multi-step action chunks. We clean BEHAVIOR-1K teleoperation demonstrations [3] into skill segments aligned with the skill contracts using the dataset’s fine-grained subtask annotations, with a per-task split of 180 training and 20 validation episodes. Training has two stages. A full-skill mid-training stage finetunes the base \(\pi_{0.5}\) checkpoint over the pooled skill data for 50k steps to preserve shared visuomotor and language grounding. From this initialization, skill-group post-training specializes separate checkpoints—navigation/orientation, switch/press, door open/close, grasping, surface placement, and containment placement—with full-parameter updates under a cosine learning-rate schedule (peak \(2.5\times10^{-5}\), global batch size 256). At deployment the agent routes each skill type to its group’s checkpoint and renders it into the skill’s language prompt (Table 5).

10pt

6pt

Additional Diagnostics↩︎

This appendix collects the per-rollout diagnostics behind the main-text attribution.

Table 6 reports the effect of tightening the navigation handoff criterion to an arm-reach test, which reclassifies handoff stalls and recovers the radio task.

6pt

The completed ablation is the arm-reach handoff criterion (Table 6). Three further ablations run on the same task suite, instances, and checkpoints but are left to future work: a no-recovery versus replanning comparison that separates VLA skill brittleness from recovery-policy limits, a verifier-cadence sweep over \(K \in \{50,100,200\}\), and a next-skill-conditioned readiness predicate that verifies \(\phi_t \wedge \rho\) directly rather than through move_to wording (Sec. 5.3).

Table 8 gives the per-task attempt, success, verifier-call, and termination counts behind these diagnostics.

Verifier example. The verifier is queried with the skill’s natural-language postcondition, plus an arm-reach clause for move_to. On the radio rollout, the first move_to(radio) check, logged under the raw token navigate_to, asks: “Has the robot successfully completed this skill: ‘navigate to the radio’? Judge only from the current camera views. For navigate_to, success requires the navigation target to appear close-up and prominent in the head camera at arm-reach distance; if it is only visible far away, small, or across the room, answer no.” It returns verdict=no, confidence=0.8, with rationale “the head camera does not show any object identifiable as a radio close-up and prominent at arm-reach distance…the robot has not successfully navigated to the radio,” and recovery_hint “re-localize or search for the radio.” This readiness verdict triggers re-navigation, and the task later succeeds once the radio is reached at arm-reach—the re-plan/retry/advance loop visualized in Fig. [fig:teaser].

Verifier audit. Because the failure attribution is read off the verifier, we audit its failure decisions. Two authors independently judged sampled verifier decisions from the same head and wrist views, blind to the verifier verdict, agreeing on 90% of non-unclear items. Of the verifier’s failure decisions they could adjudicate in consensus (21), 20 were confirmed as real failures—an over-strict rate of \(0.05\), so the verifier almost never rejects a skill that actually succeeded and a reported failure is a real failure. Task-level success is independent of this audit, being scored by the BDDL predicate. The audit covers the binary failure decision only, not the failure category, and \(N\) is small (Table 7).

6pt

Proposed Readiness Template Library↩︎

Table 9 lists the proposed next-skill readiness predicate templates \(R_{\text{name}(s_{t+1})}\) for instantiating \(\rho(s_t,s_{t+1})\) (Sec. 5.3). Each is indexed by the next skill type and is a skill-contract rule, not a per-task pairwise hand-written condition; the current system approximates them through move_to postcondition wording rather than verifying \(\phi_t \wedge \rho\) directly.

6pt

References↩︎