CoStream: Composing Simple Behaviors for Generalizable Complex Manipulation
June 24, 2026
Abstract
Long-horizon, contact-rich complex manipulation tasks, such as seating a GPU into a PCIe slot, demand both millimeter high precision and out-of-the-box generalization to new tasks. Existing paradigms struggle to satisfy both: classical pipelines use brittle, task-specific interfaces to achieve high-precision control but require costly pipeline redesigns to adapt to new tasks, whereas monolithic end-to-end policies provide better generalization but lack high precision on complex, out-of-distribution tasks unless retrained with new data. Both paradigms share an implicit assumption: once a manipulation capability is acquired, it must be deployed as a rigid pipeline or monolithic whole, rather than being freely decomposed and recomposed. In this paper, we show that complex manipulation capabilities can emerge naturally from the composition of simple, independent behaviors. Rather than deploying a monolithic policy or a rigid pipeline, we propose CoStream, a framework orchestrating foundation models and diverse sensing modalities into multiple composable core behaviors: a semantic behavior extracting spatial constraints via foundation models; a predictive behavior forecasting trajectories by tracking keypoints in imagined videos; and a reactive behavior providing high-frequency tactile and force corrections. On a shared \(SE(3)\) interface, these outputs compose by right-multiplication into a single pose command at each control step, executed by a compliant controller. We demonstrate CoStream on 8 real-world tasks spanning everyday manipulation and precision assembly, with the strongest gains in contact-rich assembly and object transfer, and show robust recovery from manual perturbations during execution. Website: https://costream-simple.github.io
1 Introduction↩︎
Inserting a GPU into a PCIe slot leaves less than a millimeter of clearance. A slight angular error can make the card catch on the slot rim and jam before seating. If the card slips inside the gripper mid-insertion, the entire trajectory jams. Long-horizon, contact-rich manipulation like GPU insertion requires a robot to understand the semantic geometry of a motherboard, predict a collision-free path, and physically react to sub-millimeter force deviations in real time.
To address these challenges, current research generally falls into two paradigms. The first paradigm employs classical, modular pipelines that explicitly decouple perception, state estimation, planning, and control. These pipelines are inherently rigid; adapting them to new tasks requires bespoke engineering and the manual redesign of perception and state estimation modules [1]–[3]. The second relies on large monolithic policies, such as vision-language-action (VLA) models. These models attempt to absorb semantic reasoning and high-frequency contact dynamics into a single network to solve tasks like GPU insertion directly. However, these end-to-end approaches are highly data-hungry, requiring extensive new human demonstrations to generalize to every novel task [4]–[15]. Both paradigms share the assumption that complex tasks require a complex and indivisible control system.
Behavior-based robotics, a pioneering framework from the 1980s, argues that complex behavior need not arise from a single complex controller; it can instead emerge from composing simple sensory behaviors through interaction with a complex environment [16], [17]. Inspired by this idea, we show that complex manipulation can be achieved by composing multiple simple, independent action components. Much like human motor control, breaking actions down ensures that each component remains simple. Consequently, whether a behavior is powered by a foundation model, a specialized neural network, or classical engineering, it becomes significantly easier to develop and naturally accommodates diverse sensor modalities. Building on this principle, CoStream establishes a common SE(3) task-space interface across three behaviors: a semantic behavior that produces object-centric geometric goals, a predictive behavior that generates nominal motion from imagined video, and a reactive behavior that produces high-frequency tactile and force contact corrections. Unlike a conventional modular pipeline, CoStream does not pass a completed plan through a sequence of fixed interfaces. Instead, these behaviors remain independent, operate at their natural rates, and compose by right-multiplication into a single pose command at every control step, which a compliant controller executes.
Our contributions are threefold. First, we introduce a behavior composition interface for contact-rich manipulation that separates semantic grounding, predictive motion generation, and tactile and force reaction into independent behaviors. Second, we propose a shared SE(3) task space representation that allows each behavior to run at its natural rate and contribute only the component it is reliable for: a task frame, a nominal motion prior, or a contact correction. Third, we validate CoStream on eight real world tasks, showing high precision assembly, manual perturbation recovery, and transfer to everyday manipulation without retraining a policy for each task.
2 Related Works↩︎
Foundation Models for Robotics. Large Language Models (LLMs) and Vision-Language Models (VLMs) have enabled robots to interpret language, reason over long-horizon tasks, and generate executable plans [4], [18]–[22]. Recent methods improve spatial precision by grounding robot actions in visual observations, using value maps [23], keypoint or affordance constraints [24]–[26], iterative visual prompting [27], or test-time simulation [28]. However, these approaches still rely primarily on vision or predictive simulation, and therefore lack the real-time tactile and force feedback needed for contact-rich manipulation.
Tactile and Force Sensing in Robotics. Touch provides physical interaction cues that vision cannot capture, especially in contact-rich tasks such as cable routing, packing, and needle threading [29]–[32]. Existing tactile methods are either analytic, using tactile signals as constraints or triggers for controllers [33], [34], or data-driven, learning skills through reinforcement learning [35], [36], imitation learning [37]–[41], or tactile-informed MPC [32], [42]. Analytic methods often require task-specific models, while data-driven methods are data-hungry and tend to specialize to the training task. In contrast, our approach uses tactile feedback to adapt object-centric behaviors online, enabling real-time adaptation and zero-shot generalization without task-specific training.
Compositional Modeling for Robotics. Compositionality is central to generalizable robot behavior. Traditional modular approaches, including Task and Motion Planning (TAMP) [43] and layered control architectures [17], provide structure but require hand-specified interfaces and constraints. Monolithic end-to-end policies [4], [44]–[46] reduce manual engineering but learn entangled mappings that struggle with compositional generalization and require retraining. Recent work composes heterogeneous, factorized, or multimodal policies [41], [47], [48], while another line composes foundation models sequentially from text to visual plans and actions [49], [50]. Our method instead composes semantic, visual, and reactive streams in parallel, avoiding both the manual engineering of TAMP and the retraining requirements of end-to-end policies.
3 Methodology↩︎
CoStream separates long-horizon contact-rich manipulation into two levels: stage compilation and per-stage action composition (Fig. 1). An open-vocabulary scene parser converts a language instruction and RGB-D observations into a Scene Summary, which a contact-aware policy compiler expands into a sequence of typed stages. Each stage specifies which behaviors to run and how their outputs should be combined. At runtime, the active stage runs three behaviors, semantic, predictive, and reactive, at different rates over a shared \(SE(3)\) interface, which an action composer fuses into one command for compliant execution. The remainder of this section formalizes the setting and then details each level.
3.1 Problem Formulation↩︎
We consider long-horizon, contact-rich manipulation under partial observability. The robot is given a free-form language instruction \(\mathcal{L}\) and must complete the described multi-stage task through closed-loop interaction. At each step \(t\), the robot receives a multimodal observation \(o_t = (I_t, \hat{\mathcal{X}}_t, \Psi_t)\), where \(I_t\) is RGB-D, \(\hat{\mathcal{X}}_t\) is an object-centric Scene Summary, and \(\Psi_t\) contains tactile and force feedback. The Scene Summary includes object poses, grasp frames, calibrated task frames, point clouds, affordances, symbolic predicates, and robot state. The policy produces actions \(a_t = ({}^{W}\!T_{\mathrm{cmd}}(t), g_t)\), with target end-effector pose \({}^{W}\!T_{\mathrm{cmd}}(t) \in SE(3)\) and gripper command \(g_t\).
3.2 Stage Compilation: Policy Compiler and Interface↩︎
Given the instruction \(\mathcal{L}\) and the initial Scene Summary \(\hat{\mathcal{X}}_0\), a structured policy compiler converts the task into a sequence of stages. For example, the GPU-insertion task in Fig. 1 illustrates a multi-stage compilation, with four representative stages highlighted: pre-grasp, grasp, insert, and home. Each stage \(k\) is then specified by a Stage Specification \(\mathcal{S}^k\) and a Composition Specification \(\mathcal{C}^k\) using a fixed schema.
The Stage Specification contains the information needed to instantiate the active behaviors: the relational objective, reference frames, execution template, guard conditions, and recovery rules. We write this as \(\mathcal{S}^k = (\Phi^k, F^k, E^k, G^k, R^k)\). The Composition Specification contains the information needed to fuse behavior outputs: the composition frame, axis ownership, residual bounds, and fallback behavior. We write this as \(\mathcal{C}^k = (\mathcal{F}^k, \mathcal{O}^k, \mathcal{B}^k, \mathcal{H}^k)\). These fields parameterize the fixed behavior-composition rule defined in Sec. 3.4.
The specifications configure built-in behavior templates and controller profiles rather than introducing free-form code, continuous gains, or force thresholds. During execution, \(\mathcal{S}^k\) parameterizes the online behaviors that produce the task-frame anchor, nominal motion, and tactile residual, while \(\mathcal{C}^k\) parameterizes how these components are framed, projected, bounded, and safely composed. The controller compiler maps the selected profiles to numerical stiffness, admittance axes, residual bounds, and force/torque limits. Sec. 3.5 describes this deterministic controller compilation process, and Appendix 9 lists the calibrated profiles used in our experiments.
3.3 Per-Stage Parallel Action Generation↩︎
The three behaviors are specified by \(\mathcal{S}^k\) and fused according to \(\mathcal{C}^k\). Each emits only the component appropriate to its role (a frame, a motion, or a residual), rather than a complete command.
Semantic behavior: task-frame anchor. The semantic behavior emits a single anchor \({}^{W}\!T_{\mathcal{I}}^{k}\) from the world frame \(W\) to a stage-specific task frame \(\mathcal{I}\), updated once per stage or on reobservation. It grounds \(\Phi^k\) and \(F^k\) by minimizing a weighted \(SE(3)\) alignment objective (Appendix 6) over LLM/VLM outputs, keypoints, calibrated fixtures, templates, or geometric constraints. The anchor is a task frame, not a trajectory, chosen so that motion and compliance decompose along its axes [51], [52]: this keeps axis ownership and stiffness diagonal and the tactile residual bounded. Because the anchor absorbs the object pose, the nominal motion and residual are relative and transfer across object poses and same-class instances (\(SE(3)\)-equivariance) [53].
Predictive behavior: nominal motion. The predictive behavior emits \({}^{\mathcal{I}}\!T_{\mathrm{traj}}^{k}(t)\), a continuous nominal motion in the task frame \(\mathcal{I}\). A video world model [54], conditioned on \(I_t\) and the step instruction \(\mathcal{L}_{\text{step}}\), samples \(K\) rollouts, and a VLM critic selects the rollout \(V^*\) that best satisfies the semantic and safety constraints. A 3D keypoint tracker [55] then lifts \(V^*\) to an object-centric reference relative to \({}^{W}\!T_{\mathcal{I}}^{k}\), forming \({}^{\mathcal{I}}\!T_{\mathrm{traj}}^{k}(t)\). This is a motion prior, not a precise plan; any source that emits a task-frame trajectory can replace it.
Reactive behavior: tactile residual and force loop. The reactive behavior runs at \(25\) Hz. It emits a tactile residual for the composer and drives a force loop realized by the compliant controller. It also flags contact events (slip, excess force, or loss of progress) for the runtime supervisor. The tactile residual rejects in-hand slip: with NormalFlow [56], a GelSight Mini tracks the object’s pose relative to its nominal in-hand pose, and the residual corrects this sliding online [57], keeping the semantic and predictive targets valid relative to the object. The force loop regulates contact from the robot’s estimated end-effector wrench: the controller applies a virtual spring to maintain a target force for sustained-contact stages such as wiping, and position-based admittance to bound the contact force during insertion (Sec. 3.5). Appendix 7 details the tactile estimator and the compliant-control choices.
3.4 Multi-Rate \(SE(3)\) Action Composer↩︎
At each controller tick the composer aligns components produced on different clocks: the anchor \({}^{W}\!T_{\mathcal{I}}^{k}\) is latched for the active stage, the nominal motion \({}^{\mathcal{I}}\!T_{\mathrm{traj}}^{k}(t)\) is interpolated, and the tactile residual \(\Delta T_{\mathrm{tac}}^{k}(t)\) uses the latest in-hand sliding estimate. It first anchors the nominal motion in the world frame, \[{}^{W}\!T_{\mathrm{nom}}^{k}(t) = {}^{W}\!T_{\mathcal{I}}^{k}\,{}^{\mathcal{I}}\!T_{\mathrm{traj}}^{k}(t), \label{eq:nominal95composition}\tag{1}\] and then applies the tactile residual by right-multiplication, \[{}^{W}\!T_{\mathrm{cmd}}^{k}(t) = {}^{W}\!T_{\mathrm{nom}}^{k}(t)\,\Delta T_{\mathrm{tac}}^{k}(t). \label{eq:residual95composition}\tag{2}\] Here \(\Delta T_{\mathrm{tac}}^{k}(t)\) is the rigid-body transform that cancels the latest measured in-hand sliding (Appendix 7), expressed in the composition frame \(\mathcal{F}^k\) from \(\mathcal{C}^k\). The force loop is not composed here: it is realized by the compliant controller (Sec. 3.5), which regulates the end-effector wrench per stage. Because \({}^{W}\!T_{\mathrm{cmd}}^{k}\) is recomputed each tick from the latest latched, sampled, and sensed inputs rather than integrated over time, drift is limited and each behavior updates at its own rate. If a component is missing or stale, the composer applies the per-stage fallback \(\mathcal{H}^k\) from \(\mathcal{C}^k\) (for example, holding the last valid anchor or zeroing the residual), so a dropped stream degrades gracefully rather than stalling the command.
3.5 Controller Compiler and Compliant Execution↩︎
The composer outputs a task-space command; the controller compiler turns \(\mathcal{S}^k\) and \(\mathcal{C}^k\) into the robot-level parameters that execute it: Cartesian stiffness and damping, impedance/admittance axis selection, force/torque and residual bounds, and guard and recovery rules, instantiated from the calibrated profile library keyed by \(\mathcal{S}^k\) rather than emitted by the VLM (Appendices 8, 9). In free space, the controller tracks the anchored nominal motion with high stiffness and loose contact gates. During contact, each stage applies either Cartesian impedance (a virtual spring, to maintain a target force) or position-based admittance (to bound force), with the per-axis stiffness and force regime read from the controller profile in \(\mathcal{S}^k\), the owned axes \(\mathcal{O}^k\) from \(\mathcal{C}^k\) (along which the tactile residual is admitted), and force limits enforced from the wrench estimate. Runtime guards advance the stage on success and trigger recovery, reobservation, or replanning on excess force, no progress, slip, or pose uncertainty.
4 Experiments↩︎
To validate our framework, we conducted real-world experiments on long-horizon, high-precision robotic manipulation tasks. Our evaluation aims to address three core research questions: (RQ1) whether the asynchronous composition of the reactive behavior effectively rejects real-time perturbations to maintain sub-millimeter precision; (RQ2) whether the behavior composition architecture enables zero-shot transitions between tasks with diverse physical constraints without retraining; and (RQ3) whether a single set of behaviors transfers across the diverse contact regimes of CPU, RAM, GPU, and drill insertion without policy retraining or behavior-module redesign.
4.1 Experimental Setup↩︎
Hardware: We utilize a robotic setup with two Franka Emika Panda arms. One arm is equipped with a GelSight Mini tactile sensor for precision manipulation, while the other uses a standard parallel-jaw gripper. External perception is provided by calibrated RealSense D415 cameras. Fig. 2 depicts the robots and sensor suite.
Tasks: We evaluate CoStream on two task groups. The first contains four high-precision assembly tasks inspired by desktop computer assembly: drill insertion with a 0.5 mm clearance, RAM insertion requiring high-force locking, CPU placement requiring gentle flat seating, and GPU insertion requiring long-edge alignment into a PCIe slot. These tasks evaluate tight-clearance contact, robustness, and sequential assembly. The second group contains four everyday manipulation tasks: turning on a lamp, wiping a whiteboard, moving a cup to a plate, and placing clothes into a box, which evaluate broader manipulation behaviors beyond precision assembly.
Baselines We compare against two representative external baselines that instantiate dominant alternatives to CoStream: VoxPoser [23], a modular vision-language planning pipeline, and \(\pi_{0.5}\) [12], a monolithic VLA policy used without task-specific training. These released foundation-model baselines with comparable tactile or force-feedback inputs are not available; we therefore evaluate these accessible methods under their native visual/language interfaces. These comparisons test whether current modular and monolithic foundation-model paradigms can solve the same long-horizon, contact-rich tasks under their standard assumptions. We use internal ablations to isolate the contribution of individual CoStream behaviors and sensing streams.
4.2 Results and Analysis↩︎
Quantitative success rates. Table ¿tbl:tab:results? reports real-world success across high-precision assembly and everyday manipulation tasks. In tight-clearance assembly, CoStream succeeds consistently, while VoxPoser and \(\pi_{0.5}\) fail to complete the tasks. On everyday tasks, CoStream improves over \(\pi_{0.5}\) across all settings, with larger gains on object transfer and smaller gains on lamp switching and whiteboard wiping. Since each setting uses 15 real-world trials, the numbers should be interpreted as task-level evidence rather than precise estimates of success probability. The large gap on assembly tasks nevertheless indicates a consistent qualitative difference between open-loop visual planning, monolithic VLA control, and closed-loop behavior composition.
Failure analysis of baselines. VoxPoser fails before contact. Its affordance maps are constructed from a single visual snapshot and never updated; once the planner commits a trajectory, deviations from the expected geometry have no path back into the system. In our tasks the maps miss the slot orientation by several degrees, and the open-loop trajectory drives the grasped part into the rim of the hole. \(\pi_{0.5}\) fails during contact: mapping observations directly to commands, it cannot separate a contact event from perceptual ambiguity, and on contact responds with high-frequency motion that destabilizes alignment. This is exactly the failure mode CoStream’s reactive behavior eliminates: contact corrections live in a dedicated behavior with their own sensing, rate, and composition term.
Failure analysis of CoStream. The single CPU failure occurred at seating, when the chip rotated outside the tactile contact patch before the reactive term registered it; the single RAM failure occurred at locking, when the downward force briefly exceeded the compiled force-limit envelope. Both sit at the boundary of the reactive behavior’s sensing or compliance envelope, not at the composition, though \(N=15\) is small.
| Method | Success Rate |
|---|---|
| Ours (Full Framework) | 15/15 |
| Ours w/o Reactive Behavior | 3/15 |
r0.5
Qualitative rollouts (RQ1 & RQ2). Fig. 3 shows two rollouts. In drill insertion, the bit stays centered in the 0.5 mm clearance by compliant adaptation while baselines jam. The second is a continuous CPU–RAM–GPU assembly without retraining, where only the object-centric constraints change between subtasks while the behaviors and control loops stay fixed from delicate CPU placement to high-force RAM locking.
Robustness to human perturbation (RQ1). We rotate the held object inside the gripper during insertion to simulate slip. The reactive behavior detects the deviation from the reference normal map and reorients the end-effector to realign (Fig. 4, left, across RAM, CPU, and GPU). The right panel breaks down recovery across the three components (thin RAM surfaces, a small CPU footprint, the GPU’s mass). Mechanism ablation (RQ1). Without the reactive behavior on Drill Insertion (no tactile or force feedback), success collapses (Table 1): the controller cannot reject the angular misalignments inside the 0.5 mm clearance, succeeding only under near-perfect initial alignment.
5 Limitations and Conclusion↩︎
Limitations. CoStream inherits its perception stack: pose estimates depend on upstream modules, and visual prediction degrades when imagined rollouts drift from the scene. Our experiments target structured rigid-object manipulation with calibrated perception, reusable templates, and explicit stage specifications; deformable and articulated objects, dexterous in-hand manipulation, and ambiguous goals remain out of scope. The reactive behavior is limited by tactile coverage and end-effector wrench estimation, so severe slip, out-of-patch contact, or inaccurate force estimates can still cause failure. Extending the composition interface to less structured settings and broader object categories is important future work.
Conclusion. CoStream decomposes long-horizon contact-rich manipulation into a semantic, a predictive, and a reactive behavior, composed by right-multiplication into one pose command per control step and run by a compliant controller. The same behaviors clear four assembly tasks, recover from manual perturbation, and transfer across components without retraining, a regime that monolithic policies and modular pipelines do not reach.
6 Semantic behavior: grounding solver↩︎
The semantic behavior grounds the task-frame anchor \({}^{W}\!T_{\mathcal{I}}^{k}\) by minimizing a weighted sum of \(M\) differentiable geometric residuals over \(SE(3)\), \[{}^{W}\!T_{\mathcal{I}}^{k\,*} = \arg\min_{{}^{W}\!T_{\mathcal{I}} \in SE(3)} \sum_{j=1}^M w_j\, \mathcal{J}_j({}^{W}\!T_{\mathcal{I}}; \hat{\mathcal{X}}_t).\] Here \(\hat{\mathcal{X}}_t\) is the Scene Summary, which supplies the object and fixture poses and detected keypoints used by the residuals. The active residuals \(\{\mathcal{J}_j\}\) and weights \(\{w_j\}\) are instantiated by the compiler from the stage objective \(\Phi^k\) and grounding sources \(F^k\); each \(\mathcal{J}_j\) encodes one geometric relation the anchor must satisfy, such as point alignment, axis or orientation alignment, or a standoff offset. A typical point-alignment residual ties a keypoint expressed in the task frame to one on the target object, \[\mathcal{J}_{\text{align}}({}^{W}\!T_{\mathcal{I}}) = \left\|{}^{W}\!T_{\mathcal{I}}\, \tilde{\mathbf{p}}_{\mathcal{I}} - {}^{W}\!T_{\text{obj}}\, \tilde{\mathbf{p}}_{\text{obj}}\right\|^2_2,\] where \({}^{W}\!T_{\text{obj}}\) is the target object (or fixture) pose from \(\hat{\mathcal{X}}_t\), and \(\mathbf{p}_{\mathcal{I}}, \mathbf{p}_{\text{obj}} \in \mathbb{R}^3\) are corresponding keypoints expressed in the task frame \(\mathcal{I}\) and the object frame, respectively, with \(\tilde{\mathbf{p}}\) denoting homogeneous coordinates. We initialize with Basin Hopping, refine with Sequential Least Squares Programming (SLSQP), and warm-start across timesteps for temporal continuity.
7 Reactive behavior: tactile residual and force loop↩︎
Tactile residual (in-hand slip rejection). We directly track the object’s pose relative to its nominal in-hand pose with NormalFlow [56], a per-frame inverse-compositional estimator [58]. Each frame solves for the warp \(\Delta \xi\) that aligns the current tactile normal map to the nominal-pose reference, \[\Delta \xi^* = \arg\min_{\Delta \xi} \sum_{\mathbf{u} \in \Omega} \left\| R(\Delta \xi)^{-1} I_{\text{obs}}(W(\mathbf{u}; \xi)) - I_{\text{ref}}(\mathbf{u}) \right\|^2,\] where \(\Omega\) is the contact region and \(W\) is the warp induced by twist \(\xi\). Because surface normal maps are scale-ambiguous in depth, we decouple the \(z\) component and recover it by averaging the depth discrepancy over the contact patch, \[\Delta z = \frac{1}{|\Omega|} \sum_{\mathbf{u} \in \Omega} \left[ D_{\text{obs}}(W(\mathbf{u}; \xi^*)) - \left( \mathbf{R}_{\xi^*} \cdot \mathbf{q}(\mathbf{u}) + \mathbf{t}_{\xi^*} \right)_z \right],\] where \(D_{\text{obs}}\) is the depth map from Poisson integration and \(\mathbf{q}(\mathbf{u})\) is the unwarped 3D surface point at pixel \(\mathbf{u}\). The recovered transform \((\Delta\xi^*, \Delta z)\) is the object’s pose relative to its nominal in-hand pose, i.e., its real-time in-hand sliding. The tactile residual is the end-effector motion that compensates this sliding online [57]; it is the rigid-body transform \(\Delta T_{\mathrm{tac}}^{k}\) applied in Eq. 2 .
Force loop (compliant contact regulation). The force loop is realized by the compliant controller from the robot’s estimated end-effector wrench, recovered from the Franka link-side joint-torque sensors mapped through the manipulator Jacobian (\(F_{\mathrm{ext}} = (J^{\top})^{+}\tau_{\mathrm{ext}}\)), with no wrist force/torque sensor. For stages that must sustain a contact force, such as wiping or pressing, a virtual-spring (Cartesian impedance) law regulates the measured wrench toward a target, giving approximate force regulation without dedicated force sensing. For insertion, where the goal is instead to bound contact force, position-based admittance maps the measured wrench to a bounded pose correction. We use position-based admittance rather than torque-level impedance because the end-effector carries additional cameras, tactile sensors, mounts, and cables: the resulting payload and friction mismatch degrade torque-level Cartesian tracking, whereas admittance preserves the robot’s inner position servo while adding bounded, force-responsive motion that lowers the risk of jamming or damaging parts.
8 Controller compilation↩︎
The controller compiler converts \(\mathcal{S}^k\) and \(\mathcal{C}^k\) into a control packet for the low-level compliant controller, \[U^k(t) = \left({}^{W}\!T_{\mathrm{cmd}}^k(t),\, \xi_{\mathrm{cmd}}^k(t),\, K^k,\, D^k,\, A^k,\, \mathcal{B}^k,\, G^k,\, R^k\right),\] where \(\xi_{\mathrm{cmd}}^k(t)\) is the commanded task-space twist, \(K^k\) and \(D^k\) are Cartesian stiffness and damping, \(A^k\) selects impedance/admittance behavior along controlled axes, \(\mathcal{B}^k\) contains force, torque, residual-translation, residual-rotation, and timeout bounds, and \(G^k, R^k\) define guard evaluation and fallback behavior. Stiffness is high in free space and lowered, or switched to admittance, on the axes the Composition Specification marks compliant.
9 Controller and guard profiles↩︎
The policy compiler chooses a per-stage controller profile and guard profile by identifier. The controller compiler then maps each symbolic level (low, medium, high) to a robot-specific numerical value, calibrated once per platform from the robot, end-effector, and object-class limits with a safety margin. The LLM/VLM never produces these numbers directly: it picks a profile, not a gain or a threshold. This keeps the action space auditable and the contact behavior within calibrated safety limits. Tables 2 and 3 give a representative library; the assembly stages in this paper use the approach, insertion, press, and wiping profiles.
| Profile | Mode | Compliant axes | Stiffness / force regime |
|---|---|---|---|
| free_space | Impedance | none | High translation and rotation |
| guarded_approach | Impedance | progress | Medium, gated on contact |
| compliant_insertion | Impedance / admittance | lateral, roll, pitch | Low lateral and roll/pitch, medium progress |
| high_force_press | Admittance | normal | Low normal stiffness, force-controlled normal axis |
| surface_wiping | Admittance | surface normal | Force-controlled normal, free tangential motion |
| Profile | Normal-force limit | Torque limit | No-progress timeout |
|---|---|---|---|
| delicate_placement | Low | Low | Short |
| rigid_insertion | Medium | Medium-low | Medium |
| high_force_locking | High (bounded) | Medium | Short |
| surface_wiping | Medium band | Medium | Disabled (sustained contact) |
10 Perception module↩︎
The Scene Summary \(\hat{\mathcal{X}}\) consumed by the policy compiler (Sec. [sec:interface]) is produced by an open-vocabulary perception stack. We adopt the two-branch design of TiPToP [59]: a 3D vision branch recovers metric geometry, and a semantic branch grounds the instruction to objects. We then upgrade both branches and extend their fused output, as detailed below.
3D vision branch. From the calibrated RGB-D streams, FoundationStereo [60] estimates a dense scene point cloud, and M2T2 [61] predicts candidate \(6\)-DoF grasps from it. Masking the dense cloud with the semantic branch below yields per-object point clouds, and convex-hull reconstruction completes object geometry, following TiPToP [59]. The resulting meshes support collision checking and grasp-frame estimation.
Semantic branch. Gemini Robotics-ER 1.6 [62], [63] detects task-relevant objects and grounds the instruction \(\mathcal{L}\) to symbolic goal propositions. SAM 3 [64] then lifts these detections to per-object masks through concept prompts that index into the point cloud. This branch supplies the object identities, language-grounded targets, and symbolic predicates of \(\hat{\mathcal{X}}\).
Scene Summary construction. Fusing the two branches yields an object-centric record per task-relevant object: pose, completed mesh, and candidate grasps. We annotate each record with three further fields. Affordances, such as an insertable bore, a slot, or a graspable edge, are attached to specific object parts. Interaction and task frames are attached to those affordances, and seed the semantic behavior’s anchor \({}^{W}\!T_{\mathcal{I}}^{k}\). Available geometry sources, such as calibrated fixtures, object-centric templates, or estimated poses, are tagged per object. These tags let the compiler judge which frames are reliable for a high-precision stage. Together, the three fields let \(\hat{\mathcal{X}}\) compile into Stage and Composition Specifications without a separate grounding pass.
Relation to TiPToP. Our stack differs from TiPToP [59] in both components and purpose. TiPToP pairs Gemini Robotics-ER 1.5 with SAM-2, and feeds the resulting meshes and symbolic goals to a TAMP planner. We instead use Gemini Robotics-ER 1.6, whose spatial and physical reasoning improves over 1.5 [62], [63]. We also replace SAM-2 with SAM 3, whose concept-promptable segmentation grounds open-vocabulary objects more reliably [64]. More importantly, our added annotations let \(\hat{\mathcal{X}}\) compile directly into Stage and Composition Specifications for contact-rich behavior composition. The same representation in TiPToP instead feeds a motion planner.
Limitations. The stack builds the Scene Summary once from the initial views, so errors in the estimated poses or frames propagate to every stage. The reactive behavior corrects only contact-observable deviations on its owned axes, not global perception errors. High-precision frames therefore still rely on calibration or object-centric templates rather than purely open-vocabulary estimation. Convex-hull completion yields coarse geometry, sufficient for collision checking but not for fine reasoning on concave parts. Finally, the semantic branch depends on a proprietary model (Gemini Robotics-ER 1.6) served through an API, which constrains reproducibility, offline use, and latency.
11 Sensing hardware and calibration↩︎
The sensing stack consists of two calibrated RealSense D415 cameras mounted at fixed external viewpoints and a GelSight Mini tactile sensor on one of the two Franka end-effectors. Camera extrinsics are obtained by an ArUco-board hand-eye calibration repeated whenever the workspace is reconfigured. Object pose estimates that feed the semantic and predictive behaviors are produced by an upstream perception module adapted from prior work; downstream behaviors treat these poses as part of the externally provided Scene Summary \(\hat{\mathcal{X}}_t\), and the composition interface is agnostic to the specific pose source. The tactile stream is read at 25 Hz (the GelSight Mini readout rate), and the surface normal map used by the reactive behavior is computed per frame via Poisson integration on the raw gel image. The reactive behavior consumes only the tactile stream and does not depend on the external cameras during contact, which is what allows the corrective loop to continue when an object is occluded by the gripper.
12 Visual Trajectory Generation↩︎
