1 Introduction↩︎

Teleoperation of robotic arms has made significant progress in recent years, improving both real-time performance and usability. Modern teleoperation systems often rely on high-bandwidth interfaces such as shadow arms [1], virtual reality devices [2], or motion capture systems [3], enabling users to control robots with high precision. Such systems provide an important pathway for assistive manipulation, allowing individuals to interact with and manipulate objects through robotic devices. However, these interfaces are often inaccessible to users with motor impairments. Instead, such users typically rely on low-bandwidth and noisy input devices, such as chin joysticks [4] or head joysticks [5]. Emerging neural interfaces [6], [7] further offer a promising alternative for users with severe motor impairments, but they remain limited by low-bandwidth and noisy control signals. Controlling high-degree-of-freedom (high DoF) robotic manipulators with these interfaces is challenging. Performing manipulation tasks such as grasping objects from specific orientations often requires numerous commands, making teleoperation slow, cognitively demanding, and prone to error. Shared control addresses this challenge by combining user input with contextual information to infer user intent and assist task execution. By integrating human commands with environmental perception, shared control systems can reduce user effort while preserving user authority, enabling more efficient human–robot collaboration.

Shared autonomy methods have demonstrated promising results in planar navigation [8], [9] and simplified manipulation tasks [10]. However, shared control for robotic grasping in realistic environments remains challenging. Two key factors contribute to this difficulty.

The complexity of the environment. Robotic manipulation often occurs in cluttered environments containing multiple objects with diverse affordances. Assistive algorithms must reason about which objects are actionable and infer the user’s intended target among several candidates. Prior work often simplifies this setting by considering only one or two sparsely placed objects [11]–[13], which does not reflect real-world scenarios. Furthermore, a single object may support multiple valid grasp poses depending on the intended task, whereas many existing approaches assume a single grasp configuration per object [10], [14]–[16]. Such simplifications limit the ability of shared control systems to accurately capture user intent.

Low-bandwidth user input on a high DoF system. Shared control systems must bridge the mismatch between low-bandwidth user input and high-DoF robotic manipulators. Manipulation tasks typically require controlling a full \(SE(3)\) end-effector pose, while assistive interfaces provide only a small number of control channels. As a result, directly commanding all degrees of freedom is often inefficient and exhausting for users. Existing work commonly reduces the problem by restricting the manipulation strategy, such as enforcing top-down grasps [10], [14] or side grasps [11], [13]. While these simplifications reduce control complexity, they limit the diversity of achievable grasps and restrict the range of tasks that can be performed.

To address these challenges, we propose Robot Trajectron V3 (RT-V3), a probabilistic shared control framework for multi-object, multi-affordance \(SE(3)\) grasping assistance under low-bandwidth user interfaces. RT-V3 formulates shared control within a Bayesian inference framework: \[p(i|u,c) \propto p(u|i)p(i|c), \label{eq:32psc}\tag{1}\] where \(i\) denotes the user’s behavioral intent, represented as a future trajectory, \(u\) denotes the current user command, and \(c\) denotes the environmental context. The prior \(p(i|c)\) models likely user behaviors given the scene context, while the likelihood \(p(u|i)\) captures the uncertainty of the user interface. Shared control is achieved by estimating the posterior intent distribution and assisting the user accordingly. The context representation \(c\) includes both geometric and affordance information. We represent scene geometry using point clouds and encode candidate grasp configurations as grasp pose clouds generated by a grasp planner. Based on this representation, the prior model predicts plausible future trajectories conditioned on the scene context. The proposed RT-V3 builds upon the Robot Trajectron V2 [17] navigation shared autonomy framework, extending it to the domain of robotic manipulation.

To effectively model this prior, we introduce a transformer-based architecture that encodes the unstructured \(SE(3)\) scene representation. The model jointly reasons over point clouds, grasp candidates, and robot dynamics through cross-attention mechanisms, enabling the system to identify relevant objects and affordances for intent inference. In addition, we factorize the action distribution into a translational component and a rotational component conditioned on translation. This factorization reduces the complexity of density estimation and enhances stability during closed-loop execution.

Finally, we introduce an asynchronous shared control mechanism that leverages periods without user input to update intent inference and shared policies. By continuously refining the posterior estimate of user intent, the system can proactively assist the user and reduce the number of required commands, alleviating fatigue associated with low-bandwidth interfaces.

The contributions of this work are threefold:

• A probabilistic shared control framework for multi-object, multi-affordance \(SE(3)\) manipulation, extending Robot Trajectron from navigation to robotic grasping.

• A transformer-based architecture for encoding unstructured \(SE(3)\) scene context, enabling reasoning over point clouds and grasp affordances for intent prediction.

• A factorized action representation and asynchronous shared control mechanism, which improves sample efficiency, closed-loop stability, and user interaction efficiency.

We evaluate RT-V3 through trajectory prediction benchmarks, closed-loop planning experiments, and human user studies using joystick teleoperation. The results demonstrate that RT-V3 achieves accurate intent prediction and significantly improves shared control performance in terms of efficiency and user agreement.

2 Related Work↩︎

2.1 Shared Control↩︎

Shared control involves the cooperative determination of a policy by both an autonomous agent and the user, with the goal of improving performance and safety in robot manipulation. This approach is widely utilized in various applications, including assistive driving [18], wheelchair control [19], and robotics manipulation [20]. Shared control for high-DoF robotic arms has been widely explored. Gottardi et al. [10] proposed an improved artificial potential field method to guide the robot toward likely goals while avoiding obstacles. Robot Trajectron [16] predicts the future end-effector trajectory from recent motion dynamics, enabling proactive support for reaching tasks. Oh et al. [13] regard shared control as a Q-value maximization problem with a divergence constraint between the robot and the shared policy, and solve the problem by natural gradients. However, these methods are limited to 2D or 3D translation space, and providing full \(SE(3)\) assistance is still challenging. Hindsight Optimization (HO) [21] provides an inverse reinforcement learning (IRL) perspective to solve the intent estimation problem, which has inspired numerous shared control approaches and shown promising performance even in complex, cluttered environments [10], [22]–[24]. However, hindsight optimization can only predict the probability for each potential target. Therefore, planning and policy blending to transform the target probabilities into real-time action guidance are still needed. Besides, HO’s predictions largely rely on Euclidean distance between the current position and the potential target poses without a consideration of motion dynamics [16] and collision avoidance, leading to inaccurate predictions sometimes.

Recently proposed Robot Trajectron V2 (RT-V2) [17] provides a probabilistic shared control framework. Instead of predicting the final target, RT-V2 seeks a behavioral modeling perspective to solve the shared control problem. It models the user’s behavior patterns, predicts collision-free future actions, and blends the user’s command via posterior estimation. However, RT-V2 is limited to 2D translation space. In this work, we propose Robot Trajectron V3, which lifts RT-V2 from 2D translation space to \(SE(3)\). Compared to RT-V2, which uses CNNs to encode the context in occupancy map format, RT-V3 leverages a transformer-based encoder to encode point clouds and grasp poses. Besides, RT-V3 factorizes the action distribution into a translation distribution and a rotation distribution conditioned on the translation.

2.2 Imitation Learning↩︎

Imitation Learning (IL), also referred to as Learning from Demonstration (LfD) or Behavior Cloning (BC), aims to train policies that mimic expert behavior. IL has been widely applied across robotics domains due to its efficiency in leveraging demonstrations for data collection and policy training [25]–[27]. Despite its appeal, several challenges remain in adapting IL to complex robotic manipulation tasks.

Modeling multimodal action distributions. Robotic tasks often admit multiple valid strategies, making the underlying action distribution inherently multimodal. Generative models are well-suited to capture such distributions, and recent work has integrated them into IL. Conditional variational autoencoders (CVAEs) [28], energy-based models [29], and diffusion models [30] have all been proposed to represent multimodal expert behaviors. These approaches have demonstrated strong performance on standard robotic benchmarks such as RLBench [31], LIBERO [32], and CALVIN [33].

Context representation. A second key design choice in IL is how the environment is encoded. RGB cameras are widely used due to their low cost and accessibility, with visual features commonly extracted using CNNs. More recently, large-scale pretrained transformers such as DINOv2 [34] have emerged as powerful visual encoders, offering compatibility with transformer-based policies [28], [35]. Depth cameras provide complementary 3D information, enabling point cloud representations that can be processed by architectures such as PointNet [36] and Point Transformer [37]–[39]. Alternatively, volumetric encodings such as truncated signed distance functions can be learned with 3D CNNs [40]. Selecting the appropriate context modality and representation remains an open challenge, as it strongly influences generalization and sample efficiency in IL.

Action representation. Finally, IL performance depends critically on how the robot’s action space is represented. Options vary along several dimensions: the space of execution (e.g., \(SE(2)\) vs. \(SE(3)\)); the type of control (position [30] vs. velocity [41]); continuity (continuous [42], [43] vs. discrete [44]); and the reference frame (absolute [30] vs. relative [40]). For \(SE(3)\) actions, rotation can be parameterized in multiple ways, such as Euler angles [45], quaternions [46], or 6D rotation vectors [42]. The choice of action representation not only affects downstream performance but also constrains model architecture and learning dynamics.

Inspired by the IL works mentioned above, RT-V3 adopts a CVAE framework to capture the multimodality of user behavior. To handle unstructured \(SE(3)\) contextual data, including grasp poses and point clouds, we employ a transformer-based encoder that integrates spatial and temporal information. Finally, to remain compatible with user twist commands and the requirements of shared control, RT-V3 defines its action space in \(\mathfrak{se}(3)\) velocity, allowing the policy to directly predict translational and rotational velocities in a geometrically consistent manner.

3 Methodology Overview↩︎

In this work, we study shared control for robotic grasping, where a human user operates a robot to grasp an object in the workspace with assistance from an autonomous controller. The controller does not know in advance which object the user intends to grasp or how the grasp should be executed. Instead, it must infer the user’s intent from past input commands, the robot’s motion history, and contextual sensory information.

We leverage off-the-shelf grasp planners [46], [47], which provide a dense set of candidate grasp poses \(\mathcal{G}\) for objects in the scene. We assume that this dense set of candidate grasp poses includes the grasp poses that the user desires to execute. This assumption allows us to formulate shared-control grasping as a goal-pose reaching task: the grasping workspace can be abstracted as a set of \(M\) candidate destinations \(\mathcal{G} \in \mathbb{R}^{M \times 3}\) and a point cloud with \(N\) points \(\mathcal{P} \in \mathbb{R}^{N \times 6}\). Operating within this defined workspace, the user and the robot collaborate through a shared-control paradigm. To explicitly delineate these collaboration dynamics—aligning with the taxonomy established by Losey et al. [48]—we distinguish between the user’s unobservable goal, the measurable interface signals, and the system’s internal prediction. During operation, the user possesses an unobserved ultimate goal (one of the candidate destinations) that they desire to reach. They express this intent by issuing continuous user commands (\(\boldsymbol{u}\)), which are instantaneous 6D translational and rotational velocity signals transmitted through a physical, inherently noisy interface (e.g., joystick, ...). The assistive controller, RT-V3, gathers these noisy commands along with the environmental context to iteratively compute an estimated intent (\(\boldsymbol{i}\)). Rather than just predicting a discrete target, this estimated intent is modeled as an intended future trajectory—a sequence of desired, noise-free velocity actions that bridge the robot’s current state to the ultimate goal. By separating the underlying signal from the interface noise, the system ultimately issues goal-directed robot velocity commands that realize the perceived user intent.

Our method approaches shared control from a Bayesian perspective. The methodology, detailed in Sections 4 through 6, is organized as follows:

Section 4: We detail the formulation of the probabilistic shared control framework described in Eq. 1 . In this framework, shared control is achieved by posterior estimation \(p(i|u,c)\), where the prior model for posterior estimation \(p(i|c)\) is obtained by training a model in a supervised way to predict how the onset of a robot motion is likely to continue. The model’s input is a “past” trajectory and contextual cues, and its output is the robot’s “future” trajectory.

Section 5: We explain the parametrization of the proposed prior model. Specifically, the prior model is based on conditional variational auto-encoder frameworks, in which we use RNNs to encode the dynamic information and use a transformer-based architecture to encode the unstructured context. Besides, in the decoding of the future trajectories, we propose a translation-conditioned rotation architecture, which simplifies density estimation and improves stability during closed-loop execution.

Section 6: This section will explain how to achieve posterior estimation \(p(i|u,c)\) of the future trajectories given the current user input in our parametrization setting mentioned in Sec. 5. Specifically, we adopt an agentic perspective, where the user provides a noisy command \(u\) at each timestep. We model the likelihood \(p(u|i)\) and combine it with the prior \(p(i|c)\) (the intent estimator) to obtain the posterior \(p(i|u,c)\).

4 Probabilistic Shared Control↩︎

In this section, we detail the theoretical formulation of the probabilistic shared control framework of RT-V3. Note that this section focuses purely on the mathematical objectives—specifically, defining the prior and the posterior distribution. The specific neural network parameterizations used to learn this prior are deferred to Sec. 5, and the computation of the posterior during real-time teleoperation is detailed in Section Sec. 6.

The prior model of RT-V3 is designed to encode the user’s inherent motion behaviors. Let us denote with the variable \(\boldsymbol{s}_t\) the state at \(t\), incorporating the current dynamics and contextual information, and with \(\boldsymbol{a}_t\) the velocity action at \(t\). Using \(t=0\) to represent the current timestep, we write the intended trajectory as \(\boldsymbol{\zeta} = (\boldsymbol{s}_{-T+1:H},\boldsymbol{a}_{-T+1:H-1})\), in which \((\boldsymbol{s}_{-T+1:0},\boldsymbol{a}_{-T+1:-1})\) is the past trajectory and \((\boldsymbol{s}_{1:H},\boldsymbol{a}_{0:H-1})\) is the future trajectory. The intended trajectories can be sampled from the interaction between the user and the environment with failure samples filtered out. The prior model is obtained via the following goal: \[\mathop{\textrm{max}}~\mathbb{E}_{\boldsymbol{\zeta}} [\textrm{log}~p(\boldsymbol{s}_{1:H}, \boldsymbol{a}_{0:H-1}|\boldsymbol{s}_{-T+1:0} )]. \label{eq:32mle}\tag{2}\] The goal of Eq. 2 is to maximize the log-likelihood of the intended trajectories. To decompose Eq. 2 , we introduce a latent state \(\boldsymbol{h}_t\), which encodes a memory of past states \(\boldsymbol{s}_{\leq t}\), then we can simplify \(p(\boldsymbol{s}_{1:H}, \boldsymbol{a}_{0:H-1}|\boldsymbol{s}_{-T+1:0})\) as \(p(\boldsymbol{h}_{0:H}, \boldsymbol{a}_{0:H-1}|\boldsymbol{s}_{-T+1:0})\). We assume that the latent state obeys a Markovian property: the latent state \(\boldsymbol{h}_t\) only depends on the previous latent state \(\boldsymbol{h}_{t-1}\) and the previous action \(\boldsymbol{a}_{t-1}\). With this assumption, we can decompose \(p(\boldsymbol{h}_{0:H}, \boldsymbol{a}_{0:H-1}|\boldsymbol{s}_{-T+1:0})\) as: \[\begin{align} & p(\boldsymbol{h}_{0:H}, \boldsymbol{a}_{0:H-1}|\boldsymbol{s}_{-T+1:0})\\ & = p(\boldsymbol{h}_0|\boldsymbol{s}_{-T+1:0}) \prod_{t=0}^{H-1} {p(\boldsymbol{h}_{t+1}|\boldsymbol{h}_{t}, \boldsymbol{a}_{t})} {p(\boldsymbol{a}_{t}|\boldsymbol{h}_{t})}, \label{eq:32decomposed32mle2} \end{align}\tag{3}\] where \({p(\boldsymbol{h}_{t+1}|\boldsymbol{h}_{t}, \boldsymbol{a}_{t})}\) and \({p(\boldsymbol{a}_{t}|\boldsymbol{h}_{t})}\) are the state transition distribution and the action distribution, respectively.

To model the high multi-modality of the user’s intent, we mimic the CVAE framework [16], [49] and introduce a discrete latent variable \(\boldsymbol{z}\), which obeys a categorical distribution, to facilitate the encoding of a low-dimensional, multi-modal representation, as: \[\begin{align} & p(\boldsymbol{h}_{0:H}, \boldsymbol{a}_{0:H-1}|\boldsymbol{s}_{-T+1:0})\\ & = p(\boldsymbol{h}_{0}|\boldsymbol{s}_{-T+1:0}) p(\boldsymbol{h}_{1:H},\boldsymbol{a}_{0:H-1}|\boldsymbol{h}_{0})\\ & = \sum_{\boldsymbol{z}} p(\boldsymbol{h}_{0}|\boldsymbol{s}_{-T+1:0}) p(\boldsymbol{h}_{1:H}, \boldsymbol{a}_{0:H-1}|\boldsymbol{h}_{0},\boldsymbol{z}) p(\boldsymbol{z}|\boldsymbol{h}_{0})\\ & = p(\boldsymbol{h}_{0}|\boldsymbol{s}_{-T+1:0}) \sum_{\boldsymbol{z}} p(\boldsymbol{z}|\boldsymbol{h}_{0})\\ &~~~~~~~~~~~~~~\prod_{t=0}^{H-1} {p(\boldsymbol{h}_{t+1}|\boldsymbol{h}_{t}, \boldsymbol{a}_{t}, \boldsymbol{z})} {p(\boldsymbol{a}_{t}|\boldsymbol{h}_{t}, \boldsymbol{z})}. \label{eq:32decomposed32mle32with32latent} \end{align}\tag{4}\] The graphical model of Eq. 4 can be represented in Fig. 1 (a). This latent variable \(\boldsymbol{z}\) denotes the maneuver class reflecting the probable maneuver directions for the user to execute [17]. Compared to Eq. 3 , Eq. 4 introduce a maneuver class prior \(p(\boldsymbol{z}|\boldsymbol{h}_{0})\), and rewrite the \({p(\boldsymbol{h}_{t+1}|\boldsymbol{h}_{t}, \boldsymbol{a}_{t})}\) and \({p(\boldsymbol{a}_{t}|\boldsymbol{h}_{t})}\) as \({p(\boldsymbol{h}_{t+1}|\boldsymbol{h}_{t}, \boldsymbol{a}_{t}, \boldsymbol{z})}\) and \({p(\boldsymbol{a}_{t}|\boldsymbol{h}_{t}, \boldsymbol{z})}\), respectively.

To train the model, we maximize the likelihood in Eq. 4 by minimizing, per CVAE practice [49], [50], the \(\beta\)-weighted evidence-based lower bound (ELBO) loss: \[\begin{align} & \mathcal{L}_{\text{ELBO}} \\ & = ~ -\mathbb{E}_{\boldsymbol{z} \sim p(\boldsymbol{z}|\boldsymbol{h}_{H})}[\textrm{log}~p(\boldsymbol{h}_{0:H}, \boldsymbol{a}_{0:H-1}|\boldsymbol{s}_{-T+1:0}, \boldsymbol{z})] \\ &~~ + \beta D_{KL}(q(\boldsymbol{z}|\boldsymbol{h}_{H})||p(\boldsymbol{z}|\boldsymbol{h}_{0})) \\ & = - \mathbb{E}_{\boldsymbol{z} \sim p(\boldsymbol{z}|\boldsymbol{h}_{H})}[\textrm{log}~p(\boldsymbol{h}_{0}|\boldsymbol{s}_{-T+1:0})+ \\ & ~~~~~~~~~~~~~~~\sum_{t=0}^{H-1} \textrm{log}~ p(\boldsymbol{h}_{t+1}|\boldsymbol{h}_{t}, \boldsymbol{a}_{t},\boldsymbol{z}) p(\boldsymbol{a}_{t}|\boldsymbol{h}_{t},\boldsymbol{z})] \\ &~~ + \beta D_{KL}(q(\boldsymbol{z}|\boldsymbol{h}_{H})||p(\boldsymbol{z}|\boldsymbol{h}_{0})). \label{eq:32ELBO} \end{align}\tag{5}\] where \(\boldsymbol{h}_{H}\) is the latent state concluding both the past and future trajectories, and we propose to use a maneuver class posterior \(q(\boldsymbol{z}|\boldsymbol{h}_{H})\) to guide the training of maneuver class prior \(p(\boldsymbol{z}|\boldsymbol{h}_{0})\) by minimizing their KL divergence. By training with the loss function described in Eq. 5 , RT-V3 models the decision-making distribution \(p(\boldsymbol{h}_{0:H},\boldsymbol{a}_{0:H-1}|\boldsymbol{s}_{-T+1:0})\), capturing the behavioral pattern of the intention policy, the ideal policy envisioned by the user.

At the current timestep, we observe the user command \(\boldsymbol{a}^{\text{u}}_{0}\), which is generated through an uncertain interface. To achieve a more accurate estimation of the intended action sequence \(\{\boldsymbol{a}_t\}_{t=0}^{H-1}\), we incorporate the observed user command \(\boldsymbol{a}^{\text{u}}_{0}\) and infer the posterior decision-making distribution as follows: \[p(\boldsymbol{h}_{0:H}, \boldsymbol{a}_{0:H-1}|\boldsymbol{s}_{-T+1:0}, \boldsymbol{a}^{\text{u}}_{0}). \label{eq:32posterior32decision32making}·\tag{6}\] Taking the argmax of the posterior estimate in Eq. 6 yields the intended action sequence \(\boldsymbol{a}_{-T+1:H-1}\), allowing us to execute the initial action \(\boldsymbol{a}_{0}\). However, computing this posterior directly is intractable without a specific parameterization of the prior and the likelihood. Therefore, we first detail the parameterization of our learned prior in Sec. 5, followed by the tractable computation of the posterior distribution in Sec. 6.

5 \(SE(3)\) Grasp-conditioned Policy↩︎

In this section, we detail the parameterization of the prior model that clones the user’s behavioral patterns, described in Eq. 4 . The proposed RT-V3’s framework is shown in Fig. 2. We assume a static environment in which the context does not change. With this assumption, we can factorize the past state trajectory \(\boldsymbol{s}_{-T+1:0}\) into a past dynamic trajectory \(\boldsymbol{x}_{-T+1:0}\) and a static context \(\boldsymbol{c}\), which consists of the point clouds of the scene \(\mathcal{P}\) and the candidate destinations \(\mathcal{G}\). Thus, \(\boldsymbol{h}_0\) in Eq. 4 can be factorized as \([\boldsymbol{h}^\textrm{d}_0, \boldsymbol{h}^\textrm{c}]\), and \(\boldsymbol{h}_H\) can be factorized as \([\boldsymbol{h}^\textrm{d}_0, \boldsymbol{h}^{+}, \boldsymbol{h}^\textrm{c}]\) in Eq. 5 , where \(\boldsymbol{h}^\textrm{d}_0\) is the past dynamic encoding, \(\boldsymbol{h}^\textrm{c}\) is the contextual encoding, and \(\boldsymbol{h}^{+}\) is the future encoding. The following subsections detail the extraction of these encodings.

All poses, velocities, and point cloud observations are expressed in the coordinate frame of the current end-effector pose. This egocentric representation reduces variability in the data and implicitly enforces \(SE(3)\)-equivariance of the learned policy. The proof for \(SE(3)\)-equivariance can be found in the Appendix.

5.1 Dynamics Encoding↩︎

Given the past dynamic trajectory \(\boldsymbol{x}_{-T+1:0}\), where \(\boldsymbol{x}_t = [X_{t}, \dot{X}_{t}] \in \mathbb{R}^{12}\) is the dynamics information of the robot (i.e., the pose and the twist of the end-effector expressed in the Lie algebra), we use LSTM to obtain the past dynamics encoding \(\boldsymbol{h}^{\textrm{d}}_0\), as: \[\boldsymbol{h}^{\textrm{d}}_0 = \textrm{LSTM}(\boldsymbol{x}_{-T+1:0}).\] Additionally, the future dynamics encoding \(\boldsymbol{h}^{+}\) is obtained by using BiLSTM to encode \(\boldsymbol{x}_{1:H}\), as: \[\boldsymbol{h}^{+} = \textrm{BiLSTM}(\boldsymbol{x}_{1:H}).\]

5.2 Context Encoding↩︎

RT-V3 encodes two sources of context: point clouds \(\mathcal{P}\) and candidate grasps \(\mathcal{G}\). The feasible grasps can be obtained from off-the-shelf grasp planners [47]. We aim to obtain a contextual encoding \(\boldsymbol{h}^{\textrm{c}}\), which consists of the point cloud encoding \(\boldsymbol{h}^{\textrm{pcl}}\) and the grasp encoding \(\boldsymbol{h}^{\textrm{grasp}}\).

Point cloud encoding. When the user controls the robot, they tend to move the robot to the desired position without a collision with objects in the workspace. Thus, the point clouds provide information about how the user would move the robot. As shown in Fig. 2, we use PT-V3 [39] as an encoder to preliminarily encode the point clouds, as: \[\boldsymbol{f}^{\textrm{pcl}}=\mathop{\textrm{PT-V3}}(\mathcal{P}),\] where \(\boldsymbol{f}^{\textrm{pcl}} \in \mathbb{R}^{\tilde{M}\times C_{\textrm{p}}}\), \(\tilde{M}\) is the number of points in the pooled point clouds, and \(C_\textrm{p}\) is the number of the encoded feature’s channels. The initial point feature \(\boldsymbol{f}^{\textrm{pcl}}\) summarizes the object geometry of the workspace. We propose to use a transformer decoder to decode \(\boldsymbol{h}^{\textrm{pcl}}\) with the past dynamics encoding as a query, and decode the point cloud encoding as: \[\boldsymbol{h}^{\textrm{pcl}}=\mathop{\textrm{TransformerDecoder}}(\boldsymbol{f}^{\textrm{pcl}},\boldsymbol{h}_0^{\textrm{d}}).\] Powered by the cross-attention mechanism in the transformer decoder, we can dynamically extract the geometry information according to the current dynamics.

Grasp encoding. We assume that the dense set of candidate grasp poses from the off-the-shelf grasp planner includes the grasp poses that the user desires to execute. Thus, providing the model with grasp information is essential for predicting the future trajectories. Similar to how point cloud encoding is obtained, we use a transformer encoder to encode the grasps, followed by a transformer decoder with the past dynamics encoding as a query, as: \[\begin{align} & \boldsymbol{f}^{\textrm{grasp}}=\mathop{\textrm{TransformerEncoder}}(\mathcal{G})\\ & \boldsymbol{h}^{\textrm{grasp}}=\mathop{\textrm{TransformerDecoder}}(\boldsymbol{f}^{\textrm{grasp}},\boldsymbol{h}_0^{\textrm{d}}). \end{align}\] Similarly, the cross-attention mechanism in the transformer decoder allows the model to adaptively focus on the most probable grasps and predict trajectories towards them. Finally, we obtain the context encoding \(\boldsymbol{h}^{\textrm{c}}=[\boldsymbol{h}^{\textrm{pcl}};\boldsymbol{h}^{\textrm{grasp}}]\).

5.3 Maneuver Class Sampling↩︎

We model maneuver class prior \(p(\boldsymbol{z}|\boldsymbol{h}_0)\) and posterior \(q(\boldsymbol{z}|\boldsymbol{h}_H)\) as categorical distributions whose parameters are generated with multi-layer perceptrons (MLPs), as: \[\begin{align} & B_{\textrm{prior}} = \textrm{MLP}(\boldsymbol{h}_{0}),\\ & B_\textrm{sec:posterior} = \textrm{MLP}(\boldsymbol{h}_{H}). \end{align}\] We sample \(\boldsymbol{z}\) from \(p(\boldsymbol{z}|\boldsymbol{h}_0)\) during the training, while from \(q(\boldsymbol{z}|\boldsymbol{h}_H)\) during the inference.

5.4 Decoding Future Trajectory↩︎

When decoding the future trajectory, we use an LSTM model to encode the latent dynamics, as: \[\boldsymbol{h}_{t+1} = \textrm{LSTM}_t(\boldsymbol{h}_{t},\boldsymbol{a}_{t},\boldsymbol{z}).\] The action \(\boldsymbol{a}_t\) is an \(\mathfrak{se}(3)\) twist \([\boldsymbol{v}_t, \boldsymbol{w}_t] \in \mathbb{R}^6\), a concatenation of translational and rotational velocity. Modeling \(\mathfrak{se}(3)\) actions as a joint 6D distribution is challenging due to the high dimensionality and strong coupling between translational and rotational components. Moreover, in many manipulation tasks, translational motion determines the global trajectory structure, while orientation adjustments are typically performed as local refinements near the target pose. Motivated by these observations, we factorize the action distribution \(p(\boldsymbol{a}_t|\boldsymbol{h}_t, \boldsymbol{z})\) into a translational component \(p(\boldsymbol{v}_t|\boldsymbol{h}_t, \boldsymbol{z})\) and a rotation component conditioned on translation \(p(\boldsymbol{w}_t|\boldsymbol{h}_t, \boldsymbol{z},\boldsymbol{v}_t)\), as: \[p(\boldsymbol{a}_t|\boldsymbol{h}_t, \boldsymbol{z})=p(\boldsymbol{w}_t|\boldsymbol{h}_t, \boldsymbol{z},\boldsymbol{v}_t)p(\boldsymbol{v}_t|\boldsymbol{h}_t, \boldsymbol{z}). \label{eq:32factorization321}\tag{7}\] This design simplifies density estimation and improves stability during closed-loop execution.

Specifically, we model the translational distribution \(p(\boldsymbol{v}_t|\boldsymbol{h}_t, \boldsymbol{z})\) as a Gaussian distribution. For a given maneuver class \(\boldsymbol{z}\), the parameters \(G^\textrm{v}_t(\boldsymbol{z}) = (\boldsymbol{\mu}_t^{z}, \boldsymbol{\Sigma}_t^{z})\) are generated via an MLP: \[(\boldsymbol{\mu}_t^{z}, \boldsymbol{\Sigma}_t^{z}) = \textrm{MLP}(\boldsymbol{h}_{t}, \boldsymbol{z}). \label{eq:velocitygmm}\tag{8}\] The translational action at time \(t\) is then sampled as \(\boldsymbol{v}^{z}_{t} \sim \mathcal{N}(\boldsymbol{\mu}_t^{z}, \boldsymbol{\Sigma}_t^{z})\).

Similarly, the rotational distribution \(p(\boldsymbol{w}_t|\boldsymbol{h}_t, \boldsymbol{z}, \boldsymbol{v}_t)\) is modeled as a Gaussian conditioned on the sampled translation \(\boldsymbol{v}_t\). The parameters \(G^\textrm{w}_t(\boldsymbol{z}) = (\boldsymbol{\omega}_t^{z}, \boldsymbol{\Gamma}_t^{z})\) are decoded as: \[(\boldsymbol{\omega}_t^{z}, \boldsymbol{\Gamma}_t^{z}) = \textrm{MLP}(\boldsymbol{h}_{t}, \boldsymbol{z}, \boldsymbol{v}_t). \label{eq:rotgmm}\tag{9}\] The rotational action is subsequently sampled as \(\boldsymbol{w}^{z}_{t} \sim \mathcal{N}(\boldsymbol{\omega}_t^{z}, \boldsymbol{\Gamma}_t^{z})\). By marginalizing over the categorical maneuver class \(p(\boldsymbol{z}|\boldsymbol{h}_{0})\), the joint action space implicitly forms a Gaussian Mixture Model (GMM), capturing the highly multi-modal nature of user intent.

During inference, RT-V3 sequentially samples \(\boldsymbol{z} \sim p(\boldsymbol{z}|\boldsymbol{h}_{0})\), followed by \(\boldsymbol{v}_t \sim p(\boldsymbol{v}_t|\boldsymbol{h}_t, \boldsymbol{z})\) and \(\boldsymbol{w}_t \sim p(\boldsymbol{w}_t|\boldsymbol{h}_t, \boldsymbol{z}, \boldsymbol{v}_t)\) at each timestep. The final \(SE(3)\) pose trajectory is integrated using rigid body dynamics: \[T_t = T_{t-1} \mathop{\textrm{Exp}}(\boldsymbol{a}_t \Delta t),\] where \(T_t\) represents the \(SE(3)\) transformation matrix at timestep \(t\), and \(\mathop{\textrm{Exp}}(\cdot)\) denotes the Lie group exponential map.

6 Posterior Decision Making↩︎

Until now, we have focused exclusively on modeling the user’s behavior with a prior model \(p(\boldsymbol{h}_{0:H},\boldsymbol{a}_{0:H-1}|\boldsymbol{s}_{-T+1:0})\), and the parametrization of this model. In this section, we expand the prior model in Sec. 4 to consider user input, in addition to historical motion and contextual cues.

At the current timestep, we receive the user command \(\boldsymbol{a}^{\text{u}}_{0}=[\boldsymbol{v}_0^{\textrm{u}}, \boldsymbol{w}_0^{\textrm{u}}]\) from an uncertain interface. To achieve a better informed estimation of the intended action sequence \(\boldsymbol{a}_{0:H-1}\), we incorporate the observed user command \(\boldsymbol{a}^{\text{u}}_{0}\) and infer the posterior decision-making distribution \(p(\boldsymbol{h}_{0:H}, \boldsymbol{a}_{0:H-1}|\boldsymbol{s}_{-T+1:0}, \boldsymbol{a}^{\text{u}}_{0})\). Following the same technique in Eq. 4 , we can decompose Eq. 6 as: \[\begin{align} & p(\boldsymbol{h}_{0:H}, \boldsymbol{a}_{0:H-1}|\boldsymbol{s}_{-T+1:0}, \boldsymbol{a}^{\text{u}}_{0}) \\ & = p(\boldsymbol{h}_{0}|\boldsymbol{s}_{-T+1:0})\\ &~~ \sum_{\boldsymbol{z}}~p(\boldsymbol{h}_{1}|\boldsymbol{h}_{0}, \boldsymbol{a}_{0}, \boldsymbol{z}) \colorbox{blue!10}{\displaystyle p(\boldsymbol{a}_{0}|\boldsymbol{h}_{0},\boldsymbol{z}, \boldsymbol{a}_0^{\textrm{u}})} \colorbox{green!15}{\displaystyle p(\boldsymbol{z}|\boldsymbol{h}_{0}, \boldsymbol{a}_0^{\textrm{u}})} \\ & ~~\cdot \prod_{t=1}^{H-1} p(\boldsymbol{h}_{t+1}|\boldsymbol{h}_{t}, \boldsymbol{a}_{t}, \boldsymbol{z}) p(\boldsymbol{a}_{t}|\boldsymbol{h}_{t},\boldsymbol{z}). \label{eq:32shared32control32goal} \end{align}\tag{10}\] Compared to Eq. 4 , the terms \(\colorbox{blue!10}{\displaystyle p(\boldsymbol{a}_0|\boldsymbol{h}_{0},\boldsymbol{z})}\) and \(\colorbox{green!15}{\displaystyle p(\boldsymbol{z}|\boldsymbol{h}_{0})}\) become \(\colorbox{blue!10}{\displaystyle p(\boldsymbol{a}_{0}|\boldsymbol{h}_{0},\boldsymbol{z}, \boldsymbol{a}_0^{\textrm{u}})}\) and \(\colorbox{green!15}{\displaystyle p(\boldsymbol{z}|\boldsymbol{h}_{0}, \boldsymbol{a}_0^{\textrm{u}})}\) in Eq. 10 , respectively. The remaining terms in Eq. 10 are identical to those in Eq. 4 . To compute \(p(\boldsymbol{h}_{0:H}, \boldsymbol{a}_{0:H-1}|\boldsymbol{s}_{-T+1:0}, \boldsymbol{a}_0^{\textrm{u}})\), we must first calculate \(\colorbox{blue!10}{\displaystyle p(\boldsymbol{a}_{0}|\boldsymbol{h}_{0},\boldsymbol{z}, \boldsymbol{a}_0^{\textrm{u}})}\) and \(\colorbox{green!15}{\displaystyle p(\boldsymbol{z}|\boldsymbol{h}_{0}, \boldsymbol{a}_0^{\textrm{u}})}\).

Following the formulation in Eq. 7 , we can factorize \(\colorbox{blue!10}{\displaystyle p(\boldsymbol{a}_{0}|\boldsymbol{h}_{0},\boldsymbol{z}, \boldsymbol{a}_0^{\textrm{u}})}\) into a translational distribution and a rotational distribution conditioned on the translation: \[\begin{align} & \colorbox{blue!10}{\displaystyle p(\boldsymbol{a}_{0}|\boldsymbol{h}_{0},\boldsymbol{z}, \boldsymbol{a}_0^{\textrm{u}})}= p(\boldsymbol{v}_0, \boldsymbol{w}_{0}|\boldsymbol{h}_{0},\boldsymbol{z}, \boldsymbol{v}_0^{\textrm{u}}, \boldsymbol{w}_0^{\textrm{u}}),\\ & = \colorbox{lime!20}{\displaystyle p(\boldsymbol{v}_{0}|\boldsymbol{h}_{0}, \boldsymbol{z}, \boldsymbol{v}_0^{\textrm{u}})} \colorbox{lightblue!70}{\displaystyle p(\boldsymbol{w}_{0}|\boldsymbol{h}_{0}, \boldsymbol{z}, \boldsymbol{v}_0, \boldsymbol{w}_0^{\textrm{u}})} \end{align}\] Based on Bayesian rule, we can express \(\colorbox{lime!20}{\displaystyle p(\boldsymbol{v}_{0}|\boldsymbol{h}_{0}, \boldsymbol{z}, \boldsymbol{v}_0^{\textrm{u}})}\) as the product of the prior and likelihood: \[\colorbox{lime!20}{\displaystyle p(\boldsymbol{v}_{0}|\boldsymbol{h}_{0}, \boldsymbol{z}, \boldsymbol{v}_0^{\textrm{u}})} \propto p(\boldsymbol{v}_{0}|\boldsymbol{h}_{0}, \boldsymbol{z}) p(\boldsymbol{v}_0^{\textrm{u}}|\boldsymbol{v}_{0}). \label{eq:32translation32factorization}\tag{11}\] We assume \(p(\boldsymbol{v}_0^{\text{u}}|\boldsymbol{v}_{0}) = \mathcal{N}(\boldsymbol{v}_0^{\text{u}}|\boldsymbol{v}_0, \boldsymbol{\Sigma}^{\text{sys}})\), where \(\boldsymbol{\Sigma}^{\text{sys}}\) represents the system covariance of the user interface for the translational velocity. Then, we can analytically calculate \(\colorbox{lime!20}{\displaystyle p(\boldsymbol{v}_0|\boldsymbol{h}_{0}, \boldsymbol{z}, \boldsymbol{v}_0^{\text{u}})}\) as: \[\begin{align} & \colorbox{lime!20}{\displaystyle p(\boldsymbol{v}_0|\boldsymbol{h}_{0}, \boldsymbol{z}, \boldsymbol{v}_0^{\text{u}})} = \mathcal{N}(\boldsymbol{v}_0|\tilde{\boldsymbol{\mu}}_0^z, \tilde{\boldsymbol{\Sigma}}_0^z),\\ & \boldsymbol{K}_{\text{v}} = \boldsymbol{\Sigma}_0^z(\boldsymbol{\Sigma}_0^z+\boldsymbol{\Sigma}^{\text{sys}})^{-1},\\ & \tilde{\boldsymbol{\mu}}_0^z = \boldsymbol{\mu}_0^z + \boldsymbol{K}_{\text{v}}(\boldsymbol{v}_0^{\textrm{u}}-\boldsymbol{\mu}_0^z), \\ & \tilde{\boldsymbol{\Sigma}}_0^z = (\boldsymbol{I}-\boldsymbol{K}_{\text{v}})\boldsymbol{\Sigma}_0^z. \end{align}\] Similar to Eq. 11 , we can also express \(\colorbox{lightblue!70}{\displaystyle p(\boldsymbol{w}_{0}|\boldsymbol{h}_{0}, \boldsymbol{z}, \boldsymbol{v}_0, \boldsymbol{w}_0^{\textrm{u}})}\) as the product of the prior and likelihood, as follows: \[\colorbox{lightblue!70}{\displaystyle p(\boldsymbol{w}_{0}|\boldsymbol{h}_{0}, \boldsymbol{z}, \boldsymbol{v}_0, \boldsymbol{w}_0^{\textrm{u}})} \propto p(\boldsymbol{w}_{0}|\boldsymbol{h}_{0}, \boldsymbol{z}, \boldsymbol{v}_0) p(\boldsymbol{w}_0^{\textrm{u}}|\boldsymbol{w}_{0})\] By assuming \(p(\boldsymbol{w}_0^{\text{u}}|\boldsymbol{w}_{0}) = \mathcal{N}(\boldsymbol{w}_0^{\text{u}}|\boldsymbol{w}_0, \boldsymbol{\Gamma}^{\text{sys}})\), where \(\boldsymbol{\Gamma}^{\text{sys}}\) represents the system covariance of the user interface for the rotational velocity, we can obtain \(\colorbox{lightblue!70}{\displaystyle p(\boldsymbol{w}_{0}|\boldsymbol{h}_{0}, \boldsymbol{z}, \boldsymbol{v}_0, \boldsymbol{w}_0^{\textrm{u}})}\) as: \[\begin{align} & \colorbox{lightblue!70}{\displaystyle p(\boldsymbol{w}_{0}|\boldsymbol{h}_{0}, \boldsymbol{z}, \boldsymbol{v}_0, \boldsymbol{w}_0^{\textrm{u}})} = \mathcal{N}(\boldsymbol{w}_0|\tilde{\boldsymbol{\omega}}_0^z, \tilde{\boldsymbol{\Gamma}}_0^z),\\ & \boldsymbol{K}_{\text{w}} = \boldsymbol{\Gamma}_0^z(\boldsymbol{\Gamma}_0^z+\boldsymbol{\Gamma}^{\text{sys}})^{-1},\\ & \tilde{\boldsymbol{\omega}}_0^z = \boldsymbol{\omega}_0^z + \boldsymbol{K}_{\text{w}}(\boldsymbol{w}_0^{\textrm{u}}-\boldsymbol{\omega}_0^z), \\ & \tilde{\boldsymbol{\Gamma}}_0^z = (\boldsymbol{I}-\boldsymbol{K}_{\text{w}})\boldsymbol{\Gamma}_0^z. \end{align}\] For \(\colorbox{green!15}{\displaystyle p(\boldsymbol{z}|\boldsymbol{h}_{0}, \boldsymbol{a}_0^{\textrm{u}})}\), we can factorize it with Bayesian rule, as: \[\colorbox{green!15}{\displaystyle p(\boldsymbol{z}|\boldsymbol{h}_{0}, \boldsymbol{a}_0^{\text{u}})} = \frac{p(\boldsymbol{a}_0^{\text{u}}| \boldsymbol{h}_{0}, \boldsymbol{z})}{p(\boldsymbol{a}_0^{\text{u}}|\boldsymbol{h}_{0})} p(\boldsymbol{z}|\boldsymbol{h}_{0}).\] Since each maneuver class \(\boldsymbol{z}\) corresponds to a component in action-GMMs, we can evaluate \(p(\boldsymbol{a}_0^{\text{u}}| \boldsymbol{h}_{0}, \boldsymbol{z})\) and \(p(\boldsymbol{a}_0^{\text{u}}|\boldsymbol{h}_{0})\) as follows: \[p(\boldsymbol{a}_0^{\text{u}}|\boldsymbol{h}_{0}, \boldsymbol{z}) = \mathcal{N}(\boldsymbol{v}_0^{\text{u}}|\boldsymbol{\mu}^{z}_0, \boldsymbol{\Sigma}^{z}_0) \mathcal{N}(\boldsymbol{w}_0^{\text{u}}|\boldsymbol{\omega}^{z}_0, \boldsymbol{\Gamma}^{z}_0)\] \[p(\boldsymbol{a}_0^{\text{u}}|\boldsymbol{h}_{0}) = \sum_{\boldsymbol{z}} p(\boldsymbol{z}|\boldsymbol{h}_{0}) p(\boldsymbol{a}_0^{\text{u}}|\boldsymbol{h}_{0}, \boldsymbol{z}).\]

The graphical model of the posterior decision making is represented in Fig. 1 (b). With \(\colorbox{green!15}{\displaystyle p(\boldsymbol{z}|\boldsymbol{h}_{0}, \boldsymbol{a}_0^{\textrm{u}})}\) and \(\colorbox{blue!10}{\displaystyle p(\boldsymbol{a}_{0}|\boldsymbol{h}_{0},\boldsymbol{z}, \boldsymbol{a}_0^{\textrm{u}})}\), we can derive the posterior distribution \(p(\boldsymbol{h}_{0:H}, \boldsymbol{a}_{0:H-1}|\boldsymbol{s}_{-T+1:0}, \boldsymbol{a}_0^{\textrm{u}})\). This distribution yields a better-informed estimate of the intended action sequence \(\{\boldsymbol{a}_t\}_{t=0}^{H-1}\), incorporating the current user’s command \(\boldsymbol{a}^{\textrm{u}}_0\).

Asynchronous Shared Control Mechanism. While RT-V3 maintains high-frequency inference (e.g., 20 Hz), real-world teleoperation often involves intermittent user input as operators pause to reassess their strategy. In RT-V2 [17], the robot remains idle if the user stops providing commands. To address this, we propose an asynchronous shared control framework that leverages RT-V3’s capacity for autonomous action execution during periods of user inactivity, thereby reducing cognitive load. Specifically, the system monitors user inputs at each timestep: if a command is detected, RT-V3 performs posterior decision-making; otherwise, it executes the most probable action from its autonomous policy. Consequently, as long as the robot’s motion aligns with the user’s intent, the operator can refrain from continuous signaling.

To prevent autonomous execution from interfering with fine-grained user adjustments during grasp acquisition, this mechanism is disabled when the end-effector is within a predefined distance of the object surface. At close proximity, operators often perform subtle corrective motions to refine the final grasp pose, whereas continued autonomous motion based on previously inferred intent may hinder such corrections. Disabling asynchronous execution in this region ensures that robot motion remains tightly coupled to ongoing user inputs during the final approach phase, while retaining the efficiency benefits of autonomous assistance during the earlier coarse-motion stages of the task.

As demonstrated in our user study, this mechanism substantially reduces the frequency of user commands and correspondingly lowers user effort without compromising task performance.

7 Experiment↩︎

We evaluate the proposed RT-V3 framework through extensive experiments in both simulation and real-world settings. Our evaluation focuses on three core capabilities: intent prediction, reactive planning, and shared control.

• Intent Prediction (Sec. 7.1): We assess trajectory forecasting accuracy by measuring the discrepancy between predicted trajectories and the ground truth within our curated dataset.

• Planning (Sec. 7.2): We evaluate the model’s ability to reach a specified target grasp pose reactively in simulation environments.

• Shared Control (Sec. 7.3 and 7.4): To demonstrate generalizability, we first conduct large-scale testing with diverse simulated agents, followed by real-world trials with human users. These experiments validate RT-V3’s assistive ability across varied user behaviors and environmental contexts.

7.1 Trajectory Prediction↩︎

We first evaluate RT-V3’s predictive performance using a simulation dataset. Following established benchmarks in trajectory forecasting [51], [52], we employ two standard metrics:

• Average Displacement Error (ADE): The mean manifold distance (translational Euclidean distance + scaled Euler angle error, see details in Appendix) between the ground truth and predicted positions over the entire trajectory.

• Final Displacement Error (FDE): The manifold distance between the final predicted position and the ground truth endpoint.

To evaluate how effectively RT-V3’s CVAE-GMM framework captures multi-modal user intent, we report both Best-of-Mode (BoM) and Most Likely (ML) metrics. BoM reflects the model’s highest accuracy among all generated Gaussian modes, while ML measures the accuracy of the model’s most likely single trajectory.

Simulation Setup and Data Collection. To train RT-V3, we curated a large-scale simulation dataset comprising point clouds, feasible grasp poses, and motion trajectories. The data generation pipeline consists of two stages:

(i) Scene Generation: Following the protocols of volumetric grasping models [46], [47], we utilized a set of 303 unique objects. Scenes were synthesized by randomly placing or dropping a subset of these objects into a \(30 \times 30 \times 30\) cm\(^3\) tabletop workspace (Fig. 3). For each scene, a depth map was captured from a randomized camera viewpoint to produce the corresponding point cloud. Feasible grasps were identified by sampling 6 rotation angles for each candidate approach vector and gripper width, resulting in a total of 833,324 valid grasps across 22,265 scenes.

(ii) Trajectory Generation: Using a Franka Research 3 robot as the experimental embodiment, we generated motion data. For each feasible grasp, a collision-free trajectory was computed from a random initial pose to the target using CuRobo [53]. These trajectories were executed via a PD controller; we discarded any instances involving collisions, out-of-joint-limit, or failed grasps. The final dataset consists of 568,277 high-quality trajectories.

Performance in the simulation dataset. The results are presented in Tab. 1. We conducted an ablation study to analyze the contribution of different components in RT-V3. The baseline model (first row) yields the highest errors in both BoM and ML metrics. The introduction of the grasp encoder provides the most substantial performance leap, further improving both BoM and ML performance (ADE-BoM 26.95, FDE-BoM 38.12, ADE-ML 30.36, and FDE-ML 44.17). This suggests that explicit goal-conditioning (grasp information) is crucial for accurate intent prediction. Finally, incorporating the point cloud encoder refines the prediction further, achieving a better BoM performance (ADE 26.70, FDE 37.90), which indicates that environmental geometric awareness helps in generating more feasible trajectories. Though adding point cloud information slightly reduces the ML performance, it helps improve planning performance, which will be explained in the next experiment. Additionally, we implement an RT-V3 variant that directly generates translation and rotation actions simultaneously (dubbed “6D pred”), rather than using a translation-conditioned rotation architecture. It achieves the best performance in both BoM and ML settings (ADE-BoM 21.05, FDE-BoM 34.55, ADE-ML 22.50, and FDE-ML 37.65). However, we do not adopt this architecture due to its inferior planning performance; this limitation is analyzed further in the next experiment.

3pt

7.2 Planning in the Simulation Environment↩︎

RT-V3 itself is a reactive planning policy, so we aim to evaluate the planning ability of RT-V3 in this experiment. The planning experiment is conducted in a simulated environment identical to the one used for data collection in Sec. 7.1. We select two grasp poses in each scene as target poses. If RT-V3 can control the robot to reach one of the target poses without colliding with the table or objects, it will be regarded as a success, while failure occurs if the robot either collides with obstacles or fails to reach a goal within the time limit. The experiments are conducted over 1,000 trials with fixed random seeds, and three metrics are recorded: success rate (SR), collision rate (CR), and time-out rate (TOR). During each round, RT-V3 samples its optimal action using the ML setting as follows: \(\bar{\boldsymbol{z}} = \mathop{\textrm{argmax}}_{\boldsymbol{z}} p(\boldsymbol{z}|\boldsymbol{h}_{0}), \bar{\boldsymbol{a}}_t = \mathop{\textrm{argmax}}_{\boldsymbol{a}_t} p(\boldsymbol{a}_t|\boldsymbol{h}_t,\bar{\boldsymbol{z}})\).

Planning performance. We evaluated the model’s reactive planning capability. RT-V3 (full version) achieves a 90.4% Success Rate (SR), significantly outperforming the configuration without the PCL encoder (78.8%). Notably, RT-V3 exhibits a lower time-out rate (TOR) (1.7%) compared to the reactive controller NeoSS [54] (11.1%), demonstrating its efficiency in reaching targets. Although RT-V3 underperforms CuRobo (95.9% SR), a dedicated global planner, CuRobo does not support multi-modal intent prediction. Therefore, as an assistive controller, RT-V3 demonstrates competitive performance in planning.

In contrast, the “6D pred” variant—which predicts translation and rotation simultaneously—exhibits poor planning performance, achieving only a 3.4% SR. Because this variant is optimized primarily for open-loop trajectory prediction, it suffers from severe compounding errors (i.e., the "drift problem") during autonomous closed-loop execution, quickly driving the system into out-of-distribution states. By utilizing a translation-conditioned rotation architecture, the model avoids learning a complex 6D joint action distribution, opting instead for more tractable 3D conditional distributions. Furthermore, the sequential two-stage sampling process (sampling translation, followed by rotation conditioned on translation) introduces beneficial stochasticity during training. This noise acts as an effective regularizer, making the learned policy significantly more robust to distributional shifts during execution.

7.3 Simulated Shared Control Experiments↩︎

To evaluate the shared control capability of RT-V3, we propose a simulated shared control experiment. Inspired by the work of Reddy et al. [55], we propose to use simulated users and conduct shared control in the simulation environment, which can comprehensively evaluate RT-V3 at scale without using real human users. These simulated users are target-pose driven policies, but they reach the target with different behaviors. Specifically, we propose four different simulated users:

• Noisy User generates the action twist towards the target pose with injected noises.

• Laggy User generates the action twist towards the target pose, but it is forced to repeat its previous action with probability 0.8.

• Mode-Switching User has two action modes: translation mode and rotation mode. In translation (or rotation) mode, the user can only generate translational (or rotational) action towards the target at a time. These two modes alternate every 10 seconds.

• Single-DoF User can only control one DoF at a time. The user will switch to another DoF that has the largest distance every 5 seconds.

We pre-generate 50 validation scenes following the paradigm mentioned in Sec. 7.1, and sample 10 to 20 feasible grasps for each scene. In each trial, we set each grasp pose as the target pose for simulated users (a total of 768 trials).

Experimental results. Tab. 2 summarizes the performance of RT-V3 and several baseline methods across four types of simulated users. We record three metrics: success rate (SR), collision rate (CR), and the number of iterations used in the success trial (Iter).

We compare RT-V3 with two other baselines: pure teleoperation (“Direct”) and Hindsight Optimization (“HO”). RT-V3 outperforms them across all user types with higher SRs, lower CRs, and fewer iterations. HO does not improve the performance because the dense candidate poses tend to attract the robot along the way to its desired pose, trapping the robot in one of the candidate poses, leading to more iterations. Besides, HO does not have a planning capability, which makes it easy to collide with objects.

We also apply a collision avoidance method NeoSS [54] to Direct and HO. We observe that NeoSS largely decreases CR for both methods. Direct with NeoSS largely improves the SR in Noisy, Mode-Switching, and Single-Dof, but their SRs are still lower than RT-V3. HO with NeoSS does not significantly improve the SR. These results demonstrate that to provide suitable assistance, the assistive algorithm must not only employ collision avoidance but also accurately estimate intent. The proposed RT-V3 combines planning and intent estimation in a unified Bayesian framework, achieving the highest SR across all user types. Combined with NeoSS, RT-V3 can achieve very low CR (\(0\%\sim6\%\)) with a small sacrifice of SR.

Besides, we conduct the ablated variant of RT-V3 “6D pred” in this shared control experiment, and its performances are inferior to the RT-V3, which further proves the effectiveness of the proposed translation-conditioned rotation architecture.

3pt

7.4 Real-world Shared Control Experiments↩︎

In this experiment, we aim to validate the performance of RT-V3 in a real-world shared autonomy setting. By having human users complete tasks in a real-world environment, assisted by RT-V3, we can assess its performance in terms of speed and user experience.

Design. In this experiment, the user will be asked to control the 7 DoF Franka Research robot with an XBox joystick to execute some grasping tasks. The left joystick, left trigger, and bumper are used to control the translation, and the right joystick, trigger, and bumper are used to control the rotation. “A, B” buttons are used to execute the grasp and release the gripper, respectively. The “Y” button is used to move along the Z-axis of the tool frame (dubbed “approach grasp”), which is designed to move from pre-grasp to grasp poses. The “X” button is used to lift the object and drop it into the box once the user grasps the object. We compared the RT-V3 method against two baselines. The first baseline was pure user control (referred to as “Direct”). The second baseline, Hindsight Optimization (HO) [21], is a well-known intent estimator. The experiment consisted of three rounds corresponding to a specific assistive method (or pure teleoperation), each comprising 4 trials. The order of the assistive methods is randomly shuffled for each user. Prior to each round, participants were instructed on the mechanics of the assistive algorithm and completed practice trials to mitigate potential learning effects. For each trial, two or three objects were placed within the workspace. Participants received a text command (e.g., “grasp the facial cleanser from the right side”) and were tasked with retrieving the corresponding object. To mitigate learning effects, participants completed 12 unique trials across 12 distinct scenes. These scenes were divided into three subsets, with each subset randomly assigned to one of the three evaluated methods. A trial was classified as a failure under any of the following three conditions: (i) the robot experienced an unrecoverable collision or joint-limit violation; (ii) the execution exceeded the 90-second limit (“time-out”); or (iii) the robot disrupted the environment to an extent that rendered the task unachievable (‘unmatched’).

Before each round, we will explain to the user how this assistive algorithm works, and the user will practice with the algorithm on some tasks to get familiar with it in order to alleviate problems caused by proficiency. In each trial, 2 or 3 objects are placed in the workspace. The user will receive a text instruction, and they need to grasp the object in the workspace that matches the text instruction. For example, we may give the user an instruction like “grasp the facial cleanser from the right side”. We prepared 12 different scenes for 12 trials, which can alleviate problems caused by proficiency. These 12 scenes were separated into 3 groups, and three methods were randomly assigned to one of the groups. A trial is considered a failure in three circumstances: (i) the robot encounters a collision or reaches the joint limit that makes it unable to recover; (ii) the operation time is out of 90 seconds (dubbed, “out of time”); (iii) the robot destroys the scene and makes it impossible to match the text instruction (dubbed, “unmatched”).

Protocol. We enrolled 21 novice participants from the local community (19 male, 2 female; age range: 18–40). The participants are able to control the robot’s translation, rotation, grasping, and releasing. The order of the controllers was randomized for each participant to mitigate the effects of novelty and practice. After each round, participants rated their agreement with the following statements on a seven-point Likert scale:

• This algorithm helped me complete the task quickly.

• This robot did what I wanted.

• This algorithm would be useful for me in teleoperation.

• I felt in control while using this algorithm.

• I would find this algorithm easy to use.

• I trust this algorithm.

Participants also completed a NASA-TLX survey [56], where they rated their workload across six subscales. Mental demand assessed the cognitive and perceptual effort required; physical demand measured the physical effort involved in the task; temporal demand evaluated the time pressure experienced by the participant; effort measured how hard the participant had to work to maintain performance; frustration assessed feelings of annoyance, stress, and irritation; and performance gauged how successful participants felt in completing the task. At the end of the three rounds, participants were also asked to indicate their preferred method and provide written comments.

All participants provided signed consent for the experimental procedure, which was approved by the Social and Societal Ethics Committee of (anonymous institution) (G-2024-8370). The collected data were processed in accordance with the General Data Protection Regulation (GDPR) of the (anonymous organization).

Metrics. Both objective and subjective metrics were used in the experiments. For objective metrics, we compared the success rate (SR), unmatched rate (UR), time-out rate (TOR), total joystick inputs, completion time, and both translational and rotational trajectory length. For subjective metrics, we compared participants’ ratings from the agreement survey and their responses to the NASA-TLX survey across the three methods.

Results of Objective Measures. Tab. 3 summarizes SR, UR, and TOR for each method. During the experiment, we did not encounter the first failure case (i.e., unrecoverable collision or out of joint limits). RT-V3 achieved the highest SR (86.90%), the lowest UR (9.53%), and the lowest TOR (3.57%). HO easily trapped the robot in an undesired grasp pose, which is also observed in the simulated shared control experiment. This is because the performance of HO largely relies on the performance of the grasp planner. If the detected poses did not include the grasp pose the user desired, this “trapped” problem became more severe, leading to a high TOR of HO. Although RT-V3 also relies on the predicted grasps from the grasp planner, it is less constrained because it predicts the next action via posterior estimation instead of guiding the user towards the predicted intent pose. Therefore, RT-V3 also has the lowest TOR.

Fig. 6 compares total joystick input, completion time, and trajectory length, excluding failure trials. The Kruskal-Wallis test was used to evaluate the effect of the method (Direct, HO, RT-V3) on these metrics. The Mann-Whitney test with a two-stage Benjamini-Hochberg post-hoc correction adjustment was applied for pairwise comparisons when significant effects were found. For total joystick input (Fig. 6 (a)), a significant difference was observed among the methods (\(p<0.0001\)), with RT-V3 and HO differing significantly from Direct (\(p<0.0001\) and \(p<0.0001\)). Although the RT-V3’s total joystick inputs are less than HO’s, we did not observe a significant difference between RT-V3 and HO. For completion time (Fig. 6 (b)), significant differences were found (\(p=0.0006\)), with both RT-V3 and HO differing from Direct (\(p=0.01\) and \(p=0.0012\)). The HO’s completion time is less than RT-V3’s non-significantly due to the more aggressive nature of HO. For translation trajectory length (Fig. 6 (c1), significant differences were found (\(p=0.0352\)), with Direct differing from HO significantly (\(p=0.0155\)). The translation length of Direct is shorter than RT-V3 non-significantly. We also observe similar effects in rotational trajectory length where significant differences were found (\(p=0.0286\)). Direct’s rotational length is significantly shorter than HO’s (\(p=0.0155\)) and non-significantly shorter than RT-V3’s, although Direct requires more joystick inputs and completion time. The user tends to control the robot degree by degree and sometimes stops to think about how to move the next step, but the length will not be largely increased. In contrast, assistive methods provide more aggressive control by moving all degrees of freedom simultaneously, but the user needs to spend extra trajectories on adjusting the robot when the intent prediction is incorrect, leading to a higher length of trajectory. Due to the asynchronous shared control mechanism, RT-V3 can automatically execute the predicted actions during the trial. Thus, we delete the trajectories when RT-V3 takes full control and plot the adjusted translation and rotation trajectory length in Fig. 6 (c2) and (d2). Significant differences were found in both adjusted translation and rotation trajectory length (\(p=0.0009\) and \(p<0.0001\)). In the adjusted figure, the Direct’s and RT-V3’s translation and rotation lengths are significantly shorter than HO’s. Notably, RT-V3’s adjusted rotation length is significantly shorter than Direct’s, which demonstrates RT-V3’s superiority in assisting users in rotation manipulation.

Results of Subjective Measures. Fig. 7 (a) illustrates the comparison of the agreement survey results. The Kruskal-Wallis H-test was employed to evaluate the effect of the method (Direct, HO, RT-V3) on these metrics. A Wilcoxon signed-rank test with a two-stage Benjamini-Hochberg correction was used for post hoc analysis of pairwise comparisons when significant effects were identified. Significant differences were observed in Quick, Wanted, and Control. Post hoc analysis revealed significant differences between Direct and HO in Quick (\(p=0.0079\)), Wanted (\(p=0.0139\)), and Control (\(p=0.0113\)). Additionally, significant differences were found between RT-V3 and Direct in Quick (\(p=0.0078\)). Furthermore, RT-V3 differed significantly from HO in Wanted (\(p=0.0375\)), Control (\(p=0.0113\)). To summarize, RT-V3 is easier to use compared with HO because it gives users a sense of control, and RT-V3 makes users feel quicker compared with pure teleoperation by providing accurate assistance.

Fig. 7 (b) demonstrates the comparison of the NASA-TLX survey. The Kruskal-Wallis H-test evaluated the effect of the method (Direct, HO, RT-V3) on these metrics. A Wilcoxon signed-rank test with two-stage Benjamini-Hochberg correction was used as post hoc analysis for pairwise comparisons when significant effects were found. RT-V3 obtains the best performance in all metrics. Significant differences were found for Mental (\(p=0.0268\)), Effort (\(p=0.0174\)). Post hoc analysis found significant differences between Direct and HO in Mental (\(p=0.0281\)), between RT-V3 and Direct in Mental (\(p=0.0298\)), Effort (\(p=0.0035\)), and between RT-V3 and HO in Mental (\(p=0.0003\)). No significant difference was found in Physical demand, as the task involved only joystick control and was not physically taxing. Similarly, the lack of urgency in achieving the grasp poses resulted in no significant difference in Temporal demand.

Visualization of Guidance. Fig. 8 (a) illustrates the real-time inference process and the corresponding uncertainty of the action distribution during a manipulation task. The text instruction is “Grasp the cap of the bottle from the left”. At the beginning (\(t=0.0\) to \(4.3\)), the robot was under shared control. The translational entropy (blue) and rotational entropy (orange) exhibited fluctuations as the model inferred user intent from multi-modal candidate grasp poses. Specifically, the rotational entropy showed a distinct peak around \(t=1.0\) and \(2.8\), reflecting the ambiguity in selecting the optimal orientation among several candidate grasps. From \(t=4.3\) to \(8.6\) (gray zone), the system entered an autonomous mode where RT-V3 predicted the future actions based on the past trajectories, and controlled the robot using the prior action distribution towards the grasp pose that it thought was most likely. Starting from \(t=8.6\) (green zone), the user decides to grasp and moves the robot from pre-grasp to grasp. By comparing the region of the zone with entropy visualized (shared control mode) and the gray zone (auto mode), the design of the asynchronous shared control mechanism decreases the number of commands required from the user to fulfill the task, which is also aligned with the result shown in Fig. 6.

Analysis of Failure Cases. Fig. 8 (b) illustrates a failure case. The text instruction is “Grasp the pill bottle from the left”. At the beginning (\(t=0.0\) to \(4.3\)), the user moves the robot towards the upper-right of the pill bottle. From \(t=4.1\) to \(9.9\), RT-V3 mispredicted that the user wanted to grasp the pill bottle from the top and approach the pill bottle with a top-down grasp pose. From \(t=9.9\) to \(21.6\), the user tried to slightly move away from the pill bottle and change the pose, but RT-V3 always dragged the robot back to the top-down grasp pose. From \(t=21.6\) to \(32.8\), the user decided to keep rotating towards the right-approaching pose, avoiding being dragged back to the top-down pose. In this case, RT-V3 mispredicted the user’s intent, providing assistance that conflicted with the user’s commands.

8 Limitations and Future Work↩︎

This paper introduced RT-V3, a probabilistic shared-control method for \(SE(3)\) manipulation that utilizes a Bayesian framework to infer user intent through posterior estimation. We validated the approach across trajectory prediction, planning, and shared-autonomy tasks. Despite these strengths, several limitations remain to be addressed in future work.

Despite these strengths, several limitations remain:

Distributional shift between behaviors of the path planner and the user. Similar to its predecessor RT-V2, RT-V3 is trained to imitate the behavior of a path planner but is deployed as a prior distribution to estimate user behavior during posterior inference. This distributional shift between the automated planner and a human operator can lead to misalignments between RT-V3’s predicted actions and the user’s true intent. This limitation is also reflected in the failure cases shown in Fig. 8 (b), where the system occasionally biases the motion toward an incorrect grasp hypothesis when the user’s control strategy deviates from the planner’s demonstrations. Future work could mitigate this issue in several ways. One promising direction is to incorporate human teleoperation data during training so that the learned prior better reflects real user behaviors rather than purely planner-generated demonstrations. Another direction is to introduce adaptive assistance mechanisms that monitor the consistency between the posterior intent estimate and the user’s control commands. For instance, when the posterior uncertainty becomes high or when the predicted assistance repeatedly conflicts with user inputs, the system could reduce the level of autonomous guidance or temporarily revert to direct user control. Such mechanisms could improve robustness to distributional mismatch while preserving the benefits of probabilistic intent inference.

Reliance on the grasp planner. RT-V3 is trained to approach a set of candidate grasp poses. Consequently, its performance is inherently bounded by the grasp planner’s accuracy. System efficacy is compromised if the planner (i) fails to generate the specific grasp poses intended by the user, or (ii) predicts the grasp poses that cannot make a successful pickup. However, we expect that this problem will be alleviated in the future as the grasp planners become stronger and stronger.

Limited modeling capacity. Learning a probabilistic reactive policy that reaches multiple candidate poses is a highly multi-modal task. Because shared control requires low-latency, real-time responses, the model size is strictly constrained. This limited capacity may prevent RT-V3 from capturing the full range of human multimodality, potentially degrading the user experience. Furthermore, assigning low probability density to action sequences that lead to collisions remains a challenge, as learning barrier functions within GMM distributions is difficult. As a result, collision-free behavior cannot be strictly guaranteed in the RT-V3 framework.

Limited to grasping task. RT-V3 is trained to approach the grasp, which means that it can only assist with the grasping task. To achieve general task assistance, the model must be trained on various tasks. Currently, vision-language-action models (VLAs) are developing rapidly and becoming increasingly accessible. We believe that the shared control mechanism proposed in the work combined with VLAs can inspire the community to develop a universal assistive controller.

9 Conclusion↩︎

In this work, we presented Robot Trajectron V3 (RT-V3), a Bayesian shared-control framework for \(SE(3)\) manipulation that models user intent via a learned prior and computes a posterior over candidate actions to produce assistance. RT-V3 encodes point clouds and grasp poses predicted by the grasp planner as contextual input via a transformer-based architecture, and leverages a CVAE-GMM framework to model the multi-modality of the user’s intended future actions. We propose to factorize the \(\mathfrak{se}(3)\) action distribution into a translational distribution and a rotation distribution conditioned on translation, reducing the complexity of modeling user behavior.

Experimental results show that RT-V3 yields accurate trajectory predictions and competitive planning capability. In the shared autonomy experiments, RT-V3 provides smoother and faster assistance across different simulated users and human users, thereby improving success rates and reducing the cognitive burdens on users.

9.1 Proof for \(SE(3)\) Equivariance of RT-V3↩︎

We can abstract RT-V3 as a function \(f\), which takes the relative past trajectory, point clouds, and grasp poses as input, as: \[\boldsymbol{a}^{\textrm{rel}}_{0:H-1} = f(\boldsymbol{T}_{\text{cur}}^{\text{base}} \boldsymbol{x}_{-T+1:0}, \boldsymbol{T}_{\text{cur}}^{\text{base}} \mathcal{P}, \boldsymbol{T}_{\text{cur}}^{\text{base}} \mathcal{G}),\] where \(\boldsymbol{T}_{\text{cur}}^{\text{base}} = (\boldsymbol{T}_{\text{base}}^{\text{cur}})^{-1}\) is the base frame expressed relative to the robot’s current end-effector frame, and the relative action sequence \(\boldsymbol{a}^{\textrm{rel}}_{0:H-1} = \boldsymbol{T}_{\text{cur}}^{\text{base}} \boldsymbol{a}_{0:H-1}\). Let us consider adding a random perturbation \(\boldsymbol{T}_{\textrm{rand}}\) to all the input data. Then the perturbed current end-effector frame becomes \(\tilde{\boldsymbol{T}}_{\text{cur}}^{\text{base}} = (\boldsymbol{T}_{\text{rand}} \boldsymbol{T}^{\text{cur}}_{\text{base}})^{-1} = \boldsymbol{T}_{\text{cur}}^{\text{base}} \boldsymbol{T}_{\text{rand}}^{-1}\), the perturbed past trajectory becomes \(\tilde{\boldsymbol{x}}_{-T+1:0} = \boldsymbol{T}_{\text{rand}} \boldsymbol{x}_{-T+1:0}\), and the perturbed point clouds and grasp poses becomes \(\tilde{\mathcal{P}} = \boldsymbol{T}_{\text{rand}} \mathcal{P}\) and \(\tilde{\mathcal{G}} = \boldsymbol{T}_{\text{rand}} \mathcal{G}\), respectively. The RT-V3 prediction based on the perturbed data will be unchanged, as: \[\begin{align} & f(\tilde{\boldsymbol{T}}_{\text{cur}}^{\text{base}} \tilde{\boldsymbol{x}}_{-T+1:0}, \tilde{\boldsymbol{T}}_{\text{cur}}^{\text{base}} \tilde{\mathcal{P}}, \tilde{\boldsymbol{T}}_{\text{cur}}^{\text{base}} \tilde{\mathcal{G}})\\ & = f(\boldsymbol{T}_{\text{cur}}^{\text{base}} \boldsymbol{T}_{\text{rand}}^{-1} \boldsymbol{T}_{\text{rand}} \boldsymbol{x}_{-T+1:0}, \boldsymbol{T}_{\text{cur}}^{\text{base}} \boldsymbol{T}_{\text{rand}}^{-1} \boldsymbol{T}_{\text{rand}} \mathcal{P}, \boldsymbol{T}_{\text{cur}}^{\text{base}} \boldsymbol{T}_{\text{rand}}^{-1} \boldsymbol{T}_{\text{rand}} \mathcal{G}) \\ & = f(\boldsymbol{T}_{\text{cur}}^{\text{base}} \boldsymbol{x}_{-T+1:0}, \boldsymbol{T}_{\text{cur}}^{\text{base}} \mathcal{P}, \boldsymbol{T}_{\text{cur}}^{\text{base}} \mathcal{G}) \\ & = \boldsymbol{a}^{\textrm{rel}}_{0:H-1} \end{align}\] Through transforming \(\boldsymbol{a}^{\textrm{rel}}_{0:H-1}\) by the perturbed current end-effector’s frame, we obtain the perturbed action \(\boldsymbol{a}_{0:H-1}= \tilde{\boldsymbol{T}}_{\text{cur}}^{\text{base}} \boldsymbol{a}^{\textrm{rel}}_{0:H-1}\). Therefore, we prove the \(SE(3)\) equivariance of RT-V3.

9.2 Distance Metric↩︎

Given two \(\textrm{SE}(3)\) poses \(T_A\) and \(T_B\), we can transform them into translation positions \(p_A\), \(p_B\) and Euler angles \(a_A\) and \(a_B\). We define the manifold distance between two poses as: \[d(T_A, T_B) = \left\| p_A-p_B\right\|_1 + \left\| \frac{0.3}{\pi}*(a_A-a_B) \right\|_1.\] Considering the size of the workspace is \(0.3 \times 0.3 \times 0.3\) m\(^3\), the scaling term \(\frac{0.3}{\pi}\) balances the translation and rotation errors.

9.3 Simulated User for Shared Control↩︎

To rigorously evaluate RT-V3’s shared control capabilities, we implemented five distinct simulated users. These users represent common limitations found in real-world human-machine interfaces, such as noise, latency, and restricted degrees of freedom (DoF). All users generate commands in the form of a 6D twist vector \(\mathbf{u} = [v_x, v_y, v_z, \omega_x, \omega_y, \omega_z]^\top \in \mathfrak{se}(3)\), representing the velocity required to move from the current end-effector pose \(\mathbf{T}_{\textrm{cur}}\) to a goal pose \(\mathbf{T}_{\textrm{goal}}\) in the local frame, as: \[\mathbf{u} = \textrm{Logmap}(\mathbf{T}_{\textrm{goal}} \mathbf{T}_{\textrm{cur}}^{-1}).\]

9.3.1 Noisy User↩︎

The Noisy User simulates a human operator with imprecise motor control or a high-variance input device. The command is generated by adding Gaussian noise to the ideal twist: \[\mathbf{u} = \mathbf{e} + \epsilon, \quad \epsilon \sim \mathcal{N}(0, \Sigma),\] where \(\Sigma\) is a diagonal covariance matrix. This tests the model’s ability to filter high-frequency noise and extract the underlying intent.

9.3.2 Laggy User↩︎

The Laggy User simulates network latency or delayed human reaction times. At each timestep, the user has an 80% probability of repeating the previous command \(\mathbf{u}_{t-1}\) rather than calculating a new error-correcting twist. This creates “sticky” behavior where the input does not immediately update as the robot moves.

9.3.3 Mode-Switching User↩︎

The Mode-Switching User alternates between purely translational and purely rotational control. (i) Translation Mode: \(\mathbf{u} = [v_x, v_y, v_z, 0, 0, 0]^\top\). (ii) Rotation Mode: \(\mathbf{u} = [0, 0, 0, \omega_x, \omega_y, \omega_z]^\top\). The user switches modes every 20 steps (2 seconds in simulation). This evaluates if the model can maintain the “passive” components of the pose while the user focuses on the “active” components.

9.3.4 Single-DoF User↩︎

The Single-DoF User represents the most constrained input scenario (e.g., a single-axis joystick or button control). Every 10 steps (1 second in simulation), the user identifies the single DOF with the largest absolute error: \[i^* = \arg\max_i | \mathbf{e}_i |\] For the duration of that period, the user only provides input for that specific index: \[\mathbf{u}_i = \begin{cases} \mathbf{e}_i & \text{if } i = i^* \\ 0 & \text{otherwise}, \end{cases}\] This requires the shared control system to provide significant assistance in the remaining 5 DOFs to ensure smooth motion.

9.4 Twelve tasks in the real-world shared autonomy experiment↩︎

Fig. 9 shows 12 tasks in the real-world shared autonomy experiment. These tasks include grasping different objects or different parts of the object from different orientations.

9.5 More visualizations of Guidance in Real-world Shared Autonomy Experiments↩︎

To further demonstrate the robustness and adaptability of RT-V3 across diverse manipulation scenarios, we provide additional visualizations of entropy and task execution for four distinct tasks in Fig. 10. Across these four tasks, “auto mode” occupies a large time area, which denotes, similar to what we discuss in the Sec. 7.4, that RT-V3 decreases the user commands by accurately predicting and automatically executing the intended actions. In addition, these demonstrations showcase how the Bayesian framework manages multi-modal intent under varying degrees of task complexity.

• A increasing trend of translational entropy: We observe that as the task proceeds, the translational entropy will have an increasing trend. This indicates that at the beginning of the trial, the reaching motion is relatively certain since the robot has to first approach the objects. Once the robot gets close to the object, many candidate grasp poses are densely distributed on the surface of the object, indicating that the robot could reach any surrounding grasp poses, leading to a high entropy.

• A decreasing trend of translational entropy: In contrast to the translational entropy, the rotational entropy will have a decreasing trend as the task proceeds. That is because when the robot gets close to the objects, many of the surrounding grasp poses, which are predicted by the grasp planner, have similar orientations. Thus, the entropy is relatively low.

• Multi-modal action prediction: In Fig. 10 (c), the difference between the entropy upper bound and lower bound is relatively large at the beginning of the trial. This implies that RT-V3 is uncertain about the user’s intent and predicts a highly multi-modal action distribution. Thus, we can also observe that the entropy values reach a high level. (Trans Entropy UB is higher than 4, and Rot Entropy UB is higher than 3).

References↩︎

---

1. Supported by Interne Fondsen KU Leuven/Internal Funds KU Leuven. Partially supported by Flanders Make (strategic research centre for the manufacturing industry).↩︎

2. \(^{1}\)KU Leuven, Dept. Mechanical Engineering, Research unit Robotics, Automation and Mechatronics, B-3000 Leuven, Belgium. Email: firstname.lastname@kuleuven.be↩︎

3. \(^{2}\)KU Leuven, Dept. Electrical Engineering, Research unit Processing Speech and Images, B-3000 Leuven, Belgium.↩︎

4. \(^{3}\)Flanders Make@KU Leuven.↩︎