[DNN]deep neural network [FCN]fully convolutional network [SDF]signed distance function [CNN]convolutional neural network [GNN]graph neural network [DL]deep learning [ML]machine learning [MC]Monte Carlo [MLP]multi-layer perceptron [DoF]degree of freedom [DoFs]degrees of freedom [VLM]vision language model [VAE]variational autoencoder [CVAE]conditional variational autoencoder [VCGS]Variational Constrained Grasp Sampler [FPS]farthest point sampling [TaI]target as input [PCA]principal component analysis [PC]principal component [AUC]area under the curve [ELBO]evidence lower bound [BPS]basis point set [MALA]Metropolis-adjusted Langevin algorithm [DP]diffusion policy [IL]imitation learning [RL]reinforcement learning [BC]behaviour cloning [OS]opposition space [OSes]opposition spaces [MCP]metacarpophalangeal [GAN]generative adversarial network [DP3]3D Diffusion Policy [DPPO]diffusion policy policy optimization [PPO]proximal policy optimization [DDIM]denoising diffusion implicit model [DDPM]denoising diffusion probabilistic model [MDP]Markov decision process [OSC]operational space controller [RRT]rapidly exploring random tree [VLA]vision-language-action

1 Introduction↩︎

To date, most dexterous grasping methods use all the end-effector’s dofs for picking objects [1]–[3]. However, these methods rarely achieve higher grasp success rates than approaches using low dof parallel-jaw grippers [4]. Consequently, some argue that a dexterous arm paired with a simple gripper is sufficient for most manipulation tasks [5]. But what if the problem is that current methods underutilize the gripper’s available dexterity? Motivated to answer this question, we propose MoDex, the first dp [6] for sequentially picking multiple objects with a dexterous gripper, as demonstrated in Fig. 1.

MoDex is trained to map partial point clouds of the scene, previously executed grasps, and an os descriptor that specifies which fingers participate in the current grasp [7] to the next desired end-effector pose. This design lets MoDex sequentially grasp multiple objects directly from raw perception, unlike previous multi-object grasping methods that either require known object models [8], decoupled grasp generation and execution [9], or near-fixed object poses [10]. MoDex is pre-trained on a new dataset containing thousands of expert sequential grasping demonstrations generated automatically by a heuristic pose generator and a controller in simulation. To extend the rather narrow state-action distribution covered in pre-training, MoDex is fine-tuned using the rl framework dppo [11] on a reward specifically designed for sequential multi-object grasping. MoDex is benchmarked against representative il and rl baselines on 15 objects of diverse geometry on a simulated Franka Emika Panda arm equipped with an Allegro Hand. We also conduct 90 real-world grasping sequences using the same physical hardware setup. The results demonstrate that MoDex outperforms dp3 by 2.92–17.92% in simulation and 6.67-17.78% in the real world.

In summary, our contributions are:

• MoDex, the first dp for sequential multi-object grasping (Section 5).

• An algorithm for collecting expert sequential multi-object grasping trajectories (Section 6).

• An extensive experimental evaluation exploring the strengths and limitations of MoDex compared to il and rl baselines (Section 7).

2 Related Work↩︎

Our work spans dexterous and multi-object grasping, as well as policy learning. Below, we review each of these separately.

2.0.0.1 Dexterous Grasping.

Traditionally, dexterous robotic grasps were generated by optimizing them with respect to a surrogate grasp quality metric, such as force closure [12]–[16]. While these methods provide theoretical guarantees, they assume known object models and often struggle with the “sim-to-real” gap due to unmodeled environmental uncertainties. To address these limitations, researchers turned to data-driven models trained to directly map partial observations to high-quality static grasp poses [3], [17]–[21]. Other, more recent methods [1], [2], [22] train il or rl policies to output continuous grasping actions rather than static grasp poses. Still, single-object grasping methods fail to fully exploit the high dof potential of dexterous hands.

2.0.0.2 Multi-Object Grasping.

Research into dexterous multi-object grasping is still in its early stages. Initial efforts [23], [24] have focused on simultaneous multi-object grasping to improve efficiency and reduce execution time by avoiding repeated arm repositioning. However, these methods require objects to be in close proximity and do not exploit the full dexterity of the gripper, as multi-object grasping is best solved by first enveloping all the objects with the hand and then closing all fingers simultaneously. More recent works [8]–[10] have explored the sequential multi-object grasping problem. To solve this problem, the number of fingers used to pick an object at a time should be minimized, ensuring that as many dofs as possible are available for subsequent grasps. However, real-world deployment remains challenging for existing sequential multi-object grasping methods, which commonly rely on collision-free path planning [8], [9]. This work overcomes these limitations by training a single multi-object grasping policy that directly maps incomplete object observations to grasp actions.

2.0.0.3 Diffusion Policies.

Diffusion models were recently applied to learning visuomotor policies [6]. These methods, called dps, treat action generation as an iterative denoising process. The main benefits over other policy learning frameworks are that dps capture multimodal action distributions more faithfully and are more stable to train [6], [11]. Therefore, many dp extensions have been proposed, such as incorporating 3D point cloud observations to improve spatial reasoning [25], adding rl fine-tuning to overcome the narrow state-action coverage of behavior cloning [11], and using it as the action head in modern vla models [26]–[30]. However, to the best of our knowledge, dps have not been used to learn sequential multi-object grasping policies, which is the problem addressed in this work.

3 Problem Formulation↩︎

In sequential multi-object grasping, the problem is to grasp the \(i\)-th target object while holding all the previously \(i{-}1\) objects grasped, where \(i \in \{1, \ldots, I\}\) is referred to as a grasp stage and \(I\) is the maximum number of objects that the gripper can possibly grasp. The kinematic structure of the Allegro hand, which is used in this work, restricts \(I \leq 3\). The goal then is to learn a control policy \({\pi_{\theta}:\mathcal{O}\rightarrow\mathcal{A}^{T_p}}\) that maps the current observation \(\mathbf{o}_{t, i}\in\mathcal{O}\) to an action chunk \({\mathbf{A}_t=\left[\mathbf{a}_t,\mathbf{a}_{t+1},\ldots,\mathbf{a}_{t+T_p-1}\right]\in\mathcal{A}^{T_p}}\) of prediction horizon \(T_p\), with the first \(T_a \leq T_p\) actions executed on the robot before the policy is queried again. Throughout, \(t\) denotes the environment timestep within grasp stage \(i\).

Observations are represented as the tuple \({\mathbf{o}_{t,i}=\{\mathbf{f}_i,~\mathbf{h}_{i-1},~\mathbf{q}_t,~\mathcal{P}_t\}}\), where \(\mathbf{f}_i\) is the os vector for the \(i\)-th grasp, \(\mathbf{h}_{i-1}\) is a grasp history, \(\mathbf{q}_t\in \mathbb{R}^{D+J}\) is the robot proprioception stacking the \(D\) arm joint angles and the \(J\) hand joint angles, and \(\mathcal{P}_t = \{\mathbf{x}_j\}_{j=1}^{M}\), with \(\mathbf{x}_j \in \mathbb{R}^3\), is a partial object point cloud of the environment. The os vector \(\mathbf{f}_i\) describes which dofs should be used to pick the current target object [8], while the grasp history \(\mathbf{h}_{i-1}\) describes which dofs are already used for picking the previous \(i-1\) objects. Each action in the action chunk is represented as the vector \({\mathbf{a}_t=[\mathbf{p}_t, \mathbf{r}_t, \boldsymbol{\phi}_t]\in\mathbb{R}^{6+J}}\), where \(\mathbf{p}_t\in\mathbb{R}^3\) is the hand base position, \(\mathbf{r}_t\in\mathbb{R}^3\) is the hand base orientation in axis-angle form, and \(\boldsymbol{\phi}_t\in\mathbb{R}^J\) denotes the hand joint angles. In this work, \(D=7\) and \(J=16\) for the Franka Panda arm and Allegro Hand, respectively. The problem then becomes: 1) how to mathematically represent \(\mathbf{f}_i\) and \(\mathbf{h}_{i-1}\), and 2) how to train the policy \(\pi_{\theta}\) from data.

4 Preliminaries↩︎

This section reviews the background on dps [6] and dppo [11] necessary for training the sequential multi-object grasping policy \(\pi_\theta\).

4.1 Diffusion Policies↩︎

A dp [6] parameterizes the policy \(\pi_\theta\) as a ddpm [31] that denoises action chunks of horizon \(T_p\). Starting from Gaussian noise \(\mathbf{A}_t^K \sim \mathcal{N}(\mathbf{0}, \mathbf{I})\), a dp iteratively denoises the chunk over \(K\) steps, \[\mathbf{A}_t^{k-1} \sim p_\theta(\mathbf{A}_t^{k-1} \mid \mathbf{A}_t^k, \mathbf{o}_{t,i}) := \mathcal{N}\!\left(\mathbf{A}_t^{k-1};\, \mu_k(\mathbf{A}_t^k, \varepsilon_\theta(\mathbf{A}_t^k, \mathbf{o}_{t,i}, k)),\, \sigma_k^2 \mathbf{I}\right),\] where \(\varepsilon_\theta\) is a neural network predicting the noise at step \(k\) and \(\sigma_k^2\) follows a fixed schedule. After denoising, the first \(T_a \leq T_p\) actions of the clean chunk \(\mathbf{A}_t^{0}\) are executed on the robot before the policy is queried again. The network \(\varepsilon_\theta\) is pre-trained via il using the behavior cloning objective: \[\label{eq:behavior95cloning95objective} \mathcal{L}_{\mathrm{BC}}(\theta) = \mathbb{E}_{(\mathbf{o}_{t,i},\, \mathbf{A}^0_t) \sim \mathcal{D}_{\mathrm{off}},\; k,\; \varepsilon^k}\left[\|\varepsilon^k - \varepsilon_\theta(\mathbf{A}^0_t + \varepsilon^k,\, \mathbf{o}_{t,i},\, k)\|^2\right],\tag{1}\] where \(\mathcal{D}_{\mathrm{off}}\) is a dataset of trajectories, \(k\) is a uniformly sampled denoising step, and \(\varepsilon^k \sim \mathcal{N}(\mathbf{0}, \mathbf{I})\) is the noise injected at that step.

4.2 Diffusion Policy Policy Optimization↩︎

A major limitation of dp and behavior cloning policies in general is that their robustness to out-of-distribution states is limited, as the training data cover a rather narrow state-action distribution [32]. Thus, to make policies more robust, a common practice is to refine them via rl fine-tuning [33]. For dps, the primary rl finetuning method is dppo [11], which treats the \(K\)-step denoising process as an mdp nested within the environment mdp. Based on this two layer mdp, dppo formulates the policy gradient updates with the ppo [34] objective: \[\label{eq:dppo} \mathcal{L}_\theta = \mathbb{E}_{\mathcal{D}_{\mathrm{itr}}} \left[ \min\!\left( \hat{A}_{\bar t} \frac{\bar{\pi}_\theta}{\bar{\pi}_{\theta_{\mathrm{old}}}},\; \hat{A}_{\bar t} \cdot \operatorname{clip}\!\left(\frac{\bar{\pi}_\theta}{\bar{\pi}_{\theta_{\mathrm{old}}}}, 1{-}\epsilon_{\mathrm{clip}}, 1{+}\epsilon_{\mathrm{clip}}\right) \right) \right],\tag{2}\] where \(\hat{A}_{\bar{t}} = \gamma_{\text{denoise}}^{k}\bigl(\bar{r} - \tilde{V}(\mathbf{o}_{t,i})\bigr)\) is the denoising-discounted advantage. The discount \({\gamma_{\text{denoise}} \in (0, 1)}\) downweights the gradient signal at noisier, earlier denoising steps (large \(k\)), where the predicted action carries little task-relevant information, and concentrates learning on the later steps (small \(k\)) that most directly shape the executed action. The return \(\bar{r}\) is the environment return associated with the rollout, and \(\tilde{V}(\mathbf{o}_{t,i})\) is the value function. To improve efficiency, only the last \(K' \leq K\) denoising steps are fine-tuned using ddim sampling [35].

5 MoDex↩︎

Fig. 2 shows an overview of MoDex. The sections below detail our specific contributions: how to architect the sequential multi-object grasping dp, how oss and grasp history are mathematically represented and integrated into the policy architecture, and which rewards are needed for successful rl fine-tuning.

5.1 os and Grasp History Context↩︎

A central challenge in single-policy sequential multi-object grasping is producing qualitatively distinct grasps that use as few dofs as possible for each successive object while leaving the dof used in prior grasps untouched. We address this through two conditioning signals: the os and the Grasp History Context.

5.1.0.1 Opposition Space.

Originally proposed by [7], an os characterizes a grasp by the set of finger pairs acting in opposition during object contact. We represent the os for the \(i\)-th grasp as a binary indicator vector \(\mathbf{f}_i \in \{0,1\}^L\), where \(L\) is the number of fingers in the hand (\(L=4\) for the Allegro Hand) and \((\mathbf{f}_i)_j = 1\) iff finger \(j\) participates in the grasp. In this work, we restrict the oss that the Allegro Hand can achieve to the four shown in Fig. 2. For example, \(\mathbf{f}_1 = [1,1,0,0]\) engages only the thumb and index finger, leaving the remaining fingers free for subsequent grasps. We include \(\mathbf{f}_i\) into MoDex by concatenating it, at every denoising step \(k\), to each action in the noisy action chunk \(\mathbf{A}_t^{k}\) , forming the os-augmented action chunk \({\tilde{\mathbf{A}}_t^{k}=\left[\mathbf{a}^k_t\;\|\;\mathbf{f}_i,~\mathbf{a}^k_{t+1}\;\|\;\mathbf{f}_i,~\ldots,~\mathbf{a}^k_{t+T_p-1}\;\|\;\mathbf{f}_i\right]}\).

5.1.0.2 Grasp History Context.

The condition \(\mathbf{f}_i\) only tells which dofs should be used to grasp the current object, but ignores which fingers have already been used by prior grasps. We therefore introduce the Grasp History Context, defined as the vector \({\mathbf{h}_{i-1} = \left[\mathbf{f}_1 \;\|\; \cdots \;\|\; \mathbf{f}_{i-1}\right] \in \{0,1\}^{(i-1)L}}\) consisting of the oss from all the preceding grasps. For the first grasp (\(i=1\)), we use a sentinel vector \({\mathbf{f}_\emptyset = [-1,-1,-1,-1]}\) to signal the absence of history. \(\mathbf{h}_{i-1}\) is appended to the observation \(\mathbf{o}_{t,i}\), enabling the policy to reason about already used dofs.

5.2 Policy Architecture↩︎

MoDex follows the encoder-denoiser architecture of dp3 [25]. The noise-prediction network \(\varepsilon_{\theta}\) is a 1D convolutional U-Net operating along the action-horizon dimension, following the standard dp architecture [6]. At every denoising step \(k\), the U-Net input concatenates the os-augmented noisy chunk \(\tilde{\mathbf{A}}_t^{k}\), with the broadcast conditioning \(\mathbf{c}_t\), and a sinusoidal embedding of \(k\). MoDex is pre-trained using the behavior cloning objective in Eq. (eq. ¿eq:eq:behavior95cloning95objective? ) on the dataset detailed in Section 6. After pretraining, the same policy is rl fine-tuned using Eq. (eq. ¿eq:eq:dppo? ), where the task-specific per-step reward \(r_t^{\mathrm{base}}\) is introduced next.

5.3 Reward for RL Fine-tuning↩︎

To make MoDex robust to states not covered by the il dataset, we rl-fine-tune it using Eq. (eq. ¿eq:eq:dppo? ) on a dense, task-specific reward. The reward is shaped to encourage the policy to: (i) establish the selected os contact on the current target object, (ii) keep every previously grasped object securely held, and (iii) avoid unnecessary off-os contacts, idle-finger motion, and hand–table collisions that would compromise sim-to-real transfer. We encode these behaviors in the reward \[r_t^{\mathrm{base}} = \begin{cases} r_{\mathrm{succ}}-\lambda_{\mathrm{ina}} r_t^{\mathrm{ina}}, & \text{if}~h_{i,t} = 1 \;\land\; c^{\mathrm{hold}}_{j,t} > 0 \;\;\forall j\in\mathcal{M}_i,\\[2pt] \operatorname{clip}\!\left( r_t^{\mathrm{grasp}} + r_t^{\mathrm{maintain}} - r_t^{\mathrm{avoid}}, r_{\min}, r_{\max} \right), & \text{otherwise}. \end{cases} \label{eq:reward95base}\tag{3}\]

The policy is updated with a large constant reward \(r_{\mathrm{succ}}\) at every step where the target object is lifted (normalized lift score \(h_{i,t}=1\)) and all previously grasped objects in \(\mathcal{M}_i\) are securely held. Because this condition is re-evaluated at every step rather than terminating the episode, the policy is rewarded for both picking the new object and holding the earlier ones. The penalty \(r_t^{\mathrm{ina}}\) discourages fingers that are neither active in \(\mathbf{f}_i\) nor recorded in the grasp history \(\mathbf{h}_{i-1}\) from drifting away from their initial pose, suppressing the idle-finger twitching that otherwise emerges under sparse reward signals and would interfere with subsequent grasps.

The reward when success has not yet been achieved is split into three terms: \(r_t^{\mathrm{grasp}}\), \(r_t^{\mathrm{maintain}}\), and \(r_t^{\mathrm{avoid}}\), each encouraging different behaviors. The first of these terms, \(r_t^{\mathrm{grasp}}\), rewards the active fingers in \(\mathbf{f}_i\) for approaching and contacting the current target, providing a smooth gradient that pulls the hand toward the commanded os. The second term, \(r_t^{\mathrm{maintain}}\), rewards keeping all previously grasped objects securely held, in contact, and stationary. The reward activates only when the new grasp contacts the target object and is scaled down by the movement of the least well-supported previously grasped objects. The scaling prevents the policy from neglecting the previously grasped objects in favor of the current object. The last term, \(r_t^{\mathrm{avoid}}\), penalizes collisions and near-collisions between the hand and the table, as well as contacts between the target object and fingers not in \(\mathbf{f}_i\) before the gripper makes contact with the object. Clipping the sum to \([r_{\min}, r_{\max}]\) keeps any single term from dominating early in training.

6 Dataset Curation↩︎

We collect a trajectory-level sequential multi-object grasping dataset in Robosuite [36] using the logic summarized in Algorithm 3. This algorithm sweeps every object \(b\in\mathcal{B}\) paired with each of the four oss \(\mathbf{f}_1,\ldots,\mathbf{f}_4\), repeating \(N\) trials per pair. The stage-indexed initializer \(\mathrm{\small Init}_i\) returns an initial simulator state \(\sigma_0\) and the prior grasp history \(\mathbf{h}_{i-1}\). For Stage 1, \(\mathrm{\small Init}_1\) resets to an empty scene with a randomized pose of \(b\) and an empty history, while, for Stage \(i{>}1\), \(\mathrm{\small Init}_i\) samples a successful trajectory from \(\mathcal{D}_{i-1}\), resumes from its final state, recovers its history \(\mathbf{h}_{i-1}\), and adds the new target \(b\) to the scene. Pairs with \(\mathbf{f}_k\in\mathbf{h}_{i-1}\) are skipped, since the same os cannot be reused. The pose generator \(\mathrm{\small Pose}(b,\mathbf{f}_k)\) produces a heuristic grasp \(\mathbf{g}\) from the object’s dimensions and the hand’s geometry, selecting a top-down approach for the finger–finger oss \(\mathbf{f}_1\), \(\mathbf{f}_2\), \(\mathbf{f}_3\) and a side approach for the ring–palm os \(\mathbf{f}_4\). An osc controller then executes the pre-grasp, closure, and lift, producing a trajectory \(\tau\). A trial is deemed successful if the robot lifts the object \(10\,\mathrm{cm}\) above the table and the hold predicate \(\mathrm{\small Hold}(\tau, \mathbf{h}_{i-1})\) confirms all previously grasped objects remain in hand.

We run Algorithm 3 with \(\mathcal{B}\) containing 15 distinct objects across three primitive geometries (spheres, cylinders, and boxes), yielding 564 trajectories for Stage 1, 592 for Stage 2, and 525 for Stage 3.

7 Experiments↩︎

We experimentally evaluate MoDex in both simulation and the real world. Our evaluation is designed to answer the following questions:

1. How well does MoDex perform compared to baselines?

2. What does each component of MoDex contribute to the overall performance?

3. Does MoDex transfer to real hardware?

7.1 Simulation Experiments↩︎

7.1.0.1 Baselines.

We compare MoDex against the following methods: BC-RNN [37], a recurrent behavior-cloning baseline that models temporal dependencies across observations; PPO [34], a standard on-policy rl method trained from scratch on the dense reward in Eq. (eq. ¿eq:eq:reward95base? ); SeqDiffuser [8], a grasp-pose generation method paired with a motion planner, which operates on full object point clouds at test time; MoDex-BC, a dp3 [25] pre-trained on point cloud input without the DPPO fine-tuning;. For all il and rl baselines, observations are kept consistent with MoDex. SeqDiffuser receives the full object point cloud as per its original formulation.

7.1.0.2 Evaluation protocol.

The simulation experiments are conducted in Robosuite [36]. For each method, we evaluate all four oss across all grasp stages, running 20 trajectories per stage for a total of 240 trials per method. The object set consists of 15 objects spanning three primitive geometries: spheres, cylinders, and boxes. The target object’s pose is randomized within \(\pm 10\) cm in translation and \(\pm 45^\circ\) in rotation. For stages 2 and 3, we initialize the robot with 1 or 2 grasped objects, respectively, from a randomly sampled successful trajectory in the dataset. This provides controlled and identical starting conditions for every method. To ensure a fair comparison, the random seed is kept consistent across all methods. A trial is deemed successful if the target object is lifted at least \(10\) cm above the table while all previously grasped objects remain securely held.

7.1.0.3 Results.

Table [[tab:sim95sr95pen]](#tab:sim95sr95pen){reference-type=“ref” reference=“tab:sim95sr95pen”} reports the per-stage and average success rates for all methods across all 15 evaluation objects. The results show that all methods degrade as the number of objects simultaneously held increases, reflecting fewer available grasp options. Across all methods, MoDex achieves the highest success rate at all evaluated stages, demonstrating that os- and grasp-history-conditioned dps effectively preserve and exploit kinematic redundancies across sequential grasps.

Additionally, the results indicate that BC-RNN and MoDex-BC achieve competitive Stage 1 performance but degrade substantially at Stages 2 and 3, where the absence of explicit os conditioning results in unstructured finger use that compromises subsequent grasps. MoDex outperforms dp at all stages, demonstrating the benefit of dppo fine-tuning. ppo struggles across all stages due to Allegro Hand’s high-dimensional action space, making reward-driven exploration from scratch impractical without a pre-trained initialization. While SeqDiffuser generates geometrically valid grasp poses, its reliance on a decoupled motion planner introduces execution failures, particularly in Stages 2 and 3, where collision-free planning becomes increasingly difficult due to previously held objects. In conclusion, MoDex outperforms all baselines thanks to its condition values and training setup, thereby answering question [[q:baselines]](#q:baselines){reference-type=“ref” reference=“q:baselines”}.

max width=0.98

max width=0.9

7.1.0.4 Ablation Study.

We also ablate MoDex to understand how the different design choices affect the performance. The results are reported in Table [[tab:ablation]](#tab:ablation){reference-type=“ref” reference=“tab:ablation”}. Removing the os condition causes the largest drop at Stages 2 and 3, confirming that explicit finger-allocation signals are essential for sequential grasping. Removing the grasp history context similarly degrades later stages, as the policy cannot adapt its motion to previously occupied fingers. The dppo fine-tuning phase yields consistent gains across all stages, validating its role in bridging the gap between demonstration coverage and deployment robustness. Finally, the os-conditioned reward proves critical during rl fine-tuning: without it, the policy converges to morphologies that violate the intended opposition space. Together, these results highlight that all the proposed design choices improve the performance in different aspects, thereby answering question [[q:ablation]](#q:ablation){reference-type=“ref” reference=“q:ablation”}.

7.2 Real-World Experiments↩︎

To assess real-world transferability, we deploy MoDex to control the real Allegro Hand and the Franka Emika Panda robot shown in Fig. 1. We use a Kinect v3 depth sensor to capture the scene point cloud and follow the same evaluation protocol as in simulation, conducting 3 trials per object per stage. Our test set comprises three everyday objects and two 3D-printed objects, four of which are unseen during training. For stages 2 and 3, we prompt Gemini [38] to determine the object and \(\mathbf{f}_i\) sequence. We use an osc controller [39] to move the arm to a predefined pose above the object before querying the policy and executing the resulting actions.

The real-world experimental results are reported in Table 4. The results demonstrate that MoDex achieves promising zero-shot real-world transfer on out-of-distribution objects, with a modest drop relative to the simulation results, consistent with typical sim-to-real gaps in contact-rich manipulation [11], [40], [41]. In contrast, MoDex-BC exhibits substantially greater sim-to-real degradation, particularly at Stages 2 and 3, strengthening the case that dppo fine-tuning yields robust policies. A few different failure cases for MoDex are demonstrated in Fig. [[fig:succ95fail95example]](#fig:succ95fail95example){reference-type=“ref” reference=“fig:succ95fail95example”}, where the top image shows a previously grasped object slipping out of the hand, while the bottom image shows the finger coming very close to securing the target but fails to close around it due to the dynamic limits of the Allegro Hand. Despite these failure cases, the real-world results demonstrate that MoDex, although only trained on simulated data, can sequentially pick real-world objects, thereby answering question [[q:realworld]](#q:realworld){reference-type=“ref” reference=“q:realworld”}.

8 Conclusion and Limitations↩︎

This work explored the question: Are we underutilizing the dofs in dexterous grippers? To answer it, we introduced MoDex, a dp trained to sequentially pick multiple objects one at a time by using a subset of the available dofs, while keeping all previously grasped objects securely held. The experimental evaluations demonstrated that the proposed os- and grasp-history conditions, and the two-phase training scheme combining il pre-training and rl fine-tuning with a task-specific reward, were important factors behind MoDex’s performance.

Despite the strengths, MoDex falls short of human-level dexterity in two key aspects. First, it does not support in-hand regrasping: humans routinely pinch an object with the thumb and index finger and then transfer it into a more secure power grasp. Picking up a pen is a perfect example. In contrast, MoDex commits to a single os per object. Second, MoDex treats the os assignment and the grasping order as given, rather than reasoning about which os best suits each object or in what order objects should be picked. Addressing both would be a promising directionfor future work.

References↩︎