¹

1 Introduction↩︎

There is great interest in using robotic hands similar to those of humans to contend with the built environment [1]. Such multi-finger hands enable a robot to perform a wide range of tasks [2], including picking up [3]–[10] and manipulating objects [11]–[14], opening doors [15], or even playing musical instruments [16]. In this paper, we look into multi-fingered grasping of novel objects given only a pointcloud from a single, fixed camera.

In the area of multi-fingered grasping, optimization-based approaches that leverage a learned model to estimate the probability of grasp success are still state-of-the-art [4], [7], [8], [17], [18] and as such tend to be sensitive to initialization and potentially computationally time-consuming. This is because optimizing over a neural network classifier is a highly nonlinear problem and multiple steps are needed to find a solution to the optimization problem. Furthermore, multiple constraints on the decision variable must exist to ensure a feasible grasp (e.g. joint limits, avoid collisions, etc.). This further complicates the process of generating grasps. To achieve faster grasp inference time, grasping as regression has been proposed [9], [19] resulting in the popular generator-evaluator-planner paradigm [17], [20]–[23].

The generator-evaluator-planner pipeline consists of generating a batch of grasps in the object frame, evaluating and ranking the proposed grasps, and planning a trajectory to the highest-ranked grasp. If the motion planner fails to find such a trajectory, it plans a trajectory to the next-best grasp. This leads to executing grasps that are less likely to succeed, and is computationally expensive due to running multiple trajectory optimizations in a row. Parallelization might improve computational efficiency, but executing a suboptimal grasp degrades the overall system performance.

We propose a practical and effective solution: 1) plan trajectories to all the target grasps, then 2) evaluate the grasps at the terminal configuration of the executable trajectories, and finally 3) execute the trajectory with the highest likelihood of grasp success. Since we swap the order of grasp evaluation and trajectory planning compared to the traditional approach, we name our approach First Plan, Then Evaluate (FPTE).

Furthermore, the traditional approach rejects trajectories that do not reach the target. This is because only the initial targets are evaluated on the grasp success likelihood, and proximity to the target is necessary for accurate grasp success estimation. To overcome the limitation of planning inaccuracy and evaluator sensitivity, we do not reject a collision-free trajectory if it doesn’t reach the target, but evaluate the likelihood of the trajectory successfully grasping the object. This allows us to execute the trajectory with the highest likelihood of grasp success instead of discarding planned trajectories based on the somewhat arbitrary measure of distance to the proposed targets.

Planning trajectories to all the targets is traditionally seen as inefficient. However, multiple works  [24]–[29] have by now addressed this issue, providing evidence for efficient computation to multiple goals. We need to plan to multiple goals only because the highest-ranked grasp is not reachable every time. Failure to successfully plan to the target grasp due to unreachability causes 20-30% lower grasp success rates [22], [23], [30] in the real world compared to simulation with a floating end effector or the floating object. There are two possible explanations for this: grounding grasps in the object frame and training data for the generator.

The problem with grounding grasps in the object frame is that the generated grasps do not take robot reachability into account. On the other hand, it allows for generalization across different environments and object locations, as we show in Figure 1. Recent RL-based approaches [26] have been successful in avoiding the object frame by learning everything in a fixed, camera frame, but it’s yet to be seen how learning in a fixed frame can generalize across different object poses and environments.

Training data for the generator is typically collected using a floating end-effector [23], [31]–[33]. This leads to collecting grasps that might not exist in the real world due to unreachability or collisions with the environment. In an attempt to overcome this problem, using simulation, we collect a dataset of 28.9M grasp attempts by running full-robot trajectories attempting to grasp and lift the object off the table. However, in our experiments, we show that the problem of unreachable grasps persists despite the large amount of realistic data.

As discussed before, this leads to the execution of grasps that are less likely to succeed than the top samples. On the other hand, if we relax the necessary proximity to the target pose for the motion planner, the initial grasp likelihood estimate is inaccurate as the learned evaluator is highly sensitive to the EE pose. This poses a tradeoff between the accuracy of the motion planner and the accuracy of the grasp success likelihood estimation. FPTE addresses this problem by evaluating grasps grounded in executable trajectories.

Our experiments in sim and real show FPTE improves over the traditional generate-evaluate-plan approach across different objects, generator architectures, and motion planners. This highlights the importance of the framework independent of the specific learner or planner used. Our primary contribution lies in a subtle change to previous grasp pipelines: we evaluate grasp configurations at where the planned trajectory ends, instead of evaluating grasps proposed by the generator that serve as targets to the planner. Finally, we show successful generalization of our approach across different real world environments (Figure 1.)

2 Related work↩︎

Grasp Planning as Optimization Papers in this category model grasping as an optimization problem and typically use a gradient-based solver to solve it, but not always [34], [35]. The metric to optimize over is either analytical [30], [34], [36]–[44] or learned [3]–[8], [45]. While these approaches are reliable and achieve remarkable success rates, they require solving several optimizations which can be time-consuming and are sensitive to initialization.

Grasping as Regression As an alternative to optimization-based approaches, researchers have proposed approaches that generate target grasps directly without running an optimization. Redmon et al. [46] reduce grasping to a computer vision problem by generating bounding boxes. Sundermeyer et al. [20] use point cloud as input and generate 6 DoF grasps, followed by execution of the most promising grasp. More recently, [47] present an approach where they generate a grasp pose in real-time. Urain et al. [48] generate trajectories to grasp objects by leveraging diffusion models. Fang et al. [49] learn a model using real-world data on a bin picking task. Finally, Xu et al. [50] propose adding keypoint detections to a deep network affordance segmentation pipeline.

Dexterous Grasping as Regression Methods here extend regression-based approaches to predict full hand configurations. Our approach falls into this category. Chen et al. [9] learn a model that predicts a preshape palm pose and final grasp pose for every fingertip from a small set of human demonstrations. While [9] is specific to a four-fingered hand (although it could easily be modified), Shao et al. [19] present an approach that generates grasp contact points for any given end-effector, similarly to Wei et al. [51]. Matak et al. [52] present an approach to predict a grasping trajectory from a partial pointcloud, similarly to [10]. Mayer et al. [21] present a GAN-like approach where they take partial pointcloud as input and generate a set of grasps candidates that are ranked using a learned classifier. The basis point set they use allows for efficient encoding of the pointcloud, which we also use. Lundell et al. [53] present a generative grasp sampling method that produces 6D grasps from an RGB-D image. Wei et al. [54] propose a generative model to generate grasps and an optimization module to refine the generated grasps. Liu et al. [30] present an algorithm for training networks to grasp novel objects by using a generalized and differentiable grasp quality metric. The network is then used at runtime to predict grasps directly from a set of pointclouds.

RL for Grasping Recent work in leveraging reinforcement learning for grasping [55]–[58] seems promising [26] when dealing with a single object in a known environment, though generalization to novel scenes and obstacles remains an open challenge. In contrast, our approach generalizes to novel environments without any retraining (Figure 1).

Planning to Distributions Our work can be seen as an extension of the ideas proposed by Pavlasek et al. [59], and Conkey et al. [60]. Instead of explicitly modeling the goal distribution, we sample from it and set the samples as the target poses for the motion planner. The grasps at the terminal configuration of the planned trajectories are then evaluated based on the likelihood of being a successful grasps, while ignoring the initial targets.

Planning to Multiple Target Poses In the area of sampling-based planners, Gammel et al. [61] explicitly address the case of multiple goals, while Faroni et al. [62] demonstrate that planning time is strongly sub-linear in the number of goals. Advances in accelerated computing enable a new generation of vectorized motion planning [24]–[29]. Such planners can plan multiple trajectories in parallel without sacrificing time. Our framework yields grasp success improvement regardless of the motion planner choice.

3 Our Approach↩︎

FPTE consists of three stages: (1) generate grasp candidates, (2) plan trajectories to all candidates, and (3) evaluate grasp success at the terminal state of each trajectory before executing the highest-scoring trajectory (Figure 2)

Step 1: Given a partial pointcloud of the object, we pass it as input to a learned grasp generator. The generator proposes a batch of grasps in the object frame constructed from the given pointcloud. A grasp consists of a pose and hand configuration, which we formalize below.

Step 2 A motion planner plans trajectories to all the grasps proposed by the generator while avoiding collisions with the environment and the object. Recent advances in planning trajectories to multiple targets allow for efficient computation of this step [24], [25], [27]. Importantly, we ignore the distance between the resulting and the target end effector pose and do not discard any executable trajectory.

Step 3: A learned evaluator estimates the likelihood of grasp success at the terminal configuration of all the planned trajectories from Step 2. The grasps proposed by the generator in Step 1 are only used as targets for the motion planner in Step 2. The proposed grasps are not evaluated on the grasp success likelihood, since they may not be reached by the trajectories from step 2. Instead, grasps resulting from the planned trajectories are evaluated. Finally, the robot executes the trajectory with the highest likelihood of grasp success.

Object and Grasp Representation Let \(\mathcal{O}\) be the object representation. We process the partial pointcloud using Basis Point Set (BPS) [63] with 4096 basis points. We encode the pointcloud as the L2 norm of the distance between each of the basis points and the closest point on the pointcloud to it. This allows for a fixed-length representation \(\mathcal{O} \in \mathbb{R}^{4096}\) that can be efficiently processed [21]. We parametrize a grasp as \(\mathcal{G} = (\boldsymbol{T}_{OH}, \theta_p, \theta_g)\) where \(\boldsymbol{T}_{OH} \in SE(3)\) is the hand pose relative to the object, \(\theta_p \in \mathbb{R}^n\) is a collision-free pre-grasp hand configuration, \(\theta_g \in \mathbb{R}^n\) is an in-grasp configuration, and \(n\) is the number of hand joints (\(n=16\) in this work).

Generator The generator neural network \(\mathcal{\hat{G}} = f_g(\mathcal{O})\) is trained exclusively on positive grasps. Given \(\mathcal{O}\), \(f_g\) it generates a batch of \(K\) grasp samples for the motion planner. During training, we apply point-cloud perturbations to improve robustness to sensor noise, similar to [26], [64].

Evaluator The evaluator neural network \(\mathcal{\hat{E}} = f_e(\mathcal{O}, \mathcal{G})\) is trained on both positive and negative samples to output the likelihood of grasp success using the cross-entropy loss. We found the evaluator struggles to discriminate between successful and unsuccessful grasps when they are close together. To combat this, we include hard negatives [20], [21] in all minibatches by perturbing positive samples (\(\pm\) 5cm and \(\pm60^{\circ}\) per axis) and labeling them as negative. We use \(\mathcal{\hat{E}}\) to evaluate the likelihood of success of the planned trajectories. Specifically, we extract \(\mathcal{G}\) from the joint positions of the robot at the last timestep of a planned trajectory.

Grasp Planning We use \(^T\) to denote target values, and \(^R\) for resulting values. Given a set of \(K\) target grasps \(G_{OH}^T = \{\mathcal{G}_1^T, \mathcal{G}_2^T, ..., \mathcal{G}_K^T\}\) proposed by the generator \(\mathcal{\hat{G}}\) for object \(\mathcal{O}\), we transform each of the \(K\) target grasp poses \(\boldsymbol{T}_{OH}^T\) from the object frame to the robot frame and obtain \(K\) different \(\boldsymbol{T}_{RH}^T\). We set \(G_{RH}^T=\{(\boldsymbol{T}_{RH}, \theta_p)_1^T, (\boldsymbol{T}_{RH}, \theta_p)_2^T, ... (\boldsymbol{T}_{RH}, \theta_p)_K^T\}\) as targets for the motion planner. The motion planner plans \(K\) trajectories from the current robot state (\(\boldsymbol{q}, \dot{\boldsymbol{q}}\)) to the targets \(G_{RH}^T\) while avoiding the fixed environment obstacles and the estimated object mesh.

Given a set of resulting trajectories \(\{\tau_1, \tau_2, ..., \tau_K\}\), we extract end effector (EE) poses \(\boldsymbol{T}_{RH}^R\) and EE configurations \(\theta_p^R\) at the last timestep of each trajectory \(\tau_i\). We transform the EE poses \(\boldsymbol{T}_{RH}^R\) to the object frame to obtain a set of resulting EE poses \(\boldsymbol{T}_{OH}^R\) and pre-grasp configurations \(\{(\boldsymbol{T}_{OH}, \theta_p)_1^R, (\boldsymbol{T}_{OH}, \theta_p)_2^R, ... (\boldsymbol{T}_{OH}, \theta_p)_K^R)\}\) which we combine with \(\theta_g\) values from \(G_{OH}^T\) to create a set of resulting grasps \(G_{OH}^R = \{\mathcal{G}_1^R, \mathcal{G}_2^R, ..., \mathcal{G}_K^R\}\). A learned evaluator \(\mathcal{\hat{E}}\) estimates grasp success likelihood for each grasp in \(G_{OH}^R\), and we execute \(\tau_i\) that corresponds to \({G}_i^R\) with the highest likelihood. Finally, we fit a cubic spline between \(\theta_p\) and \(\theta_g\) to close the hand, followed by a heuristic for increasing the stiffness as in [8] before lifting the object. Incidental table contact may occur while closing the hand from \(\theta_p\) to \(\theta_g\).

Grasp Generation Our dataset contains objects with associated grasps, rendered depth images, and grasp success labels. The grasp data is stored as tuples \(\{(\mathcal{G}^j, \mathcal{O}^j, y^j)\}\) where \(\mathcal{O}\) is pointcloud rendered from the single, fixed depth camera, and \(y\) is a binary label. To synthesize a sample, we spawn an object from the BigBird dataset [65] at a random orientation around the z axis at a random position on the table. We render the depth camera and save the pointcloud. We generate \(K\) target grasp poses around the object and pre-grasp hand configurations using a heuristic [8]. We then use a vectorized motion planner [27] to move the arm to the target poses and configurations while avoiding collisions with the object and table. Each environment in the simulator executes one of the \(K\) trajectories. We opted for this solution over a floating end effector because this approach generates realistic grasps for the whole embodiment.

Given the resulting collision-free hand configuration \(\theta_p\), we plan for a hand configuration \(\theta_g\) such that the fingertips of the hand are in contact with the object surface as in [8]. We solve the problem in parallel for all the environments in the simulator using a vectorized motion planner [27]. Once we compute the configuration \(\theta_g\), we fit a cubic spline between \(\theta_p\) and \(\theta_g\) and execute the trajectories, followed by a heuristic for increasing the stiffness as in [8] before lifting.

During data collection, we use fabrics [27] as the motion planner [12], [26]. We plan to a batch of \(K\) different grasps \(\mathcal{G}\). During data collection, these \(K\) target grasp poses are generated using a heuristic described previously. We execute all \(K\) trajectories, one per environment, and label the data.

Our grasp generation pipeline is motivated by Matak et al. [8], but heavily modified to (1) leverage parallelized simulation while executing full-robot trajectories, and (2) plan precision grasps during data generation instead of at runtime as in [8]. We generated 28.9M grasp attempts in simulation using full-robot trajectories planned to heuristic grasp proposals. Successful and failed grasps were retained for training, yielding approximately 180k successful grasps.

4 Experiments↩︎

We conduct two sets of experiments to help us understand the system: in simulation and in the real world. In simulation, we focus on understanding the efficiency of our system in a known environment. In the real world, we focus on generalization across novel objects and novel environments. All of our experiments are performed on a KUKA LBR4 (7 DoF) with a four-fingered Allegro hand (16 DoF).

4.1 Simulation: understanding the system↩︎

To understand the impact of our approach on the overall robot performance, we compare our approach to the traditional generate-evaluate-plan approach. In the traditional approach, after the generator proposes grasp targets and the evaluator ranks the grasp targets, the motion planner plans a trajectory to the highest-ranked grasp. If the planner succeeds, the robot executes the trajectory. If not, the motion planner plans to the next-best ranked grasp. We use default position (0.5 cm) and rotation (14 degrees per axis) threshold values from Curobo [25] to determine if the planned trajectory succeeds. In Fig. 4, we refer to this method as “trad”.

Instead of planning trajectories one by one, we use a vectorized motion planner to plan trajectories to all grasp targets simultaneously. This does not slow down planning since all computations are done in parallel on a GPU [25], [27]. After planning, we rank all the trajectories based on their grasp likelihood. This is in contrast to the “trad” method that ranks grasp targets proposed by the generator; hence, FPTE – First Plan, Then Evaluate. Importantly, we rank all the resulting trajectories, not only successful trajectories. This allows the robot to execute the trajectory that is most likely to succeed in grasping the object, even if distinct from the initially generated grasp.

We conduct experiments across two different vectorized motion planners (Curobo [25] and Fabrics [27]), three different generator architectures (cVAe [21], Diffusion [22], [23], and MDN [66]), and 20 different object shapes from the training dataset across 100 poses on the table. We set the batch size to 128 for Fabrics [27] and 64 for cuRobo [25] since it requires more GPU memory. We keep the object shape encoding method (BPS) and the learned evaluator model the same throughout all the experiments.

We use Isaac Sim as our physics simulator. A single object is placed in isolation on the table at one of the 100 poses, and a fixed depth sensor is used to observe the partial pointcloud. The observed pointcloud is used as the input for the generator, and the loaded mesh of the object is used for collision modeling for the motion planner.

Results are presented in Figure 5. We observe that FPTE-fabrics achieves higher success rates than FPTE-curobo. We hypothesize this is because fabrics was used during data collection, thus limiting the gap between data collection and deployment. We observe FPTE-rrt has lower success rates than other FPTE approaches. This is because of coarse collision checking, which allowed more experiments but resulted in more collisions during execution.

Importantly, our approach, First Plan Then Evaluate, consistently achieves a higher success rate irrespective of the motion planner choice or the generator architecture. To understand why this is the case, we ablate the importance of distance to the target during motion planning on grasp success while keeping the generator (cVAE), the evaluator, and the rotation threshold fixed.

We compare three different motion planners: RRT [67], fabrics [27], and teleportation directly to the IK solution. We use a custom IK solver for RRT and teleportation. We visualize the results in Figure 5. Analyzing results for the teleportation method, we observe FPTE yields higher grasp success rates compared to the traditional method regardless of the motion planner, as the robot is directly teleported to the collision-free IK solution. Results using RRT and fabrics confirm that finding.

Furthermore, we analyze the evaluator’s precision as the allowed IK position increases and observe degradation in average precision as the IK position error increases only when the traditional method is used, as visualized in Figure 6. The observed evaluator sensitivity to IK error in the traditional approach, but not in FPTE, explains why FPTE outperforms the traditional approach.

We observe unreachable grasps across different generators and motion planners, despite using physically realistic data for training and the generated grasp targets being distributed around the object, as shown in Figure 7. The unreachability of the proposed grasps stems from grounding grasps in the object frame. However, grounding grasps in the object frame allows for generalization across different object poses and environments, which we analyze next.

4.2 Real World Generalization↩︎

We evaluate our approach across 11 novel objects shown in Figure 3. We place an object on the table, segment it, reconstruct its mesh, plan a grasp trajectory, and execute it.

We use Grounded SAM [68], on the RGB image from the depth observation with the prompt object that can be picked up with one hand, followed by matching the RGB segmentation with depth values. We use BRRP [69] to reconstruct the object mesh from the partial view and model it as an obstacle for the motion planner. We use the best-performing duo from the simulator: cVAE [21] as the generator, and fabrics [27] as a motion planner with batch size 512. We place every object across five different locations on the table. If the baseline cannot find a successful trajectory after three attempts, we count it as a failed attempt. Our approach, FPTE, achieves an 80% success rate compared to 22% with the traditional approach. It is worth noting that the baseline didn’t find a trajectory for 23 attempts. When counting only executed trajectories, the success rate of the baseline was 38%. The difference in success rate comes from 1) the baseline not finding a trajectory (23 out of 55 times), and 2) the executed grasps are less likely to succeed as evaluated by the learned evaluator. On average, the predicted likelihood of success of executed grasps using our approach was 90% compared to 26% when using the baseline. That is because the baseline ignores trajectories that would lead to a good grasp if the distance to the target pose is above the goal threshold. Some of the resulting grasps are shown in Figure 8.

Finally, to assess the soundness of our evaluator, we plot Figure 9. We report (top) the precision-recall (PR) curve of our evaluator used to estimate the likelihood of grasp success. In plotting the PR curve, we filter out samples where the robot hit the object while moving to the pre-grasp configuration. We further visualize (bottom) predictions of the evaluator in the real world for the executed grasps.

Figure 9 shows us that the evaluator 1) performs well in the real world, despite being trained on simulated data only, and 2) grasp success rates can be further increased by executing grasps only if the evaluator’s prediction is above a high threshold. For example, we could execute grasps only if the likelihood of success is above 90% and call for human intervention otherwise. To construct the plots, in our experiments we didn’t filter out any grasps based on the estimated probability of success.

Generalization to Novel Environments Object-centric grasp planning enables us to grasp the target object even if there are multiple objects on the table. Furthermore, it enables our approach to grasp objects in novel environments, such as different table heights and shelves. Without measuring quantitative results, we show different environments and grasps in Figure 1. We model all the detected objects and the environment (hand-coded) as collisions for the motion planner. Due to inaccuracies in the learned object shape reconstruction [69], the robot occasionally made unintended contact with the object while approaching the target grasp. We observed 8 such contacts across 25 executed trajectories, and 2 of these contacts led to grasp failure. Additionally, the robot’s fingertips contacted the table during hand closure in 4 trials; however, none of these contacts led to grasp failure. Note that all of the learned models have been trained solely on data collected by grasping a single object in isolation from a fixed table height in the simulator. The real-world objects used are not part of the training data, and clutter has never been simulated during training.

Note that all of the learned models have been trained solely on data collected by grasping a single object in isolation from a fixed table height in the simulator. The real-world objects used are not part of the training data, and a cluttered environment has never been simulated during training.

5 Conclusion & Limitations↩︎

In this work, we propose a novel framework that improves learning-based grasping pipelines. By evaluating the likelihood of success at the terminal configuration of planned trajectories, instead of evaluating proposed grasp targets that may not be reachable, we improve grasp success rates.

Limitations include limited training-object diversity, and segmentation inaccuracy that leads to incorrect collision modeling and thus potential contact with objects modeled as collisions in a cluttered scene. The object shape diversity can be addressed using a larger dataset [17], potentially combined with active learning [6]. On the perception side, the system runs in open loop based on a single RGBD reading. In our experiments, we show that our approach leads to improved grasp success rates over the traditional approach. We show our improvements persist across different objects, generator architectures, and motion planners. Furthermore, we show the ability of our method to generalize in the real world across novel objects and scenes that may contain multiple objects, despite training only by grasping objects in isolation off a fixed table height in simulation.

ACKNOWLEDGMENT↩︎

This work was supported by DARPA under grant HR0011-24-9-0423.

References↩︎