July 07, 2026
Robust robotic grasping of novel objects requires datasets that simultaneously provide photorealistic RGB-D observations, physically validated grasp quality annotations, and a principled bridge between simulation and the real world, which existing datasets lack to provide jointly. GraspIT addresses this gap: tabletop scenes in NVIDIA Isaac Sim are annotated via a four-stage physical slip-test on parallel Franka Panda instances, producing trajectory-reachability checks and continuous quality scores beyond force-closure.Of \({\sim}\)2.3M candidates, 83% pass as good (\(s{\geq}0.50\)); the 17% that passed force-closure but failed the slip-test provide graded hard negatives. A Real\(\leftrightarrow\)Sim loop back-projects these labels onto 100 real-world scenes. The release provides \({\sim}\)316k annotated RGBD frame sets across 1 035 sim and 100 real scenes, with instance masks, 6-DoF poses, physical object properties, and scored 6-DoF grasps. All tools are open-source and Docker-containerized. The trajectory planning within Isaac Sim further allows streaming of high resolution demonstrations for tabletop manipulation policy learning and behavior cloning.
Robotic manipulation of everyday objects — picking, placing, stacking, and handing over — remains an unsolved challenge despite decades of research [1], [2]. Recent large-scale imitation learning systems such as RT-2 [3], \(\pi_0\) [4], and Octo [5] demonstrate multi-task instruction-following in laboratory settings; yet, their generalization collapses when confronted with objects, scenes, or lighting conditions outside the training distribution. A central root cause is the absence of physically grounded visual representations: the encoders driving these systems are trained on semantic similarity objectives (CLIP [6], SigLIP [7]) that do not encode metric geometry, 3D object understanding, or the physical grounding that determines whether a proposed grasp or object manipulation will succeed [8], [9].
Collecting the training data needed to close this gap is expensive. Accurate 6-DoF grasp annotation requires either real-robot execution at scale or a physics simulator; real RGBD observations necessitate a physical setup; linking the two so that simulation annotations are valid for real sensor data requires a careful calibration pipeline. No existing dataset provides all of these simultaneously (see Tab. 1).
We introduce GraspIT (Fig. 1), a dataset and generation system that addresses this gap through four principled design decisions (Fig. 2): 1) Quality over quantity. Every grasp candidate is validated by a four-stage physical slip test executed by a simulated Franka Panda arm (Fig. 6), producing a continuous quality score \(s \in \{0, 0.25, 0.50, 0.75, 1.0\}\) that captures failure modes (slip under gravity, oscillation, pendulum swing) that are absent from force-closure metrics. 2) Robot-in-the-loop reachability filtering. Every candidate is validated for collision-free trajectory reachability by the robot arm before the slip test, ensuring that all positive labels are actionable. 3) Real\(\leftrightarrow\)Sim bidirectional link. Real-world scene geometry is captured via a robot-mounted camera and registered into Isaac Sim; simulation-validated labels are back-projected onto all real camera frames. This means a single real-world capture session yields both real RGB-D images and unlimited additional photorealistic synthetic views with full annotation (Fig. 5). 4) Open, containerized, and extendable. All generation tools are packaged as Docker containers, enabling any lab to reproduce, extend, or contribute to the shared GraspIT data foundation.
GraspIT initial release provides \(\sim316\)k annotated RGBD frame sets across 1 035 Isaac Sim scenes and 100 real-world scenes, with \({\sim}\)2.3M slip-test validated scene unique grasp candidates spanning \({\sim}7.4\)k+ unique object identities drawn from the ABC dataset [10] and our own real assets. Every frame carries per-instance segmentation masks, 6-DoF object poses, scored visible/hidden grasping poses, camera calibrations, complete 3D surface point clouds from CAD, analytically derived physical properties (centroid, extent, volume), and robot trajectories to grasping attempts.
(A) GraspIT: the first dataset providing photorealistic RGBD observations, robot-reachability-filtered grasps, and staged slip-test quality scores simultaneously. (B) A Real\(\leftrightarrow\)Sim registration pipeline that makes simulation annotations valid for real sensor data. (C) A scalable, open, containerized generation system enabling unlimited new scene generation with minimal manual labeling.
The BOP benchmark [11] consolidates eight pose estimation datasets (LineMOD [12], YCB-Video [13], T-Less [14], HomebrewedDB [15], and others), providing calibrated RGBD frames with CAD-model annotations and 6-DoF ground truth. These datasets are invaluable for detection and shape reasoning but carry no manipulation annotations: grasping labels, robot reachability, and physical object properties are absent. The MVIP dataset [16] extends benchmarking to industrial multi-view settings with 308 object classes, including real object weight and language property annotations —– but similarly focuses on recognition rather than manipulation.
Cornell [17] and Jacquard [18] provide 2D planar grasp rectangles — a useful simplification but incapable of representing tilted or sideways approach configurations required in real manipulation.
GraspNet-1Billion [2] is the dominant 6-DoF benchmark, providing real RGBD tabletop imagery with 1B analytically scored grasp candidates. However, its force-closure metric [2] assumes rigid, perfectly known contact geometry, which has been shown to correlate poorly with real-robot success under dynamic perturbation [19]. Graspnet further reports only a moderate grasping success correlation between a subset of “good” grasp candidates and tested robot success. Critically, GraspNet carries no robot-reachability filtering: many high-scoring candidates require the arm to reach through the support surface or exceed workspace limits.
ACRONYM [20] advances beyond analytic scoring by executing 17.7M grasp candidates in NVIDIA FleX under gravity-free shaking, but it provides no visual observations, uses non-textured ShapeNet meshes [21] that produce a large sim-to-real gap, and does not model robot kinematics. GraspIT directly addresses all three limitations.
Dex-Net 2.0 [19] introduced the Grasp Quality Convolutional Neural Network (GQ-CNN), trained on synthetic depth images paired with analytic grasp-quality metrics to predict parallel-jaw grasp success from a single depth image. While demonstrating strong sim-to-real transfer for planar grasps, Dex-Net 2.0 relies on force-closure as its simulation ground truth, inheriting the correlation gap between analytic scores and real execution that motivates our slip-test design. Dex-Net 4.0 [22] extended this to ambidextrous policies combining suction and parallel-jaw grasping, reporting \({\sim}9\)–\(98\%\) real-robot success depending on quality score — the correlation range that motivates GraspIT’s staged validation. Contact-GraspNet [23] moves beyond quality prediction to direct 6-DoF grasp generation from point clouds in cluttered scenes, using GraspNet-1B as its training corpus. Its strong real-world performance highlights the value of scene-level RGBD observations — a modality absent from ACRONYM and present in GraspIT. Critically, none of these methods validates candidate reachability via robot trajectory planning, nor provides a Real\(\leftrightarrow\)Sim registration link; GraspIT is designed to supply exactly these missing components as a pretraining foundation for next-generation object manipulation models.
Large-scale imitation learning datasets — BridgeData V2 [24], DROID [25], and Open X-Embodiment [26] — provide paired observation-action sequences and have driven the development of generalized manipulation policies [3], [4], [27]. However, they carry no explicit 3-D geometry, CAD models, or grasp quality scores; they teach what to do but not why it works physically. ManiSkill2 [28] provides simulation-based demonstrations with object metadata but focuses on task-success trajectories rather than physical object properties or the Real\(\leftrightarrow\)Sim link. GraspIT is positioned as a complementary pretraining foundation whose physically grounded labels can improve the visual representations consumed by VLAs and diffusion policies [8], [29], [30].
Recent work has shown that explicitly injecting geometric knowledge into visual encoders improves downstream robotic perception. Here, Yang [9] recently quantified a \(2{\times}\) depth-estimation gap between the VGGT geometric foundation model and the GR00T-N1.5 VLA’s encoder, confirming that semantic pretraining leaves a measurable physical reasoning deficit. GraspIT is the first dataset designed to support pretraining that fills this object-level physical grounding gap with directly manipulable, physics-validated labels for robot tabletop manipulation tasks.
Synthetic data generation with domain randomization [31] and photorealistic rendering [32], [33] has become standard practice for pose estimation and detection training. Robot-assisted annotation pipelines [34] demonstrate that real-world 6-DoF labels can be collected with minimal user interaction. Our Real\(\leftrightarrow\)Sim loop extends this philosophy to grasp quality: annotations generated in simulation are explicitly validated for transfer to real sensor frames through metric 3D registration, addressing the domain gap documented in the BOP challenge [11].
| Dataset | Year | Scenes | Frames | A | B | C | D | E | F | G | H | I |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| NYU-Depth V2 [35] | 2012 | 464 | 1 499 | |||||||||
| LineMod [12] | 2013 | 15 | 15k | \(\approx\) | ||||||||
| YCB-Video [13] | 2018 | 92 | 134k | \(\approx\) | ||||||||
| GraspNet-1B [2] | 2020 | 190 | 97k | \(\approx\) | \(\approx\) | |||||||
| ACRONYM [20] | 2020 | – | – | \(\approx\) | \(\approx\) | |||||||
| ManiSkill2 [28] | 2023 | 20+ | – | \(\approx\) | \(\approx\) | |||||||
| Open X-Emb. [26] | 2023 | – | 1M+ ep | |||||||||
| DROID [25] | 2024 | – | 76k ep | |||||||||
| GraspIT (ours) | 2026 | 1 035 Sim + 100 Real | \(\sim316\)k | \(\approx\) | ||||||||
| = fully provided; \(\approx\) = partially or generate-able; = absent. | ||||||||||||
GraspIT is designed not as a static benchmark but as a living, expandable generation system: a closed Real\(\leftrightarrow\)Sim loop (Fig. 2) that produces arbitrarily large, fully annotated RGBD corpora from a single consistent pipeline. Its design is guided by the observation that downstream robot manipulation models suffer most from two compounding deficits—the lack of physically grounded pretraining data and the lack of a principled bridge between simulation labels and real sensor observations [8], [9]. All tools are packaged as Docker containers and hosted at https://github.com/KochPJ/GraspIt.
GraspIT rests on four design pillars:
Rather than generating billions of analytically scored candidates, GraspIT validates each through a four-stage physical slip test (Sec. 3.5), yielding a continuous quality score \(s \in [0,1]\). The base score \(b\) is determined by the last stage successfully completed:
\[b = \begin{cases} 0.00 & \text{trajectory planning to closure failed} \\ 0.25 & \text{trajectory OK; lift failed} \\ 0.50 & \text{lift OK; horizontal oscillation failed} \\ 0.75 & \text{horizontal OK; pendulum swing failed} \\ 1.00 & \text{all four stages passed} \end{cases} \label{eq:base95score}\tag{1}\]
The final quality score additionally penalizes excessive slip, even within a passed stage. For each stage \(i\) that is executed, if the measured positional slip \(|\Delta\mathbf{p}_i| > 2\,\text{cm}\) or the measured orientational slip \(|\Delta\boldsymbol{\theta}_i| > 10°\), a penalty of \(\tfrac{1}{12}\) is subtracted: \[s = b \;-\; \frac{N_{\text{xyz}>2\text{cm}}}{12} \;-\; \frac{N_{\text{rpy}>10°}}{12}, \qquad s \in [0, 1], \label{eq:quality95score}\tag{2}\] where \(N_{\text{xyz}>2\text{cm}}\) (resp.\(N_{\text{rpy}>10°}\)) counts the number of executed stages in which the positional or orientational slip exceeds the threshold. A grasp is annotated good if \(s \geq 0.50\) and bad otherwise.
This two-level design is intentional: a grasp that nominally passes the lift stage but displaces the object by 3 cm can be penalized below the “good” threshold, reflecting real-world unreliability. The denominator 12 is calibrated so that a single maximal-slip violation on a 0.75-base grasp still keeps it marginally good (\(0.75 - 1/12 \approx 0.67\)), while two violations on a 0.50-base grasp push it to bad (\(0.50 - 2/12 \approx 0.33\)). We clamp scores between \([0, 1]\) when reaching a theoretically reachable negative score.
This design is motivated by the finding that Dex-Net 4.0’s [22] force-closure scores of \(s{=}0.1\) yield only \({\sim}9\)–\(23\%\) real-robot success, whereas \(s{=}1.0\) yields \({\sim}93\)–\(98\%\). Each stage of our slip test exposes a distinct failure mode absent from static force-closure analysis.
Every candidate passes collision-free trajectory planning by the Franka Panda arm via cuRobo [36] before the slip test. Candidates requiring approaches through obstacles or beyond workspace limits are discarded, ensuring all positive labels are actionable for a deployed system.
Real scene geometry is registered into Isaac Sim; simulation-validated annotations are back-projected to all real camera poses (see Fig. 2). A single capture session therefore yields both real RGB-D images and unlimited additional photorealistic synthetic views at novel viewpoints, all sharing a consistent annotation coordinate frame (Fig. 5).
The Docker containerized pipeline accepts any registered scene, enabling the progressive integration of existing datasets (BOP, GraspNet) into the GraspIT framework and extension to new robot models, environments, or sensor modalities.
Tabletop scenes are assembled in NVIDIA Isaac Sim [37] by a probabilistic scene sampler that draws 10–20 objects uniformly at random from the ABC dataset [10] (\({\sim}10\)k accessible curated CAD meshes spanning industrial, organic, and household geometries). Each object per scene is randomly scaled to a maximum side length of \([2, 20]\)% of the maximum table side length—ensuring object scale diversity, graspable and non-graspable object instances, and disconnecting grasping candidates from object IDs. The sampled and scaled objects are then randomly oriented, spawned \([20, 40]\)cm above the table, and dropped with gravity physics enabled to generate fully randomized, physically plausible scene configurations. We use ten table models with domain-randomized PBR textures, HDR lighting, and ambient occlusion, so each generated Sim scene presents a unique visual distribution (Fig. 3).
For each scene, 256 camera poses are sampled from a hemispherical trajectory at 1–2 m from the table center. Each frame is rendered at \(1920{\times}1080\) and exported with: photorealistic RGB; metric depth with structured-light noise; per-instance semantic segmentation from the scene graph; ground-truth 6-DoF object pose; and complete CAD point cloud. Because of the large 10k ABC object pool, every new generation run introduces novel object combinations, preventing object-identity overfitting.
Every GraspIT frame carries the full annotation stack shown in Fig. 4: RGB-D: photorealistic color image and metric depth with structured-light noise applied to match real sensor characteristics [33]. Instance segmentation: per-object binary masks derived directly from the Isaac Sim scene graph. 6-DoF object pose: ground-truth \(SE(3)\) transform for every visible object. Camera Calibration: per view extrinsic and intrinsic calibrations. 3D Point Cloud: full CAD surface geometry. Physical properties: geometric centroid (center-of-mass proxy under uniform density), spatial extent, visibility (%), and convex-hull volume fraction.Validated 6-DoF grasps: slip-test quality scores per candidate grasp with translation and orientation (the pose origin is placed between the fingertips). Grasping and Slip-Test Trajectories: each grasp attempt has a robot trajectory with respect to the reached slip-test stage, which could be followed in Isaac Sim with full demonstration capturing (robot states, camera observations, and action commands) for policy learning.
Figure 4: GraspIT Annotation Modalities. RGB, metric depth with sensor noise, per-instance segmentation, and validated 6-DoF grasp candidates overlaid on the 3D point cloud in camera coordinates.. a — RGB, b — Depth, c — Masks, d — Grasp+Shape
Physical objects are 3D scanned using TRELLIS [38], a generative image-to-3D reconstruction model. A set of high-resolution RGB images is captured under controlled lighting from multiple viewpoints; TRELLIS reconstructs a high-fidelity textured 3D asset with PBR material attributes (base colour, metallic, roughness). Absolute scale is recovered by fitting the reconstructed mesh to real-world scene geometry via multi-view ICP using metric camera poses from the robot’s high precision forward kinematics. GraspIT initial release includes 100 real objects scanned via this pipeline (See Fig. 5 for example illustration).
A UR-5 arm equipped with an Intel RealSense D405 (\(1280{\times}720\)) executes a hemispherical scanning trajectory of 256 predefined joint configurations. Eye-in-hand calibration provides the rigid transform \(T_\text{cam}^\text{EE}\), so every camera pose in world coordinates is obtained from forward kinematics alone.
For every object, grasp candidates are sampled by antipodal point sampling on the down-sampled surface point cloud (1000 points). From all initial points, geometric antipodal filtering (\({\sim}10\%\) pass) and collision checking with Franka Panda trajectory planning via cuRobo [36] and Lula [39] discards kinematically infeasible configurations before physics validation (passes \(\sim88.54\)% of remaining) – reducing the pool to \({\sim}307\) candidates per object. We use the Franka Hand gripper with \(50\)mm finger length and \(80\)mm opening width. The center grasping point between the fingertips marks the pose SE(3) origin, and the orientation defines the approach direction of the gripper.
Surviving candidates enter the Robot School (Fig. 6): parallel Franka Panda instances in Isaac Sim simultaneously execute each grasp through four sequential perturbations. Position slip \(>2\) cm or orientation slip \(>10°\) at any stage constitutes failure, yielding the continuous quality score defined in Eq. 2 . The four stages are: 1) Trajectory + Close: trajectory to gripper fingers antipodal force-closure contact; failure indicates a geometric collision, insufficient contact, or slip. 2) Lift: 20 cm vertical displacement under gravity; failure indicates insufficient friction force between the fingertips, the Franka Hand closure-force (\(70\)N), and the object material. We use the default friction coefficient of the Isaac Sim material for all objects and the fingertips (\(\mu_s 1.0\)), and Isaac Sim scales object weight linearly with object volume (1000 kg/m\(^3\)). 3) Horizontal oscillation: linear perturbation in the horizontal plane; exposes rotational instability. 4) Pendulum swing: rotation of joint 5; tests resistance to moment arms created by off-center mass.
This staged protocol captures failure modes invisible to static force-closure: an object that passes vertical lift may still slip under pendulum torque, and the quality score encodes which failure mode occurred (Fig 6) and to what extent slip occurred.
In the top-down reference frame, ground-truth object poses are obtained via ICP-refined CAD fitting and propagated to all 256 real views through relative camera transforms. The registered scene is then imported into Isaac Sim; additional synthetic views are rendered at the real camera intrinsics, and per-instance segmentation masks are back-projected to real frames, and grasps are annotated in the slip test (see Fig. 7).
For policy learning (behavior cloning), successful demonstrations are required, while failures can be used for reinforcement, out-of-distribution detection, and data curation (filtering and uncertainty estimation) [40]. The GraspIT Robot School is capable of capturing and streaming robot states, camera observations, and goal conditioning into generated successful and failure demonstrations. In combination with photorealistic Real\(\leftrightarrow\)Sim assets, this yields an unlimited exploration foundation that can be finetuned with minimal real data demonstrations to address real applications efficiently and at scale.
Tab. 2 summarizes the GraspIT initial release. The dataset comprises 1,035 simulated scenes, each rendered at \(1920{\times}1080\) from 256 hemispherical camera poses, yielding 264,960 annotated frame sets (each with RGBD, Mask, etc). Scenes contain \(13.9 \pm 2.7\) objects on average (range 8–19), from 7,396 randomly drawn unique object identities from 10k ABC object pool (\(\sim73\%\) of ABC pool used).
In total, 2,275,352 grasp candidates were evaluated by the Robot School, of which 1,887,185 (82.94%) are annotated good (\(s \geq 0.50\)) and 388,167 (17.06%) bad (\(s < 0.50\)). The mean score across all grasps is \(\bar{s} = 0.72 \pm 0.30\); good grasps average \(\bar{s}_\text{good} = 0.85\) while bad grasps average \(\bar{s}_\text{bad} = 0.13\).
We use Nvidia RTX6000 GPUs to generate GraspIT. A single GPU can create a batch of 100 scenes in less than a week’s compute. Generating GraspIT takes \(\sim2-3\)month on a single GPU (with the majority of the time being used for robot trajectory computation and grasp evaluation).
The pipeline ablation stats (Tab. 2) show that 88.54% of attempted candidates obtain a planned trajectory; of those, 88.48% survive horizontal perturbation, and 94.26% survive the pendulum test. The largest attrition is at the trajectory-planning stage (11.46% of all candidates receive \(s{=}0.00\)), confirming that robot-reachability filtering is a genuine bottleneck.
| Property | Value |
|---|---|
| Sim scenes | 1 035 (256 sim views each) |
| Real scenes | 100 (256 real + 256 synth views each) |
| Sim annotated frame sets | (1 035 + 100)\(\times\)256 = 290 560 |
| Real annotated frame sets | 100\(\times\)256 = 25 600 |
| Total annotated frames | 290 560 + 25 600 = 316 160 |
| Image resolution | 1920\(\times\)1080 (Sim) 1280\(\times\)720 (Real) |
| Unique objects | 7 396 (Sim) + 100 (Real) |
| Objects per scene | \(13.9 \pm 2.7\)(range 8–19) |
| Grasp Pipeline | |
| Total candidates passed force-closure | 2 275 352 |
| w/ trajectory | 2 014 689 (88.54%) |
| Total Good grasps (\(s \geq 0.50\)) | 1 887 185 (82.94%) |
| Total Bad grasps (\(s < 0.50\)) | 388 167 (17.06%) |
| Mean Grasps per Object& Scene | 307\(\pm\)649 & 2198\(\pm\)1866 |
| Good & Bad Grasps per Scene | 1823\(\pm\)1612 & 375\(\pm\)423 |
| Good & Bad Grasps per object | 255\(\pm\)567 & 52\(\pm\)155 |
| Mean score \(\bar{s}\) (Good & Bad) | 0.72 \(\pm\) 0.30 (0.85 & 0.13) |
| No Trajectory to closure (\(s = 0.00\)) | 260 663 (11.46%) |
| Fail lift (\(s = 0.25\)) | 33 029 (1.45%) |
| Penalized large slip (\(s < 0.5\)) | 94 475 (4.15%) |
| Fail horizontal (\(s = 0.50\)) | 167 549 (7.36%) |
| Fail pendulum (\(s = 0.75\)) | 883 733 (38.84%) |
| Full success (\(s = 1.00\)) | 835 903 (36.74%) |
The 82.94% good-grasp rate reflects the stringency of slip-test validation relative to analytic metrics. In particular, the penalized “bad” bucket (94,475 grasps) demonstrates the value of Eq. 2 : these candidates nominally pass two or more stages yet exhibit excessive slip, making them unreliable in practice; a pure stage-count score would incorrectly label them as good. The fine granular grasp scoring system allows for future work on novel grasp critic functions that can anticipate the quality of a physically plausible grasp based on, e.g., the generated object properties of center-of-mass, shape extend, or volume.
Tab. 3 compares GraspIT’s annotation strategy against competing datasets.
| Dataset | #Obj. | #Grasps | Phys. | Reach. | Scores | Obs. | Sim\(\leftrightarrow\)Real |
|---|---|---|---|---|---|---|---|
| Cornell [17] | 240 | 5.1k | 1 | ||||
| Jacquard [18] | 11k | 1.1M | 1 | ||||
| GraspNet-1B [2] | 88 | 1.1B | 10 | ||||
| ACRONYM [20] | 8.8k | 17.7M | \(\approx\) | 2 | |||
| GraspIT (ours) | 7.4k | 2.3M | 5 |
GraspIT is the first dataset to simultaneously provide: photorealistic RGBD observations at scale; robot-reachability-filtered grasps; staged, physically-validated quality scores with explicit slip-penalty calibration (Eq. 2 ); complete 3D object shape, including occluded geometry; and a bidirectional Real\(\leftrightarrow\)Sim registration link (Tab. 1, 3).
The 17.06% bad-grasp fraction (388,167 candidates) is a deliberate feature rather than a limitation. The penalized bucket (94,475 grasps that pass two or more stages yet score \(< 0.50\) due to excessive slip) provides particularly informative hard negatives for training grasp-quality regressors: these candidates look geometrically plausible but fail under realistic perturbation—exactly the failure mode that force-closure metrics miss. Out-of-distribution detection, reinforcement-learning negative examples, and data curation algorithms [40] all benefit from this granular annotation.
Recent analyses [8], [9] show that current VLAs inherit a depth-estimation deficit of \({\sim}2{\times}\) relative to geometric foundation models at the encoder level — a gap that downstream fine-tuning can only partially close. GraspIT’s physically grounded labels (shape, centroid, grasp quality) are designed to reduce this deficit when used as a pretraining stage for VLAs [4], [41], [42] and diffusion policies [29], [30] before task-specific fine-tuning — enabling the grounding of the encoder stages to understand object properties and affordance.
The ABC CAD library [10] (\({\sim}10\)k meshes) ensures highly diversity. With 7,396 unique objects across 1,035 sim scenes (mean 13.9 objects/scene) and 100 real objects and scenes, each new generation run introduces novel combinations that prevent object-identity overfitting. This scalability stands in contrast to GraspNet (88 objects) or YCB-Video (21 objects), where models risk learning object-specific shortcuts.
The Real\(\leftrightarrow\)Sim pipeline can enhance rather than replace existing datasets. BOP sub-datasets and GraspNet scenes with known object meshes can be imported into Isaac Sim and their grasp candidates generated or validated via the Robot School, converting analytic annotations to slip-test-validated labels while preserving the original real RGBD observations.
(i) No Downstream Evaluation. At this stage, we do not provide a downstream evaluation of GraspIT as the dataset has no single purpose that has an established evaluation. A comprehensive downstream benchmark is beyond the scope of this initial release; we leave comparative evaluation to future work. (ii) Parallel-jaw gripper only. GraspIT covers only parallel-jaw configurations; multi-finger and suction grasps are out of scope. (iii) Single robot model. Only the Franka Panda arm is used for reachability filtering; transferred labels may not be directly valid for different kinematic chains. (iv) No task demonstrations. GraspIT provides grasp-level annotations with trajectories, but no full task-level demonstration trajectories yet. However, the trajectories can be recaptured with the full robot state, camera observation, and action space tracking. Extending the Robot School infrastructure to pick-and-place demonstrations with language annotation is a direct next step and can be adopted by GraspIT users depending on their setup and modality needs. (v) generated physical properties. The center-of-mass is derived from CAD geometry, while material dependent friction and volume are ignored. To further improve the current approximations, real object weighting, material simulation, and volume estimations have to be implemented into the object corpus to narrow the gap between the simulated manipulations and real world behaviors. (vi) Real\(\leftrightarrow\)Sim scan and registration offset scanning, scaling, and registration of the real objects create inevitable unique and method- or hand-precision-dependent offsets that are inherent in the projected annotations. Truly measuring them in real world evaluations is impossible and can only be approximated with high precision scanning that exceeds our project budget, or their severity can be reflected in the future work downstream task evaluations and pretraining ablations.
Extending the Robot School to full manipulation demonstrations (grasp \(\to\) lift \(\to\) place) with language annotation would create a grounded policy-learning dataset bridging GraspIT’s physical labels and demonstration datasets such as DROID [25] and Open X-Embodiment [26]. Additional robot models, new sensor noise profiles, and multi-camera configurations can be parameterized at the Isaac Sim level without changes to the export tooling. We release GraspIT initially as an open foundation starting point and invite the community to contribute new scenes, objects, and annotations.
We presented GraspIT, a living dataset and generation system for physically grounded robot grasping manipulation. By combining photorealistic Isaac Sim rendering with a four-stage physical slip test, robot-reachability filtering via cuRobo, a continuous quality score that additionally penalizes excessive positional and orientational slip (Eq. 2 ), and a bidirectional Real\(\leftrightarrow\)Sim registration loop, GraspIT provides annotation coverage that no existing dataset achieves simultaneously.
The initial release comprises \(\sim316\)k annotated RGB-D frames across 1,035 simulated and 100 real scenes spanning \(\sim7.4\)k unique object geometries, with \(\sim\)2.3M slip-test-validated grasp candidates and fine grained scores \([0, 1]\) and slip-test stage pass annotations — of which 82.94% are annotated as good (\(s \geq 0.50\)) and the remaining 17.06% provide graded negative examples, including a penalized bucket of physically ambiguous near-misses (94,475 (4.15%) candidates). The mean quality score of \(\bar{s} = 0.85\) across all validated “good” grasps reflects the genuine physical rigor of the staged slip-test, in contrast to the binary or force-closure-only labels of prior datasets.
With our staged slip-test, we demonstrated again that antipodal force-closure is not sufficient to yield high-quality “good” grasp candidates. Within our slip-tests, we showed that adding path planning for physical plausibility (11.46% filtered from force-closure accepted candidates) and staged slip-test filtering (grasped but slip filtered 17.06% from force-closure accepted candidates) yields a better approximation for “good” grasp candidates than force-closure alone. Moreover, filtering “bad” grasping candidate targets adds the opportunity to investigate future work on critic functions and grasping quality aware grounding.
All tools, Docker containers, and the GraspIT release are publicly available at https://github.com/KochPJ/GraspIt. We invite the community to contribute new objects, environments, robot models, and task demonstrations to grow the GraspIT database as a shared foundation for generalized robotic manipulation.