1 Introduction↩︎

\(^{\dagger}\)Task counts follow source-specific definitions and are not directly comparable. \(^{\ast}\)Ego4D uses the current project-page counts (923 participants; seven cameras); its CVPR 2022 paper reports 931 camera wearers. OpenEgo’s six denotes source datasets, not participants.

Embodied foundation models are increasingly constrained by the scale and quality of real-world interaction data [7], [8]. Unlike language models, which can learn from internet-scale text, robot learning requires physical demonstrations that capture how humans perceive scenes, move their hands, contact objects, and complete temporally extended tasks [8]. Such data must be not only large and diverse, but also structured in geometry, time, and semantics for manipulation learning [8]. Empirical scaling studies further show that diversity across environments and objects can matter more than simply increasing the number of demonstrations within fixed conditions [7].

Egocentric video provides a natural and scalable modality for this problem [1], [2]. By recording interaction from the actor’s viewpoint, it captures hands, objects, scenes, and action progress in the same visual stream [1], [2]. Compared with third-person video, egocentric video is closer to the perceptual structure of robot execution [3], [6]. Compared with robot teleoperation, it can be collected more naturally, continuously, and cheaply in everyday environments [8], [9]. Recent datasets such as Ego4D, EPIC-KITCHENS, EgoDex, EgoLive, OpenEgo, and EgoScale have advanced egocentric data along multiple axes, including larger-scale daily activity capture, finer-grained hand pose annotation, richer manipulation scenarios, and stronger alignment with robot learning [1]–[6].

However, a central gap remains. The community does not simply need more egocentric video. It needs open data infrastructure that can continuously collect, systematically process, and directly support model training. High-quality egocentric datasets often rely on specialized head-mounted devices, AR/VR hardware, or controlled capture protocols [3], [4]. Specialized hardware can improve pose and camera sensing, but it narrows accessibility relative to commodity-smartphone collection [3], [4], [9]. In contrast, large-scale passive video is easier to obtain, but usually lacks the hand motion, camera trajectory, action boundaries, and physical structure required for robot training [3], [5]. Recent efforts have consequently developed dataset-specific pipelines for hand-pose unification, human-to-robot transfer, and raw-video processing [5], [6], [9]. However, these pipelines remain fragmented and have yet to form a unified infrastructure. This fragmentation underscores the distinction between releasing videos and releasing data that can be used directly for training.

To address this gap, we introduce Open-AoE, a community-oriented egocentric manipulation dataset and toolchain. Open-AoE builds on the AoE consumer-smartphone collection framework [9] and releases the full pipeline from raw capture to model training. The first release contains approximately 2,000 hours of egocentric human manipulation data collected in natural environments by 500+ contributors using 400+ smartphone models. The dataset provides structured signals including videos, text descriptions, MANO hand poses, camera trajectories, and atomic action segments. For dataset production, Open-AoE releases the cloud-side processing pipeline that converts uploaded smartphone clips into structured samples through quality screening and privacy erasure, temporal segmentation, semantic labeling, camera-trajectory estimation, hand reconstruction, atomic-action annotation, and multi-stage quality inspection. Separately, the downstream data consumption toolchain starts from the released corpus and supports synchronized visual inspection, 4D hand-object interaction reconstruction, cross-embodiment motion retargeting, and robotized-video generation. The Training-ready toolchain then maps the aligned video, hand, camera, and language signals into action representations and training interfaces for three model directions: Vision-Language-Action (VLA) policies, World Action Models (WAMs), and World Models.

The goal of Open-AoE is not to release an isolated video collection, but to open a reusable capture-process-reconstruct-train data production loop. Starting from the same egocentric corpus, researchers can inspect data, reconstruct 4D interactions, replay motions on robots, train vision-language-action models, learn world action models, or study world model learning without rebuilding a private post-processing stack. By lowering both the barrier to contributing data and the barrier to using data, Open-AoE aims to turn low-cost smartphone-captured human manipulation video into practical open infrastructure for embodied intelligence research.

Our contributions are three-fold:

• We release approximately 2,000 hours of real egocentric human manipulation data collected with consumer-grade smartphones in natural environments, covering diverse participants, devices, and everyday manipulation scenarios.

• We open an end-to-end data processing pipeline that converts raw smartphone videos into structured training samples with hand poses, camera trajectories, action segments, and language descriptions.

• We provide a downstream embodied learning toolchain for data visualization, reconstruction, and cross-embodiment retargeting, together with training-ready representations and interfaces for three major model directions: VLA policies, WAMs, and World Models.

2 Related Work↩︎

2.1 Open Egocentric Data Sources for Embodied Learning↩︎

Open egocentric corpora differ chiefly in the supervision they expose above raw video. EPIC-KITCHENS-100 [2] and Ego4D [1] provide broad coverage of daily activities, environments, and language. Geometry-rich releases make manipulation more directly observable: HOI4D [10] provides RGB-D sequences with hand and object poses, segmentation, and reconstructed geometry; HOT3D [11] adds motion-capture ground truth for multi-view hand, object, and camera poses; EgoMimic [12] pairs camera motion and 3D hand tracking for imitation learning; and EgoDex [3] scales native hand tracking and language annotation to 829 hours of dexterous tabletop manipulation.

Other releases address normalization, scale, or sensor fidelity. OpenEgo [5] consolidates 1,107 hours from six public datasets into standardized hand poses and localized action primitives. EgoScale [6] studies dexterous policy scaling with more than 20,000 hours of in-the-wild video, while EgoLive [4] contributes 1,680 hours of 60-FPS stereo recordings from service scenarios with camera motion, 3D hands, depth, masks, and sub-task descriptions. Open-AoE [9] complements these efforts with a 2,000-hour smartphone-captured release spanning 500+ contributors, 400+ device types, 400+ scenes, and 8,000+ tasks, together with atomic action descriptions, MANO hand motion, camera trajectories, and reusable processing and model-integration tools.

2.2 Embodied Applications of Egocentric Data↩︎

Reconstruction and cross-embodiment transfer. Egocentric video becomes actionable when implicit motion is converted into geometric or control-oriented supervision. HaWoR [13] recovers world-space hand motion from monocular video, while EgoInfinity [14] reconstructs metric 4D hand-object trajectories and supports robot retargeting. EgoAERO [15] estimates contact-consistent trajectories from a single RGB-D demonstration, and EgoEngine [16] jointly generates robotized observations and feasible robot actions. These outputs support several downstream uses: reconstructed hand-object assets expose contact and object motion, retargeted trajectories provide robot-facing supervision, and robotized videos preserve scene context while narrowing the visual embodiment gap. Visual adaptation, shared action spaces, active vision, and whole-body retargeting further address viewpoint and morphology differences across embodiments [17]–[21].

Vision-Language-Action policies. VLA methods learn action-aware representations directly from human video rather than requiring every example to be replayed by a robot. Being-H0 [22] tokenizes human motion for vision-language-action pretraining; EgoVLA [23] predicts wrist and hand actions before robot adaptation; and VITRA [24] derives robot-aligned supervision from real-life activities. EgoScale [6] further shows benefits from scaling diverse egocentric pretraining. The route from human observation to robot policy can therefore combine language-aligned task semantics with embodiment-aware action targets, but it depends on temporally stable segments, geometry, and action interfaces that remain meaningful after embodiment conversion.

World Action Models. WAMs occupy a middle ground between direct policy learning and unconstrained video generation: they learn action variables together with observation dynamics, including settings where explicit robot actions are unavailable. Latent-action learning can infer compact motion surrogates from video, although visual distractors may require additional supervision [25]. LaWAM [26] uses compact latent visual subgoals to reduce pixel-space generation latency, while DreamZero [27] jointly predicts future video and action and uses the learned dynamics as a zero-shot policy. This route makes temporal alignment between the current observation, the inferred or supplied action, and the resulting visual change especially important.

World Models. Predictive world models use egocentric video to learn reusable scene and interaction dynamics beyond a single robot policy. DreamDojo [28] pretrains a generalist robot world model on 44,000 hours of egocentric video with continuous latent actions; iVideoGPT [29] studies scalable interactive video prediction; and Ctrl-World [30] introduces controllable generation for robot manipulation. Such models can exploit both weak latent actions and explicit hand, camera, or robot controls, provided that long videos are segmented into coherent transitions and paired with stable conditioning signals. Across reconstruction, retargeting, VLA, WAM, and world-model learning, the shared requirement is an interface between diverse human video and model-consumable supervision. Open-AoE is designed to provide both sides of that interface through a broad smartphone corpus and its associated processing and conversion tools.

3 Open-AoE Dataset↩︎

3.1 Data Processing Pipeline↩︎

As shown in Figure [fig:data-pipeline], the pipeline comprises an online capture stage and an offline processing stage. It proceeds through four stages, namely edge-side online detection, offline quality checking with scene labeling, reconstruction and annotation, and quality inspection with data delivery, and finally produces the anonymized training corpus.

3.2 Edge-side Online Detection↩︎

On the capture device, lightweight on-device vision models gate recording in real time and govern capture quality, so that only clips containing meaningful hand-object interaction are retained [9]. A hand-visibility check starts recording automatically once hands appear and stops it when they leave the frame, and prompts the user by voice to readjust whenever a hand reaches the image border or becomes truncated, which keeps the manipulating subject fully visible. Operating and wearing compliance are monitored at the same time. The operating-spec check combines gesture detection with scene detection to flag non-standard actions and non-compliant capture scenes, while the wearing-spec check performs both a wearing check and a stability check to correct improper wearing poses and unstable views; either kind of issue is reported to the user through real-time voice feedback. The device also adapts to the lighting environment by turning on a constant fill light together with global auto-exposure under dim light, and it sustains real-time performance through a fixed focal length, motion stabilization, deblurring, and an adaptive frame rate, halting recording when storage becomes insufficient or the device overheats. This lightweight front-end design protects privacy at the source, saves storage and bandwidth, and uploads only valid and permitted clips to the cloud for offline processing.

3.3 Offline Quality Check & Scene Labeling↩︎

3.3.1 Image-based Detection↩︎

The first gate operates on sampled frames without deep models, relying on image-analysis algorithms and lightweight detectors. For video quality, a bank of rule-based detectors runs in parallel to flag the common defects that make recordings unusable, covering aspect ratio and rotation, file-header and stream integrity, pure-black or over- and under-exposed frames, insufficient duration, and decoding integrity, so that clearly corrupted videos are discarded early. A valid-recording check then removes clips without visible hands, non-egocentric recordings such as fixed-mount or third-person side views, and segments with severe camera jitter, leaving only valid first-person manipulation. Security-information erasure detects faces and other sensitive content and masks (pixelates) them in the frames, and it further replaces the personally identifiable information in the metadata, including the collector’s name and contact, with anonymized identifiers while rewriting the corresponding fields and directory names; the mapping is stored in full and supports incremental reuse, so the process can be safely re-run and the anonymization reversed when authorized. A hand re-check then verifies that both hands remain clearly visible by screening for residual third-person views, hand truncation, and an excessive occlusion ratio; a defect is confirmed only when it persists across several consecutive frames, which suppresses single-frame noise. Calibration finally validates the camera intrinsics for completeness and validity, since reliable calibration is a prerequisite for later pose reconstruction.

3.3.2 Video Slicing and Frame-Rate Fixation↩︎

After image-based detection, the video is split along contiguous qualified and unqualified spans and its frame rate is fixed, so that a long recording becomes several semantically independent segments (termed Parts), with qualified segments that are too short flagged separately. Each Part stays linked to its source video through a unified process-tag file that records the data lineage from the raw video to each Part and accumulates multi-level labels as the pipeline advances, establishing a data contract that holds throughout the pipeline.

3.3.3 Large-Model Detection↩︎

Qualified Parts then move to the large-model stage, where the vision-language model Qwen3.7-Plus performs higher-level semantic detection and scene labeling. Compliance detection re-checks at the semantic level whether a segment follows the collection specifications and identifies violations that are difficult to detect at the image level, while valid-action detection judges whether the action is valid and forms a complete, purposeful manipulation, complementing the earlier image-level screening. The same model then constructs a four-level label tree under a closed-set design, assigning a representative frame exactly one domain, one scene, and one task from predefined sets together with the salient objects it contains, after which these outputs are standardized into unified identifiers and accumulated over the domain, scene, task, and object dimensions to form a hierarchical label system.

3.4 Reconstruction & Annotation↩︎

Each qualified Part enters reconstruction, where camera-trajectory estimation, hand reconstruction, and atomic-action annotation are produced together. Camera poses are recovered by DROID-W [31], whose robust kernels are re-tuned for handheld and wearable capture to keep the 6-DoF trajectory stable under severe camera shake and torso-motion interference. Hand reconstruction starts from a two-hand detector trained on large-scale egocentric data collected through AoE, which supplies per-frame bounding boxes, links detections across frames, and fills in missing or occluded hands; building on these boxes, HaWoR [13] recovers 3D MANO [32] meshes that are metric-scale aligned through SLAM and brought into world coordinates, with sliding-window optimization enforcing temporal consistency. A global bundle adjustment over the HaWoR reconstruction and the dynamic-object masks from DROID-W then optimizes the hand meshes and the camera trajectory jointly in a single world coordinate frame, removing the scale drift and coordinate mismatch that arise across segment-wise reconstructions. From the recovered meshes, 21-joint hand keypoints are extracted, and each video is finally segmented into semantically coherent atomic clips labeled in English, with a human-in-the-loop review correcting model hallucinations.

3.5 Quality Inspection & Data Delivery↩︎

The reconstructed and annotated Parts pass a three-gate quality inspection before archiving, from which about 2,000 hours are curated from the full data pool for open-source release. The completeness gate retains Parts whose hand-reconstruction valid-frame ratio is high enough for reliable pose annotation; the correctness gate retains those whose inverse-kinematics failure rate stays low when retargeting to the 28-DoF joint space, which preserves downstream usability; and the consistency gate retains Parts with smooth, continuous camera trajectories free of abrupt jumps between adjacent frames. Finally, a randomly sampled subset undergoes human inspection to confirm annotation accuracy and to remove edge cases such as severe occlusion or extreme lighting, yielding the anonymized training corpus that constitutes the final output.

3.6 Data Distribution Analysis↩︎

Figure 2 summarizes the diversity observed in a randomly sampled 100-hour subset of the Open-AoE release. All panels use this same sampling scope, including semantic content, collection context, anonymous contributor coverage, consumer-phone models, and horizontal field of view (FOV). Wedge area, bar height, and word size encode relative prevalence or frequency rather than absolute duration.

3.6.0.1 Broad scenes, actions, and interaction objects.

The updated 100-hour audit used in Section 5 contains 32,407 distinct natural-language action descriptions, 175 action verbs, 8,030 object strings, and 135 scene labels. As reported in Figure 5, the updated temporal annotations achieve 99.99% temporal coverage, with a mean segment duration of 9.64 s and an annotation density of 13.97 segments per minute. Temporal coverage is computed from the union of annotated time spans, whereas mean duration and density are computed over annotation records; these quantities should therefore not be multiplied as if the annotations formed a strictly non-overlapping partition. These values supersede the interval and vocabulary counts from an earlier preprocessing snapshot. These distributions cover reusable manipulation primitives together with a long tail of foods, tools, containers, craft materials, garments, and household objects. Kitchens, tabletop or indoor settings, offices or desks, and bedrooms account for 24.8%, 23.3%, 17.8%, and 9.1%, respectively, while the remainder spans living rooms, workshops, bathrooms, outdoor areas, laundry spaces, and retail or service settings. The joint diversity of scene, action, and interaction object provides a substrate for compositional generalization.

3.6.0.2 Distributed contributor coverage.

The same 100-hour sample exhibits a long-tailed distribution across anonymous collector IDs. The median contribution is 2.6 minutes per ID, and the ten largest contributors together account for only 13.7% of sampled duration. This long-tailed coverage indicates that the sample is not dominated by a small group of collectors, preserving broader variation in manipulation habits, hand appearance, and recording environments.

3.6.0.3 A broad consumer-camera domain.

The random 100-hour sample covers 400+ consumer-smartphone market models. The nested donut groups brands in the inner ring and exposes leading models plus the aggregated model tail in the outer ring. To our knowledge, this is the broadest reported consumer-phone device coverage for an egocentric manipulation dataset. Restricting device statistics to the same random sample makes the model, semantic, context, and contributor distributions directly comparable under one evaluation scope. The resulting breadth provides substantial consumer-camera domain diversity within the sampled subset. Device identity is also a practical proxy for the underlying camera stack, including the lens, sensor, image-signal processor, resolution mode, exposure pipeline, and capture firmware. Compared with a single-headset collection, Open-AoE therefore introduces naturally occurring camera-domain variation at capture time.

3.6.0.4 Optical diversity for robust visual transfer.

Within the random 100-hour sample, the user-weighted horizontal-FOV distribution is multimodal: its three dominant 5-degree bins account for 30.7%, 30.2%, and 31.7%, concentrated around \(65^{\circ}\)–\(75^{\circ}\) and \(90^{\circ}\)–\(95^{\circ}\). The multimodal profile reflects recurring camera and lens configurations rather than one optical system. A downstream visual policy must recover hand-object geometry and motion across different projections, crops, distortions, and imaging pipelines. This variation provides a form of natural sensor-domain randomization that may improve robustness when transferring to unseen cameras, deployment rigs, and robot viewpoints.

Takeaway. Open-AoE is differentiated not only by scale, but by diversity at three coupled levels: what people manipulate, where and how they act, and the consumer-camera domain through which every interaction is observed. Together, these properties create a stronger basis for visually robust, cross-domain manipulation learning.

Scope and claim boundary. Every distribution reported in this subsection is computed from the same random 100-hour sample. The downstream benefit of camera-domain diversity remains a training hypothesis that should be validated through controlled ablations.

4 Open-AoE Toolchain↩︎

The Open-AoE Toolchain turns synchronized egocentric video, hand geometry, camera motion, and action semantics into research assets that can be inspected, reconstructed, retargeted, and used for learning. The released implementation, documentation, and model-integration recipes are available in the Open-AoE GitHub repository. Each AoE sample aligns undistorted RGB, camera intrinsics and trajectories, bilateral MANO reconstruction [32], validity masks, and atomic-action annotations on one timeline. These signals form a common evidence layer, but no single action vector is appropriate for human-motion modeling, robot control, and visual-dynamics learning at the same time. The toolchain therefore preserves the physical meaning of the source signals and selects a representation level according to the downstream problem.

4.1 AoE-Visualization↩︎

AoE-Visualization is the entry point for synchronized inspection and error diagnosis. It uses the undistorted egocentric video as the canvas, overlays MANO meshes, 21-keypoint skeletons, future wrist trajectories, and atomic actions, and presents a world-frame 3D view and timeline alongside the image. This joint view exposes reprojection error, missing-hand intervals, SLAM drift, and semantic-boundary misalignment before a downstream model can absorb them as apparent motion regularities. The same visual grammar is used for dataset examples and for reviewing intermediate toolchain outputs.

4.2 AoE-Reconstruct-Retarget↩︎

AoE-Reconstruct-Retarget connects an egocentric observation to three complementary outputs: 4D hand-object assets, robot-executable trajectories, and robotized video, as summarized in Figure [fig:reconstruct-retarget]. The outputs are coupled: reconstruction supplies object and contact references, retargeting supplies executable motion, and robotized video places that motion back into the real visual context.

4.2.0.1 4D hand-object reconstruction.

EgoInfinity [14] and Do-as-I-Do [33] lift monocular video into time-varying hand geometry, object pose, and interaction assets. Here, AoE contributes more than additional RGB frames: calibrated intrinsics and metric-scale camera trajectories help separate observer motion from scene motion, MANO supplies hand-scale and pose priors, and action segments delimit the temporal support of an interaction. When combined with object 6-DoF estimates, the timing of finger approach, grasp, and release can provide contact candidates, object-tracking constraints, and priors over manipulation phases. This allows large-scale egocentric data to convey knowledge about object affordances and hand-object coordination rather than only scene appearance.

4.2.0.2 Cross-embodiment retargeting.

This route asks how motion demonstrated by a human can become motion executable by a target robot. Phantom combines arm inverse kinematics with dex-retargeting [34] to map camera-local wrist and finger motion to Unitree G1 with Dex3 or Inspire hands. Retarget Galbot converts bimanual palm targets into Galbot/Galaxea TCP trajectories and gripper control, while Retarget Lab organizes comparable kinematic and physics-aware routes including EgoInfinity/G1, Do-as-I-Do/Sharpa, and SPIDER/XHand [35]. AoE’s bilateral MANO trajectories, validity masks, and atomic-action boundaries jointly provide supervision for wrist paths, finger synergies, bimanual role assignment, and grasp timing. Compared with isolated end-effector poses, these signals are better suited to learning cross-embodiment motion priors because they preserve the structure among reach, contact, manipulation, and release.

4.2.0.3 Robotized video.

This route removes the human arm and composites a simulated robot back into the original egocentric scene. Phantom and Retarget Galbot use MANO keypoints as geometric prompts for segmentation and combine video inpainting with MuJoCo rendering to produce temporally aligned robot overlays. Because it preserves the real environment, manipulated object, illumination, and task progression while changing the actor appearance, it can provide paired signals for human-to-robot visual-domain transfer, retargeting diagnosis, and robot-scene consistency learning. Its value is to create aligned observations of the same interaction under different embodiments, not to treat synthesized video as a substitute for real robot execution.

4.3 AoE-Training-Ready↩︎

AoE-Training-Ready adapts the same synchronized source signals to model-specific action semantics rather than claiming a universal training format; Figure [fig:training-ready] summarizes this representation spectrum and its downstream model connections. Multiple representations are necessary because human-motion learning, robot control, and video generation preserve different invariants, as summarized in Table 3: human modeling benefits from finger articulation, cross-embodiment control needs end-effector or joint targets, and egocentric dynamics must account for camera self-motion.

For recipes that require dense hand articulation, Open-AoE concatenates two 55D per-hand vectors into a 110D representation: \(2\!\times\![\mathrm{valid}(1)+\mathrm{wrist\;translation}(3)+\mathrm{wrist\;axis-angle}(3)+\mathrm{velocity}(3)+\mathrm{MANO\;pose}(45)]\). Translations and finite-difference velocities are expressed in meters and meters per second in the current-frame OpenCV camera coordinates (\(x\) right, \(y\) down, \(z\) forward), while rotations are represented in radians; because this frame moves with the wearer, the velocity contains both hand motion and camera ego-motion rather than purely inertial wrist motion.

4.3.0.1 Policy learning and cross-embodiment control.

This route compresses egocentric human motion into targets that a policy can predict and a robot can execute. VITRA [24], H-RDT [36], GR00T N1.7 [37], SmolVLA [38], and LeRobot-based [39] \(\pi_{0.5}\), ACT, and Diffusion Policy recipes consume dense MANO, wrist–fingertip, robot-joint, or hand-as-EEF targets according to their control abstraction. Atomic-action language supplies task intent, framewise hand geometry supplies motion realization, bilateral trajectories supply coordination structure, and long-tailed objects and behaviors in natural scenes supply affordance and behavioral diversity. Together, these signals support a hierarchy from observing and describing an interaction to predicting embodiment-constrained motion, without requiring robot joint labels for every human video.

4.3.0.2 World-action and action-conditioned video learning.

This route models how the next action or future visual state changes given the current observation and an action condition. DreamZero [27], LingBot-VA [40], Ctrl-World [30], and iVideoGPT [29] use either dense next-hand states or compact hand-plus-camera actions to learn action prediction and visual dynamics. Large image motion in egocentric video may arise from hand-object interaction or from motion of the head or phone; because AoE synchronizes hand and camera trajectories, these two causal sources can be represented as separate conditions or controlled variables. This supplies direct signals for controllable visual prediction, for testing whether a model genuinely uses its action condition, and for learning action representations that are robust to a moving camera.

4.3.0.3 Latent-action and world-model learning.

This route discovers action variables from visual change and uses limited structured supervision to determine what those variables should represent. GenieRedux [41] and AdaWorld [42] learn latent actions from videos or frame pairs, LAOM [25] uses limited hand-action supervision to resist camera ego-motion as a strong distractor, and DreamDojo [28] injects explicit MANO motion into its action condition. AoE therefore contributes three forms of knowledge: scene dynamics from unlabeled video, nuisance-factor identification from camera trajectories, and controllable weak supervision from MANO motion and action segments. The same data can move from discovering latent factors of change, to grounding those factors as hand actions, to controlling future video with explicit motion, thereby connecting unsupervised world modeling with embodied control.

Together, AoE-Visualization, AoE-Reconstruct-Retarget, and AoE-Training-Ready connect data inspection, interaction reconstruction, cross-embodiment conversion, and model learning in one traceable path. The central design is not the number of interfaces, but the reuse of the same egocentric evidence at geometric, action, and visual-dynamics levels.

5 Dataset Analysis↩︎

To assess whether Open-AoE exhibits a distinctive data profile beyond dataset scale, we compare it with OpenEgo [5], EgoDex [3], and EgoXtreme [43] across four complementary dimensions: visual representation diversity, semantic and temporal supervision, annotation consistency, and downstream training utility. Approximately 100 hours of data are included for Open-AoE, OpenEgo, and EgoDex, whereas EgoXtreme is evaluated using its entire available release because its total duration is substantially smaller. The analysis therefore focuses not only on the breadth of the visual and semantic distributions, but also on the completeness, consistency, and practical usability of the associated supervision.

5.1 High-Dimensional Visual Diversity↩︎

We use CLIP [44] image embeddings to compare the visual feature distributions of the four datasets. Visual frames are sampled from each video sequence and encoded using the CLIP image encoder. Each embedding is then \(\ell_2\)-normalized, yielding \(\mathbf{z}_i\in\mathbb{R}^{D}\), where \(i\) indexes a sampled visual observation and \(D\) is the CLIP feature dimension. To control for unequal dataset sizes, we conduct 3,000 balanced random trials, each sampling 2,000 embeddings without replacement from every dataset. All reported values are averaged across these trials.

We first examine the covariance spectrum of each dataset in the original normalized CLIP space. For dataset \(s\), let \(\lambda_{s,j}\) denote the \(j\)-th eigenvalue of its embedding covariance matrix, where \(j=1,\ldots,D\), and let \(p_{s,j}\) be the corresponding proportion of total variance. Effective Rank is the exponential entropy of the normalized covariance spectrum, whereas Participation Ratio measures its inverse concentration: \[p_{s,j}=\frac{\lambda_{s,j}}{\sum_r\lambda_{s,r}}, \qquad R_{\mathrm{eff}}^{(s)} =\exp\!\left(-\sum_j p_{s,j}\log p_{s,j}\right), \qquad R_{\mathrm{PR}}^{(s)} =\frac{\left(\sum_j\lambda_{s,j}\right)^2}{\sum_j\lambda_{s,j}^{2}}.\] Both metrics estimate the effective number of feature directions that carry substantial variance. Higher values therefore indicate that visual variation is distributed across a larger set of active directions rather than being concentrated in a few dominant modes.

We next measure how broadly and evenly each dataset occupies a shared visual feature partition. In each trial, the pooled balanced embeddings are transformed using a common PCA representation and partitioned into a shared codebook of \(K=50\) cells. Let \(n_{s,c}\) denote the number of embeddings from dataset \(s\) assigned to cell \(c\), and let \(q_{s,c}\) denote the corresponding occupancy proportion. Meaningful Coverage counts cells containing at least \(\tau\) samples, while Normalized Cluster Entropy and Effective Clusters quantify the balance of the occupancy distribution: \[\begin{align} q_{s,c} &=\frac{n_{s,c}}{\sum_{u=1}^{K}n_{s,u}}, & C_{s,\tau} &=\sum_{c=1}^{K}\mathbf{1}\!\left[n_{s,c}\ge\tau\right], \\ H_{\mathrm{norm}}^{(s)} &=-\frac{\sum_{c=1}^{K}q_{s,c}\log q_{s,c}}{\log K}, & N_{\mathrm{eff}}^{(s)} &=\exp\!\left(-\sum_{c=1}^{K}q_{s,c}\log q_{s,c}\right). \end{align}\] Here, \(C_{s,\tau}\) denotes the number of visual regions with sufficiently stable sample support, with \(\tau=5\) used throughout our experiments. \(H_{\mathrm{norm}}^{(s)}\) measures how evenly the samples are distributed across the shared codebook, while \(N_{\mathrm{eff}}^{(s)}\) expresses the same entropy as an equivalent number of uniformly occupied cells. Higher values indicate broader and more balanced coverage of the shared visual feature space.

Finally, we compute kNN Domain Mixing directly in the original normalized CLIP space, without codebook discretization. Let \(y_i\) denote the dataset identity of embedding \(i\), let \(N_s\) be the number of sampled embeddings from dataset \(s\), and let \(\mathcal{N}_k(i)\) denote the set of its \(k\) nearest neighbors in the pooled sample. The metric is defined as \[M_{\mathrm{kNN}}^{(s)} = 1-\frac{1}{N_s} \sum_{i:y_i=s} \frac{1}{k} \sum_{j\in\mathcal{N}_k(i)} \mathbf{1}\!\left[y_j=s\right],\] where \(\mathbf{1}[\cdot]\) is the indicator function and \(k=20\). The inner term measures the proportion of same-dataset neighbors around an embedding; therefore, \(M_{\mathrm{kNN}}^{(s)}\) is the average proportion of neighboring samples originating from other datasets. A higher value indicates stronger local connectivity to a broader range of visual domains.

As shown in Fig. 4, across the 3,000 balanced trials, Open-AoE achieves the highest mean in all six metrics: Effective Rank (\(97.43\)), Participation Ratio (\(40.47\)), Meaningful Coverage (\(19.89\)), Normalized Cluster Entropy (\(0.712\)), Effective Clusters (\(16.29\)), and kNN Domain Mixing (\(0.0775\)). The largest advantages are observed in Effective Rank and kNN Domain Mixing, indicating that Open-AoE spans a larger number of active feature directions while maintaining stronger local connectivity to other visual domains. Moreover, Open-AoE ranks first in Effective Rank, Participation Ratio, and kNN Domain Mixing in all 3,000 trials. Its advantages in the coverage- and entropy-based metrics are smaller, particularly relative to OpenEgo, but their repeated-sampling means remain the highest among all four datasets.

5.2 Semantic Breadth and Temporal Completeness↩︎

We next examine the semantic breadth and temporal structure of the released action annotations. Temporal coverage is defined as the fraction of the evaluated sequence timeline covered by at least one action segment, measuring the continuity of semantic supervision over the underlying video. The released action vocabulary is quantified by the number of unique entries in each dataset’s primary action-annotation field. Because the datasets follow different native annotation schemas, this measure reflects the breadth of semantic supervision exposed by each release rather than a controlled comparison under a shared action ontology. We further characterize annotation structure using mean segment duration and annotation density, defined as the number of annotated segments per minute. The former captures the typical temporal extent of individual action annotations, whereas the latter reflects their frequency along the video timeline. Taken together, these metrics capture complementary aspects of supervision: vocabulary size reflects semantic breadth, temporal coverage measures annotation completeness, and segment duration and density characterize the granularity of localized action annotation.

As illustrated in Fig. 5, Open-AoE combines broad natural-language supervision with near-complete temporal coverage. Its primary action field comprises 32,407 distinct natural-language descriptions, compared with 26,864 native labels in OpenEgo and 111 categorical action classes in EgoDex. Beyond these descriptions, Open-AoE includes structured semantic fields with 8,030 object strings, 175 action verbs, and 135 scene labels, enabling each action interval to be represented at multiple levels of semantic abstraction. Its annotations cover 99.99% of the evaluated timeline, substantially exceeding the 50.1% coverage of OpenEgo, while retaining a high annotation density of 13.97 segments per minute. EgoDex achieves complete clip-level label coverage and a slightly higher segment density, but relies on a substantially smaller categorical action vocabulary. Overall, Open-AoE combines open-vocabulary semantic breadth with near-continuous temporal alignment and densely localized action supervision, without compromising annotation completeness.

5.3 Image-Annotation Consistency↩︎

We assess whether the released action annotations are supported by the corresponding visual content using Idefics2 [45] as a shared vision-language evaluator. Let \(a_{q,t}\in[1,5]\) denote the consistency score assigned to the \(t\)-th sampled frame of sequence \(q\). The sequence-macro score is defined as \[S_{\mathrm{macro}} = \frac{1}{Q} \sum_{q=1}^{Q} \left( \frac{1}{T_q} \sum_{t=1}^{T_q} a_{q,t} \right),\] where \(Q\) is the number of evaluated sequences and \(T_q\) is the number of sampled frames in sequence \(q\). Frame-level scores are first averaged within each sequence and then across sequences, thereby assigning equal weight to each sequence and preventing longer recordings from dominating the dataset-level estimate. Higher scores indicate stronger agreement between the released annotations and the sampled visual evidence.

As shown in Fig. 6, Open-AoE achieves a sequence-macro score of \(4.583/5\), substantially exceeding those of OpenEgo (\(3.029\)), EgoDex (\(2.916\)), and EgoXtreme (\(2.015\)). Its 95% bootstrap confidence interval is \([4.557,\,4.608]\). This advantage is also evident in the score distribution: 85.4% of Open-AoE sequences attain a mean consistency score of at least 4, whereas only 2.2% score at or below 2. The higher aggregate score therefore reflects a broad concentration of sequences near the upper end of the scoring range rather than a small subset of exceptionally high-scoring cases. Because the datasets differ in their native annotation schemas and levels of textual detail, these results should be interpreted as a cross-dataset audit of image-annotation support under a shared evaluator rather than as a fully controlled benchmark of annotation accuracy.

5.4 Training-Window Retention and Multimodal Completeness↩︎

We finally assess how efficiently the released sequences can be converted into temporally aligned training samples and whether the principal supervision signals required by downstream embodied-learning pipelines are jointly available. For a sequence \(q\) of duration \(L_q\), we construct candidate samples using a history horizon of \(h=2\) s, a future horizon of \(f=2\) s, and a temporal stride of \(\Delta=2\) s. The number of candidate windows is defined as \[N_q = \max\!\left( 0,\, \left\lfloor \frac{L_q-h-f}{\Delta} \right\rfloor + 1 \right).\]

The history-future window yield and normalized retention are then defined as \[Y_{\mathrm{window}} = \frac{\sum_q N_q}{\sum_q L_q/3600}, \qquad \eta_{\mathrm{window}} = \frac{Y_{\mathrm{window}}}{3600/\Delta},\] where \(\eta_{\mathrm{window}}\) measures retention relative to the no-boundary ceiling of \(3600/\Delta=1{,}800\) windows per hour. This metric isolates losses induced by sequence boundaries and temporal discontinuities, excluding task-specific filtering imposed by individual downstream models. We additionally evaluate native hand-signal availability using the released validity information, distinguishing between entries with at least one valid hand and those with valid signals for both hands. At the sequence level, we record the availability of five principal supervision modalities: action annotations, bounding boxes, annotation confidence, hand pose, and camera pose.

As shown in Fig. 7, Open-AoE yields approximately 1,760 candidate history-future windows per hour, corresponding to 97.8% of the theoretical ceiling. This small gap from the no-boundary limit indicates that its temporal continuity enables dense extraction of fixed-context training samples with minimal loss at sequence boundaries. Open-AoE also exhibits substantially higher native hand-signal availability than OpenEgo, with 98.93% of evaluated entries containing at least one valid hand signal and 98.02% containing valid signals for both hands. Moreover, it is the only evaluated release in which all five supervision modalities are available with near-universal sequence-level coverage. Bounding-box quality is similarly high, with 100% valid boxes, 98.39% fully contained within the image boundaries, a mean confidence of 0.947, and a 10th-percentile confidence of 0.950. These results show that Open-AoE combines high temporal sample retention with dense, jointly available semantic, geometric, hand-motion, and camera-motion supervision, thereby reducing the modality filtering and temporal alignment required before downstream VLA, world-model, and human-to-robot training.

6 Conclusion↩︎

We presented Open-AoE, an open data infrastructure that couples approximately 2,000 hours of real-world egocentric manipulation data captured with consumer smartphones with a complete path from data to models. On the production side, the capture and processing pipeline turns raw smartphone video into structured samples aligned across vision, language, hand motion, camera trajectories, and action boundaries; on the consumption side, the open-source toolchain supports synchronized visualization, 4D reconstruction, cross-embodiment retargeting, and training representations for Vision-Language-Action policies, World Action Models, and World Models. Its broad coverage of scenes, actions, interaction objects, participants, device models, and fields of view allows this infrastructure to connect low-cost real-world collection with multiple embodied-learning paradigms.

Open-AoE is intended not as a static data package, but as an open ecosystem co-developed by data contributors, universities, research institutions, robot developers, and model teams. We invite the community to contribute new real-world scenes and tasks, reconstruction and retargeting methods, training adapters, and evaluation results, allowing data, tools, and models to evolve together in an open loop. Our goal is to make Open-AoE the lowest-barrier egocentric data infrastructure for embodied intelligence—making egocentric data easy for anyone to collect and use, so that data is no longer the bottleneck for embodied foundation models.

Contributors↩︎

The following contributors are listed by module contribution.

Dataset and Visualization↩︎

Zishuo Li\(^{*,1}\), Qingze Guan\(^{*,5}\), Zhengxing Wu\(^{*,7}\), Wanke Zhan\(^{*,1}\), Yang Sun\(^{*,1}\), Zhiyi Huang\(^{*,1}\), Zitong Shan\(^{*,1}\)

Toolchain and Experiments↩︎

Bowen Yang\(^{*,1}\), Changtao Miao\(^{*,1}\), Zhenchao Jin\(^{*,3}\), Jiadong Hong\(^{*,2}\), Taowen Wang\(^{*,4}\), Yushi Feng\(^{*,3}\), You Liu\(^{*,6}\), Yibo Wang\(^{*,6}\), Yifan Yang\(^{*,7}\), Zhaowen Zhou\(^{*,1}\), Man Luo\(^{*,1}\), Hao Cheng\(^{*,8}\)

Project Leaders↩︎

Zishuo Li\(^{1}\), Bowen Yang\(^{1}\), Changtao Miao\(^{\ddag,1}\), Kai Zhu\(^{\ddag,1}\), Hao Chen\(^{\ddag,2}\)

Supervisors↩︎

Bo Zhang\(^{1}\), Jianshu Li\(^{1}\), Jiansheng Cai\(^{1}\), Guocai Yao\(^{6}\), Jize Zhang\(^{8}\), Chenhao Lin\(^{9}\), Renjing Xu\(^{4}\), Lequan Yu\(^{3}\), Chao Shen\(^{9}\), Chunhua Shen\(^{2}\), Zhe Li\(^{1}\)

\(^{*}\)Equal contribution. \(^{\ddag}\)Corresponding authors.

Organizations↩︎

\(^{1}\)Ant Digital Technology, Ant Group

\(^{2}\)Zhejiang University

\(^{3}\)The University of Hong Kong

\(^{4}\)The Hong Kong University of Science and Technology (Guangzhou)

\(^{5}\)National University of Singapore

\(^{6}\)Beijing Academy of Artificial Intelligence

\(^{7}\)University of Chinese Academy of Sciences

\(^{8}\)The Hong Kong University of Science and Technology

\(^{9}\)Xi’an Jiaotong University

Appendix↩︎

Privacy and Ethical Statement↩︎

The collection and release of the Open-AoE dataset were conducted with explicit authorization from all data contributors. Before participating, each contributor was informed of the purpose of data collection, the types of data involved, the intended scientific use of the collected data, and its subsequent public release as part of the Open-AoE dataset. All contributors voluntarily consented to the collection, processing, research use, and open-source distribution of their contributed data.

Before uploading, contributors reviewed the collected data locally on their mobile devices and explicitly confirmed each upload. After the confirmed data were uploaded, privacy-preserving processing was performed in a controlled cloud environment. This processing included masking visual regions that might reveal personal or sensitive information and removing or desensitizing metadata unrelated to embodied-intelligence research. The released data do not contain account information, payment information, or other personal metadata. Only privacy-processed, anonymized, and task-relevant data are made publicly available.

Open-AoE is released exclusively to support scientific research and technological development in embodied intelligence, robotics, and related fields. The dataset is not intended for identity recognition, personal profiling, surveillance, re-identification, or other uses that may infringe upon individual rights. Dataset users are expected to comply with the applicable license, privacy regulations, and generally accepted research-ethics principles, and must not attempt to recover or infer the identities of data contributors.

Based on the informed-consent process, explicit contributor authorization, local review and upload confirmation, controlled cloud-based privacy processing, data minimization, and anonymization measures described above, we have identified no unresolved material privacy or ethical concerns associated with the research use and open release of the Open-AoE dataset.

References↩︎