1 Introduction↩︎

Video Referring Expression Comprehension (Video REC) is a critical task in computer vision that involves localizing a specific object or region within a video based on a natural language query, known as referring expression. The objective of this task is to perform spatio-temporal localization of the target object indicated by a textual query such as “the blue chair spinning on the left side of the screen” across all frames of a given video. This is a fundamental and challenging multimodal problem that requires not only recognizing an object but also understanding its attributes, spatial relationships, and dynamic context, including temporal changes and movements, thereby “grounding” language to visual information. While significant progress has been made in REC for static images, extending this task to videos introduces new dimensions of complexity. It demands that a model possess advanced spatio-temporal reasoning capabilities to capture appearance changes of the object over time and the dynamics of complex interactions between multiple objects.

Among the various forms of video, the importance of the egocentric recordings of the first-person views has grown rapidly in recent years [1]–[4], driven by the anticipated proliferation of AR/VR devices, smart glasses [5], and collaborative robots [6]. In contrast to third-person videos that provide observers’ viewpoint, egocentric videos provide the camera wearers’ subjective experience. The camera motion directly reflects the wearer’s focus of attention, while the appearance of their hands in the frame captures their physical interactions with the environment, providing an unparalleled source of information for understanding user intent. As such, REC in the egocentric view is a foundational technology for the seamless integration of human-centric AI systems into our daily lives and professional workspaces [7], [8]. However, research in this domain is still in its nascent stages, with a notable lack of benchmarks that reflect the complexity of the real world. The unique and challenging characteristics of the egocentric perspective—such as drastic viewpoint changes, frequent occlusions by the wearer’s hands, and long-duration tasks—pose significant challenges to existing methods.

While previous research has made significant progress, existing benchmarks do not adequately capture the complex situations that humans encounter in the real world. Specifically, there are three main limitations or gaps: 1. Lack of Temporal Context: Many existing benchmarks are composed of short video clips, typically lasting from a few to tens of seconds. However, real-world tasks often require long-form memory and contextual understanding. 2. Simplified Scenarios: The tasks tend to be oversimplified, with the target object often being clearly visible for the majority of the video. In the real world, it is common for an object to appear only transiently or be situated among many confusing objects (distractors). 3. Neglect of Interaction: Most importantly, there is a significant lack of referring expressions that focus on dynamic and complex human-object interactions, such as a person manipulating an object or collaborating with others. This capability is essential for the next generation of AI assistants to be truly useful.

To overcome these challenges, in this work, we propose LongEgoRefer, a new egocentric video REC benchmark based on more realistic and challenging scenarios (see Figure 1). This benchmark was created by combining long-form videos from Ego4D with the precise object tracking annotations from EgoTracks [8]. The high-quality referring expressions were constructed using a hybrid approach: automatic generation by a Large Language Model (LLM) followed by careful verification and refinement by humans. As illustrated in the comparative analysis in Figure 1, our benchmark exhibits distinct characteristics that directly address the aforementioned limitations: 1. Long-Form Videos: By leveraging the large-scale egocentric video dataset Ego4D, we target long-form videos averaging 45 minutes in length to evaluate the capability for long-form contextual understanding (Figure 1a). 2. Sparse Appearance: We intentionally keep the appearance frequency of the referent low (increasing sparsity) to create a more realistic and difficult REC task (Figure 1b). 3. Linguistic Complexity: As such extensive footage inevitably contains numerous visually ambiguous scenes, we provide highly descriptive referring expressions to uniquely disambiguate the target from similar-looking temporal contexts. This results in significantly increased expression lengths compared to prior benchmarks (Figure 1c). 4. Interaction-Centric: The majority of our referring expressions are grounded in complex human-object interactions, demanding a high level of scene understanding (Figure 1, top).

To quantitatively demonstrate the difficulty of our proposed benchmark and to provide a starting point for the research community, we evaluate existing video REC models alongside a training-free baseline we construct. Instead of proposing a definitive end-to-end solution, our training-free baseline modularizes the task into two stages: temporal grounding, leveraging state-of-the-art Vision–Language Models (VLMs) to determine when an event occurs, and spatial grounding, utilizing Grounded SAM2 [9] to localize where the referred object appears. Our comprehensive experiments reveal a significant performance gap between open-source and closed-source models. Notably, while GPT-5 significantly outperforms all other models, the overall performance of both existing video REC models and our advanced VLM-based baselines remains modest, particularly under stricter evaluation metrics (e.g., mtIoU and mvIoU). This strictly indicates that the task of accurately localizing rare events and objects in long-form videos continues to be a significant and unresolved challenge even for current state-of-the-art models. The contributions of this research can be summarized as threefold:

• We introduce LongEgoRefer, a novel and challenging egocentric video REC benchmark. It specifically addresses the limitations of prior work by featuring hour-scale temporal contexts, extreme target sparsity, and complex interaction-centric referring expressions, thereby accurately reflecting realistic first-person visual experiences.

• We establish standardized baseline strategies to effectively evaluate models on extreme long-form videos. Specifically, we formulate two distinct approaches: (1) adapting existing short-clip video REC models via a sliding window inference strategy, and (2) constructing a training-free, two-stage pipeline that explicitly decouples the task into high-level temporal grounding (via VLMs) and low-level spatial tracking (via Grounded SAM2).

• We provide a comprehensive empirical analysis exposing the fundamental bottlenecks of current paradigms. We reveal that existing video REC models applied in a sliding-window manner lack global temporal awareness. Meanwhile, the two-stage pipeline suffers from critical error propagation, where inaccuracies in identifying when an event occurs strictly constrain localizing where it happens.

2 Related Work↩︎

2.1 Video-based REC Datasets↩︎

Video-based REC extends image-based grounding [10]–[14] to the spatio-temporal domain, aiming to localize target objects in videos based on natural language queries. Early video REC studies primarily adopted short, trimmed clips (3–10 seconds) and focused on detecting objects or events within brief frame sequences [15]–[19]. Concurrent work explored dynamic human activities and object interactions in third-person videos [18], [20]–[23]. Recent research has moved toward egocentric viewpoints to capture everyday activities. MeViS [24] scales referring video object segmentation to multi-event, motion-centric scenarios in third-person videos. RefEgo [7] introduces first-person REC over short clips lasting about 12 seconds on average, with temporal dependencies. EgoMask [25] benchmarks fine-grained spatio-temporal grounding in egocentric videos but covers segments of about 90 seconds with frequently appearing targets. However, these benchmarks are limited in temporal scale and do not address the challenge of localizing rare object occurrences across extended temporal contexts. LongEgoRefer advances this direction by introducing hour-scale first-person recordings with sparse target appearances, requiring models to maintain long-term memory while performing precise spatial grounding.

2.2 Egocentric Video Benchmarks↩︎

Egocentric video benchmarks are essential for understanding human activities from a first-person perspective, directly relevant to embodied AI and human-robot interaction. Large-scale wearable-camera datasets [1], [2], [26], [27] have captured daily activities with multi-modal data including RGB, depth, audio, and gaze. Recent benchmarks further incorporate multi-view synchronization and 3D scene geometry [3], [4], [28]–[30]. Building on these datasets, particularly Ego4D, language-conditioned egocentric understanding has emerged across tasks such as video summarization [31], [32], video question answering [33]–[35], and video visual grounding [7], [25], [36]. While these works have made significant progress, existing grounding benchmarks focus on short, densely annotated clips where target objects appear frequently and predictably. In contrast, real-world egocentric videos contain sparse, irregular object occurrences distributed across hours of continuous recording. LongEgoRefer addresses this gap by requiring models to perform spatio-temporal grounding in hour-long videos where targets may appear only briefly and unpredictably, better reflecting realistic egocentric scenarios.

2.3 Long-form Video Understanding↩︎

Understanding long-form videos is critical for this work, as it requires models to integrate information across extended time spans rather than within isolated clips. Early video understanding focused on short, trimmed clips (10–100 seconds) with localized reasoning [37]–[39]. Subsequent datasets increased narrative complexity but remained limited to several minutes [40]–[42]. Recent benchmarks extend to long-form comprehension spanning tens of minutes to hours [31], [33], [43]–[47], emphasizing multi-event reasoning and cross-scene consistency. However, these works primarily address event-level temporal reasoning without spatial grounding. In contrast, LongEgoRefer requires models to localize rare objects across extended temporal contexts while maintaining precise frame-level spatial grounding. This setting challenges models to combine long-term memory for tracking rare occurrences with accurate language–visual alignment for fine-grained localization in long-form videos.

3 Benchmark↩︎

3.1 Task↩︎

We address the task of Spatio–Temporal Grounding of Object Occurrences in long-form egocentric videos. Given an untrimmed video \(V\) of length \(T_V\) and a natural language query \(Q\), the model must localize a single, continuous appearance of the referred object within the untrimmed video. The prediction is represented as a tuple \((t_{\text{start}}, t_{\text{end}}, \mathcal{B})\), where \([t_{\text{start}}, t_{\text{end}}]\) denotes the temporal boundaries satisfying \(0 \le t_{\text{start}} \le t_{\text{end}} \le T_V\), and \(\mathcal{B}\) is the sequence of bounding boxes corresponding to that duration. The key challenge is to find a short, language-relevant moment within hours of egocentric footage and precisely localize the target object throughout that interval, jointly testing temporal localization and spatial grounding under realistic, first-person conditions.

3.2 Benchmark Creation↩︎

3.2.0.1 Dataset and annotation preparation

We build our benchmark upon the annotations from the EgoTracks dataset [8], which provides long-form object tracking annotations on the large-scale egocentric video corpus Ego4D [2]. EgoTracks contains per-frame bounding boxes for each object occurrence, along with instance-level attribute information. While EgoTracks primarily consists of six-minute clips extracted from the original Ego4D videos, our benchmark instead utilizes the full-length Ego4D videos to evaluate spatio–temporal grounding under truly long-form conditions. Although six-minute clips are longer than those in prior grounding datasets, untrimmed hour-scale videos are crucial to evaluate robustness to rare events and long-term temporal dependencies in realistic scenarios. Importantly, EgoTracks retains timestamp metadata linking each clip to its position in the original video, allowing us to accurately remap existing annotations back to the full-length Ego4D sequences.

3.2.0.2 Annotation Pipeline

To ensure the quality, diversity, and uniqueness of our referring expressions, we designed the semi-automated annotation pipeline illustrated in Figure 2, which consists of a data selection stage and a two-stage language generation process that guarantees each expression does not match other parts of the long video.

(i) Data filtering. To capture realistic long-term scenarios, we first perform strict filtering on the Ego4D dataset. We exclude all videos shorter than 20 minutes to ensure sufficient temporal context for long-form reasoning. To avoid trivial segments, we retain only object tracks lasting at least 10 seconds where the camera wearer actively manipulates the object (annotated as \(\mathtt{is\_active}=\mathtt{True}\) in EgoTracks). This criterion prioritizes dynamic, interaction-rich episodes that demand temporal grounding.

(ii) Referring expression generation. For each filtered clip, we generate referring expressions using Gemini 2.5 Flash through a two-stage process. In the first stage, the model is prompted to produce 10 detailed captions describing the target object’s interactions, state changes, and spatial relations. In the second stage, these captions are aggregated and refined into a single, natural referring expression that captures the temporal context. This step removes redundancy and produces a refined, cohesive referring expression that encapsulates the temporal context.

3.2.0.3 Human annotation

All generated expressions undergo rigorous human verification to ensure their quality and correctness. Because automatically generated referring expressions can be semantically inconsistent or temporally ambiguous (that is, applicable to multiple moments in the video), we adopt a four-stage human curation pipeline conducted by expert annotators. As a result of this meticulous process, our annotators substantially revised 95.39% of the Gemini-initialized descriptions to precisely match the video content and ensure referential exclusivity. This extensive human intervention effectively mitigated potential model-induced biases, transforming the initial LLM-generated seeds into high-quality, human-curated grounding queries that provide a fair evaluation platform for all VLM families. Starting from 2,000 candidate triplets consisting of an Ego4D video, an object occurrence with bounding boxes, and a generated referring expression, we perform the following steps:

(i) Validating and filtering. We first check the video segment and temporal boundaries of each occurrence. Samples are removed if (a) the corresponding segment is blank or corrupted, or (b) the object remains visible before or after the annotated timestamps, which makes precise temporal grounding unreliable.

(ii) Correcting expressions. For each valid sample, an annotator verifies whether the referring expression accurately describes the target object and its associated actions. If inaccuracies are found, the expression is rewritten to be factually and semantically correct.

(iii) Disambiguating references. The annotator then ensures that the refined expression uniquely identifies the target moment. If the same description fits other segments in the video, additional visual or temporal cues (such as object attributes, spatial relations, or nearby actions) are added until the expression becomes exclusive to the intended instance.

(iv) Quality assurance. A second annotator independently reviews the results from the previous stages, returning 52.20% of them for re-annotation until they fully pass the strict quality check. Ten percent of all annotations are selected for an additional verification round to ensure consistency and final quality.

Through this rigorous pipeline, 25.1% of the initial candidates were filtered out, resulting in a final dataset of 1,498 high-quality curated samples.

3.3 Statistics↩︎

The proposed LongEgoRefer benchmark provides a more realistic and challenging setting than previous video REC benchmarks. Table 1 compares LongEgoRefer with existing video referring expression and grounding benchmarks. Our proposed LongEgoRefer substantially increases both temporal and linguistic complexity compared to existing video grounding benchmarks. It contains 433 egocentric videos with average length of 2,714.9 seconds (approximately 45 minutes) and 1,498 referring expressions with an average of 63.5 words per query, which are significantly longer and more descriptive than those in prior datasets. While most existing benchmarks capture short segments where the referred object remains visible throughout the clip (with an appearance rate of 60.3% or higher), the target objects in LongEgoRefer appear in only 1.3% of frames, requiring models to perform long-range temporal search, contextual reasoning, and memory maintenance over extended sequences. Figure 1 highlights clear differences between RefEgo and LongEgoRefer. RefEgo contains short clips with frequent object appearances, whereas LongEgoRefer consists of much longer videos with sparse targets and longer descriptions. Furthermore, LongEgoRefer explicitly annotates human–object interactions (HOI), creating a challenging setting for spatio-temporal grounding in real-world, continuous egocentric videos. These differences make LongEgoRefer a comprehensive benchmark for evaluating models’ ability to perform robust, fine-grained spatio-temporal grounding over extended egocentric video sequences, bridging research and real-world applications.

4 Baseline↩︎

To comprehensively evaluate the difficulty of the LongEgoRefer benchmark and provide a starting point for future research, we construct and evaluate two types of baselines: existing end-to-end video REC models adapted for long videos, and a training-free two-stage pipeline leveraging modern visual foundation models.

4.1 Video REC Models↩︎

Existing video REC models are predominantly designed and trained on short, trimmed clips (typically spanning a few seconds). Consequently, they are fundamentally constrained by memory limitations and context window sizes, making them infeasible to process hour-long videos natively.

To evaluate these conventional models on the extensive sequences of LongEgoRefer, we adopt a sliding window inference strategy. Specifically, we partition the untrimmed long video into a sequence of short temporal windows. We then independently feed each window, along with the referring expression, into the end-to-end video REC model. The model generates a sequence of bounding boxes and an associated confidence score for each window.

The aggregation strategy for the final prediction depends on the output capabilities of the specific model. For models capable of generating confidence scores for their predictions, we select the temporal window and its corresponding spatio-temporal track that yields the highest confidence score across the entire video. For models that do not provide such evaluation metrics, we instead aggregate the results by temporally concatenating the predicted bounding boxes from all windows to form a single continuous sequence. This sliding window approach serves as a straightforward baseline to gauge how well current short-clip REC models generalize to extreme long-form search scenarios.

4.2 Training-free Two-stage Pipeline↩︎

Spatio-temporal grounding in long-form videos poses significant computational challenges for end-to-end models. To address this without extensive retraining, we construct a hierarchical pipeline that decouples the task into temporal and spatial reasoning. This design leverages a VLM for global temporal reasoning (“when”) and Grounded SAM2 [9] for local spatial localization (“where”). We adopt Grounded SAM2 for the spatial stage because of its SoTA performance on EgoMask [25].

Specifically, we first employ a pre-trained VLM to identify the temporal interval that matches the query. This coarse estimate is then refined into a precise trajectory using Grounded SAM2. We extract the middle frame of the identified clip and use Grounding DINO [48] to detect the target object and generate an initial bounding box. This box serves as a visual prompt for SAM2 [49], which tracks the object across the segment to produce the final spatio-temporal track. This modular approach provides a clear reference point for the LongEgoRefer benchmark.

5 Experiments↩︎

5.1 Experimental Settings↩︎

5.1.0.1 Metrics

For the evaluation on LongEgoRefer, we adopt four metrics. tIoU and vIoU are commonly used metrics for video REC. Following RefEgo, we also employ STIoU and IoU+n.

5.1.0.2 Implementation Details

We evaluate 4 video REC models, and 14 open-source and 5 closed-source VLMs on LongEgoRefer. The video REC models are VideoLISA [50], Sa2VA [51], Grounded SAM2 [9], and SAM3 [52]. The open-source models are InternVideo 2.5 Chat [53], MiMo-VL [54], TimeChat [55], LITA [56], Video-Chat [57], mPLUG-Owl3 [58], LLaVA-OneVision [59], Video-LLaMA3 [60], LongVA [61], LongVILA R1 [62], InternVL 3.5 [63], and Qwen3-VL [64]. The closed-source models include Gemini-family (2.0 Flash, 2.5 Flash, and 2.5 Pro) [65] and GPT-family (GPT-4o [66] and GPT-5 [67]). See Appendix for detailed information.

5.2 Main Results↩︎

Table 2 presents the quantitative evaluation results of video REC models and leading open-source and closed-source VLMs on our proposed LongEgoRefer benchmark. A key finding from our experiments is that even state-of-the-art models struggle with referring expression comprehension in long-form egocentric videos. The generally low scores across temporal and spatio-temporal metrics highlight the inherent difficulty of accurately localizing a specific clip, often lasting only tens of seconds, within a video that spans tens of minutes to an hour, based solely on referring expressions.

Notably, the poor performance of video REC models highlights that existing approaches are technically incapable of handling LongEgoRefer directly. The sliding window strategy, while a natural baseline, fails because it lacks the global temporal awareness needed to filter out irrelevant segments across an entire hour, leading to high false-positive matches. This underscores the necessity of our two-stage pipeline: by decomposing the task into temporal and spatial grounding, we leverage the VLM’s extensive context window to prune the vast search space. This “temporal filtering” allows the video REC model to focus its fine-grained spatial localization on a manageable, contextually relevant clip, which would otherwise be buried in hours of irrelevant data.

Comparisons across model categories reveal a substantial performance gap between open-source and closed-source models. Even relatively large and high-performing open-source models, such as InternVL and Qwen3-VL, remain in the single digits for tIoU@0.1. Furthermore, it is noteworthy that models specifically designed for long video understanding, such as LongVILA, also show limited effectiveness on our benchmark. In contrast, the closed-source models achieve significantly better performance. Notably, GPT-5 sets a new state of the art, with a tIoU@0.1 of 37.31 and an mtIoU of 16.19, outperforming all other models by a wide margin. Importantly, the fact that GPT-5 consistently outperforms the Gemini series, which was used to generate the initial referring expressions, demonstrates that the benchmark does not suffer from a Gemini-specific bias. This result, coupled with the rigorous human curation process where 95.39% of initial descriptions were substantially revised (see Section 3.2), confirms that LongEgoRefer evaluates objective spatio-temporal reasoning rather than rewarding model-specific linguistic patterns. This advantage extends to spatio-temporal grounding. While most open-source models struggle to produce accurate object tracks (reflected in mSTIoU scores below 1.0), the closed-source models demonstrate stronger, though still developing, capabilities. GPT-5 again leads, achieving an mSTIoU of 8.76 and an mvIoU of 7.89. Within the Gemini family, Gemini 2.5 Pro performed best, and 2.5 Flash followed, suggesting that both the model scale and architectural sophistication contribute directly to the performance improvements on complex long-form context understanding.

Despite achieving the best results, the absolute performance of GPT-5 remains relatively low, indicating substantial room for improvement. Temporal localization accuracy appears to be the primary limiting factor, constraining subsequent spatio-temporal grounding metrics. Overall, these results empirically demonstrate that LongEgoRefer presents a highly challenging problem, underscoring the need for novel approaches capable of efficiently processing and deeply understanding long-form video content.

5.3 Analysis↩︎

5.3.1 Effect of Temporal Resolution↩︎

Figure 3a investigates the impact of temporal sampling density on GPT-5 by reducing the maximum frames from the default 500 (constrained by API limits). We observe a significant performance decay as the temporal resolution decreases. Specifically, tIoU@0.5 drops sharply from 12.55 to 3.60 when the input is limited to 64 frames. This suggests that sparse sampling leads to the loss of fine-grained visual cues essential for precise temporal localization in long-form videos. These results confirm that high temporal resolution is critical for accurate grounding and that GPT-5 effectively leverages dense 500-frame contexts to model complex temporal dependencies.

5.3.2 Effect of Reasoning Effort↩︎

Figure 3b investigates the impact of reasoning effort levels on GPT-5. Specifically, “Minimal” corresponds to zero reasoning effort, whereas “High” represents the maximum available reasoning capacity. The results demonstrate that increasing reasoning effort yields substantial performance gains, with tIoU@0.5 improving from 8.7 to 12.61. Notably, we observe that performance largely saturates at the “Medium” level, which serves as the API’s default setting. This suggests that while more intensive reasoning generally benefits temporal localization, there may be a point of diminishing returns or increased noise.

5.3.3 Spatial Grounding↩︎

We further investigate the importance of temporal grounding. We evaluate spatial grounding alone with the annotated (oracle) and Grounded-SAM2-based temporal segmentation for the object appearance of each video. With the oracle temporal segmentation, we evaluate the segmentation model of SAM2.1 Base and Large. We use the ground-truth bounding box in the first frame for the prompts of SAM2.1. Figure 3c presents the results in terms of vIoU. Compared to the results in Table 2, the scores have improved dramatically with the oracle temporal segmentation. For example, while GPT-5 achieves an mvIoU of 7.89 when using its own temporal grounding results, Grounded SAM2 attains 43.54 (+35.65). This demonstrates that the accuracy of temporal grounding has a substantial impact on the overall performance of spatio-temporal grounding. Furthermore, when comparing Grounded SAM2 with SAM2.1 Base and SAM2.1 Large, mvIoU increases to 67.21 (+23.67) and 67.82 (+24.28), respectively. These results indicate that the precision of the bounding box used as the prompt for SAM2 significantly contributes to the performance of spatial grounding.

5.3.4 Results on EgoTracks Videos↩︎

To better understand the challenges of long-form videos in video REC, we evaluate Gemini on LongEgoRefer using the EgoTracks dataset instead of the original Ego4D videos. EgoTracks is constructed by trimming the Ego4D videos to an average length of six minutes. Although shorter than the original Ego4D videos (45 minutes on average), EgoTracks still consists of considerably longer videos than those in RefEgo or EgoMask. The results are summarized in Table 3. Compared to the results on the original long-form Ego4D videos, the performance improves substantially. For instance, Gemini 2.5 Pro achieves a gain of +12.24 mtIoU and +5.39 mvIoU. These results indicate that while long-form video REC is an important and practical problem, it remains highly challenging. We hypothesize that the main factors behind the performance degradation on long videos are the repeated occurrence of visually similar events and the decreased frequency of target object appearances.

5.4 Qualitative Results↩︎

We present prediction examples of InternVL 3.5 38B, GPT-5, and ground truth in Figure 4. In the example (a), both InternVL and GPT predict the correct bounding boxes. In the example (b), GPT predicts the correct bounding boxes, but InternVL predicts the wrong temporal duration and bounding boxes. In the example (c), both InternVL and GPT predict the wrong bounding boxes, even though the temporal duration of GPT is close to the ground truth.

6 Conclusion↩︎

In this work, we introduced LongEgoRefer, a new benchmark for egocentric video referring expression comprehension that addresses key limitations of existing benchmarks. It substantially expands the temporal scale (averaging 45 minutes per video) and introduces extreme target sparsity, where referents are visible for only 1.3% of the video, requiring models to identify brief and infrequent visual occurrences within long, continuous streams. Furthermore, LongEgoRefer is the first egocentric benchmark explicitly centered on complex human-object interactions (HOI), accompanied by long, descriptive expressions averaging over 63 words, which demand advanced spatio-temporal reasoning and language grounding. Our experiments show that these characteristics present significant challenges to current vision-language models. Even the strong training-free baselines we constructed, which combine state-of-the-art VLMs with the powerful spatial tracker, achieve limited accuracy. LongEgoRefer thus establishes a new standard for real-world complexity and provides a foundation for developing models with robust long-term memory, interaction reasoning, and the ability to ground complex language in sparse and cluttered visual environments, advancing research toward more capable human-centric AI systems.

This study focuses on constructing a rigorous benchmark for spatio-temporal grounding in long-form egocentric videos. While LongEgoRefer provides a critical testbed, developing a comprehensive, large-scale training dataset for videos spanning tens of minutes remains an important direction for future work. For practical applications such as human-robot interaction in daily-life scenarios, extending the benchmark to multi-hour or day-long continuous videos will be indispensable.

Acknowledgements↩︎

This work was supported by JST PRESTO Grant Number JPMJPR22P8, JST K Program, Grant Number JPMJKP25V2, JST CRONOS, Grant Number JPMJCS24K6, and JST BOOST, Grant Number JPMJBY24C5. This work used the TSUBAME4.0 supercomputer at Institute of Science Tokyo.

7 Word Cloud↩︎

Figure 5 visualizes the linguistic diversity of referring expressions in LongEgoRefer. The prominent size of action verbs such as “placed,” “lifted,” and “held,” alongside the noun “hand,” illustrates the benchmark’s strong emphasis on
Human-Object Interaction (HOI). This distribution confirms that objects in egocentric video are primarily identified through dynamic manipulation and their spatial relationship with the user’s hands and surfaces. The benchmark also relies heavily on visual attributes (e.g., “white,” “black,” “dark”) and spatial context (e.g., “surface,” “positioned”). Furthermore, temporal indicators such as “subsequently” and “initially” highlight the need for models to track object state changes over time.

8 Object Distribution↩︎

Figure 6 presents the object distribution of LongEgoRefer annotated by EgoTracks [8]. As LongEgoRefer is interaction-centric, the most frequent objects are everyday items such as phones and knives, reflecting its practical and application-oriented nature.

9 Interaction Distribution↩︎

Figure 7 presents the distribution of Human-Object Interactions in LongEgoRefer. We employ GPT-5 to extract interaction verbs from the referring expressions in our benchmark. The results highlight the linguistic and semantic richness of the dataset, covering a diverse range of egocentric activities.

10 EgoTracks Split and Annotation Cost↩︎

We use the train and validation splits of EgoTracks for LongEgoRefer benchmark construction. The total annotation cost amounts to $4,700. All 1,498 curated queries are designed exclusively as an evaluation benchmark (test set). As modern foundation models and VLMs are increasingly evaluated on their zero-shot generalization capabilities, we do not provide a specific training split. Instead, LongEgoRefer serves as a pure testbed for zero-shot long-form spatio-temporal reasoning.

11 Training-Free Two-Stage Pipeline↩︎

Our baseline adopts a two-stage pipeline that sequentially performs temporal and spatial grounding, as illustrated in Figure 8.

11.0.0.1 Stage 1: Temporal Grounding

The primary objective of this stage is to identify the temporal interval \(T = [t_{\text{start}}, t_{\text{end}}]\) within a potentially long and untrimmed video \(V\) that semantically corresponds to a given textual query \(Q\). To achieve this, we leverage a pre-trained VLM, capitalizing on its advanced capabilities for processing and reasoning over extended video sequences.

We formulate the temporal grounding task as a structured generation problem. The VLM must output the start time \(t_{\text{start}}\) and end time \(t_{\text{end}}\) that define the temporal boundaries of the described event.

This process can be formally expressed as: \[t_{\text{start}}, t_{\text{end}} = f_{\text{VLM}}(V, Q),\] where \(f_{\text{VLM}}\) represents the prompted VLM. A key advantage of this approach is the VLM’s inherent ability to process long contexts. This allows the baseline to ground queries that may refer to events, objects, or their relationships separated by large temporal distances within the video.

11.0.0.2 Stage 2: Spatial Grounding

The second stage refines the coarse temporal estimate from the VLM into a precise spatio-temporal track of the object. For this purpose, we employ Grounded-SAM2, a two-step pipeline consisting of Grounding DINO [48] and SAM2 [49], which are renowned for their exceptional zero-shot, prompt-guided object detection and segmentation capabilities, respectively.

First, we extract the clip \(C\) corresponding to the timestamp \([t_{\text{start}}, t_{\text{end}}]\) identified by the VLM. To initialize the track, we provide the original referring expression \(Q\) and the middle frame of the clip, \(I_{\text{mid}}\), which is likely to contain a representative view of the target object, to Grounding DINO. This generates a high-quality initial bounding box for the target object, which we will use as a prompt: \[B_{\text{prompt}} = f_{\text{GD}}(I_{\text{mid}}, Q),\] where \(f_{\text{GD}}\) denotes the Grounding DINO model and \(I_{\text{mid}}\) is the frame at timestamp \((t_{\text{start}} + t_{\text{end}}) / 2\).

This initial bounding box, \(B_{\text{prompt}}\), is then used as a spatial prompt to guide SAM2 to track the object throughout the entire clip \(C\). The operation is formulated as: \[\mathcal{B} = f_{\text{SAM2}}(C, B_{\text{prompt}}),\] where \(f_{\text{SAM2}}\) denotes the SAM2 model operating in video tracking mode, and \(\mathcal{B}\) is the final bounding box sequence of the target object.

This two-stage design creates a modular workflow where the VLM handles the high-level when and Grounded-SAM2 handles the low-level where. While this decomposition is conceptually intuitive, it inherently faces the challenge of error propagation, where Stage 1 inaccuracies inevitably constrain Stage 2 performance. By evaluating this pipeline, we aim to expose the fundamental difficulties of the LongEgoRefer benchmark and provide a clear baseline for future work to improve upon, particularly in bridging the gap between temporal and spatial reasoning.

12 Metric↩︎

For the evaluation on LongEgoRefer, we adopt four base metrics. tIoU and vIoU are commonly used metrics for video REC. Following RefEgo [7], we also employ STIoU and IoU+n, as LongEgoRefer contains a large number of background frames without target objects.

• mtIoU: This metric evaluates the temporal localization accuracy. It is computed as the Intersection over Union (IoU) between the predicted temporal segment and the ground-truth temporal segment.

• tIoU@R: This metric, also known as localization recall or success rate, measures the percentage of queries for which the tIoU exceeds a certain threshold \(R\). It provides a binary measure of successful temporal localization. Following standard practice, we report the recall for multiple thresholds, \(R \in \{0.1, 0.5\}\).

• mvIoU: This metric assesses the spatio-temporal localization accuracy by measuring the overlap between a sequence of predicted and ground-truth bounding boxes. It is calculated by averaging the spatial IoU of the bounding boxes over the frames where both the predicted and ground-truth objects are present.

• vIoU@R: Similar to tIoU@R, this metric evaluates the percentage of samples where the vIoU score is above a threshold \(R\). This serves as a strict measure of successful spatio-temporal localization. We report the recall for multiple thresholds, \(R \in \{0.1, 0.5\}\).

• mSTIoU: This metric is adopted in RefEgo [7]. While vIoU is based on frame-by-frame IoU, STIoU is a multi-frame-based IoU.

• mIoU+n: This metric adopted in RefEgo is similar to mvIoU, but IoU is set to 1 for frames that are true negatives (i.e., when the frame is negative in both the prediction and ground truth).

13 Prompt↩︎

We present prompts for referring expression generation in Figure 9 and the experimental prompts in Figure 10.

14 Implementation Details↩︎

For video REC models, we aggregate predictions over 1-minute video clips. Among these models, only SAM3 is capable of predicting confidence scores for its outputs, while the other models do not provide such scoring metrics. For Gemini, videos are sampled at 1 FPS. Each frame is converted into 258 tokens. When video durations exceed 1 hour, we enable the low-resolution mode, where each frame is represented by 66 tokens. For GPT, videos are temporally sampled at equal intervals to comply with the API limits, and up to 500 frames are extracted. Each frame is resized while preserving the aspect ratio so that the longer side is 512 pixels. For open-source VLMs, frames with a shorter edge of approximately 400 pixels are input up to 128 frames or the maximum window size of each model. As VLMs sometimes predict very long temporal duration, such as over 1,000 seconds, we set the maximum temporal duration in the spatio-temporal evaluation as 300 seconds to reduce the computational cost. We use two NVIDIA H100 GPUs.

15 More Analysis↩︎

15.0.0.1 Effect of Video Duration

Table 5 presents experimental results of mvIoU on LongEgoRefer with respect to the video duration. We split the duration into three ranges: 20-40 minutes, 40-60 minutes, and 60-120 minutes. The results demonstrate that for most models, including GPT-5, the performance is highest in the relatively shorter 20-40 minutes range and drops as the video length increases. This performance degradation is likely due to the strict limit on the maximum number of input frames (e.g., 500 frames for GPT-5 and 128 frames for open-weight VLMs), which significantly reduces the temporal resolution for longer videos. Interestingly, Gemini 2.5 Pro exhibits a distinct trend, achieving its best performance in the 60-120 minutes range. We attribute this robustness to Gemini’s massive context window, which allows it to maintain a dense sampling rate of 1 FPS regardless of the video duration. By preserving fine-grained temporal information even in extreme long-form videos, Gemini effectively avoids the temporal information loss that severely hinders other models.

15.0.0.2 Effect of Appearance Rate

Table 6 presents experimental results of mvIoU on LongEgoRefer with respect to the target object’s appearance rate. We split the rate into three ranges: 0-1%, 1-2%, and 2-25%. The results reveal a stark correlation between the appearance rate and the localization performance. All evaluated models struggle significantly when the object appears sparsely within the video. For instance, even the state-of-the-art GPT-5 and Gemini 2.5 Pro experience a drastic performance drop of over 50% when the appearance rate falls below 1% compared to the >2% range, while open-weight models fail almost entirely. This confirms that precisely localizing rare objects remains a fundamental bottleneck for current VLMs, highlighting the rigorous nature of our benchmark.

15.0.0.3 Effect of Referring Expression Length

Table 7 presents experimental results of mvIoU on LongEgoRefer with respect to the referring expression length. We split the length into three ranges: 0-50 words, 50-70 words, and 70-110 words. The results demonstrate that linguistic complexity has a significant impact on the difficulty of the task. Specifically, as the referring expression becomes longer, the performance of the VLMs, including GPT-5 and Gemini 2.5 Pro, consistently degrades. Longer expressions typically contain more complex constraints, such as detailed visual attributes, fine-grained object states, and sequential interactions, which challenge the models’ ability to align extended linguistic queries with long-form visual contexts. Notably, while GPT-5 maintains the best performance across all lengths, its score drops by over 40% from the shortest to the longest queries, highlighting the necessity for stronger fine-grained multimodal alignment in future foundation models.

15.0.0.4 Effect of Temporal Position

To further investigate the impact of event timing on grounding performance, we analyzed the mvIoU with respect to the temporal position of the target event within long-duration videos. We divided each video into three segments: early (0, 1/3], middle (1/3, 2/3], and late (2/3, 1]. As shown in Table 8, compared to the early segment, all evaluated models exhibit a significant performance drop when the target event occurs in the middle or late segments of the video.

15.0.0.5 Effect of Object Scale

Table 9 presents experimental results of mvIoU on LongEgoRefer with respect to the object scale. As shown in the table, both models exhibit a consistent trend where performance scales with the size of the target object. For GPT-5, mvIoU increases from 5.03 for small objects to 10.76 for large objects, more than doubling across the scale spectrum. These results suggest that detecting and tracking smaller objects remains a significant challenge for existing VLMs, likely due to the limited visual resolution and information density of smaller regions of interest in egocentric videos.

15.0.0.6 Effect of Velocity

Table 10 presents experimental results of mvIoU on LongEgoRefer with respect to the target’s velocity. We observe an inverse relationship between performance and object velocity. For GPT-5, mvIoU drops from 10.94 for slow-moving objects to 4.84 for fast-moving objects. The fast-moving objects often introduce motion blur and rapid spatial displacement between frames, which appears to substantially hinder the models’ ability to maintain accurate temporal association and precise localization.

15.0.0.7 Grounded-SAM2 vs SAM3

Table 11 presents experimental results on LongEgoRefer where Grounded-SAM2 was replaced with SAM3. Performance actually dropped, possibly because Grounded-SAM2’s independent grounding module handles complex referring expressions more robustly.

2pt

15.0.0.8 Cross-Benchmark Comparison

To explicitly demonstrate the unique challenges of LongEgoRefer, we compare our results with the performance reported in the EgoMask paper [25] (Table 12). Existing models achieve reasonable performance on EgoMask’s Short clips (12.15s on avg.), but their accuracy consistently decreases on the Medium (116.30s) and Long (361.32s) splits. On LongEgoRefer, where videos are substantially longer (45 minutes on average), performance further drops to near zero (e.g., 49.95% to 0.67% for Grounded-SAM2). This substantial drop highlights LongEgoRefer’s distinct challenges, particularly extreme video duration and target sparsity.

4pt

16 Time Consumption↩︎

We present the time consumption for different VLMs in Figure 11.

17 Limitations and Future Work↩︎

Beyond the directions discussed in Section 6, we note several additional limitations. First, the extreme temporal length of our videos imposes heavy computational demands on current models. Consequently, high-resolution evaluations (e.g., higher frame rates like 5 FPS) and more exhaustive spatio-temporal inferences were constrained by these computational bottlenecks. Second, we did not conduct a separate, controlled human-performance study, and therefore do not report a formal human-ceiling baseline. While our rigorous multi-tier quality assurance workflow was designed to ensure that the finalized queries are clear and consistently interpretable by human annotators, establishing a precise, quantitative human performance ceiling remains an important objective for future iterations of this benchmark. Third, a potential risk is data contamination, as LongEgoRefer is built upon the widely adopted Ego4D dataset. It is possible that both the evaluated open-weight and closed-source models have been exposed to the underlying videos during their large-scale pre-training or fine-tuning phases. Although our interaction-centric referring expressions are entirely novel, this prior visual exposure remains a common limitation when evaluating modern foundation models. Future research should therefore not only aim to improve grounding accuracy on LongEgoRefer but also explore more computationally efficient architectures and robust evaluation protocols tailored for extreme long-form video understanding.

References↩︎

---

1. Woven by Toyota was not involved in this work. The views and opinions expressed in this paper are those of the authors and do not necessarily reflect the official policy or position of Woven by Toyota.↩︎