1 Introduction↩︎

With the rapid advancement of multimodal large language models, embodied agents are increasingly expected to perceive, remember, and reason about complex real-world environments [1]. Central to this capability is memory: the ability to retain and retrieve past observations to support informed decision-making and long-horizon task execution [2], [3]. However, existing memory frameworks for embodied perception are predominantly built upon egocentric observations [3], [4], capturing the world exclusively from the agent’s own first-person viewpoint. While egocentric memory has been widely studied for tasks such as episodic event recall [5]–[7], scene understanding [8], [9], and navigation [10], it remains fundamentally limited by the agent’s physical vantage point, leaving out-of-view regions and their spatial relationships inconsistently observed. This raises a critical question: is egocentric memory alone sufficient for comprehensive spatial understanding and temporal reasoning in embodied environments?

Real-world scenarios reveal fundamental limitations of purely egocentric memory. For holistic scene understanding, an agent’s egocentric stream captures its own interactions but offers limited visibility into other agents’ interactions within the shared space. While equipping all agents with smart glasses has been proposed [5], [11], this is impractical due to device cost and wearable burden [12]. Egocentric recording is also ill-suited for capturing whole-body movements, particularly in healthcare contexts such as fall detection [13]. Given the growing prevalence of exocentric infrastructure, including hospital surveillance [14], [15] and smart home cameras [16], [17], we propose complementing egocentric memory with exocentric observations to address these limitations.

Humans naturally encode experiences from two complementary perspectives: field memory, which relives events through a first-person viewpoint, and observer memory, which recalls the same event from a third-person vantage point [18]. These perspectives mirror the brain’s parallel egocentric and allocentric reference frames [19], and their interplay supports situation awareness [20]. Yet, existing ego-exo works focus on imitation learning or unidirectional knowledge transfer [21], with methods treating the two streams independently [22] and benchmarks evaluating only local view-clip matching [23] or view-invariant temporal consistency [24], [25], rather than reasoning over the full, synergistic memory formed by both views.

To address this gap, we introduce EgoExoMem, a large-scale benchmark comprising \(2.6K\) human-verified Multiple Choice Questions (MCQs) designed to evaluate memory-based reasoning over synchronized egocentric and exocentric videos sourced from EgoExo4D [26] and LEMMA [27]. We characterize memory-based reasoning along eight QA types covering temporal, spatial, and cross-view dimensions: Habitual Location, Instantaneous Position, Resulting Location, Egocentric Direction, Object State, Allocentric Relation, Third-Person Activity, and Temporal Ordering, with an example illustrated in Fig. [fig1:banner]. A text-only check is applied to ensure the vision-dependency of all answers. Motivated by the view-asymmetric and temporally synchronized nature of EgoExoMem, we further propose E\(^2\)-Select, a simple yet effective frame selection method with relevance-based budget allocation and k-DPP sampling, explicitly designed to handle view asymmetry and cross-view temporal consistency beyond single-view memory methods.

Extensive experiments are conducted to validate the reasonableness of the EgoExoMem task, benchmark the dataset, demonstrate the effectiveness of E\(^2\)-Select, and analyze failure cases. Proprietary [28] and open-source MLLMs [29]–[31] are evaluated on EgoExoMem as zero-shot baselines, confirming the task’s challenge. Memory mechanisms spanning frame selection [32], [33], text retrieval [34], [35], visually-grounded retrieval [36], [37], and structured memory retrieval [7], [11] are used to benchmark EgoExoMem, while our E\(^2\)-Select achieves state-of-the-art performance with an averaged MCQ accuracy of \(58.2\%\). Comprehensive ablation studies support the design choices of E\(^2\)-Select. Failure case analysis reveals a view-preference conflict between question framing and answer grounding, highlighting the complementary potential of synergistic ego-exo views.

2 Related Work↩︎

\begin{table}[t]

Benchmark	Venue	#QA	#Videos	#Tasks	View	Sync	Memory QA	Cross-view QA
EgoSchema [38]	NeurIPS’23	5.1k	5.1k	1	Ego	–
VSI-Bench [8]	CVPR’25	5.0k	288	8	Ego	–
EgoMemoria [6]	ICLR’25	7.0k	629	–	Ego	–
EgoPlan-Bench [39]	ECCV’24	4.9k	–	1	Ego	–
EgoExoLearn [22]	CVPR’24	2.2k	747	3	Ego+Exo	Async
EgoExoBench [23]	NeurIPS’25	7.3k	–	11	Ego+Exo	Both
EPFL-Smart-Kitchen [40]	NeurIPS’25	17.7k	49	6	Ego+Exo	Sync
MA-EgoQA [11]	arXiv’26	1.7k	–	5	Multi-ego	–
EgoExoMem (Ours)	2026	2.6k	390	8	Ego+Exo	Sync

\vskip

-2ex \end{table}

Ego-Exo Video Understanding and Reasoning. Cross-view understanding spans view translation [21], [41]–[44], correspondence learning [22], [45], cross-view grounding [46]–[48], and audio-visual association [49], [50]. View-conditioned generation has also advanced rapidly, including ego-exo synthesis [51]–[53], viewpoint-aware anticipation [54], and analysis of active versus passive perspectives [55]. Beyond generation, view-invariant representation learning [56] and imitation error detection [57] further demonstrate the value of cross-view alignment for fine-grained activity understanding. Recent benchmarks have also expanded the ego-exo scope: EPFL-Smart-Kitchen [40] evaluates action and motion understanding in multi-modal ego-exo settings, and SAW-Bench [58] probes situated awareness in real-world environments. Closest to our setting, Ravi et al. [59] show that off-screen context can be recovered from egocentric observations, and EgoExoLearn [22] benchmarks skill assessment across paired viewpoints. Yet, existing methods treat the two streams as inputs to a shared representation rather than as asymmetric sources of complementary evidence, a distinction that prior cross-view benchmarks do not evaluate. Relevant benchmarks are summarized in Tab. [tbl:tab:comparison].

Memory-Augmented Video Understanding. Episodic memory QA over egocentric video is well established [60]–[64]. Recent work pushes toward longer temporal horizons [5], [65], [66] and richer memory structures, including spatial mind palaces [67], dynamic memory refinement [68], and multi-agent memory [11], [69]. Fine-grained spatiotemporal grounding [70]–[72] and egocentric captioning [73], [74] further underpin retrieval capabilities. However, egocentric streams capture fine-grained interactions but miss the spatial layout and third-person activities. Exocentric streams offer the reverse perspective but lack close-up manipulation detail. EgoExoMem exploits this asymmetry with QA types that are unresolvable from either view alone.

Spatial and Procedural Video QA Benchmarks. Spatial QA benchmarks probe object size, distance, room layout [8], embodied cognition [75], and broad multi-modal egocentric perception [76]–[78]. Procedural and task-oriented QA is addressed across activity sequences [62], instructional video settings [79], and long-horizon retrieval tasks [5], [66]. Chain-of-thought and cross-modal reasoning in egocentric settings are targeted by Egothinker [80] and EgoCross [81]. Object-centric and referential understanding [82]–[84] connects directly to EgoExoMem’s localization and relation-change question types, while visual query localization [63] formalizes the where did I last see X task that underlies our displacement queries. Scene graph understanding [85]–[87], egocentric text QA [88], [89], low-light settings [90], and assistive scenarios [91]–[99] further round out the evaluation landscape. Taken together, these benchmarks span a rich set of reasoning skills but do not require joint ego-exo memory, which is the gap EgoExoMem targets.

3 EgoExoMem↩︎

3.1 Video Selection↩︎

We select synchronized videos from two datasets, LEMMA [27] and EgoExo4D [26]. Since our goal is to capture spatial changes over time through memory, the camera wearer must interact extensively with the environment. We therefore retain only the cooking and bike repair scenarios from EgoExo4D, which involve substantial environmental interaction, and exclude the remaining skilled activity categories. To capture activities of persons other than the camera wearer, we use all multi-agent videos from LEMMA, supplemented by single-agent videos from its test set. The video length distributions are shown in Fig. 4 (a): LEMMA (\(M=111.3s\), \(\sigma=74.8s\)) and EgoExo4D (\(M=425.0s\), \(\sigma=339.3s\)).

3.2 QA Types↩︎

We refer to QA types from egocentric episodic memory [4], [5] and spatial intelligence benchmarks [8], [9], [100], adapting them to the synergistic nature of ego-exo memory as shown in Fig. 2.

Habitual Location (HL) tests persistent spatial map retention by asking where an object is habitually located, requiring semantic spatial memory integrated over the full video. Ego and exo views contribute complementarily: the egocentric stream resolves fine-grained proximity, while the exocentric view anchors answers in global scene layout. Instantaneous Position (IP) probes the model’s ability to bridge egocentric action observations and allocentric spatial positions, requiring a latent mapping between first-person and world-centered reference frames. Resulting Location (RL) evaluates spatial memory across an event boundary, combining temporal grounding with spatial memory update to track an object’s location after manipulation. Egocentric Direction (ED) assesses fine-grained spatial awareness within the first-person reference frame, asking where a target lies relative to the camera wearer (e.g., back-left). While ego-view-anchored, exocentric frames provide complementary global context to resolve directions occluded from the wearer’s perspective. Models must maintain a continuous spatial representation as the wearer moves. Object State (OS) tests whether the model identifies the physical state of a manipulated object at a specified moment (e.g., uncut/sliced, whole/eaten). Object states change incrementally as procedural actions unfold, demanding frame-accurate understanding of manipulation sequences from either or both views. Allocentric Relation (AR) probes allocentric spatial reasoning by asking where one object lies relative to another (e.g., left of, directly behind). Both ego and exo views are viable but imperfect, as each provides a different vantage on the same configuration, potentially requiring cross-view integration. Third Person Activity (TPA) requires simultaneous spatial tracking of two people: while one agent performs a stated action, the model must locate the second agent in the scene. It benefits substantially from the exocentric view, which captures the full room layout, including agents outside the egocentric field of view. Temporal Ordering (TO) evaluates episodic memory for procedural sequences by requiring models to arrange sub-actions into their correct temporal order. The primary challenge is temporal-episodic: models must recall event ordering across potentially long video segments, where the egocentric view provides richer action-level detail and the exocentric view aids in disambiguating simultaneous or spatially separated sub-events.

3.3 Benchmark Construction↩︎

Fig. 3 illustrates the benchmark construction pipeline: MCQs are first generated, then human-edited and filtered for accuracy, and finally subjected to a text-only check to ensure vision dependency.

MCQ Generation. QA pairs are generated programmatically from structured annotations and then verbalized using GPT-5.4 [101]. The pipeline is shared across both datasets but draws on different annotation sources. Each question is formatted as a four-option multiple-choice question with exactly one correct answer.

For LEMMA, the primary annotation source is per-frame HOI labels, which record tuples of (action verb, interacted object, furniture anchor location) alongside person bounding boxes in the exocentric view. We extract contiguous action segments and derive structured ground-truth facts (object resting location, person position, object displacement, egocentric object direction, state transitions) as seeds for question generation. For EgoExo4D, we use keystep annotations (step name, step description, start/end timestamps) to identify activity segments, and relation annotations (per-frame object bounding box tracks) to accurately localize objects in the scene. Egocentric object directions are computed from the median of bounding box centroid across a track sequence.

For both datasets, each candidate is passed to GPT-5.4 with up to two keyframe images and a structured ground-truth context, producing a natural-language question and four answer options. People are identified by visual appearance rather than abstract labels. The keyframe selection varies by question type: HL uses a representative exocentric frame at the dominant get/put action; IP uses a window of ±5 surrounding exocentric frames around the action timestamp; RL uses an egocentric frame at pickup and an exocentric frame at placement; ED uses an egocentric frame, with the egocentric direction derived from the interacted object’s bounding box centroid; OS uses keyframes at both the state-changing event and the later query action; AR uses a single exocentric keyframe, with the spatial relation between two objects determined entirely by the model from the image; TO is derived directly from annotation timestamps. TPA (simultaneous two-person localization) is generated for LEMMA only, as EgoExo4D lacks concurrent multi-person spatial annotations.

Human Editing and Filtering. The generated MCQs are not guaranteed to be fully accurate. To ensure dataset quality, three human annotators reviewed all generated MCQs and either corrected them (for instance, fixing confusion over human appearance, location ambiguity, or incorrect give/get relations) or discarded them entirely (such as those with ambiguous reference objects or objects that never appear in the video). The interface used for human verification is shown in Fig. 6.

Text-only Check. The label noise in the MCQs is minimized through human editing and filtering. To ensure the dataset remains challenging and vision-dependent, we prompt GPT-4o [102] at temperature values of 0 and 1 to answer each MCQ based solely on the question text, without any video input. If GPT-4o answers correctly at both temperatures, the question is deemed common-sense answerable; if incorrect at both, vision-dependent; if mixed, potentially chance-correct. We discard all common-sense answerable questions and remove a subset of chance-correct ones to ensure that the text-only accuracy of GPT-4o (\(Temperature=0\)) for each QA type remains below \(45\%\). The resulting distribution of QA types and the word cloud of questions are shown in Fig. 4 (b-c).

4 E\(^2\)-Select: EgoExo Frame Selection↩︎

Unlike prior ego-exo benchmarks that focus on view matching or local clip understanding, EgoExoMem treats synchronized egocentric and exocentric videos as a persistent dual-view memory. It requires models to integrate complementary evidence across time: ego views provide fine-grained hand-object and wearer-relative cues, whereas exo views offer global layout, other-agent activity, and out-of-view context.

To address this setting, we propose E\(^2\)-Select, a training-free frame selection method with independent view scoring, relevance-based budget allocation, and per-view k-DPP sampling. This design preserves view-specific evidence, reduces temporal redundancy, and implicitly routes each query to the most informative ego-exo memory evidence.

Independent View Scoring. We first conduct pre-selection of individual views according to the query-visual relevance. We compute query-frame relevance independently for each view to avoid cross-view interference. Given a query \(\mathbf{q}\), ego frames \(\{\mathbf{f}_i^e\}_{i=1}^{N_e}\), and exo frames \(\{\mathbf{f}_j^x\}_{j=1}^{N_x}\) where \(N_e\) denotes the number of frames of egocentric video and \(N_x\) denotes the number of frames of exocentric video, we compute CLIP [103] cosine similarities separately according to Eq. 1 and Eq. 2 . \[\label{eq:1} \mathbf{s}_e(i) = \text{sim}\!\left(\phi_{visual}(\mathbf{f}_i^e),\, \phi_{text}(\mathbf{q})\right), \quad i = \left[1, \ldots, N_e\right],\tag{1}\] \[\label{eq:2} \mathbf{s}_x(j) = \text{sim}\!\left(\phi_{visual}(\mathbf{f}_j^x),\, \phi_{text}(\mathbf{q})\right), \quad j = \left[1, \ldots, N_x\right],\tag{2}\] where \(\phi_{visual}(\cdot)\) and \(\phi_{text}(\cdot)\) denote the CLIP visual encoder and text encoder. Each view is evaluated independently in its own semantic subspace, which avoids cross-view score interference and prevents a more dominant view from overwhelming the other during joint retrieval. When the two views are synchronized, \(N_e\) is equal to \(N_x\).

Relevance-Based Budget Allocation. To address view asymmetry, we allocate the total frame budget \(K\) proportionally to the aggregated query relevance of each view (Eq. 3 ): a view whose frames are collectively more relevant to the query receives a larger frame budget, providing a soft and query-dependent alternative to hard view selection. \[\label{eq:3} K_e = \left\lfloor K \cdot \frac{\sum_{i} \mathbf{s}_e(i)}{\sum_{i} \mathbf{s}_e(i) + \sum_{j} \mathbf{s}_x(j)} \right\rceil, \quad K_x = K - K_e,\tag{3}\] where \(\lfloor \cdot \rceil\) denotes rounding to the nearest integer. This formulation implicitly performs view selection while maintaining a better balance than hard-selecting between ego and exo views at each timestamp.

Per-View k-DPP Sampling. To address temporal redundancy within each view, we replace coverage-based selection with a Determinantal Point Process (DPP), which provides a principled diversity guarantee via determinant geometry.

For the ego view, we construct a quality-diversity kernel matrix \(\mathbf{L}^e \in \mathbb{R}^{N_e \times N_e}\), as shown in Eq. 4 . \[\label{eq:4} \mathbf{L}^e_{ij} = \mathbf{s}_e(i) \cdot \text{sim}\!\left(\phi_{visual}(\mathbf{f}_i^e),\, \phi_{visual}(\mathbf{f}_j^e)\right) \cdot \mathbf{s}_e(j),\tag{4}\] The diagonal entries \(\mathbf{L}^e_{ii} = \mathbf{s}_e(i)^2\) encode per-frame relevance; off-diagonal entries penalize the simultaneous selection of pairs of frames with similar visual content. We then draw a subset \(S_e\) of size \(K_e\) from the k-DPP defined by \(\mathbf{L}^e\) as in Eq. 5 . \[\label{eq:5} P(S_e) \propto \det\!\left(\mathbf{L}^e_{S_e}\right), \quad |S_e| = K_e,\tag{5}\] where \(\mathbf{L}^e_{S_e}\) denotes the principal submatrix of \(\mathbf{L}^e\) restricted to the frame indices in \(S_e\). Because the determinant measures the volume spanned by the selected frame embeddings, maximizing it encourages a subset that is both relevant to the query and visually diverse, rather than merely well-spaced in time.

We apply the same k-DPP sampling procedure to the exocentric stream, obtaining a subset \(S_x\) with \(|S_x|=K_x\). For efficiency, we use approximate Cholesky-based k-DPP [104] inference, which has complexity \(\mathcal{O}(N^2K)\).

Timestamp-Ordered Merge. After sampling view-specific subsets, we merge the selected ego and exo frames according to their original timestamps as shown in Eq. 6 . \[\label{eq:6} \mathcal{F} = \operatorname{Sort}_{t}\!\left( \{\mathbf{f}_i^e \mid i \in S_e\} \cup \{\mathbf{f}_j^x \mid j \in S_x\} \right),\tag{6}\] where \(t\) denotes the original video timestamps. This chronological merge preserves the temporal structure of the synchronized ego-exo memory while retaining view-specific evidence. The resulting sequence \(\mathcal{F}\) contains \(K\) frames and is used as the visual input to the MLLM.

5 Experiments↩︎

Since we propose the first benchmark dedicated to egocentric and exocentric memory, EgoExoMem, we dedicate ourselves to answering four research questions: (1) Can egocentric and exocentric streams serve as complementary memory sources for spatial and temporal reasoning? (2) How do existing MLLMs perform on EgoExoMem? (3) How does E\(^2\)-Select compare to existing memory retrieval strategies on EgoExoMem? (4) under what conditions does joint ego-exo memory retrieval deteriorate, and what are the underlying causes?

\begin{table}[t]

Model	Frame Input	Views	HL	IP	RL	ED	OS	AR	TPA	TO	Avg
Gemini 2.5 Flash [28]	concat	Ego	58.1	62.5	63.5	38.9	56.4	54.4	48.9	34.7	52.2
Gemini 2.5 Flash [28]	concat	Exo	51.4	72.2	59.6	34.6	48.8	44.6	45.5	38.9	49.5
Gemini 2.5 Flash [28]	concat	Ego+Exo	62.9	70.9	66.7	38.7	53.7	56.3	48.0	44.9	55.3
InternVL3.5 [30]	concat	Ego	50.7	55.5	55.1	38.2	55.5	43.3	47.7	39.3	48.2
InternVL3.5 [30]	interleaved	Ego	53.8	57.7	57.7	36.1	56.7	42.5	49.5	41.9	49.5
InternVL3.5 [30]	concat	Exo	47.2	65.4	50.6	33.0	50.0	39.2	36.0	37.0	44.8
InternVL3.5 [30]	interleaved	Exo	46.9	66.3	51.3	34.0	44.5	37.9	35.7	38.0	44.3
InternVL3.5 [30]	concat	Ego+Exo	53.1	65.2	64.1	36.4	57.9	42.1	44.8	41.9	50.7
InternVL3.5 [30]	interleaved	Ego+Exo	50.7	63.1	58.3	35.3	55.5	44.2	39.2	42.2	48.6
LLaVA-OV [29]	concat	Ego	48.3	53.6	52.6	32.7	50.0	37.9	36.9	36.3	43.5
LLaVA-OV [29]	interleaved	Ego	48.6	55.3	56.4	35.6	51.8	38.3	37.0	35.3	44.8
LLaVA-OV [29]	concat	Exo	47.2	64.2	42.9	31.2	50.0	37.9	34.1	36.6	43.0
LLaVA-OV [29]	interleaved	Exo	46.2	62.5	42.3	29.6	50.6	36.7	34.8	35.0	42.2
LLaVA-OV [29]	concat	Ego+Exo	51.0	63.6	50.0	34.8	52.4	41.2	34.6	35.6	45.4
LLaVA-OV [29]	interleaved	Ego+Exo	47.6	60.1	53.2	35.3	53.7	41.2	33.1	36.0	45.0
Qwen2.5-VL [105]	concat	Ego	50.0	49.3	53.2	36.9	43.3	38.3	38.5	37.0	43.3
Qwen2.5-VL [105]	interleaved	Ego	51.7	50.4	57.1	34.0	40.9	38.3	35.3	35.0	42.8
Qwen2.5-VL [105]	concat	Exo	50.7	57.1	51.9	30.4	41.5	40.0	35.7	36.6	43.0
Qwen2.5-VL [105]	interleaved	Exo	50.0	56.1	53.2	30.1	37.8	40.4	35.6	37.0	42.5
Qwen2.5-VL [105]	concat	Ego+Exo	53.1	58.2	58.3	31.7	42.7	40.4	34.3	36.0	44.3
Qwen2.5-VL [105]	interleaved	Ego+Exo	53.5	57.4	54.5	34.0	45.7	43.3	32.9	36.0	44.7

\vskip

-2ex \end{table}

\begin{table}[t]

Method	Views	HL	IP	RL	ED	OS	AR	TPA	TO	Avg
AKS [32]	Ego	64.7	64.2	62.8	37.7	56.1	47.9	55.9	44.8	54.2
AKS [32]	Exo	50.0	68.5	54.5	33.3	46.3	42.9	42.2	38.3	47.0
BOLT [33]	Ego	62.9	63.6	62.2	40.3	58.5	51.3	55.6	45.9	55.0
BOLT [33]	Exo	51.0	69.5	59.6	34.3	44.5	43.8	41.7	39.6	48.0
BM25 [34]	Ego	51.7	59.6	62.8	34.3	51.8	44.6	46.0	45.2	49.5
BM25 [34]	Exo	47.9	66.0	57.7	30.6	43.9	44.2	39.8	37.6	46.0
BM25 [34]	Ego+Exo	54.2	66.3	62.8	37.4	51.8	45.8	43.8	42.9	50.6
DPR [35]	Ego	52.8	60.6	64.7	34.5	48.2	44.2	45.8	44.9	49.5
DPR [35]	Exo	48.6	66.3	54.5	29.1	42.1	41.2	39.7	37.6	44.9
DPR [35]	Ego+Exo	56.3	68.2	67.3	36.9	45.7	40.8	45.8	41.9	50.4
VideoRAG [37]	Ego	47.2	53.6	62.8	36.6	53.0	40.8	48.0	40.3	47.8
VideoRAG [37]	Exo	47.2	69.0	60.3	29.6	46.3	37.9	41.4	36.0	46.0
VideoRAG [37]	Ego+Exo	49.0	66.6	62.2	33.5	57.9	43.8	41.0	39.6	49.2
Video-RAG [36]	Ego	56.3	55.3	50.0	36.9	50.6	47.1	48.8	35.6	47.6
Video-RAG [36]	Exo	49.3	63.9	51.9	34.0	42.7	43.8	40.6	35.3	45.2
Video-RAG [36]	Ego+Exo	53.1	66.0	57.1	35.8	50.6	45.8	43.0	35.0	48.3
EgoMAS [11]	Ego+Exo	45.5	56.6	51.3	28.6	62.8	35.8	37.0	38.3	44.5
WorldMM [7]	Ego+Exo	36.5	44.8	39.4	23.1	36.5	36.3	35.1	31.5	35.4
E\(^2\)-Select (ours)	Ego+Exo	59.8	72.8	64.1	34.3	57.9	51.7	47.9	46.2	56.3

\vskip

-2ex \end{table}

\begin{table}[]

Model	Frame Input	Views	HL	IP	RL	ED	OS	AR	TPA	TO	Avg
Gemini 2.5 Flash [28]	concat	Ego+Exo	62.9	70.9	66.7	38.7	53.7	56.3	48.0	44.9	55.3
Gemini 2.5 Flash [28]	k-DPP	Ego	67.8	69.5	68.6	38.4	57.9	56.3	48.6	45.9	56.6
Gemini 2.5 Flash [28]	k-DPP+hard selection	Ego+Exo	67.1	73.0	69.0	34.1	57.3	54.4	52.3	46.7	56.7
Gemini 2.5 Flash [28]	k-DPP+soft allocation (E\(^2\)-Select)	Ego+Exo	72.5	77.0	66.0	36.8	60.4	54.2	51.6	47.4	58.2
InternVL3.5 [30]	concat	Ego+Exo	53.1	65.2	64.1	36.4	57.9	42.1	44.8	41.9	50.7
InternVL3.5 [30]	k-DPP	Ego	63.3	63.3	62.2	35.3	54.9	52.1	55.1	44.6	53.8
InternVL3.5 [30]	AKS [32]+soft allocation	Ego+Exo	58.0	71.7	68.6	37.1	58.5	46.2	45.8	42.6	53.6
InternVL3.5 [30]	BOLT [33]+soft allocation	Ego+Exo	56.6	68.2	64.1	37.9	54.3	47.9	44.1	43.6	52.1
InternVL3.5 [30]	k-DPP+hard selection	Ego+Exo	60.5	71.7	63.5	38.2	52.4	53.8	45.1	43.6	53.6
InternVL3.5 [30]	k-DPP+soft allocation (E\(^2\)-Select)	Ego+Exo	59.8	72.8	64.1	34.3	57.9	51.7	47.9	46.2	56.3

\vskip\caption{Ablation study of E$^2$-Select.}\label{tab:ablation}-2ex

\end{table}

5.1 Implementation Details↩︎

We evaluate a proprietary MLLM, Gemini 2.5 Flash [28], and open-source ones, including InternVL3.5 [30], LLaVA-OneVision [29], and Qwen2.5-VL [31], to investigate the synergy of egocentric and exocentric memory and their performance on EgoExoMem. For this part, we explore two strategies for inputting frames from two video sources: concatenation [11], [23], which preserves intra-video temporal consistency but provides weaker inter-video frame correspondence; and interleaving [106], which improves inter-video frame correspondence but disrupts intra-video temporal consistency. Based on the best-performing open-source MLLM, InternVL3.5, we further experiment with different memory mechanisms, RAG-based [34]–[37] and frame selection for single video [32], [33], and our E\(^2\)-Select for two synchronized videos.

For fair evaluation, all reasoning models receive \(32\) frames as input. For MLLMs, \(16\) frames are uniformly sampled from either or both videos. For memory mechanisms, all videos are first processed at \(1\) FPS. For the RAG-based methods, videos are segmented into clips, and the top-\(k\) (\(k=8\)) most relevant clips are retrieved, each represented by \(4\) frames. The captions used for retrieval are generated with Gemini 2.5 Flash [28]. All other settings follow the original configurations of the methods. Our experiments are conducted on 4 NVIDIA A100 (40GB) GPUs.

5.2 Results↩︎

Reasonability of EgoExoMem and Performance of MLLMs. Tab. ¿tbl:tab:all95frame95concat? evaluates MLLMs with two simple video frame input strategies: concatenation and interleaving. To assess the complementarity of the two views, we keep the total number of frames constant by duplicating the same video for single-video input, ensuring any performance gain stems from view complementarity rather than additional temporal information. Overall, combining both views consistently outperforms single-view input across all evaluated models. This supports the reasonability of leveraging both views in EgoExoMem. For instance, Gemini 2.5 Flash improves from \(52.2\%\) (Ego) and \(51.4\%\) (Exo) to \(55.3\%\) with Ego+Exo, and InternVL3.5 improves from \(48.2\%\) (Ego) and \(44.3\%\) (Exo) to \(50.7\%\) with Ego+Exo under concatenation. For Habitual Location, Resulting Location, Object State, and Allocentric Relation, the cross-view combination is clearly preferred across models. Instantaneous Position favors the exocentric view, and Egocentric Direction favors the egocentric view. Notably, Third Person Activity consistently favors the egocentric view across all models, with InternVL3.5 showing a gap as large as \(13.8\%\) (\(49.5\%\) vs. \(35.7\%\)), which is counterintuitive and will be analyzed in the following section. Temporal Ordering shows no significant difference across egocentric, exocentric, and mixed inputs, with scores remaining comparable across all input strategies.

Despite being state-of-the-art MLLMs, all models achieve relatively low average scores, with the best-performing model, Gemini 2.5 Flash, reaching only \(55.3\%\), highlighting the challenge of EgoExoMem. Among open-source models, InternVL3.5 achieves the best performance and is therefore adopted as the reasoning model for the subsequent memory mechanism evaluation.

Performance of Memory Mechanisms. Tab. ¿tbl:tab:baselines? demonstrates the performance of memory mechanisms in order of increasing complexity, including frame selection (AKS [32] and BOLT [33]), text retrieval (BM25 [34] and DPR [35]), visually-grounded retrieval (Video-RAG [36] and VideoRAG [37]), and structured memory retrieval (EgoMAS [11] and WorldMM [7]). Consistent with the frame combination results in Tab. ¿tbl:tab:all95frame95concat?, retrieving from both egocentric and exocentric streams yields better performance than single-view inputs across RAG-based methods. However, despite their strong performance on long-video benchmarks, structured retrieval methods underperform on EgoExoMem, whereas simpler approaches, frame selection and text retrieval, prove more effective. We attribute this to the minute-level duration of our videos: for such short clips, complex retrieval pipelines introduce unnecessary overhead and may be redundant [107].

As frame selection methods demonstrate superior performance on single-view understanding, we propose E\(^2\)-Select, the first frame selection method for dual-view ego-exo inputs. As shown in Tab. ¿tbl:tab:baselines?, E\(^2\)-Select achieves superior performance over all baselines. A comprehensive ablation study is further conducted to verify the design choices in Tab. ¿tbl:tab:ablation?. Though AKS [32] and BOLT [33] achieve better performance on single-view ego input, k-DPP [108] combines the two views more effectively. As AKS and BOLT rely on temporal structure estimation and single-video saliency calibration, their performance degrades under cross-view budget allocation. In contrast, k-DPP natively supports any frame budget \(k\) and selects frames by maximizing diversity in feature space, making it distribution-agnostic and robust to the domain shift between ego and exo views. We also compare soft allocation with hard selection, which picks the more query-similar view at each timestep and then applies k-DPP. It performs on par with single-view k-DPP, failing to fully exploit the complementarity of dual views.

Failure Cases and Potential Reasons. Among all baselines, it is counterintuitive that Third Person Activity relies significantly on the egocentric view. By examining the keyframes used to generate the MCQs, we find that many questions can be answered from either view due to the small room settings in LEMMA [27] and the collaborative nature of the tasks, as shown in Fig. 5 (b). This raises the question of whether the view most relevant to the answer is also the most relevant to the question. To investigate this, we report the question-aware view-dependency measured by CLIP for all question types in Fig. 5 (a). We observe that the view preferences of the question and the answer for Third Person Activity (Tab. ¿tbl:tab:all95frame95concat? and Tab. ¿tbl:tab:baselines?) are significantly different: the question favors the exocentric view, whereas the answer favors the egocentric view. This discrepancy causes severe degradation of TPA performance when frame selection is based solely on question-aware similarity, highlighting the necessity of synergy between both views.

6 Conclusion↩︎

We present EgoExoMem, the first benchmark for memory-based reasoning over synchronized ego-exo video. Spanning eight QA types across spatial, temporal, and cross-view memory, it reveals that neither view alone suffices for comprehensive understanding, and that existing MLLMs and memory mechanisms fail to fully exploit dual-view complementarity. To fill the gap in multi-view frame selection, we propose E\(^2\)-Select, which achieves superior performance via relevance-based budget allocation and k-DPP sampling that accounts for view asymmetry and cross-view temporal consistency. Failure analysis further exposes a systematic view-dependency mismatch for Third Person Activity, motivating joint query-answer view routing in future work. We hope EgoExoMem and E\(^2\)-Select serve as a foundation for cross-view memory reasoning in embodied AI.

Acknowledgment↩︎

This work was performed on the HoreKa supercomputer funded by the Ministry of Science, Research and the Arts Baden-Württemberg and by the Federal Ministry of Education and Research. The authors also acknowledge support by the state of Baden-Württemberg through bwHPC and the German Research Foundation (DFG) through grant INST 35/1597-1 FUGG. The project is funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) – SFB 1574 – 471687386. This project is also supported in part by the National Natural Science Foundation of China under Grant No. 62473139, in part by the Hunan Provincial Research and Development Project (Grant No. 2025QK3019), and in part by the State Key Laboratory of Autonomous Intelligent Unmanned Systems (the opening project number ZZKF2025-2-10). This research was partially funded by the Ministry of Education and Science of Bulgaria (support for INSAIT, part of the Bulgarian National Roadmap for Research Infrastructure).

7 Social Impact and Limitations↩︎

Memory is essential for reasoning with MLLMs as a form of auxiliary cognition. However, existing memory-based approaches rely solely on the egocentric stream of mobile agents and their interactions with the environment. This limits their ability to capture full-body movements of other agents and track their interactions with the environment, both of which are critical to the holistic understanding of the scene. With the advancement of surveillance systems, wearable devices for human agents, and visual sensors for embodied agents, memory can increasingly be constructed from cross-view sources to enable more comprehensive retrieval, e.g., in homes [27], hospitals [109], and parking facilities [110].

As the first benchmark for ego-exo memory, EgoExoMem has several limitations. First, the minute-level video duration limits the scope of our benchmark, where simple retrieval methods may already suffice. Since real-world observations can span weeks or months, datasets supporting long-term ego-exo memory are necessary, and structured retrieval methods could prove more beneficial in such settings. Second, to standardize the task, we currently restrict input to one egocentric and one exocentric stream. However, the underlying datasets contain multiple egocentric and exocentric views, which means that the same MCQs could be repurposed for multi-view settings. Future work could explore this direction and investigate how multiple egocentric or exocentric streams individually contribute to benchmark performance.

8 Human Editing and Filtering↩︎

Fig. 6 shows the verification user interface.

9 Evaluation Prompts↩︎

The prompt used to generate captions for RAG-based methods with Gemini 2.5 Flash is shown in Fig. 7. The evaluation template is provided in Fig. 8.

None

None

10 Further Ablation Studies↩︎

Following standard practice [111], we uniformly sample \(32\) frames as input. For RAG-based methods, we retrieve the top-\(k\) clips (\(k=8\)) and sample \(4\) frames per clip. An ablation study on the effect of \(k\) is provided in Tab. 1, which demonstrates that performance is largely insensitive to the choice of \(k\).

References↩︎

---

1. Corresponding author: kunyu.peng@kit.edu; First author: ruiping.liu@kit.edu↩︎