1 Introduction↩︎

Long-term egocentric memory assistants enable users to answer questions about interactions captured through wearable cameras, supporting applications such as autobiographical memory assistance [2], personal analytics, and cognitive support. Unlike conventional video understanding [3]–[5], which answers over seconds to minutes of videos, egocentric memory assistants must answer queries about events that have occurred hours or days before. To enable efficient retrieval over such long videos, existing methods [1], [6] first transform raw video into unstructured captions and then into hierarchical textual summaries.

While effective for describing individual observations and reducing memory volume compared to videos, identify recurring person–action–object patterns and aggregating evidence across long time horizons is challenging in textual summaries. This limitation becomes particularly apparent for behavioral reasoning queries such as “Have I been skipping lunch more often recently?” (see Figure 1), which require tracking and comparing interactions distributed across multiple days rather than retrieving a single relevant event. A possible solution is to record interactions explicitly as structured records without summarization. However, this approach introduces a scalability challenge, as wearable cameras generate thousands of observations per day many of which correspond to repeated interactions. Consequently, three requirements for effective long-horizon egocentric memory emerge: preserving interaction structure, supporting temporal aggregation across recurring interactions, and remaining scalable as observations accumulate.

We address these requirements through Imprint, a structured memory framework in which each interaction is represented as an Interaction Record containing the interacting person, action, object, and the start and end timestamps of the interaction. While this representation preserves the interaction semantics required for long-horizon reasoning, storing every Interaction Record is impractical even for a day (e.g. >10,000 interactions in 8 hours [1]) with rapidly growing and highly redundant memories. To address this challenge, we introduce an online compression approach where memory compression occurs as interaction records arrive preventing the need to store all the interaction records. Incoming interaction records are organized into recurring event patterns, an importance score for each interaction record is assigned to determine its priority, and important interaction records are consolidated into compact long-term memories. The importance score is designed using the principles of frequency, redundancy and distinctiveness inspired by human long-term memory consolidation [7]. The consolidated memory is able to long-horizon behavioral query such as "Have I been skipping lunch more often recently?" with evidence as shown in Figure 1.

We evaluate Imprint on EgoLifeQA [1], an egocentric benchmark with seven-day of videos containing questions that require reasoning over interactions occurring from minutes to multiple days before the query. Compared with hierarchical caption summaries, Imprint achieves higher question answering accuracy while increasing evidence-grounded answers by \(6\times\). This suggests that cognitively inspired importance signals improve long-horizon memory by preferentially retaining interactions that are most likely to support future retrieval and reasoning. Furthermore, we show that Imprint substantially reduces memory footprint and retrieval latency, demonstrating that interaction-centric memory compression is a scalable alternative to hierarchical text summarization for long-horizon egocentric reasoning. Our contributions are as follows:

• We introduce Imprint, a cognitively inspired online memory compression framework that discovers recurring interaction patterns, estimates their long-term importance, and selectively consolidates redundant interactions while preserving retrieval-relevant evidence.

• Imprint showing that it retains relevant interactions better than hierarchical caption summarization, yielding higher QA accuracy, and \(6\times\) better evidence grounding with the same LLM.

• We also show that Imprint requires \(2.3\times\) lower memory and is \(11.8\times\) faster at answering than hierarchical caption summaries.

2 Related Work↩︎

2.0.0.1 Long-Horizon Egocentric QA.

Early egocentric datasets such as CharadesEgo [8], EGTEA Gaze+ [9], EPIC-KITCHENS [10], and Ego4D [11] focus on activities ranging from seconds to a few hours, but do not capture the multi-day interaction dynamics required for persistent memory tasks. Recent work has begun addressing this limitation. EgoLife [1] introduces a 300-hour multimodal egocentric dataset together with the EgoLifeQA benchmark spanning five reasoning categories across seven days of continuous recording. Castle [12] introduces a 600-hour multimodal egocentric dataset recorded over four days and includes the point of view of 10 participants. Recent instruction-tuned multimodal LLMs  [3]–[5], have improved video-language alignment and grounding. Retrieval-augmented approaches store past observations as captions or summaries and retrieve relevant entries during inference. EgoRAG [1] follows this paradigm by generating per-clip captions and organizing them hierarchically (clip \(\rightarrow\) group \(\rightarrow\) day).

2.0.0.2 Memory-Augmented Retrieval for Long-Context Reasoning.

Recent work in long-context RAG highlights the limitations of unstructured text retrieval. Retrieval quality depends on preserving document structure and contextual continuity [13], while chunk-level retrieval often breaks semantic dependencies between related evidence [14]. To improve scalability and coherence, recent systems incorporate graph-based indexing with dual-level retrieval [15], hierarchical coarse-to-fine abstraction [16], and external structured memory for persistent agents [17]. Together, these advances highlight the importance of structured memory representations for scalable long-horizon retrieval and reasoning.

2.0.0.3 Cognitive Memory Models.

Cognitive memory theory suggests that scalable retrieval emerges from selective and structured representations rather than uniform storage, with consolidation influenced by repetition, recency, distinctiveness, and salience [18]. Repeated interactions are gradually compressed into structured semantic representations, enabling efficient long-term retention while preserving salient contextual information [7], [19]. At the same time, distinctive or behaviorally significant events tend to resist abstraction and remain individually retrievable in episodic memory [20]. However, existing egocentric QA pipelines largely treat observations uniformly, overlooking principles of selective consolidation and salience-driven retention. In contrast, our framework models interactions as structured action–object memories with importance-aware weighting, enabling selective compression while preserving distinctive events for long-horizon retrieval.

3 Imprint: Interaction-centric Memory Framework↩︎

3.1 Interaction Records↩︎

Free-form captions often obscure the interaction structure required for long-horizon reasoning. For example, repeated observations such as pick up mug, fill mug with coffee, and drink coffee may be summarized as a generic description such as having coffee. While sufficient for describing an individual event, such summaries make it difficult to track recurring interactions, aggregate behavior over time, and answer questions about habits and routines. To preserve interaction-level semantics, we represent each observation as a structured interaction record: \[f_i = (p_i, a_i, o_i, t_i^{s}, t_i^{e}) \label{eq:fio}\tag{1}\] where \(p_i\) denotes the interacting person, \(a_i\) the action, \(o_i\) the interacted object, \(t_i^{s}\) and \(t_i^{e}\) the interaction start and end timestamps. We parse each caption into an interaction record using an LLM ( Qwen2.5-7B-Instruct) as shown in Figure 2. The extraction prompt and schema are provided in Supplementary B.

3.2 Memory Compression↩︎

Unlike offline compression methods that operate after all observations have been collected, Imprint performs memory consolidation online for every incoming interaction record obtained from the caption. This streaming design eliminates the need to retain the complete set of extracted interaction records, allowing memory to evolve continuously as new observations arrive.

3.2.1 Event Prototype Assignment↩︎

Interaction records generated from continuous egocentric observations often exhibit recurring patterns due to routine human behavior. For example, a user may repeatedly perform interactions such as pick up mug, fill mug with coffee, and drink coffee every morning. Treating each occurrence as an independent memory introduces substantial redundancy while obscuring recurring behavioral patterns. To identify such patterns and estimate their long-term significance, we organize incoming interaction records into a set of event prototypes \(\mathcal{G}\). Each event prototype \(g_n \in \mathcal{G}\) maintains a recurrence count \(\nu_n\) and last-seen timestamp \(T_n^{\mathrm{last}}\). These statistics are later used to compute frequency and recency signals for interaction importance estimation.

For each incoming record \(f_t\), we first encode its action and object as \(e_t=\mathrm{Encoder}(a_t,o_t)\) and augment it with short-term interaction history of \(k\) recent interactions to form a contextualized embedding \(\tilde{e}_t\). The contextualized embedding of the interaction record is compared against all prototypes in \(\mathcal{G}\) and assigned to the nearest prototype subject to the similarity exceeding \(\delta_g\). Otherwise, a new prototype is created to represent the unseen interaction pattern. To maintain bounded memory growth, the prototype bank is capped at \(N_{\max}\). Once this capacity is reached, incoming interactions are assigned to their nearest existing prototype, trading representational granularity for scalability. Event prototypes are auxiliary structures rather than retrievable memories - they summarize recurring interaction patterns and maintain the recurrence and recency statistics used for downstream importance estimation. Implementation details are provided in Supplementary C.

3.2.2 Interaction Importance↩︎

Not all recurring interactions contribute equally to long-term memory. For instance, a coffee-drinking interaction observed every morning over several days is likely to be more informative for future behavioral reasoning than an isolated interaction such as searching for a receipt once. Inspired by cognitive theories of memory consolidation [7], [19], [20], we prioritize retrieval-relevant interactions by assigning an importance score to each interaction record based on cognitive concepts of frequency, recency and diversity. The importance score for an interaction record \(f_t\) is computed as follows:

\[w(f_t) = \overbrace{\log(1+ \nu_{n_t})}^{\text{\small frequency}} \cdot \overbrace{ e^{-\lambda \Delta T}}^{\text{\small recency}} {\overbrace{\frac{1}{1+\log(1+N_t)}}^{\text{\small distinctiveness}}}\] where \(n_t\) denotes the event prototype assigned to the interaction record \(f_t\), \(\nu_{n_t}\) denotes the recurrence count of the event prototype \(g_{n_t}\). \(\Delta T\) shows the elapsed time since its previous occurrence, and \(\frac{1}{1+\log(1+N_t)}\) reflects the diversity of event prototypes (\(N_t\)) currently in memory. Distinctiveness is significantly higher for the initial interaction records and gradually decreases when the prototypes are populated.

The importance score \(w(f_t)\) estimates the long-term relevance of an interaction by combining recurrence and recency signals and \(\lambda\) controls the rate at which importance decays with temporal distance. Records with \(w(f_t)\) greater than a threshold (\(\tau\)) are retained. In addition to the retention decision, the importance score is recorded along with the retained interaction record, and then reused for Retrieval and Question Answering (Section 3.3).

3.2.3 Interaction Consolidation↩︎

Even after importance filtering, we may have multiple instances of the same interaction pattern across days. For example, repeated records corresponding to drink coffee observed across several mornings would be stored independently despite describing essentially the same recurring behavior. Storing such interactions separately increases memory redundancy and retrieval cost.

For each retained interaction record \(f_i\) from the previous stage, we compare its action \(a_i\), object \(o_i\), and contextual embedding \(\tilde{e}_i\) against existing entries in the compressed memory \(\mathcal{M}\). An interaction record is merged with an existing memory entry when their action-object representations and contextual embeddings are similar. For example, interactions such as open book, read notes, and write on paper may be consolidated into a single event sequence representing a studying episode. When no suitable match exists, a new memory entry is created for the interaction record and added to \(\mathcal{M}\). For instance, the consolidated studying interactions above form a single entry \(m = (\langle f_1, f_2, f_3 \rangle,\, w)\), where \(f_1 = (\textit{Jake},\textit{open},\textit{book},t^s_1,t^e_1)\) and similarly, \(f_2, f_3\) are read notes and write on paper records, and \(w\) is their aggregated importance. Each memory entry preserves the temporal ordering, aggregated importance, and contextual evidence of the interactions it represents, enabling reconstruction of events during retrieval.

3.3 Retrieval and Question Answering↩︎

Given a question \(q\) asked at time \(t_q\), interaction records whose start time precedes the question timestamp are obtained \(\mathcal{M}_q = \{m_k \in \mathcal{M} \mid t_k^{s} \le t_q\}\). The question is then analyzed using an LLM to extract retrieval cues, including object \(o_q\), action \(a_q\), and temporal constraints \(r_q\) (if present). For example, for the question “When did Jake cook pasta in the evening?”, the extracted retrieval cues are \(o_q=\textit{pasta}\), \(a_q=\textit{cook}\), and \(r_q=\textit{evening}\). The extracted object and action are encoded using the same sentence encoder employed during memory construction. Each candidate memory entry \(m_k\) is then ranked according to a weighted combination of object and action similarity:

\[s_k = \gamma_1 \cos(e_{o_q}, e_{o_k}) + \gamma_2 \cos(e_{a_q}, e_{a_k}) +\gamma_3 w(f_k), \label{eq:3}\tag{2}\]

where \(e_{o_k}\) and \(e_{a_k}\) denote the object and action embeddings and \(w_k\) importance score associated with the memory entry \(m_k\). The top-\(k\) highest-scoring memories are retrieved as evidence \[R = \mathrm{TopK} \bigl(\{(m_k,s_k)\} \forall {m_k \in \mathcal{M}_q}\bigr).\] For questions containing temporal constraints, the retrieved evidence is further filtered according to \(r_q\). Specifically, first_time and last_time select the earliest and latest matching interactions, respectively, while before and after retain only interactions satisfying the corresponding temporal condition. The resulting evidence set \(R'\) is then provided to an LLM for answer generation along with the question: \(r = \mathrm{LLM}(q,R')\). More details about the retrieval mechanism are in Supplementary D.

4 Experiments and Results↩︎

4.1 Setup↩︎

4.1.0.1 Dataset.

EgoLifeQA [1] is a long-horizon egocentric QA benchmark of 3,000 multiple-choice questions, derived from seven consecutive days of wearable-camera recordings of multiple participants. We evaluate on the published 500-question benchmark for the participant Jake. Questions are distributed across five reasoning categories-EntityLog (125), EventRecall (126), RelationMap (125), HabitInsight (61), and TaskMaster (63).

4.1.0.2 Implementation details.

We compare Imprint against EgoRAG [1], a hierarchical caption-retrieval framework that uses GPT-4o as the answering LLM. To isolate the contribution of memory representation from LLM capacity, we reproduce EgoRAG [1] with Qwen2.5-7B-Instruct, the same LLM used throughout our pipeline. This reproduced version serves as our primary baseline. We use EgoGPT [1] for caption generation, identical to the captioner used in EgoRAG [1], ensuring that performance differences arise from memory representation and retrieval design rather than caption quality. We use a query-aware prompting strategy for caption generation. Details are in the Supplementary A. For all subsequent stages, including extraction of interaction record, compression and answer generation, we use Qwen2.5-7B-Instruct. For event prototype assignment, each interaction record is augmented with a short-term history of the previous \(k=5\) interactions to form the contextualized embedding \(\tilde{e}_t\). Historical interactions are incorporated using a context-history weight of \(\alpha_h=0.5\), and only interactions whose similarity exceeds \(\delta_h=0.3\) contribute to the contextual representation. Incoming interaction records are assigned to the nearest event prototype when cosine similarity exceeds \(\delta_g=0.75\); otherwise, a new prototype is created. The prototype bank is capped at \(300\) entries.

For Interaction Importance, the recency decay coefficient is set to \(\lambda =1.14\) and retain interaction records whose importance score satisfies \(w(f_i) \geq \tau\), where \(\tau = 0.2\). During Interaction Consolidation, retained records are merged into existing memory entries when their action-object representations and contextual embeddings exceed a cosine similarity threshold of \(\delta_m = 0.75\).

At query time, candidate memory entries are retrieved from the compressed memory \(\mathcal{M}\) and ranked using the retrieval score defined in Eq. 2 , with weights \(\gamma_1 = 0.53\), \(\gamma_2 = 0.29\), and \(\gamma_3 = 0.18\) details are in Supplementary E.4. The \(TopK=5\) memories are provided to Qwen2.5-7B-Instruct for answer generation. All experiments are conducted on a single NVIDIA L40 GPU.

4.1.0.3 Metrics.

For long-horizon egocentric assistants, accuracy alone is insufficient because language models can answer plausibly using parametric priors or semantic guessing without retrieving supporting memory evidence. This is particularly problematic for personal reasoning queries such as “Do my recent activities suggest changes in my health?”, which require aggregating temporally distributed interactions rather than relying on generic behavioral assumptions. To evaluate whether answers are genuinely grounded in retrieved memory, we introduce grounded accuracy in addition to QA accuracy.

Grounded Accuracy (GA) measures the proportion of correct predictions supported by retrieved evidence rather than inferred solely from the model’s parametric knowledge:

\[\mathrm{GA} = \frac{ \#\;\text{grounded correct answers} }{ \#\;\text{correct answers} }\] An answer is grounded when it is obtained directly from a retrieved interaction and its explanation matches that interaction’s fields. For the question What happened the last time I was at the claw machine?, selecting the correct option Distribute coins is grounded when its explanation matches a retrieved record, e.g. \(\langle \textit{Jake}, \textit{operated}, \textit{claw machine}, \textit{coins} \rangle\), whose coins field matches the answer. The same correct option is not grounded when its explanation matches no retrieved field and instead relies on a general assumption (e.g.that arcades involve coins) but not evidence-supported. More examples are in Supplementary E.5.

4.2 Interaction-Centric Retrieval Performance↩︎

Table 1 reveals that the gains of Imprint arise from both from the use of structured interaction representation and memory compression. Replacing hierarchical caption summaries in EgoRAG [1] with Interaction Records dramatically improves grounding (GA: 10.8% \(\rightarrow\) 41.5%), indicating that explicit person-action-object representations allow better evidence retrieval than textual summaries. However, this improvement does not translate directly into higher QA accuracy, which slightly decreases from 31.0% to 28.4%. We hypothesize that while raw Interaction Records preserve relevant evidence, they also introduce substantial redundancy, retrieval noise, and competition among repeated observations of the same behavior. Imprint’s memory compression improves both grounding and QA accuracy, demonstrating that effective long-horizon memory requires not only preserving interaction structure but also organizing that structure into a retrieval-efficient representation.

2pt

Figure 3 provides a category-wise analysis across the five EgoLifeQA question types. Imprint improves grounded answering across all categories, with the largest gains observed on EntityLog, EventRecall, RelationMap, and HabitInsight. These categories require tracking entities across time, recovering relationships between people, actions, and objects, and aggregating evidence across multiple interactions. Such reasoning benefits directly from interaction records, where interaction participants and objects are represented explicitly rather than embedded within natural-language summaries. Improvements on HabitInsight further suggest that Imprint preserves recurring interaction patterns that are critical for habit-level reasoning. In contrast, the gains on TaskMaster are comparatively smaller, indicating that they need planning and multi-step reasoning, so better memory alone is not enough to improve performance. Collectively, these results support the central hypothesis of Imprint: long-horizon egocentric QA benefits from memories that preserve interaction structure and consolidate interaction records around behaviorally salient recurring interactions.

4.3 Long-Horizon Retrieval and Efficiency↩︎

Figure 4 (a) evaluates retrieval grounding using the temporal-gap partitions introduced by EgoRAG [1], which group questions according to the time elapsed between a query and its supporting evidence. Across all temporal gaps, Imprint substantially outperforms EgoRAG in Grounded Answers (GA). Notably, Imprint’s grounded accuracy remains between 53-64% for evidence occurring up to 24 hours before the query, whereas EgoRAG remains below 22% across all gaps. The performance drop in the >24h gap likely reflects the increased difficulty of retrieving and aggregating evidence distributed across multiple days of experience. Nevertheless, Imprint retains a clear advantage over EgoRAG, indicating that interaction-centric memories remain more robust to temporal separation than summary-based representations.

Figure 4 (b) examines the efficiency implications of memory consolidation. Despite achieving substantially higher retrieval grounding, Imprint uses less memory and retrieves evidence significantly faster than both All Interaction Records1(All Int. Rec.) and EgoRAG. Memory size decreases from 267 MB for all interaction records to 109 MB after consolidation, compared with EgoRAG’s 254.2 MB consisting of both caption-level memories and hierarchical summary. More importantly, retrieval latency decreases from 20.1 s/query in EgoRAG to 1.7 s/query in Imprint, yielding an 11.8\(\times\) speedup. These results indicate that Imprint simultaneously improves retrieval grounding while reducing storage and retrieval cost, making it well-suited for long-horizon egocentric memory, despite continuously growing recordings.

4.4 Ablations↩︎

4.4.0.1 Stage-wise Memory Compression.

Table 2 evaluates the contribution of each stage in the Imprint memory formation pipeline. Removing any stage reduces both Grounded Answers (GA) and QA Accuracy, indicating that event-prototype assignment, interaction importance, and interaction consolidation each contribute to effective long-horizon reasoning. The largest reduction in GA occurs when event-prototype assignment is removed (64.8% \(\rightarrow\) 58.6%), accompanied by a notable drop in accuracy (35.8% \(\rightarrow\) 31.4%). This suggests that identifying recurring interaction patterns is critical for estimating memory relevance and distinguishing routine behaviors from isolated observations. Removing interaction importance yields the largest accuracy degradation after prototype assignment (35.8% \(\rightarrow\) 34.2%) while reducing GA to 57.3%, indicating that prioritizing interactions according to recurrence, recency, and distinctiveness improves both evidence selection and answer quality. Similarly, removing interaction consolidation decreases GA to 58.3% and accuracy to 32.6%, demonstrating that preserving interaction structure alone is insufficient; repeated observations must also be organized into compact memory entries to reduce redundancy and improve retrieval efficiency. The final row further shows that interaction consolidation alone is insufficient. Notably, the relative variation across configurations is larger for GA than for accuracy, suggesting that these stages primarily improve the quality of retrieved evidence, with improved QA accuracy emerging as a consequence of better-grounded retrieval.

4.4.0.2 Interaction Importance Signals.

Table 3 reveals a consistent trend across both Grounded Accuracy (GA) and Accuracy: removing any component of the importance score degrades performance, indicating that frequency, recency, and distinctiveness each contribute useful information for memory selection. Removing frequency produces the largest performance degradation indicating that recurrence is the strongest signal for identifying retrieval-relevant interactions. Removing recency also substantially reduces performance, reflecting the fact that many questions emphasize recent interactions rather than equally frequent events from the distant past. Distinctiveness has the smallest impact, but its removal still degrades both GA and Accuracy, indicating its role in distinguishing similar interaction patterns. Notably, the effect of all three signals is substantially larger on GA than on accuracy, indicating that their primary benefit is improving retrieval quality and evidence grounding. These findings support the central premise of Imprint that it is able to selectively retain behaviorally relevant interactions.

4.4.0.3 Memory Growth and Prototype Behavior.

Figure 5 illustrates the growth of memory over the seven-day EgoLife [1] recording. Imprint grows substantially more slowly than EgoRAG’s summaries (day, hour, minute) as recording length increases. The gap widens after Day 3, when recurring interaction patterns become increasingly common and are consolidated into existing memories rather than stored as independent entries. By Day 7, Imprint maintains approximately 2.4\(\times\) fewer interaction records in memory as seen in its comparison with all interaction records. Importantly, this improved scalability does not incur significant pre-processing overhead: compressing the full seven-day EgoLife recording requires 3 h 4 min, comparable to EgoRAG’s 2 h 54 min hierarchical summarization pipeline.

Figure 6 provides additional insight into the consolidation mechanism. As shown in Figure 6 (a), too few prototypes (100-200) force semantically distinct interaction patterns share prototypes, reducing the quality of importance estimation and subsequent consolidation. Increasing the budget beyond 300 provides little additional benefit and slightly degrades performance, suggesting that excessive prototype fragmentation weakens the recurrence statistics needed for reliable memory consolidation. Figure 6 (b) shows that at event prototype merging threshold \(\delta_g=0.75\), more than 90% of incoming interactions are assigned to existing prototypes, while only a small fraction create new prototypes. This suggests that most everyday observations correspond to recurring behavioral patterns rather than novel interaction types.

4.4.0.4 Impact of Hyperparameters.

We study the effect of hyperparameters on GA (see Figure 7). As shown in Figure  7 (a) at \(\lambda=1.14\) gives the highest GA. We observe that a higher value of \(\lambda\) suppresses interactions that remain relevant over longer horizons, while a lesser value of \(\lambda\) reduces the ability to prioritize recent evidence. Next, we show the effect of importance score threshold \(\tau\) on the retention of records, as shown in Figure 7 (b). The performance at \(\tau=0.2\) strikes a balance between retaining interaction records and limiting redundant interactions. Finally, we study the impact of \(\alpha_h\) in Figure 7 (c). We find moderate \(\alpha_h=0.5\) produces the best result, while both weaker and stronger history weighting reduce retrieval quality. Thus, over-weighting historical context biases prototype assignment towards outdated observations, degrading adaptation to evolving egocentric video streams. Other hyperparameter ablations on the history length \(k\) and history-similarity threshold \(\delta_h\) are provided in Supplementary Section E.3.

4.4.0.5 Qualitative Analysis

Figure 8 shows that structured interaction memory improves intra-day episodic recall. For the question “What happened the last time I was at the claw machine?” (DAY3 16:01), Imprint retrieves a temporally grounded interaction sequence from DAY3 16:01–16:32, preserving the progression walked towards \(\rightarrow\) watched \(\rightarrow\) operated around the claw-machine area. Crucially, the retrieved interaction records retain discriminative interaction objects such as coins and claw machine, directly grounding the correct answer “Distribute coins.” In contrast, EgoRAG [1] retrieves semantically related but shallow location-centric captions (e.g., “standing near,” “visible nearby,” “looking at claw machines”) and defaults to the prototypical arcade prior “Grab a doll.” These examples highlight how interaction-centric memory better supports temporally grounded episodic retrieval in long-horizon egocentric QA.

5 Conclusion↩︎

In this work, we introduced Imprint, a structured memory framework for long-horizon egocentric question answering. By consolidating recurring interactions using a cognitively inspired importance score, Imprint stores fewer interactions while producing more evidence-grounded answers. Experiments on EgoLifeQA [1] show that Imprint improves grounded answers while substantially reducing memory footprint and retrieval latency. However, its performance remains dependent on caption quality, as errors in identifying people, objects, or locations can propagate through memory formation and retrieval. While our importance score is based on cognitively motivated heuristics, future work could explore learnable memory heuristics for adaptive memory.

6 Query-Aware Captioning↩︎

To improve the extraction of interaction records, we define set of meta-questions based on EgoLifeQA [1]. Conditioning caption creation on these meta-questions enables the model to generate more comprehensive and retrieval-focused descriptions that retain details regarding participants, actions, objects, places, temporal context, and user intentions, hence enhancing memory quality for long-term question answering.

6.1 Meta-Question Taxonomy↩︎

To guide caption generation toward retrieval-relevant content, we organize 64 representative meta-questions into five functional categories — Episodic, Social, Behavioral, Temporal, and Goal-Oriented — each capturing a distinct facet of long-horizon egocentric reasoning targeted by EgoLifeQA [1], as shown in Figure 9. The meta-questions are not explicitly posed to the captioning model for response. Rather, they are integrated as implicit guidance during caption generation, prompting the model to assimilate information regarding participants, actions, objects, places, temporal context, and objectives that may subsequently be necessary for memory retrieval and question responding.

6.2 Query-Aware Captioning Prompt↩︎

Figure 11 shows the full prompt used to generate a caption for each video segment. The prompt guides EgoGPT [1] to produce a first-person narrative using ‘I’ as the subject while leveraging the meta-questions as implicit guidance rather than explicit questions to answer.

None

6.3 Query-Aware Captioning Ablation↩︎

Figure 10 shows that query-aware captioning consistently enhances downstream QA performance for both caption-based and interaction-based retrieval methods. The enhancement is notably significant for Imprint, with Grounded Accuracy rising from 5.41% to 64.80%, indicating that preserving retrieval-relevant details during memory creation improves downstream retrieval and reasoning.

7 Interaction Record Extraction↩︎

Following query-aware caption generation, each caption is parsed into one or more interaction records. Each interaction record contains the core fields defined in Section 3.1: person, action, object, start time, and end time. In addition, we extract auxiliary metadata, including tools, applications, locations, speech content, and contextual attributes. These fields are retained as contextual metadata and are used during the answering stage to improve interaction disambiguation and provide better-grounded answers.

Figure 12 shows the extraction prompt. The prompt defines a fixed JSON schema containing persons, action, object, tool, app, location fields, speech_content, and attributes, and instructs the model to generate one Interaction Record per distinct interaction while preserving speech verbatim. To resolve first-person references, the camera wearer’s identity is prepended to the caption before extraction, enabling consistent participant identification.in the persons field correctly.

None

8 Memory Compression↩︎

Algorithm 13 computes an importance score \(w(f_i)\) for each Interaction Record using frequency, recency, and distinctiveness. Low-importance records are discarded, while redundant records are consolidated into existing memory entries with temporal history preserved. The resulting records form the compressed memory \(\mathcal{M}\), followed by a final recency re-scoring step.

9 Retrieval and Question Answering↩︎

The evidence retrieval and answer generation algorithm is shown in Algorithm 14. Questions are decomposed into structured components, candidate Interaction Records preceding \(t_q\) are ranked using entity similarity, action similarity, and importance score , and answers are resolved either directly from retrieved records or through LLM reasoning over the top-5 retrieved records depending on the question type.

None

9.1 Answering Prompt↩︎

The retrieved evidence set \(R\) is passed to Qwen2.5-7B-Instruct using the prompt shown in Figure 16. The prompt provides the original question and its decomposed sub-questions, the inferred query intent, the four answer choices, and the ranked interaction memories retrieved from the compressed memory \(\mathcal{M}\). The model is instructed to select a single answer option (A–D) and provide a brief evidence-grounded explanation.

None

10 Additional Experiments↩︎

In this section, we provide additional analyses covering stage-wise ablations over seven days, the effect of context history size, retrieval score ablations, ablation on \(\mathrm{TopK}\) retrieval, EgoRAG’s [1] hierarchical summaries, and qualitative examples to further examine the behavior and robustness of the proposed memory framework.

10.1 Per-Category Performance on EgoLifeQA↩︎

Table 4 shows the per-category Grounding Accuracy (GA) and QA Accuracy results summarized in Figure 3 of the main paper across the five EgoLifeQA [1] question types. Imprint improves grounding in all categories, with the largest gains observed for EntityLog, EventRecall, RelationMap, and HabitInsight.

10pt

10.2 Stage-wise Ablation Over Seven Days↩︎

We track the stage-wise ablation as the recording horizon grows from Day 1 to Day 7 to verify that each compression stage contributes consistently and that the benefits do not arise from compression alone. Table 5 shows GA / Acc. per configuration for each cumulative day.

12pt

10.3 Ablation on Historical Context Parameters↩︎

Interaction records are contextualized using a short history of preceding interactions before prototype assignment and consolidation. The history window provides additional temporal context that helps distinguish semantically similar actions occurring in different situations. Results for different history lengths are reported in Table  6. Increasing the history size from 3 to 5 improves Grounded Answers, showing that more local interaction context helps retrieval. Increasing history size to 10, however, reduces Grounded Answers, suggesting that longer histories might introduce less relevant contextual information. We follow this trend and use k = 5 for all experiments. The results show that the local interaction context is useful for memory formation, but most of the retrieval-relevant information is contained within a relatively short temporal neighborhood.

5pt

10pt

10.4 Ablation on Retrieval score↩︎

To analyze the contribution of each term in Eq. 3 (Section 3.3), we assigned a weight of zero to that term and re-normalized the remaining weights and re-ran retrieval as shown in Table 8. Removing any component reduces grounding accuracy (GA) compared to the full score (64.8%), with the largest drop observed for action similarity (48.2%), followed by importance weighting (51.3%) and entity similarity (52.7%). This indicates that action cues are particularly important for distinguishing interactions involving similar objects, while entity similarity and importance score provide complementary retrieval signals.

10.5 Qualitative Analysis↩︎

10.5.0.1 Temporal disambiguation (Figure 17).

In case of contradictory evidence, caption-based retrieval retrieves both without preserving their temporal order. Imprint instead retrieves the relevant interaction records and orders them chronologically, resolving the apparent conflict cleanly: Tasha handed the knife (\(20{:}34{:}30\)) \(\rightarrow\) Jake started cutting (\(20{:}34{:}30\)) \(\rightarrow\) Jake placed the knife (\(21{:}03{:}30\)) \(\rightarrow\) Jake picked up the knife (\(21{:}03{:}30\)).

10.5.0.2 Grounded co-action retrieval (Figure 18).

For questions involving co-actions at a specific time, Imprint retrieves the grounding interaction evidence (e.g., Jake and Shure interacting with the whiteboard at the target timestamp) and answers directly from retrieved evidence. While EgoRAG [1] retrieves wrong evidence and produces the correct answer only through language priors rather than grounded retrieval.

10.5.0.3 Salient-object bias (Figure 19).

When retrieval returns broad caption spans containing multiple objects and activities, the language model often relies on the most prominent object mention rather than the interaction specified by the query. In this example, EgoRAG [1] retrieves evidence dominated by the activity “working on a laptop” and consequently predicts laptop, overlooking the temporal constraint of the interaction immediately preceding the last skipped lunch. In contrast, Imprint preserves the chronological structure of interactions and retrieves the relevant sequence of events: Jake used a laptop (\(\text{DAY2}~12{:}15{:}30\)) \(\rightarrow\) Jake drank coffee (\(\text{DAY2}~12{:}31{:}01\)) \(\rightarrow\) lunch skipped \(\rightarrow\) next eating event (\(\text{DAY3}~13{:}00{:}30\)). This temporal ordering correctly identifies coffee as the interaction immediately preceding the skipped meal.

10.6 EgoRAG’s Hierarchical Summaries↩︎

Figure 20 shows EgoRAG’s [1] hierarchical summary growing across the seven-day EgoLife recording. The number of summaries almost increases linearly with recording duration, and most of these are L1 (minute-level) summaries. This makes storage and retrieval increasingly demanding as recording duration increases.

10.7 Ablation on Top-K Retrieval↩︎

We analyze the effect of the number of retrieved evidence \(\mathrm{TopK}\) fed into the response LLM as shown in Table 9. GA is higher at \(\mathrm{TopK}=5\) (64.8%). Retrieving fewer evidence may miss essential supporting context, whereas retrieving more may introduce distracting information that weakens grounding and provides little benefit to QA accuracy.

10pt

References↩︎