1 Introduction↩︎

Foundation-model robot planners can translate language goals, scene observations, and retrieved demonstrations into action sequences. Yet long-horizon planning in familiar environments also depends on memory formed from accumulated embodied experience. Appropriate action may depend on where a participant tends to keep tools, how objects have changed state, which procedures have appeared before, and what regularities past behavior reveals. For an embodied agent, such experience is not merely extra context; it is part of what makes a future goal interpretable for a particular person and environment.

Human planning reflects this dependence on memory. When preparing a meal, cleaning a workspace, or arranging tools, people use memory formed through observation and repeated action. Research in cognitive science separates episodic traces of particular events from semantic regularities abstracted across experience, with consolidation turning repeated episodes into reusable knowledge [1]–[3]. For embodied agents, this suggests a planning-facing memory capability beyond larger context windows: event evidence, changing state, and recurring procedures need to be maintained as experience unfolds and reused when new goals are posed. We call this capability Embodied Action Memory (EAM).

Egocentric video is a natural source for EAM because it records experience from the actor’s perspective. Across sessions, it captures a participant moving through familiar spaces, manipulating objects, and repeating everyday procedures. Recent egocentric-to-robot efforts use such first-person human video to scale imitation, dexterous manipulation, and cross-embodiment robot learning [4]–[6]. MEMORA studies a complementary use of the same signal. Instead of converting egocentric video directly into a policy, action prior, or robot-compatible trajectory, it converts accumulated experience into persistent semantic and procedural memory that helps a planner identify which routine, object, preference, or prior state is relevant. The challenge is to transform a continuous stream into memory that remains useful after individual episodes have passed, while still preserving experience-grounded context and temporal change.

This motivates a lifecycle view of memory. Classical cognitive architectures, differentiable memory agents, and recent LLM-agent memories all distinguish memory formation, maintenance, retrieval, and consolidation in different forms [7]–[15]. For egocentric embodied experience, the key question is how experience becomes memory as it unfolds. A useful memory system has to encode the perception stream into structured observations, revise the persistent state as new evidence arrives, consolidate repeated episodes into reusable regularities, and retrieve the right mixture of procedural and grounding evidence for planning. This lifecycle also has to respect the different continuities present in embodied experience: places persist, objects change state while retaining identity, actions unfold in temporal order, and routines emerge only across events. These requirements are difficult to satisfy by storing clips, flattening observations into text, or maintaining a single static entity graph alone [16], [17]. MEMORA therefore centers a lifecycle view: embodied experience becomes useful for planning when the perception stream is encoded, edited, consolidated, and retrieved as typed memory rather than treated as a passive archive.

MEMORA implements this lifecycle with typed memory stores and active maintenance mechanisms (Figure 1). Its memory state maintains four typed stores: Environment Memory for spatial context, Entity Memory for object identity and state, Activity Memory for temporally ordered action evidence, and Inferred Knowledge for regularities consolidated across repeated experience. Online, a Memory Editor actively maintains entity memory through Add, Update, Delete, and Noop operations, preserving object identities and state histories as new observations arrive. Offline, consolidation abstracts routines, habits, and preferences; at inference time, typed retrieval exposes both procedural regularities and physical grounding to the planner.

We evaluate EAM along two axes: whether memory preserves past experience and whether it supports future action. MEMORA-Bench contains 45 hours of EPIC-KITCHENS-100 extension video across 18 participants [18], paired with MEMORA-Embodied Memory Assessment and MEMORA-Planning. Planning includes Replay tasks grounded in observed workflows and Generalize tasks requiring transfer, composition, or new goals. Across four open-weight language models, full MEMORA achieves the strongest aggregate results among the evaluated memory conditions. It improves memory-assessment accuracy by up to 20.5 points over the strongest controlled baseline and improves out-of-distribution Robot-Grounded Plan score by up to 16.6% relative. A qualitative two-task physical-robot deployment check shows the resulting language-level plans grounding remembered action choices for downstream execution. This places MEMORA on the planning side of the egocentric-to-robot pipeline: policy-learning methods can learn how actions are executed, while MEMORA supplies remembered context for deciding and grounding what to do.

Contributions. (1) A formulation of Embodied Action Memory (EAM) (§3) as a memory substrate for embodied reasoning and planning, with a taxonomy separating spatial, entity-state, temporal-action, and consolidated-regularity continuities. (2) MEMORA (§4), a framework that instantiates this dependency chain on real egocentric video through typed stores, active editing, offline consolidation, and typed retrieval. (3) MEMORA-Bench, a 45-hour evaluation suite pairing retrospective memory faithfulness with prospective memory-grounded planning. (4) A controlled empirical study isolating the contributions of online editing and offline consolidation and showing improvements in both memory assessment and planning, with the largest gains out of distribution.

2 Related Work↩︎

2.0.0.1 Foundation-model planning for embodied agents.

Language models and vision-language models now connect perception, reasoning, and action in embodied systems. Prompted planners, affordance-aware language models, code-generation approaches, and inner-monologue agents translate language goals and scene context into executable plans or skill calls [16], [19]–[21]. Vision–language–action models further learn policies that map multimodal observations and instructions directly to robot actions [22], [23]. MEMORA studies the memory layer that can condition such planners and policies: prior embodied experience is written, revised, consolidated, and queried when a later goal needs experience-grounded context.

2.0.0.2 Egocentric human video for robot learning.

Egocentric human video is a major data source for robot learning, including large-scale imitation, dexterous manipulation, vision–language–action pretraining, active-vision imitation, and methods that convert human demonstrations into robot-compatible observations or trajectories [4]–[6], [24]–[32]. These approaches typically use first-person video as demonstration data for learning policies, action representations, hand or head motion, active viewpoint control, or cross-embodiment alignment. MEMORA uses the same kind of signal for a different but compatible object of study: memory formation over the participant’s places, entities, actions, routines, and preferences. This makes egocentric policy learning and MEMORA naturally composable: one learns how actions can be executed on an embodiment, while the other supplies experience-grounded context for choosing and grounding what to do.

2.0.0.3 Embodied and spatial memory.

Several embodied agents maintain spatial, semantic, or 3D memory to support navigation, grounding, and long-horizon interaction. Recent systems build 3D scene memories, metric-aligned video memories, or multi-memory robotic agents for interactive environmental learning [33]–[35]. Such memories support localizing objects, grounding language in physical scenes, and acting over extended horizons. Recent robot-memory systems often accumulate an agent’s own task trials, scene observations, or interaction history for later reuse, whereas MEMORA forms typed memory from longitudinal egocentric human experience and evaluates its use for both retrospective memory questions and prospective planning. MEMORA differs from geometric scene mapping and visuomotor memory: its units are typed records of embodied experience, including spatial context, object identity and state, action traces, and consolidated regularities. The resulting memory can be queried by planners or policies as semantic-procedural context, leaving metric prediction, control, and action generation to the corresponding robot modules.

2.0.0.4 Memory formation in language agents.

Agent memory has been modeled as parametric storage in model weights [36] or as explicit stores retrieved as documents, embeddings, summaries, graphs, or edited records [17], [37]–[42]. Recent work further revisits the formation–consolidation–retrieval lifecycle for LLM agents, introducing explicit encoding and consolidation stages over text or simulated experience [12]–[15]. MEMORA brings this lifecycle to real egocentric video. It is revised online through write-time active editing and organized by embodied continuities instead of a single transcript, summary, or graph. This design draws on cognitive distinctions between episodic, semantic, and consolidative memory [1]–[3], [43], but instantiates them as typed stores used by a planner.

2.0.0.5 Benchmarks for embodied memory.

Long-video QA benchmarks evaluate comprehension of video supplied at query time [44]–[46], while conversational-memory benchmarks isolate memory in text-only interactions [47], [48]. Egocentric datasets provide rich embodied activity, but they do not evaluate whether an agent can form memory across sessions and later use it for planning without receiving the relevant video at test time [18], [49]. MEMORA-Bench targets this experience-to-memory setting: questions and planning goals are grounded in experience-dependent evidence remembered across sessions.

3 Problem Formulation: Embodied Action Memory↩︎

We formulate Embodied Action Memory (EAM) as a setting in which an embodied agent’s decision context evolves with experience. Later decisions depend on a memory state formed over time, together with any observations supplied when the decision is requested. Let \(o_t\) be the observation at time \(t\), \(\mathcal{M}_t\) the memory after processing observations through \(t\), and \(x_\tau\) a later language question, goal, or decision context. EAM captures this setting as \[\mathcal{M}_t = F(\mathcal{M}_{t-1}, o_t), \qquad d_\tau = G(\mathcal{M}_\tau, x_\tau),\] where the formation function \(F\) updates memory online and the decision function \(G\) acts on the memory available at time \(\tau\). Our benchmark enforces this setting through time-restricted memory questions and post-observation planning goals.

The decision \(d_\tau\) is language-level: a multiple-choice answer for memorization, or a sequence of grounded action–entity–location triples for planning. In our setting, memory is experience-dependent: decisions can use routines, preferences, action history, objects, and locations observed for that participant instead of a generic action script. EAM therefore treats \(F\) as a structured formation process, not an archival write operation. It preserves the spatial, entity-state, temporal-action, and consolidated-regularity continuities instantiated by MEMORA in §4.

4 Method↩︎

We present MEMORA as an implementation of the EAM formulation. §4.1 describes memory formation, where observations are encoded, edited into typed stores, and consolidated across repeated experience. §4.2 describes how the formed memory is exposed for retrieval and planning, and §4.3 defines the evaluation suite. The two central design choices are four typed stores matched to embodied continuities, and store-specific write/update rules that combine deterministic maintenance, LLM-based entity editing, and offline consolidation.

4.1 Memory Formation: Encoding, Editing, and Consolidation↩︎

MEMORA operationalizes EAM with four typed stores: Environment Memory, Entity Memory, Activity Memory, and Inferred Knowledge (Table 1).

The stores separate continuities that change at different rates: spatial context is relatively stable, object identity persists through state change, actions provide ordered event evidence, and routines or preferences emerge across repeated experience. Motivated by multi-store views of memory [7] and distinctions between episodic traces and semantic generalisations [1], [2], MEMORA maintains \(\mathcal{M}_t = (\textcolor{envGreen}{\mathcal{M}_{t}^{\mathrm{env}}},\, \textcolor{entBlue}{\mathcal{M}_{t}^{\mathrm{ent}}},\, \textcolor{actOrange}{\mathcal{M}_{t}^{\mathrm{act}}},\, \textcolor{infPurple}{\mathcal{M}_{}^{\mathrm{inf}}}{})\). The memory-store assignment determines the update rule for each observation: deterministic operations handle stable structure, while the Memory Editor performs semantic revision.

3pt

During online memory formation, the write path first encodes the current segment with working context, then revises the persistent memory state via Edit, which instantiates the formation function \(F\) from Section 3: \[\begin{align} (z_t, c_t) &= \phi(v_t, c_{t-1}), \\ \mathcal{M}_t &= \mathrm{\small Edit}(\mathcal{M}_{t-1}, z_t). \end{align}\] Here \(\phi\) is the Segment Encoder, a multimodal model that converts each 10 s video segment \(v_t\) into text-structured observations \(z_t = (\textcolor{envGreen}{z_{t}^{\mathrm{env}}}, \textcolor{entBlue}{z_{t}^{\mathrm{ent}}}, \textcolor{actOrange}{z_{t}^{\mathrm{act}}})\), and \(c_t\) carries a short working-memory summary across segment boundaries. Inside Edit, following recent active memory-editing systems [41], [42], the Memory Editor selects \(\delta_j \in \Delta = \{\mathrm{\small Add},\, \mathrm{\small Upd},\, \mathrm{\small Del},\, \mathrm{\small Noop}\}\) per object observation while preserving previous states in state_history; Environment Memory and Activity Memory are updated deterministically (merge by detected place; append by segment), and Inferred Knowledge is populated offline by consolidation [3] over repeated entity and activity evidence. This editable state lets later observations update object histories or delete inconsistent identifications, reducing the chance that early segment-level errors persist unchanged into downstream retrieval. JSON-style records, entity-edit diagnostics, and memory-store composition are in Appendix 8.

MEMORA therefore maintains an action-relevant memory state rather than an append-only observation log; Figure 2 quantifies a typical \(18\times\) reduction relative to the unedited observation stream on the 18-participant EPIC-KITCHENS-100 extension corpus (state histories preserved).

4.2 Memory-Guided Retrieval and Planning↩︎

Once memory has been formed, MEMORA is exercised through two interfaces. Retrieval is an evaluation interface: given a question \(q\) at time \(t_{\text{ask}}\), the agent recovers experience-grounded evidence from the available memory snapshot \(\widetilde{\mathcal{M}}_{t_{\text{ask}}}\) and selects an answer \(a\) only when the memory supports it. Planning is an application interface: given a goal \(g\), the agent turns the memory state into a grounded plan \(\pi = ((a_1, e_1, \ell_1), \ldots, (a_n, e_n, \ell_n))\), where each step specifies an action \(a_k\), an entity \(e_k\), and a location \(\ell_k\) drawn from memory.

The two interfaces share an iterative ReAct loop [50] whose primary read tool is a type-aware search: under the hood it dispatches to four typed indices — objects (Entity Memory), activities (Activity Memory), environment (Environment Memory), and patterns (Inferred Knowledge) — with an optional category argument when the agent already knows which store to query. Additional memory-read tools expose the structured access patterns enabled by typed memory: get_state_at_time and get_object_history support time-restricted reasoning by rolling entities back through their state_history and excluding Inferred Knowledge records that post-date \(t_{\text{ask}}\), so the agent cannot use evidence consolidated after the question time; get_routine_skill and get_preferences expose consolidated Inferred Knowledge for planning. The agent uses the same search tool across conditions, while MEMORA internally routes each query to the appropriate typed memory index.

At read time the four stores partition by role: Inferred Knowledge/Activity Memory supply procedural templates (“how to do it”), and Entity Memory/Environment Memory supply physical grounding (“which object, where, with what attributes”) — a partition that matters most when no observed routine matches the goal (Generalize). Tool signatures, prompts, iteration budgets, and an end-to-end trace are in Appendix 8.

4.3 MEMORA-Bench: An Evaluation Suite for Embodied Action Memory↩︎

Probing embodied action memory requires both faithfulness to past experience and support for future action over memory constructed from continuous embodied experience. This combination is absent from long-video QA [44], [45] and conversational-memory benchmarks [47], [48]. MEMORA-Bench therefore pairs two interfaces over the same experience-grounded memory state, formed from 45 hours of EPIC-KITCHENS-100 extension video across 18 participants. MEMORA-Embodied Memory Assessment (EAM-QA) probes retrospective faithfulness with multiple-choice questions over four EAM types (SPref: preferences; SHabit: habits; SRoutine: ordered procedures; ERecall: within-video recall), each item including an explicit information-not-available option to separate missing memory evidence from forced guessing. MEMORA-Planning probes prospective utility by producing grounded plans for experience-grounded requests, split into Replay (observed workflows) and Generalize (transfer, composition, or fully novel goals). The benchmark goals follow an experience-dependent view of embodied memory: they ask whether an agent can retain preferences, habits, routines, episodic action evidence, and the observed context needed for future action. We instantiate these goals from human-annotated EPIC action sequences, task templates, and verification rules; evaluated agents are defined separately by the memory interface exposed at test time, with MEMORA following an experiential consolidation design that separates episodic evidence, entity-state memory, spatial context, and consolidated routines or preferences. EAM-QA and Generalize references are annotation- or task-grounded; Replay metrics use matched participant routines, objects, and preferences when those axes are defined. Released benchmark items are static artifacts and the MEMORA pipeline itself runs entirely on open-weight models (Appendix 12.1.0.11), so reproducing the headline results does not require closed-weight LLM access. Construction and quality-control details are in Appendices [app:embodied-memory], 10.5.0.2, and 9; dataset, conditions, and metrics follow in §5.1.

5 Experiments↩︎

5.1 Experimental Setup↩︎

5.1.0.1 Evaluation logic.

The experiments isolate three parts of the MEMORA design. Online editing is tested by measuring whether it reduces noisy entity accumulation and improves memory use. Offline consolidation is tested by comparing full MEMORA with MEMORA-Episodic on cross-event memory and out-of-distribution planning. Typed memory is tested against flat text and entity-graph memories under the same planner backbones. The robot study then checks whether the generated language-level plans can ground two tabletop executions.

5.1.0.2 Data and protocols.

We evaluate on the 45-hour EPIC-KITCHENS-100 extension video subset used by MEMORA-Bench [18], covering all \(18\) participants for whom consolidated Inferred Knowledge memory banks are built (\(8\)–\(15\) sessions each), with tasks grounded in each participant’s observed action history. The subset contains \(201\) egocentric videos and yields \(5{,}653\) object-state entries with \(1{,}957\) unique object-name strings in the constructed memory artifacts, providing substantial variation in kitchen layouts, object instances, and recurring routines within the tabletop manipulation domain. MEMORA-Planning is the primary robotics-facing protocol, with \(207\) in-distribution Replay tasks and \(153\) out-of-distribution Generalize tasks; Replay is scored against matched memory-derived routine/object/preference references when defined, while Generalize uses verified task-level order references. Subsets are reported in Appendix 10.5.0.2. EAM-QA complements planning with \(N{=}2{,}212\) multiple-choice items (four content options A–D plus an explicit information-not-available option E) across the four EAM types defined in §4.3, testing whether the formed memory faithfully preserves the experience-grounded evidence that planning later relies on.

5.1.0.3 Conditions and metrics.

We compare a controlled seven-condition ladder: Parametric with no memory, Flat-1D chronological text memory [16] and Graph-2D entity-relation memory [17] in both raw and MEMORA-edited variants, MEMORA-Episodic without offline consolidation, and full MEMORA. All matched comparisons fix perception, retrieval/tool interface, and backbone LLM; task specifications come from the benchmark-construction pipeline, while agent variants differ only in the memory interface exposed at test time. Thus raw/edited Flat-1D and Graph-2D comparisons isolate write-time memory construction, while MEMORA-Episodic versus full MEMORA isolates offline consolidation under the same downstream agent loop. EAM-QA emphasizes experience-dependent memory questions missed by Parametric, where the answer is determined by observed action history rather than generic kitchen priors; complementary evaluations on the full benchmark and memory-grounded questions are in Appendix 9.5. Planning keeps Parametric as a real prior baseline and reports Robot-Grounded Plan score (RGP), the mean of OrderExec (step order), KeyObj (required objects), and PrefAdh (matched preferences); reference provenance and auxiliary planning metrics are in Appendices [app:planning-full] and 10.6. Unless noted, perception and editing use Qwen2.5-Omni-7B and Qwen3-30B-A3B-Instruct-2507; both interfaces use the same four open-weight answer backbones, with per-role model identifiers, vLLM serving, and context budgets in Appendix 13. Table 2 links the benchmark probes to planning use; Figures 3 and 4 show the main planning and robot-deployment results.

3pt

5.2 MEMORA-Planning Results↩︎

Planning tests whether formed memory can guide future action beyond retrospective question answering. Replay is scored against matched memory-derived routine, object, and preference references when defined; Generalize requires transfer or composition when no stored episode matches the goal and uses verified task-level order references.

5.2.0.1 MEMORA is strongest on Generalize tasks requiring transfer and composition.

A MEMORA condition achieves the highest RGP in each Replay and Generalize setting across the four answer backbones. The gain is consistently larger on Generalize than on Replay, indicating that structured memory is especially useful when the planner must compose remembered routines, objects, locations, and preferences for a goal not directly observed as a single prior workflow. This is the planning version of the behavior illustrated in Table 2: the agent first retrieves reusable procedural evidence, then grounds it in remembered entities and places. Figure 3 pairs a real Generalize trace with the Gemma-4-31B-it case: all non-MEMORA retrieval baselines fall below the no-memory prior, while MEMORA improves RGP by \(+16.4\%\). The same ranking pattern holds on Qwen3.6-35B-A3B; detailed results for each backbone and cross-backbone scaling are in Appendix [app:planning-full], Figure 18. Additional MEMORA-Planning examples generated from EPIC-KITCHENS-derived tasks are shown in Appendix [fig:planning-tool-trace].

5.2.0.2 Offline consolidation helps when planners can compose it.

Comparing MEMORA with MEMORA-Episodic isolates the contribution of offline consolidation. On Replay, MEMORA-Episodic matches or exceeds full MEMORA for several backbone settings because the relevant routine is already present as episodic evidence. On Generalize, full MEMORA helps the two strongest reasoning backbones, Qwen3.6-35B-A3B (\(+.025\)) and Gemma-4-31B-it (\(+.022\)), where reusable routines and preferences in Inferred Knowledge can be composed with entity and location evidence. Smaller backbones are neutral or negative under the same comparison, indicating that consolidated regularities help when the planner can combine them with episodic evidence. Thus the effect is not simply “more memory”: consolidation helps most when the answer model can use it as an intermediate abstraction for a new goal. On Gemma-4-31B-it, per-axis tables attribute the largest relative gains to OrderExec (recovering reference steps in order; \(+43\%\)) and PrefAdh (reflecting matched preferences; \(+14\%\)), while KeyObj coverage remains competitive with the parametric prior (Appendices [app:planning-full], 10.6).

5.2.0.3 Diagnostics identify where the planning gains come from.

The central mechanism is that MEMORA gives the planner separable evidence for procedure, preference, object state, and location, instead of one undifferentiated retrieved chunk. Tool logs reflect this division of labor: Flat-1D primarily issues keyword search queries, while MEMORA uses a typed cascade over routines, preferences, entities, and activities; Appendix [app:planning-full], Figure 19 gives the store-level breakdown. This helps explain why flat retrieval can help Replay yet hurt Generalize: it exposes evidence, but does not separate reusable procedure from physical grounding. Ablations further test whether these gains depend on memory construction. Reducing the Memory Editor from Qwen3-30B-A3B to Qwen3-14B costs \(5.8\) EAM-QA points and raises information-not-available selections from \(42.8\%\) to \(62.3\%\), suggesting that weaker write-time editing makes memory less usable, not merely smaller. Replacing Omni perception with VL-only perception costs \(2.7\) EAM-QA points, and removing Inferred Knowledge shifts \(35\%\) of planning calls toward episodic activity search rather than removing the need for consolidated routines; full ablations are in Appendix 12.1 and Table 15. Additional breakdowns, alternative grounding metrics, and worked examples are in Appendix [app:planning-full].

5.3 MEMORA-Embodied Memory Assessment Results↩︎

6pt

5.3.0.1 MEMORA improves both cross-video and single-video memory probes.

Table 3 reports per-type accuracy under the experience-dependent, memory-grounded evaluation subset; Appendix 9.5 reports complementary evaluations on the full benchmark and memory-grounded questions. MEMORA is best on all four question types and overall, leading the strongest controlled baseline, Flat-1D after online editing, by \(+20.5\) points on Gemma-4-31B-it. The lead holds on cross-video probes SPref, SHabit, and SRoutine, and on the single-video probe ERecall, indicating that the same memory state supports aggregation across sessions and retrieval within a session. The pattern follows the benchmark design in Table 2: repeated evidence benefits from consolidation into explicit regularities, while within-video recall benefits from edited object histories rather than duplicated observations.

5.3.0.2 Both online editing and offline consolidation contribute.

The raw and processed baseline rows in Table 3 separate the two effects. Online editing before graph construction lifts Graph-2D by \(+7.0\) points but Flat-1D by only \(+2.0\) points: chronological text cannot preserve typed state revisions and is vulnerable to noisy passages [51], [52]. Offline consolidation, measured as MEMORA minus MEMORA-Episodic, adds \(+14.4\) points on top of the multi-store online memory. The gain is concentrated on cross-event types, with at least \(+9.4\) points on each of SPref, SHabit, and SRoutine; the factorial store/editing ablation and a negative result on overly elaborate consolidation are in Appendix [app:embodied-memory]. The larger gains on these cross-event probes suggest that consolidation is most useful where single-episode lookup is insufficient and repeated observations have to be summarized into participant-level patterns.

5.3.0.3 The gains are consistent across backbones.

The Table 3 pattern replicates on the other three open-weight backbones: MEMORA leads the strongest controlled baseline for every backbone and question type, and the offline-consolidation lift remains positive throughout. Per-backbone tables, analyses of model capacity and memory-construction effects, and complementary evaluation views are in Appendix 9.6 and Appendix 9.5.

5.4 Qualitative Robot Deployment↩︎

MEMORA’s language-level plans are executed on a Unitree G1 with a fixed rule-based low-level controller. The tasks require experience-grounded choices: Prepare Drink selects Coca-Cola and the orange cup despite distractors, while Breakfast selects the yellow bowl, Cinnamon Tea, yellow spoon, and bread in the remembered order. Figure 4 shows the qualitative deployment check on two prepared tabletop tasks using memory-grounded plans. Full traces are in Appendix 11, Figures 21 and 22.

6 Discussion↩︎

6.0.0.1 Memory as a substrate for robot planning.

MEMORA treats memory as part of the planning interface. The core separation is procedural versus grounding evidence: Inferred Knowledge and Activity Memory provide routines and temporal traces, while Entity Memory and Environment Memory resolve the objects, states, and places needed for the current plan. This separation becomes most visible in Generalize tasks, where no single stored episode fully specifies the future goal.

6.0.0.2 Reading the evidence.

The controlled comparisons make the role of the memory lifecycle visible. Online editing changes the write path, reducing duplicated entity evidence while preserving state histories. Offline consolidation changes the memory state, improving cross-event questions and Generalize planning when the answer backbone can compose reusable regularities with episodic evidence. Typed retrieval changes the read path, giving the planner distinct procedural and grounding evidence instead of one undifferentiated context.

6.0.0.3 Relation to robot policies.

Modern robot policies increasingly combine language goals, visual observations, and learned action priors. This includes VLA policies that map instructions and images to actions, world-action models (WAMs) that reason over action-conditioned futures, and lower-level controllers that execute motions in a particular embodiment. MEMORA addresses the memory side of this stack: long-horizon context accumulated through prior embodied experience. It records what tends to remain stable, what has changed, which procedures have been observed, and which regularities are specific to the participant. Such semantic-procedural memory can guide policy-level action selection, while VLA policies, WAMs, dynamics models, or controllers handle embodiment-specific execution. In this sense, egocentric video can support two complementary roles: learning action-capable robot policies and forming the memory those policies or planners can consult before acting.

6.0.0.4 Why egocentric experience is the source.

Egocentric video gives MEMORA a first-person record of how a participant actually uses a space, beyond how a generic task could be performed. This matters for planning because many goals in familiar spaces are under-specified by the instruction alone; in our tabletop examples, prepare drink depends on the remembered cup and beverage, and breakfast depends on remembered object choices and ordering. The benchmark and robot demonstration use this property in different ways. MEMORA-Bench measures whether memory formed from many sessions improves retrieval and planning at scale, while the robot demonstration illustrates the same kind of participant grounding in two physical tabletop tasks.

6.0.0.5 Why both benchmark protocols are needed.

Together, EAM-QA and MEMORA-Planning test whether formed memory is both faithful to past experience and useful for future goals: EAM-QA probes remembered evidence, while planning stresses prospective composition across remembered routines, entities, and locations. Within planning, Replay checks recovery of observed workflows, while Generalize asks whether typed memory exposes reusable structure for new goals.

7 Conclusion↩︎

We introduce MEMORA, a typed memory framework that turns egocentric experience into editable, consolidated memory for planning, and MEMORA-Bench, a 45-hour evaluation suite for memory faithfulness and memory-grounded planning. Across four open-weight backbones, MEMORA outperforms flat-text and entity-graph memories, with the largest planning gain on out-of-distribution Generalize tasks: up to \(+16.6\%\) relative RGP on Qwen3.6-35B-A3B. A qualitative two-task robot deployment check uses the same memory to generate remembered-context language plans for downstream execution. Together, the results show that how embodied experience is formed, edited, consolidated, and queried matters when future goals require more than replaying a matching episode. This memory layer can complement egocentric imitation learning and vision–language–action systems: those methods learn executable behavior from human video, while MEMORA organizes the same kind of experience into experience-grounded context, routine cues, and procedure choices for planning.

Limitations↩︎

7.0.0.1 Domain and horizon.

The current benchmark is built from kitchen activity in the EPIC-KITCHENS-100 extension videos [18], with at most \(15\) sessions per participant. This is long enough to expose recurring places, objects, and routines, but not enough to study months-long drift, seasonal reorganization, or rare events. Extending EAM beyond this kitchen-centered benchmark may require policies for capacity, forgetting, and deciding when repeated observations revise rather than reinforce a regularity.

7.0.0.2 Pipeline dependence.

MEMORA depends on the quality of perception and write-time editing. The ablations in Table 15 show that weaker perception models or weaker memory editors reduce memory-assessment performance. Because memory is formed from perceived segments, missed objects cannot be recovered unless they appear in later evidence, and repeated perception errors may enter consolidated knowledge if they are not contradicted by subsequent observations.

7.0.0.3 Robot deployment and privacy.

The Unitree G1 experiment studies language-level memory grounding in a prepared tabletop scene with a fixed controller. Future work can connect MEMORA to full closed-loop robot autonomy, including navigation, deformable-object manipulation, online perception updates during execution, and recovery from failed grasps. Because persistent embodied memory is participant-specific, practical deployment also requires consent, inspection and deletion tools, access control, and clear limits on memory sharing.

7.0.0.4 Appendix roadmap.

The appendix is organized around construction, evaluation, and controlled analysis. Appendix 8 specifies the memory-bank construction pipeline from egocentric video. Appendix [app:embodied-memory] defines the retrospective Embodied Memory assessment and its temporal-control protocol. Appendix 10 specifies the MEMORA-Planning benchmark and rule-based evaluation panel. Appendix 11 documents the physical robot demonstration used as a qualitative deployment check. Appendix 12 defines controlled ablations over processor and answer-agent choices. Appendix 13 consolidates the model registry, inference stack, perception preprocessing, and retrieval/agent-loop settings shared across all of the above.

8 Memory-Bank Construction Pipeline↩︎

Appendix 8 specifies how MEMORA constructs the typed memory bank used by the Embodied Memory and Planning evaluations. The pipeline maps EPIC-KITCHENS-100 extension videos into the four stores defined in Section 4: Environment Memory, Entity Memory, Activity Memory, and offline Inferred Knowledge. The presentation follows processor roles rather than engineering stage names.

8.0.0.1 Terminology (Inferred Knowledge and MEMORA-Episodic).

In the main paper, cross-episode routines, habits, and preferences are typed as Inferred Knowledge and populated by offline consolidation (Table 1). This appendix decomposes that offline path into two processors: an Offline Consolidation Processor (per-video storage preferences, organizational habits, workflow patterns, and repeated action sequences) and an Inferred Knowledge Enrichment Processor (participant-level routine-skill indices, generated preferences, and retrieval text). Together they form the consolidated Inferred Knowledge bank used at evaluation time. Full MEMORA loads the enriched participant memory bank: EAM-QA reaches consolidation outputs through search_patterns and, under full MEMORA, also indexes enriched routine skills and generated preferences in the same unified retrieval interface; Planning additionally advertises dedicated get_routine_skill and get_preferences entry points over the enriched consolidated Inferred Knowledge slice. The label MEMORA-Episodic matches the main-paper ablation on online Environment Memory/Entity Memory/Activity Memory: EAM-QA evaluates the per-video memory bank before enrichment through the standard layered search tools; Planning evaluates the same episodic stores through search_objects, get_object_history, and search_activities.

8.1 Implementation Overview↩︎

Each video is divided into non-overlapping 10-second turns. This appendix uses implementation names for the two main method components: the Perception Processor implements the Segment Encoder \(\phi\), and the Online Memory Processor implements the write-time Edit update. For turn \(t\), the Perception Processor emits a layered observation \(z_t=(z_t^{\mathrm{env}},z_t^{\mathrm{act}},z_t^{\mathrm{ent}})\). The Online Memory Processor then updates a persistent memory state \(\mathcal{M}_t=(\mathcal{M}_t^{\mathrm{env}},\mathcal{M}_t^{\mathrm{act}},\mathcal{M}_t^{\mathrm{ent}},\mathcal{M}^{\mathrm{inf}})\) in timestamp order: \[z_t = \phi(v_t, c_{t-1}), \qquad \mathcal{M}_t = \mathrm{\small Edit}(\mathcal{M}_{t-1}, z_t).\] In this notation, \(v_t\) is the 10-second clip and \(c_{t-1}\) is a bounded textual summary of earlier turns in the same video. Previous video frames are not retained in the model KV cache across turns.

4pt

Each participant memory bank comprises four typed stores: turn-local perception records, per-video environment and activity logs, an edited entity registry with operation history, and a participant-level consolidated Inferred Knowledge slice.

8.2 Memory Update Semantics↩︎

The Perception Processor produces turn-local observations only; it does not reconcile duplicate entities or maintain long-horizon state. Cross-turn memory formation is handled by the Online Memory Processor. Activity memory is append-only because it represents the event stream. Environment memory is merged by stable location identifiers and retains a history of updates. Entity memory is edited by the online memory processor: for each incoming object observation, the model emits one of Add, Update, Delete, or Noop against the compact current registry. When the entity registry is too large for the prompt budget, dense retrieval (Appendix 13) selects the most relevant existing entities for the edit decision.

The Offline Consolidation Processor reads completed memories for a participant and writes per-video consolidation summaries (storage preferences, habits, workflow patterns, and action sequences). These summaries are useful for preference, habit, and routine reasoning, but they are withheld for retrospective QA whenever they would expose evidence after the query anchor. The Inferred Knowledge Enrichment Processor subsequently derives planning-oriented memory from the participant slice. It creates routine-skill structures and retrieval text deterministically, then uses the main-paper enrichment LLM in Table ¿tbl:tab:impl-models? to produce evidence-grounded generated preferences; a closed-weight alternative is reported as a sanity check in Appendix 12.1.0.11.

8.3 Editor Activity and Memory Compression↩︎

This subsection documents the data behind Figure 2 in Method §4.1. The 18-participant figure traces the cumulative size of Entity Memory under two write policies on the same main-paper perception stream and online memory processor (Table ¿tbl:tab:impl-models?): a hypothetical without-editing policy that would persist one record per entity observation (every state_history event the perception layer emits with a turn-aligned timestamp), and the MEMORA policy that emits one record per unique entity and folds repeat encounters into its state_history list. The compression ratio reported per participant is the ratio of these two record counts at end-of-session; segment-level cumulative curves use turn-aligned events only (events without a resolved turn_id are dropped from both numerator and denominator so they cannot inflate either side). Panel (b)’s editor decision distribution is read off the released editor operation logs; these decision counts are fully available for the P01–P04 calibration cohort and partially available for the 18-participant evaluation cohort, so the distribution panel is reported on P01–P04 alone while panels (a) and (c) use the full 18-participant cohort.

8.3.0.1 Per-participant size and compression.

Figure 2, panels (a) and (c), report the per-participant figures. Every participant achieves at least a \(9\times\) reduction; the typical (median) participant achieves \(18.1\times\), and three participants exceed \(30\times\). PIDs with longer sessions and higher entity reuse (P12, P23, P25, P35) reach the largest ratios because the editor turns repeated encounters with the same kitchen objects into state updates rather than new records; participants with shorter sessions (P09) sit at the floor of the range.

8.3.0.2 Editor decision distribution (P01–P04).

The calibration-cohort editor logs (P01–P04) report the editor’s decisions over \(n{=}37{,}648\) object observations: Add \(=2{,}995\) (8.0%), Update \(=16{,}718\) (44.4%), Delete \(=691\) (1.8%), Noop \(=17{,}244\) (45.8%). Two readings are worth recording. First, Noop \(+\) Update together account for \(90.2\%\) of decisions: most of the editor’s work is recognizing that an entity is already in memory and either revising its state or letting it stand, not adding new records. Second, Delete is rare (1.8%) but never zero — the editor does retract identifications that subsequent observations disconfirm, which a frozen, append-only memory could not. This decision profile is the per-event mechanism behind the \(18.1\times\) aggregate compression in Figure 2: every Noop on a previously-seen object is one entity observation that does not become a new record, and every Update is one observation that updates an existing record rather than creating one.

8.4 Processor Prompt Templates↩︎

The construction pipeline in Figure 6 uses fixed prompt templates with runtime variables filled from the current segment, current memory state, or participant memory slice. Figure 7 summarizes the four prompted processors, and Figure 8 gives compact records for the four MEMORA memory types. Deterministic rule components in the Inferred Knowledge Enrichment Processor have no prompt.

8.5 Memory Type Examples↩︎

Figure 8 provides abbreviated records for the four memory types used by MEMORA: Environment Memory, Entity Memory, Activity Memory, and Inferred Knowledge. Field names are preserved from the released records; values are shortened to make the representation readable in print.

8.6 Memory Stores and Leakage Control↩︎

8.6.0.1 Leakage control.

In retrospective Embodied Memory, leakage control means the agent may retrieve only evidence available strictly before the question anchor \(t_{\mathrm{ask}}\): activity and environment entries are time-filtered, entity states are rolled back through state_history, and offline Inferred Knowledge summaries are withheld whenever they would expose post-anchor observations. In Planning, scope control restricts both benchmark tasks and retrieval to videos already present in the participant’s consolidated memory slice, so the agent is not evaluated on episodes it never encoded. Table 5 summarizes these controls for each memory store.

4pt

8.7 Limitations↩︎

The memory bank inherits errors from the perception stream: missed objects, weak audio cues, and ambiguous hand-object interactions can propagate into later processors. The Online Memory Processor can repair cross-turn state, but visually similar objects may still be duplicated or stale attributes may persist when evidence is weak. Offline consolidation improves participant-level summaries but must be disabled or sliced for retrospective questions to prevent temporal leakage. Finally, consolidated Inferred Knowledge is optimized for planning and preference retrieval; it should not be interpreted as an exhaustive transcript of every observed event.

9 MEMORA-Embodied Memory Assessment↩︎

Appendix [app:embodied-memory] defines the retrospective QA arm of MEMORA-Bench. The MEMORA-Embodied Memory Assessment measures whether experience-derived memory is faithful, accessible, and temporally controlled. The annotation-grounded release supports the headline four-way split (ERecall, SPref, SHabit, SRoutine) on 45 hours of EPIC-KITCHENS-100 [18] extension video across 18 participants. Generation is architecture-agnostic: questions are mined from action annotations and do not presuppose MEMORA’s four-substrate layout. §9.1 specifies benchmark construction and question grounding; §9.4 specifies the evaluation protocol. The EAM-QA setup evaluates retrieval from the constructed memory bank rather than generic video question answering.

9.1 Benchmark Construction↩︎

9.1.0.1 EPIC-KITCHENS inputs.

The construction builds on EPIC-KITCHENS-100 [18], an egocentric kitchen corpus in which each participant records multiple videos in the same home over time. The public release supplies:

• RGB video for every session (not used in generation; required at evaluation by the perception front-end);

• Dense action annotations—one row per atomic action, time-aligned to the video (used in generation);

• Participant and session identifiers (participant_id, video_id) so longitudinal behavior can be grouped (used in generation);

• Timestamps for when each action occurs (narration_timestamp, plus frame- and clock-based start/stop fields in the official CSV; used in generation, with start/stop fields optional);

• Structured labels (verb, noun, and their taxonomy classes) and a free-text narration describing the action in context (used in generation).

The Embodied Memory generator consumes only the annotation table (not pixels): it needs a time-ordered log of who did what, when, and in which video.

The release is further restricted to an EPIC-KITCHENS-100 extension video whitelist (45 hours across 18 participants), so every item remains within the same longitudinal setting.

9.2 Question Types and Grounding↩︎

9.2.0.1 Design.

Each item is experience-dependent: it concerns one person’s observed kitchen behavior across longitudinal sessions in the same home, not generic cooking knowledge, so the benchmark tests memory formed from lived action history rather than generic cooking priors. Released questions require evidence-grounded stems and keys: every item must cite observable actions (timestamps, video_ids, verb–noun labels) drawn from EPIC narrations, so items remain auditable and cannot be answered from world knowledge alone. Because LLM generation is noisy and many candidates fail rule checks, we overgenerate then curate (target \(\times 3\) per type before verification) to hit per-type quotas without hand-writing questions; the surplus absorbs parse failures, weak evidence, and judge rejections. The pipeline applies a cross-model judge (generator gpt-5.5 vs.verifier gpt-4o-mini) so acceptance is not decided by the same model that proposed the item. Every released item must pass mandatory verifiability prompts at generation time plus post-hoc quality verification (semantic deduplication, per-item verification, and answer re-derivation).

3pt

9.2.0.2 Scope of this appendix.

This appendix specifies the annotation-grounded pipeline that produces the four reported EAM-QA types in Table 6. For each participant, we aggregate in-scope narrations, summarize them into compact statistics and per-video digests, prompt a generator LLM for structured multiple-choice items, and curate with a separate verifier model. Two earlier pixel-grounded types (object localization ELoc, temporal ordering EOrder) are excluded from the released benchmark because parametric and perception-heavy baselines already saturate those probes.

9.2.0.3 Construction steps.

For each participant, the pipeline applies the following steps; each step is selected to keep generation grounded while fitting in context windows.

(i) Index. We aggregate verb/noun marginals, verb–object pairs, consecutive verb bigrams, and trigger\(\rightarrow\)next histograms (within-video only). Marginals surface stable tool/object preferences for SPref; trigger–response tables directly support SHabit (“after \(X\), what next?”); verb bigrams expose coarse workflow regularities for SRoutine without sending the full action log to the LLM. These statistics are a lossy but faithful summary: they highlight recurring structure the generator should mine, while staying far smaller than raw narrations.

(ii) Summarize. Each in-scope video is compressed into a short Goal / key-sequence / object digest so cross-video types see thematic context (e.g., “checks pan by stirring”) without loading thousands of raw rows.

(iii) Single-hop generate. Per-type prompts combine statistics, a sparse action sample, video digests, and shared instructions requiring timestamp and video_id citations, balanced distractors, and guards against tautological or commonsense-only items. SPref, SHabit, and SRoutine use a lower decoding temperature than multi-hop variants to limit hallucinated counts while preserving phrasing diversity.

(iv) Multi-hop variants (within each type). Multi-hop is not a fifth question family: every headline type also receives 2-hop and 3-hop items. The pipeline first extracts 4–6 step action chains within a video, then prompts the generator to select a 2- or 3-step subchain and phrase a question whose answer depends on that subchain only. Deterministic chain extraction guarantees the hops exist in the log; the model selects and words the probe.

(v) Post-process. Rule deduplication removes exact duplicates; keyword topic deduplication is disabled in the reported release because it was too aggressive (\({\sim}22\%\) loss). LLM curation and contradiction resolution on SPref/SHabit drop near-duplicate or conflicting claims. A per-participant cap (50% per type) keeps the release balanced across cognitive classes. Long runs can resume from intermediate save points after interruption.

9.2.0.4 Per-type inputs, flow, and prompts.

Figure 10 and Table 6 summarize the pipeline and per-type inputs and evidence thresholds for each headline type. Every generation prompt appends the same three instruction blocks: verifiability (mandatory timestamps, video_ids, verb–noun citations), answer format (balanced distractors, varied correct-letter positions), and anti-failure guards (tautology, triviality, hallucination, stem leakage). Figure 11 gives the auditable prompt skeletons.

9.3 Quality Control and Diagnostics↩︎

9.3.0.1 Summarization and multi-hop wording.

The summarization prompt requires each video digest to list a one-sentence goal, 2–5 titled arrow sequences, and key objects/tools. For multi-hop items in SPref, SHabit, SRoutine, and ERecall, the generator receives a timestamped within-video chain and must select a 2- or 3-step subchain before writing the question; released items are tagged with reasoning_depth\({\in}\{2,3\}\) but keep the parent type’s schema.

9.3.0.2 Automated verification before release.

Every candidate passes three verification stages, each targeting a different failure mode: (1) Semantic dedup (embedder in Table 7; cosine \(\geq 0.95\)) groups near-duplicate stems before expensive judging; an LLM picks which cluster members to keep so we remove redundancy without collapsing legitimately distinct probes. (2) Quality verifier (\(T{=}0\)): a type-aware yes/no prompt rejects unclear stems, weak distractors, or answers not supported by cited evidence; a soft retry at \(T{=}0.3\) rescues borderline items if the first pass is overly strict (\({<}30\%\) pass rate). (3) Answer re-derivation: the judge sees the question, choices, and evidence but not the stored label and must independently recover the same letter—catching cases where the generator assigned a key inconsistent with its own evidence. Rule-based gates at generation time enforce minimum support (e.g., \(\geq 2\) videos for cross-video types, minimum trigger counts for SHabit) so obviously weak candidates never reach the judge. On a reference run for P06, 173 candidates survived in-generator steps and 129 were released after verification (27 removed by the quality judge alone). Representative numeric thresholds are listed in Table 7.

9.3.0.3 OpenAI API configuration.

The reported annotation-grounded release is produced through the OpenAI Chat Completions API (https://api.openai.com/v1). Each request is a single-turn user message (no system role, no dialogue history). We use gpt-5.5 for generation and gpt-4o-mini for verification so acceptance is not self-judged. Structured outputs use JSON mode when the prompt requests JSON; the verifier uses low-temperature decoding (\(T{=}0\), top_p\(=0.9\)). The generator is a reasoning-style model: the API ignores custom temperature and instead allocates a larger completion budget so hidden reasoning tokens do not crowd out the visible answer. Operational settings are 3 concurrent requests, 240 s timeout, up to 3 retries with exponential backoff, and a 128k-token context budget with prompt truncation when needed.

3pt

Embedding dedup uses the embedder in Table 7 before any verifier API calls.

9.3.0.4 Reproducibility.

We generate one bundle per participant on the EPIC-KITCHENS-100 extension video whitelist; per-type targets are 8 single-hop and 8 multi-hop items per semantic type (SPref, SHabit, SRoutine, before \(\times 3\) overproduction) and one ERecall item per in-scope video. Full launch parameters and record schemas are recorded in the MEMORA-Bench reproduction appendix. The EAM-QA release is automatically generated and verified rather than separately human-validated. Its reproducibility comes from fixed prompts, schemas, thresholds, and annotation-grounded evidence requirements; its quality control comes from cross-model verification, answer re-derivation, and released audit metadata. This design supports scalable extension to new annotated corpora, while manual human validation remains complementary future quality control.

9.4 Evaluation Protocol↩︎

9.4.0.1 Task and metrics.

At query time \((q, t_{\mathrm{ask}})\) the agent must choose one of five options \(\{\mathrm{A},\ldots,\mathrm{E}\}\). Options A–D are contentful answers; E denotes “information not available,” so a model must recognize when the memory state lacks sufficient experience-grounded evidence rather than guess among four plausible kitchen actions—otherwise parametric priors inflate accuracy on underspecified probes. Accuracy is reported per type and macro-aggregated over participants; the Pool column pools all headline items with count weighting. Each question carries an ask_turn_id (or equivalent temporal anchor) so retrieval uses only observations strictly before \(t_{\mathrm{ask}}\); this matches embodied deployment where the agent cannot read the future and isolates memory quality from leakage. See the main method for the time-restricted snapshot \(\widetilde{\mathcal{M}}_{t_{\mathrm{ask}}}\).

9.4.0.2 Controlled comparison.

All memory conditions share the same perception front-end, retrieval stack, and answer-agent backbone within a column (Appendix 13), so differences are not confounded by vision or embedding capacity. The only deliberate variable is the memory substrate exposed to a ReAct agent [50]: Parametric (no external memory), Flat-1D [16] (chronological text memory exposed via a flat search tool), Graph-2D [17] (entity-relation graph memory), or MEMORA (typed substrates with consolidation variants). Accuracy gaps are therefore attributed to how experience is organized and retrieved, not to unequal perception or prompting budgets.

9.4.0.3 Agent loop.

The answering agent follows the shared Embodied Memory ReAct budget in Appendix 13 (up to five search calls, then one letter). For Embodied Memory evaluation, the time-restricted snapshot \(\widetilde{\mathcal{M}}_{t_{\mathrm{ask}}}\) is applied: entity states roll back to \(t_{\mathrm{ask}}\) and offline consolidation summaries are withheld so answers must come from observations strictly preceding the question anchor, not sleep-time rules that would trivialize SPref/SHabit items. Tool schemas and per-condition system prompts are fixed within each block; Graph2D receives schema-equivalent per-type prompts so routing verbosity does not confound substrate comparisons.

9.4.0.4 Reporting diagnostics.

A Parametric Memory column is always reported alongside structured memory: types where Parametric already exceeds \({\sim}70\%\) are flagged as prior-saturated in footnotes rather than excluded post hoc. The rate of E selections is tracked separately because flat retrieval can overproduce information-not-available answers on some commercial backbones.

9.4.0.5 Illustrative items.

Figure 12 provides one released record per headline type from participant P06. Fields reasoning and full evidence chains are audit metadata used during construction and quality verification; the answering agent receives only the question, four content options, and (at evaluation time) option E appended to the choice list.

9.5 Experience-Dependent Memory Evaluation Views↩︎

The Embodied Memory benchmark mixes answerable questions (ground truth \(\in\{\mathrm{A,B,C,D}\}\)) with unanswerable probes (ground truth \(\mathrm{E}\)), the latter required so a model cannot inflate accuracy by guessing on underspecified items. Our headline view focuses on experience-dependent memory: questions missed by the parametric (no-memory) baseline, where the answer should be determined by observed action history rather than generic kitchen priors. Within that view, we report memory-grounded answerability, the operating point where MEMORA can ground a contentful answer in retrieved memory evidence (selects one of \(\mathrm{A}\)–\(\mathrm{D}\) rather than \(\mathrm{E}\)). All compared conditions are evaluated on the same question subset so the comparison is matched. This follows two memory-evaluation principles:

(i) the experience-dependent view isolates questions whose answer should come from the observed history of a particular person, rather than from a closed-book prior or generic kitchen knowledge [54]–[58];

(ii) memory-grounded answerability asks whether the formed memory contains enough retrievable experiential evidence to support a concrete answer, rather than rewarding guesses when the memory state is incomplete [59], [60].

This is a memory-centric diagnostic: it separates two failure modes that are otherwise conflated, namely whether the memory state surfaces relevant evidence at all, and whether the answer agent can discriminate the correct option once such evidence is available. Table 8 reports MEMORA’s overall accuracy and lead over the strongest controlled baseline under three views: the full benchmark, memory-grounded answerability, and the experience-dependent answerability headline view. The Parametric (no-memory) baseline is \(0\%\) in the experience-dependent view by construction and is omitted from the per-backbone tables; its full-benchmark accuracy is \(\sim 20\%\) on all four backbones (random guess on \(5\)-option MC).

9.5.0.1 Memory-grounded answerability rate and precision.

To check that absolute-accuracy differences across backbones reflect answer-agent capacity rather than pipeline failure, we decompose MEMORA’s accuracy within the experience-dependent view into a memory-grounded answerability rate \(\mathrm{P}(\text{MEMORA picks A--D} \mid \text{parametric wrong})\) and a memory-grounded precision \(\mathrm{P}(\text{correct} \mid \text{A--D}, \text{parametric wrong})\); the latter equals the headline experience-dependent, memory-grounded accuracy by construction. Memory-grounded precision increases monotonically with backbone capacity — Gemma-4-26B-A4B \(54.1\%\), Qwen3.6-27B \(69.1\%\), Gemma-4-31B \(74.5\%\) — whereas the answerability rate varies non-monotonically (Gemma-4-26B-A4B aggressive at \(67.5\%\), Qwen3.6-27B also high at \(70.3\%\), Gemma-4-31B conservative at \(50.7\%\)) and stays \(\geq 50\%\) everywhere, indicating that every backbone engages with the retrieved memory; what differs across backbones is how well they discriminate among grounded options once they answer. We read this as a capacity-bound rather than pipeline-bound cross-backbone pattern: absolute differences across backbones reflect the answer-agent’s discriminative capacity once it has sufficient evidence to answer, not a failure of the memory pipeline that feeds it.

3.6pt

Subset sizes per backbone (\(n_{\text{full}} / n_{\text{evidence}} / n_{\text{headline}}\)): Q3.6-35B-A3B \(1{,}202 / 819 / 721\); G4-26B-A4B \(2{,}763 / 1{,}770 / 1{,}497\); Q3.6-27B \(2{,}763 / 1{,}753 / 1{,}556\); G4-31B \(2{,}763 / 1{,}234 / 1{,}122\). \(\Delta\)best \(=\) MEMORA \(-\) best of {Socratic raw/processed, Graph-2D raw/processed} on the same subset. ^\(\sharp\) Q3.6-35B-A3B panel is reported on the \(8\)-PID intersection (\(n_{\text{full}}{=}1{,}202\)) for which this diagnostic was first computed; the headline row agrees with the \(12\)-PID Qwen3.6-35B-A3B intersection in Table [tbl:tab:mc5-per-backbone-raw] (MEMORA \(63.5\%\), \(+11.8\) points over best controlled baseline) within \(\pm 0.8\) points.

9.6 Per-backbone \(\times\) per-type Results↩︎

The main paper headlines a \(2{\times}2\) open-source cross-backbone design (Qwen 3.6 / Gemma 4 \(\times\) MoE / dense; Table 3 displays the Gemma-4-31B-it column on the full \(18\)-PID panel). This subsection unpacks the remaining three backbones — Qwen3.6-35B-A3B, Qwen3.6-27B, and Gemma-4-26B-A4B-it — under the unified experience-dependent, memory-grounded EAM-QA view of Appendix 9.5 (Table ¿tbl:tab:mc5-per-backbone-raw?; the per-backbone \(\Delta_{\text{consol}}\) rows summarise the per-type lift).

9.6.0.1 Cross-backbone consistency of the consolidation lift.

Under the unified experience-dependent, memory-grounded EAM-QA view (Table ¿tbl:tab:mc5-per-backbone-raw?, \(\Delta_{\text{consol}}\) rows), the consolidation lift \(\Delta_{\text{consol}} =\) MEMORA \(-\) MEMORA-Episodic is positive on every question type and every backbone. The SRoutine lift is the most robust as expected for the aggregation-requiring type: \(+4.9\) on Qwen3.6-35B-A3B, \(+9.9\) on Gemma-4-26B-A4B, \(+10.7\) on Qwen3.6-27B, and \(+24.6\) points on Gemma-4-31B. SHabit, the other cross-event aggregation type, lifts by \(+6.1\) to \(+13.7\) points across the four backbones; the single-fact retrieval types lift by \(+7.0\) to \(+11.3\) points (SPref) and \(+4.7\) to \(+10.5\) points (ERecall). The four-backbone means are \(+12.5\)/\(+10.1\)/\(+9.2\)/\(+7.4\) points on SRoutine/SHabit/SPref/ERecall, with every per-backbone, per-type cell positive, consistent with consolidation strengthening retrieval whenever the question has enough evidence for a contentful answer.

3.6pt

Bold: best per column within each backbone block. \(\Delta_{\text{consol}}\) values are bolded when they exceed the \(\pm 4\)–\(5\) points per-type binomial \(95\%\) CI. ^\(\flat\) Experience-dependent, memory-grounded view (Appendix [sec:app:selective-eval]): Gemma-4-31B-it on the full \(18\)-PID EAM-QA panel (\(N{=}1{,}122\); mirrors Table [tbl:tab:memorization-main]); Qwen3.6-35B-A3B on the \(12\)-PID intersection (\(N{=}1{,}141\)); Qwen3.6-27B on the full \(18\)-PID panel (\(N{=}1{,}556\)); Gemma-4-26B-A4B-it on the full \(18\)-PID panel (\(N{=}1{,}497\)). The Qwen3.6-35B-A3B row is reported on the \(12\)-PID intersection because that is the largest subset on which all seven memory conditions are currently materialized; the full \(18\)-PID Qwen3.6-35B-A3B panel will be released alongside the code and benchmark, and the \(8\)-PID evaluation-view panel reported in the \(^{\sharp}\) footnote of Table [tbl:tab:filter-sensitivity] indicates that MEMORA’s accuracy and its \(+11.8\)-point lead over the strongest controlled baseline on Qwen3.6-35B-A3B are preserved within \(\pm 0.8\) points across the \(8\)-PID and \(12\)-PID subsets.

9.6.0.2 Scaling to larger corpora.

The pipeline in Figure 10 is participant-agnostic: adding more benchmark volume means repeating steps 1–7 for additional people or sessions, not redesigning the question types. To extend beyond the current EPIC-KITCHENS-100 extension subset, a new corpus must supply at minimum:

• Participant identifiers that are stable across sessions (so cross-video SPref, SHabit, and SRoutine items remain meaningful);

• Video/session identifiers mapping each file to exactly one participant;

• Time-ordered action annotations per video: timestamps plus verb–object (or equivalent) labels, and preferably short natural-language descriptions for audit trails;

• A defined video whitelist (or split policy) stating which sessions enter generation;

• For evaluation on new footage: synchronized RGB video and the same perception/retrieval stack as in the main experiments (generation can remain annotation-only).

Optional enrichments—frame-level bounding boxes, object tracks, or a parallel VLM-augmented track—are not required for the headline annotation-grounded release but would enable spatial types (ELoc) or pixel-grounded verification at larger scale. As annotation density and participant count grow, the same overgenerate-and-curate recipe applies; cost scales with LLM verifier calls per candidate, which is why semantic dedup and rule-based gates precede the cross-model judge.

10 MEMORA-Planning: Construction and Evaluation Protocol↩︎

Appendix 10 specifies the prospective planning evaluation in MEMORA-Bench. MEMORA-Planning evaluates whether the constructed memory bank supports future action, rather than only retrospective question answering. In contrast to the Embodied Memory assessment, which asks an agent to select a discrete answer about past experience, MEMORA-Planning requires the agent to synthesize an ordered kitchen plan for a specific participant. Each item is grounded in EPIC-KITCHENS-100 annotations [18]: the reference sequence is an observed segment of narrated actions, but the planner is evaluated on whether it can produce an executable, experience-grounded procedure rather than reproduce the annotation verbatim.

The reported benchmark uses the released MEMORA-Planning split. Its tasks are generated deterministically from EPIC action annotations, restricted to videos available in the participant’s consolidated Inferred Knowledge, and enriched with object metadata from the same memory. No LLM is used to generate task labels or task queries in this release. The evaluation compares no-memory, linear-memory, and structured-memory agents under the same planner model, and reports rule-based metrics to avoid judging an LLM agent with another LLM.

10.1 Task Construction↩︎

10.1.0.1 Task definition.

An instance consists of a natural-language request \(x\), a participant ID, a video ID, and a reference sequence of EPIC actions. The request has the form of a user instruction to a household robot, e.g., “Help P03 clean the washing up bowl in the sink.” The reference sequence stores the ordered EPIC narrations, verbs, nouns, and timestamps for the segment from which the request was derived. The benchmark therefore tests procedural generation under verifiable evidence: every released item can be audited against the annotation table, but the model’s output remains open-ended.

10.1.0.2 Design principles.

Planning items are experience-dependent: they should reward agents that retrieve the person’s observed routines, object locations, and preferences, not agents that merely know generic cooking scripts. The generation process is model-agnostic: segmentation, classification, query templating, diversity selection, filtering, and memory grounding are rule-based in the reported release. Finally, the benchmark is memory-scoped: a task is generated only from videos present in that participant’s consolidated Inferred Knowledge, so a memory-enabled agent is not penalized for lacking access to the relevant episode. Figure 13 summarizes the deterministic generation stages.

10.1.0.3 Segmentation and task typing.

Within each video, action rows are sorted by timestamp. A hard split is placed when adjacent narrations are separated by more than 30 seconds. Soft splits are deterministic: verbs map to cleanup, cooking, prep, or general, and action \(i\) starts a new segment only if the \(i{-}1\!\to\!i\) gap exceeds 10 seconds, the category changes between two non-general categories, and action \(i{+}1\) confirms the new category. Segments with fewer than three actions are merged, segments with more than 15 actions are split into near-even chunks, and the fixed task set is shared by all memory conditions. The resulting candidates are classified into the five task types in Table 9. Filler pick-and-place verbs are removed before classification so that superficial object motion does not dominate the task label.

5pt

10.1.0.4 Selection and memory-grounded enrichment.

Candidate tasks are scored by proximity to the target length, verb diversity, and the fraction of non-filler actions. For each participant we greedily select up to 12 diverse tasks, with a per-type cap of four tasks under the main-paper configuration. A subsequent improvement pass computes duplicate-narration rate, \(1-|\mathrm{unique\;narrations}|/|\mathrm{steps}|\), and moves tasks above 0.70 to an audit-only filtered bucket. For retained tasks, each EPIC noun is matched against the same video’s object_registry using exact, substring, token-overlap, and synonym matching. Successful matches add an enriched narration and a structured matched_object record containing memory object ID, name, color, material, and location. This enrichment is strictly additive: the original EPIC fields remain unchanged.

10.1.0.5 Consolidated Inferred Knowledge.

The MEMORA conditions consume a per-participant consolidated Inferred Knowledge bank. For planning, the most important fields are skill_memory.routine_skills, which stores routine goals, canonical steps, supporting episodes, and key objects; and skill_memory.generated_preferences, which stores evidence-grounded preferences, keywords, confidence scores, and supporting episodes. These fields make the consolidated Inferred Knowledge more than a searchable transcript: they expose consolidated procedures and preferences that a planner can retrieve and adapt.

10.1.0.6 Participant coverage.

The reported generation covers 18 EPIC participants with released consolidated Inferred Knowledge banks: P01, P02, P03, P04, P06, P07, P09, P11, P12, P22, P23, P25, P26, P27, P28, P30, P35, and P37. P05 is excluded because its EPIC identifier convention does not match the three-digit suffix pipeline used for this release.

10.2 Memory Conditions and Tools↩︎

10.2.0.1 Controlled comparison.

The evaluation compares three memory conditions while holding the planning model, benchmark tasks, and shared runtime stack (Appendix 13) fixed. This isolates the effect of memory organization and tool interface. The linear-memory baseline follows the Socratic-models style of exposing a text memory through language-model interaction [16]. Table [tab:planning-conditions] summarizes the conditions.

10.2.0.2 Planner and runtime.

Planning-specific runtime settings beyond the shared stack in Appendix 13 are:

• Task-list loading: the planner is loaded once per 72-specification run so weights stay resident for the full batch.

• ReAct budget: at most \(8\) iterations, including a forced-answer final iteration with tools disabled.

• Tool parsing: a local parser handles Qwen-style XML/tool-call formats inside the agent wrapper.

• Optional judge: gpt-4o-mini is available for ablations but is not used in the reported rule-based panel.

10.2.0.3 ReAct loop and plan parser.

Memory-backed conditions run a ReAct-style loop [50] under the shared agent-loop budget in Appendix 13 (Planning: \(8\) iterations including forced answer). At each iteration the environment supplies conversation state and registered tools; valid tool calls are executed and appended as observations. On the final iteration, the environment enters forced-answer mode: tools are cleared, previous tool calls are summarized as text, and the prompt explicitly requires a final Plan: rather than another tool call. The parser extracts the generated plan from numbered lines, bullet lists, or an explicit Plan: block.

10.2.0.4 Structured planning tools.

The MEMORA prompt is designed to make the consolidated Inferred Knowledge operational. It asks the agent to begin with get_routine_skill(goal_query, top_k), optionally query get_preferences, and then ground concrete objects through Entity Memory search. get_routine_skill retrieves routine memories by embedding similarity over routine goals, canonical steps, and key objects. get_preferences retrieves evidence-grounded preferences by query similarity or confidence.

10.2.0.5 Planning prompt templates.

Figure [fig:planning-prompt-templates] summarizes the prompt templates used by the planning conditions and forced-answer mode. The no-memory, Socratic, and MEMORA conditions differ in available memory and tools.

10.3 Metrics↩︎

10.3.0.1 Rule-based metrics.

The reported panel uses rule-based metrics rather than an LLM judge. For a generated plan \(P=(p_1,\ldots,p_m)\), task query \(x\), participant routines \(R\), and participant preferences \(Q\), we compute four scores.

10.3.0.2 Reference provenance.

Replay asks whether an agent recovers a workflow already present in participant memory, so OrderExec, KeyObj, and PrefAdh use matched consolidated routine, object, and preference references. Generalize has no matching stored routine by construction; procedure order is therefore scored against verified task-level ground_truth_steps, with the chosen source recorded in metric_source.

Executability (Exec). A step is executable if it contains an action, an object, and a location phrase according to rule-based predicates. Let \(A(p)\), \(O(p)\), and \(L(p)\) denote the action, object, and location predicates: \[\mathrm{Exec}(P)= \frac{|\{p_i \in P: A(p_i)\wedge O(p_i)\wedge L(p_i)\}|}{|P|}.\]

Procedure order (OrderExec). The metric compares the normalized action-verb sequence of the generated plan with a per-task reference verb sequence. On Replay, the reference is the matched routine’s canonical-step verbs (selected as the participant’s routine_skills entry with highest Jaccard overlap with the task query, threshold \(0.10\)). On Generalize, where no matching routine exists by construction, the reference is the task’s verified ground_truth_steps verb sequence; the metric router (scripts/compute_rule_metrics.py) auto-detects Generalize tasks by their denovo_ prefix and records the chosen reference in a per-task metric_source field. \[\mathrm{OrderExec}(P,s)= \frac{\mathrm{LCS}(v(P),v(s))}{|v(s)|},\] where \(s\) is the per-task reference sequence and \(v(\cdot)\) denotes a synonym- and inflection-normalized verb sequence. Tasks with no qualifying reference are undefined and skipped in the metric mean.

Key-object coverage (KeyObj). For the matched routine \(r\) with key-object set \(K_r\), we compute: \[\mathrm{KeyObj}(P,r)= \frac{|\{k\in K_r: k \in P\}|}{|K_r|}.\]

Preference adherence (PrefAdh). Preferences are first filtered by Jaccard overlap between the task query and the preference text or keywords, with threshold 0.05. For each relevant preference, we measure the fraction of preference tokens appearing in the plan, then average over relevant preferences.

Robot-Grounded Plan score (RGP). The headline planning aggregate is the unweighted per-task mean of three robot-followability axes: \[\mathrm{RGP}(P) = \operatorname{mean}\bigl(\mathrm{OrderExec}, \mathrm{KeyObj}, \mathrm{PrefAdh}\bigr).\] Each axis captures one execution requirement (order, task-relevant object coverage, personalization). Raw Exec is reported separately because it rewards plan length regardless of correctness. KBGround is reported as a separate diagnostic in §10.6 rather than folded into RGP, for the metric reasons discussed there.

Verb normalization. OrderExec uses the V2 verb extractor for both the plan and reference sequences. This normalization corrects two systematic under-estimation errors: canonical memory steps often begin with “Person ...” rather than “The person ...”, and routine or GT-step descriptions often use inflected verbs such as places, washing, or preparing. V2 strips article-free and pronominal subjects, maps common inflections to canonical verb classes, and suppresses non-verb fallback tokens such as person. The LCS scoring formula itself is unchanged.

10.4 Results and Diagnostics↩︎

10.4.0.1 Cross-backbone summary.

Figure 18 summarises the eight (backbone, benchmark) cells from the main paper. Grey = best non-MEMORA baseline; orange = best MEMORA variant. MEMORA wins every cell, with \(\Delta\) RGP rising from \(+0.012\!-\!+0.056\) on Replay to \(+0.035\!-\!+0.075\) on Generalize. Two recurring patterns underlie the two observations summarised in §5.2. First, MEMORA’s gain is at least as large on Generalize as on Replay for three of the four backbones, indicating that structured, consolidated memory matters most when no verbatim segment is available to retrieve. Second, the two larger reasoning backbones (Gemma-4-31B-it and Qwen3.6-35B-A3B) extract more value from the same memory state on Generalize, both delivering \(\geq +0.064\) absolute \(\Delta\) RGP, consistent with a reasoning-bound rather than memory-bound ceiling on the harder split.

10.4.0.2 Per-condition numerical results: Gemma-4-31B-it (main-paper RGP figure).

Figure 3 reports per-condition RGP on Gemma-4-31B-it for Replay (\(n{=}207\)) and Generalize (\(n{=}153\)); the per-axis OrderExec / KeyObj / PrefAdh decomposition referenced in §5.2 is available in the figure companion panel.

10.4.0.3 Per-condition numerical results: Qwen3.6-35B-A3B.

Table 10 reports the same per-condition decomposition on Qwen3.6-35B-A3B. The qualitative pattern matches Gemma-4-31B-it: typed retrieval lifts the same three axes, with the magnitude consistently larger on Generalize, where chunk-retrieval baselines no longer benefit from verbatim segment recall.

5pt

10.4.0.4 Per-store tool-use distribution.

Figure 19 aggregates every logged tool_call across all \(1{,}440\) planning tasks (\(4\) backbones \(\times\) \(2\) suites \(\times\) \(18\) PIDs \(\times\) \(12\) tasks, \(7{,}163\) calls under the main MEMORA configuration) and maps each tool to the typed store it queries (Entity Memory: search_objects, get_object_history; Inferred Knowledge: get_routine_skill, get_preferences; Activity Memory: search_activities). Two patterns recur. First, backbones differ sharply in how they spend their tool budget: Inferred Knowledge share ranges from \(27\%\) (Qwen3.6-27B) to \(54\%\) (Gemma4-26B-A4B-it), so any architecture that exposes only one store would systematically disadvantage one backbone family. Second, comparing MEMORA against the MEMORA-Episodic ablation (identical agent loop and KB, but agents rely on Entity Memory and Activity Memory tools only) shows that Inferred Knowledge’s \(40\%\) share does not simply vanish: \(35\%\) of all tool calls instead migrate to search_activities over Activity Memory, suggesting that consolidated routines absorb queries that episodic activity search alone answers noisily. Across both configurations the unified search dispatcher remains rare (\(<\!4\%\)); agents prefer the typed entry points the prompt advertises, and \(\approx 98.5\%\) of MEMORA’s tool calls go through the typed cascade (search_objects / get_routine_skill / get_preferences / search_activities), with the cascade invoked on \(96\)–\(100\%\) of Generalize tasks per backbone. For contrast, Flat-1D conditions on the same Generalize split issue an average of \(5.8\) search queries per task, overwhelmingly keyword lookups that can surface a verbatim chunk when the task matches an observed video but offer no structured affordance for composition.

4pt

10.4.0.5 Complementary evidence from EAM-QA.

The planning-side distribution above is partial because the planning prompt does not advertise an environment-search tool; spatial information rides on Entity Memory object returns. On EAM-QA, where the agent is given a single unified search entry point with optional category\(\in\{\)objects,activities,environment,patterns\(\}\), the same backbone uses that one tool for \(95.9\%\) of all calls (Figure 20, aggregated from \(7{,}068\) questions and \(17{,}220\) tool calls across four backbones and four EAM-QA question types: ELoc, EOrder, SPref, ERecall). That call pattern is consistent with Environment Memory being read on essentially every query: the unified search routes internally through all four typed indices and merges the results, so Environment Memory contributions surface inside an environment-category result list rather than as a separate tool invocation. The two views together show that all four stores are accessed in practice — planners reach them through typed entry points the prompt advertises, while EAM-QA agents reach them through a single dispatching call.

10.4.0.6 Illustrative trace.

Figure [fig:planning-tool-trace] provides a compact excerpt from a real MEMORA trace in the reported evaluation. The task is plan_P03_101_019: “Help P03 clean the washing up bowl in the sink” (cleanup_organization, 4/8 iterations). The agent first retrieves a generic cookware-washing routine, notices that the retrieved key objects do not match the requested bowl, refines the routine query, grounds the bowl through object search, and then emits a grounded plan.

10.5 Generation pipeline reproducibility↩︎

10.5.0.1 Reproducibility.

The generation pipeline is deterministic under the fixed EPIC CSV, participant list, and released consolidated Inferred Knowledge banks. Task extraction targets 12 tasks per participant, followed by a rule-based improvement pass that computes duplicate-narration rate and moves high-repeat segments to an audit-only bucket. The evaluation task list contains one specification for each participant and condition, for \(18 \times 4 = 72\) specifications. An optional gpt-4o-mini judge can be run as a separate phase, but it is not used in the reported rule-based panel.

10.5.0.2 Generalize benchmark.

The Generalize split is generated by a separate, OOD-by-design pipeline that combines three rule-based families with an LLM verification pass that filters out tasks the KB could have answered as Replay. The first family, transfer, draws on a participant’s consolidated Inferred Knowledge to identify “observed action \(a\) on object \(o_{\mathrm{obs}}\)” pairs (e.g.P01 has been observed cutting mushrooms three times), then substitutes a target object \(o_{\mathrm{tgt}}\) that the same participant has been observed but never in combination with \(a\) (e.g.cutting potato); the task query asks the agent to apply the skill to the substituted target. The second family, composition, samples a high-level goal that is not present in routine_skills but whose sub-procedures individually are (e.g.prepare a vegetable stir-fry dish decomposes into operations such as heating a pan, picking up vegetables, and stirring, each of which appears as a routine but never as a parent skill). The third family, fully-novel, asks for plans whose surface goal has neither a matching routine nor a matching sub-procedure decomposition in the KB. An LLM verifier (gpt-4o-mini) is used at generation only: it writes a \(5\)-step ground-truth plan for each candidate task and scores it on five binary criteria (novelty relative to KB routines, sub-procedure grounding, object grounding, physical correctness, and step-quality coherence); tasks that fail any criterion are dropped. The released split contains \(153\) verified tasks across the \(18\)-PID panel. At evaluation time, the rule-based panel substitutes the task-level PlanFid_GT (verb-LCS against ground_truth_steps) for the KB-routine reference, because a Jaccard-matched nearest KB routine for a novel goal is only lexically close and penalises agents that compose correctly; the metric router (scripts/compute_rule_metrics.py) auto-detects Generalize tasks by their denovo_ task_id prefix and records the chosen reference in a per-task metric_source field.

10.6 Object-grounding diagnostic↩︎

We separately report object grounding as a diagnostic because no token-overlap definition of object grounding cleanly captures what memory contributes to robot-followable planning. This subsection (i) defines four candidate object-grounding metrics, (ii) reports them across all six completed (backbone, benchmark) cells, and (iii) discusses why we ultimately exclude object grounding from RGP while still tracking it.

10.6.0.1 Four candidate definitions.

Let \(P\) be the set of distinct kitchen-noun stems extracted from a generated plan, \(G\) the corresponding set for the ground-truth plan, and \(K\) the participant’s KB-vocabulary noun set (§[app:planning-full]). We consider: \[\begin{align} \mathrm{KBGround}(P,K) &= |P \cap K|/|P|, \\ \mathrm{ObjGT\text{-}R}(P,G) &= |P \cap G|/|G|, \\ \mathrm{ObjGT\text{-}P}(P,G) &= |P \cap G|/|P|, \\ \mathrm{ObjGT\text{-}F1} &= \frac{2\,\mathrm{ObjGT\text{-}P}\,\mathrm{ObjGT\text{-}R}}{\mathrm{ObjGT\text{-}P}+\mathrm{ObjGT\text{-}R}}, \\ \mathrm{GR}(P,K,G) &= \frac{|P \cap K \cap G|}{|K \cap G|}, \\ \mathrm{GP}(P,K,G) &= \frac{|P \cap K \cap G|}{|P|}, \\ \mathrm{GF1} &= \frac{2\,\mathrm{GP}\,\mathrm{GR}}{\mathrm{GP}+\mathrm{GR}}. \end{align}\] KBGround answers “does the plan mention things known to the participant?” ObjGT-{R,P,F1} answers “does the plan mention the same objects as the GT plan?” GR / GP / GF1 (Grounded Recall / Precision / F1) answer the more robot-relevant question “does the plan mention the right task objects and are those objects findable in the participant’s kitchen?”

10.6.0.2 Results across all completed cells.

Table ¿tbl:tab:planning-grounding-allcells? reports the four grounding metrics for every (backbone, benchmark) cell. The pattern is consistent across cells: chunk-based retrieval baselines win KBGround by a wide margin (their plans inherit verbatim text from the chunks used to build the KB), and the winner under ObjGT-F1 or GF1 is typically Flat-1D or Graph-2D rather than MEMORA.

5pt

10.6.0.3 The chunk-copying signature: KeyObj and KBGround disagree on baselines.

The contrast between KeyObj (in RGP) and KBGround (this diagnostic) is informative. On Qwen3.6-35B-A3B / Generalize, Flat-1D (raw) scores KBGround \(=.920\) but KeyObj \(=.570\); MEMORA scores KBGround \(=.883\) but KeyObj \(=.650\) — the chunk-copying signature. The Pearson correlation between KeyObj and KBGround at the per-task level is essentially zero on this cell (range \(r{=}{-}.116\) to \({+}.220\) across conditions), confirming that the two metrics measure complementary properties: KeyObj rewards plans that name the per-task gold objects, while KBGround rewards plans that mention any KB-known noun. Folding both into a single composite would (i) double-weight “object naming” relative to procedure / preferences, and (ii) reward chunk-copying baselines on the KBGround side without rewarding task-relevant retrieval. RGP therefore keeps only KeyObj.

10.6.0.4 Why no token-overlap metric cleanly favours MEMORA.

Across all four grounding-metric variants, MEMORA wins outright on at most one of six cells (Gemma-4-31B-it / Replay, ObjGT-F1 and GF1 tied at \(+.001\)). Three structural reasons explain this: (1) The Generalize task query directly names the key objects (“Help P01 cut the cucumber”), so any LLM with kitchen prior — including the No-Memory parametric baseline — produces a plan whose nouns largely match the GT plan; this is a verbosity confound that ObjGT-R rewards (No Memory wins ObjGT-R on every Generalize cell). (2) Flat-1D copies EPIC narrations verbatim, so its plans inherit both KB-membership and GT-noun-membership “for free” on Replay; Graph-2D injects graph-resolved entities sparsely and wins ObjGT-Precision by being terse. (3) MEMORA paraphrases retrieved objects into natural-language plan steps (e.g., “the green cucumber that P01 keeps on the counter”) and adds procedural elaboration, both of which dilute token-overlap scores even when the underlying entity is correct. None of these metrics tests whether the specific instance named is correct (which cucumber, on which counter); answering that question requires execution-grounded evaluation in simulation or on a physical platform, which we leave to future work.

11 Physical Robot Demonstration↩︎

This appendix documents the qualitative robot case study referenced in Section 5.4 and visualized in Figure 4. The case study checks whether the experience grounding measured by MEMORA-Planning remains visible when a memory-conditioned instruction is executed in a physical scene. The demonstration is a transfer setting: the memory is formed from a human demonstrator’s egocentric video, then retrieved as procedural, preference, temporal-order, and entity-grounding guidance for a robot acting in the same environment. We therefore report the setup, task design, success criterion, and video traces for two memory-grounded robot tasks.

11.1 Setup and Protocol↩︎

11.1.0.1 Platform and scene.

The demonstration uses a Unitree G1 operating in a kitchen-like tabletop scene with visually similar distractor objects. Before robot execution, the human demonstrator records two egocentric videos in the same scene: one for a drink-preparation routine and one for a breakfast routine. These videos are processed before robot execution, so the robot plan is conditioned on memory formed from human first-person experience rather than on direct access to the demonstration video at execution time.

11.1.0.2 Task design.

Each instruction is intentionally underspecified unless the system can use the participant’s prior experience. Task A: Prepare Drink tests preference and entity grounding by checking whether the robot selects bottled Coca-Cola over a yellow Fanta pineapple can and the demonstrator’s orange plastic cup among cups with other colors. Task B: Breakfast tests preference, grounding, and sequence by checking whether the robot selects the yellow bowl rather than a gray bowl, places Cinnamon Tea rather than Lipton Black Tea into the bowl, places the yellow spoon rather than a metal spoon into the bowl, and then takes the bread. The intended test is whether a memory system can transfer experience-derived context into a robot plan that resolves preferences, temporal order, entity identity, and attributes, rather than following a generic task script.

11.1.0.3 Evidence.

The demonstration consists of four videos: the human egocentric demonstration for Task A, the human egocentric demonstration for Task B, the Unitree G1 execution of Task A from MEMORA’s plan, and the Unitree G1 execution of Task B from MEMORA’s plan. The two robot videos provide the qualitative success traces reported below.

11.1.0.4 Execution interface.

MEMORA produces a language-level plan for each experience-grounded instruction. The generated plan is then converted into grasp and placement primitives by a fixed rule-based low-level controller. This design holds motor execution fixed, so the demonstration isolates memory-grounded language planning as the variable of interest.

4pt

11.2 Results and Traces↩︎

Table 12 gives the task-level plan and success criterion, and Figures 21 and 22 show the full visual traces used to construct the compact main-paper figure. Each panel includes the human egocentric demonstration and the corresponding Unitree G1 execution, making explicit which observed routines, preferences, and choices are transferred into robot behavior.

Both tasks succeed on the first attempt (2/2): Task A selects bottled Coca-Cola and the orange plastic cup; Task B selects the yellow bowl, Cinnamon Tea, yellow spoon, and bread in the remembered order.

11.2.0.1 Limitations.

This demonstration is best interpreted as a deployment-oriented sanity check. It involves two tasks in one physical scene and is not intended to establish statistical robot generalization. Its role is to make the planning metric concrete by illustrating how memory formed from human egocentric observation can guide physical action selection, entity grounding, and sequencing in a robot execution setting.

12 Controlled Ablation Studies↩︎

Appendix 12 defines three orthogonal controlled ablations over the MEMORA memory-bank construction pipeline and its downstream answer agents. CTRL uses the main-paper processor settings in Appendix 13 (processor contracts: Table 4); the released pipeline contains no closed-weight model in any role. Cohort split. We partition the 18-participant MEMORA-Bench panel into a calibration cohort and a held-out evaluation cohort, and run Axes 1–2 on the held-out cohort to avoid leakage of any processor tuning or prompt iteration into the ablation result:

• Calibration cohort (P01–P04, 4 PIDs, \(\sim\)​80 EPIC-KITCHENS-100 extension videos, \(746\) EAM-QA questions = \(27.0\%\) of the full EAM-QA panel). This subset was used during pipeline development to iterate perception- and memory-processor prompt templates and to fix the MEMORA pipeline; it is therefore not a clean test set for processor swaps.

• Held-out ablation cohort (P06, P07, P09, P11, P12, P22, P23, P25, P26, P27, P28, P30, P35, P37 – 14 PIDs, \(\sim\)​121 videos, \(2{,}017\) EAM-QA questions = \(73.0\%\) of the benchmark). Memory-processor outputs for the three Axis-1/Axis-2 arms (Omni+Q30B, Omni+Q14B, VL+Q30B) were materialized in a single “rest” merge over exactly this PID set, so the comparison covers a fixed video population for every arm.

Isolation protocol: each study varies exactly one of {online memory processor, perception processor (and audio), answer agent}; the other two axes remain at the main-paper settings in Appendix 13 and the CTRL row of Table 13. Axes 1–2 use the MEMORA-Episodic memory bank (without offline Inferred Knowledge enrichment) so all three arms are compared on identical memory-processor outputs, isolating the perception\(\to\)memory-encoding contribution before enrichment. All ablations reuse the processor prompt templates in Figure 7 and the planning prompt templates in Figure [fig:planning-prompt-templates]; only the ablated model, modality, or answer-agent backbone changes.

12.1 Master Condition Matrix↩︎

4pt

12.1.0.1 Axis 1: Online Memory Processor.

12.1.0.2 Question.

How sensitive are downstream scores to the capacity of the memory-editing language model when segment perception is held constant?

12.1.0.3 Design.

The Perception Processor is fixed at Qwen2.5-Omni-7B, and the same per-segment JSON is reused for both conditions; only the Online Memory Processor differs (A1-14B vs.CTRL in Table 13). No Inferred Knowledge enrichment is applied to either tree, so the enrichment LLM is not part of the comparison. The answer agent is held fixed across conditions so EAM-QA isolates encoding quality rather than answerer capacity; we report results on two answer backbones (Gemma-4-26B-A4B-it and Qwen3.6-35B-A3B) over the same 14-participant cohort in Table 15, to verify that the encoding-level conclusion is not specific to a particular answerer.

12.1.0.4 Memory-bank shape under the A1 swap.

Before turning to downstream accuracy, we compare the shape of the Entity Memory bank produced by the two memory processors on the same Omni-7B perception stream (14-PID held-out cohort). Table 14 reports, for each participant, the append-only entity-observation count (No-edit), the number of unique entities retained after editing (MEMORA), and the resulting compression ratio.

5pt

Two observations. First, the smaller editor’s bank is not the result of tighter deduplication: the larger Q30B editor reaches a higher compression ratio on \(13/14\) participants (median \(20.6\times\) vs \(18.2\times\); mean \(+2.5\times\) paired difference). Second, the dominant axis of shrinkage is the number of state changes recorded: A1-14B commits \(24.5\%\) fewer turn-aligned entity observations to memory (\(61{,}478\to46{,}438\)) while keeping a similar number of unique entities (\(-5.0\%\), \(2{,}800\to2{,}660\)). The smaller editor therefore produces an under-written memory — it sees the same perception output but issues Update and Add less often — rather than a more compact one. This shape change is the mechanism behind the \(-5.8\) point accuracy drop and the E-rate surge in Table 15 (Gemma block): temporal questions (SHabit, SRoutine) lose the most because they depend on cross-segment state-update density, not on the entity inventory.

12.1.0.5 Axis 2: Perception Processor and Audio.

12.1.0.6 Question.

How much do perception backbone and audio modality contribute to encoded memory when the memory processor and answer agent are fixed?

12.1.0.7 Design.

The Online Memory Processor is fixed at Qwen3-30B-A3B-Instruct-2507; we report results on two answer backbones in Table 15. CTRL uses the Omni multimodal path with model-native audio so each 10 s segment is encoded jointly from video frames and the synchronized audio stream. The ablation swaps in Qwen2.5-VL-7B-Instruct (vision-only); because this backbone has no audio modality, A2-VL7 simultaneously tests (a) the multimodal \(\to\) vision-only perception swap, and (b) the audio-absent condition for memory encoding, without requiring a separate audio-strip pass. Temporal preprocessing matches Appendix 13: each 10 s segment is decoded to video frames at 2 FPS and capped at \(\leq\)​24 frames per segment; under Omni the native audio stream of the clip is additionally passed in, and under VL only the frame stack is passed in. The only thing that varies across conditions is therefore the perception-processor backbone and, by construction, its access to audio (A2-VL7 vs.CTRL in Table 13).

12.1.0.8 Axis 3: Answer agent.

12.1.0.9 Question.

How much do MEMORA-Bench and Planning scores depend on the answer-agent backbone when the encoded consolidated Inferred Knowledge bank is frozen to the main-paper pipeline output?

12.1.0.10 Design and reference to existing panels.

Axis 3 is structurally identical to the cross-backbone evaluation already reported in the main supplementary material: all conditions load the same CTRL consolidated Inferred Knowledge bank (Omni-7B perception, Qwen3-30B-A3B-Instruct-2507 memory) and only the answer agent changes. We therefore do not introduce a new Axis-3 table here; the four backbones {Qwen3.6-35B-A3B (CTRL), Qwen3.6-27B, Gemma-4-26B-A4B-it, Gemma-4-31B} are scored on MEMORA-Bench EAM-QA in Table ¿tbl:tab:mc5-per-backbone-raw? and on MEMORA-Planning in Figure 18 (eight cells over Replay \(\times\) Generalize \(\times\) four backbones; per-axis detail in Table 10). The 18-participant cross-backbone Planning panel is a superset of the Axes 1–2 cohort (it includes the calibration PIDs and uses identical rule-based metrics and task prompts). For the Qwen3.6 family we disable extended thinking; Gemma models use the project tool-call parsers without a standardized think mode.

12.1.0.11 Sanity check: closed-weight enrichment alternative.

The main-paper pipeline performs Inferred Knowledge enrichment with Qwen3.6-35B-A3B, so the released system contains no closed-weight model. For completeness we also produce an alternative consolidated Inferred Knowledge bank using gpt-4o-mini, holding every other component (perception, online memory processor, retrieval encoder, answer agent) fixed. On the two backbones where the full cross-bank evaluation has finished, the two enrichment LLMs yield statistically equivalent EAM-QA accuracy: Gemma-4-26B-A4B-it scores \(35.76\%\) with the Qwen3.6 bank and \(35.92\%\) with the gpt-4o-mini bank (\(\Delta=-0.16\) points on a triple-paired subset of \(N{=}1{,}843\) questions); Gemma-4-31B scores \(36.4\%\) with the Qwen3.6 bank and \(35.8\%\) with the gpt-4o-mini bank (\(\Delta{=}{+}0.6\) points, the only cell where the open-weight bank exceeds the closed-weight one). Both gaps lie inside the typical seed/condition variance of EAM-QA at this \(N\), so we conclude that the open-weight enrichment used in the main-paper pipeline does not sacrifice headline accuracy and that the entire MEMORA pipeline is reproducible without any closed-weight LLM.

12.2 Evaluation Protocol↩︎

Axes 1–2 are scored on the MEMORA-Bench EAM-QA harness over the held-out 14-participant cohort (P06–P37, \(2{,}017\) questions = \(73.0\%\) of the full EAM-QA panel) using the MEMORA-Episodic memory bank (without Inferred Knowledge enrichment), so perception- and memory-encoding effects are not confounded with Inferred Knowledge enrichment. Metrics are overall accuracy over five options (A–E, with E denoting insufficient information) and per-type accuracy on SPref/SRoutine/SHabit/ERecall. Axis 3 is read off of the existing cross-backbone panels (Table ¿tbl:tab:mc5-per-backbone-raw?, Figure 18) without re-running.

12.3 Results↩︎

12.3.0.1 Axes 1–2 results on two answer backbones (14 PIDs, 2,017 EAM-QA questions).

Table 15 reports overall accuracy, E-rate (the rate at which the answer agent selects the explicit “E” insufficient evidence option), and per-type accuracy for the three arms on the MEMORA-Episodic memory bank, scored on two answer backbones to verify that the encoding-level conclusion is not specific to a particular answerer.

5pt

12.3.0.2 Findings.

On the same answer agent (Gemma-4-26B-A4B-it) and the same 14-PID MEMORA-Episodic bank: (i) Memory-processor capacity dominates: dropping the online memory processor from Qwen3-30B-A3B-Instruct-2507 to Qwen3-14B costs \(-5.8\) points of overall accuracy and is accompanied by a sharp rise in E-rate (\(42.8\% \to 62.3\%\)), indicating that the smaller model produces memory-bank records that less often support a contentful answer. Table 14 traces this to the shape of the memory bank rather than to over-deduplication: A1-14B writes \(24.5\%\) fewer turn-aligned entity observations into Entity Memory while keeping a comparable entity inventory (\(-5.0\%\)), so the answer agent selects E because the underlying state-change record is sparser, not because the bank is more compact. (ii) Perception backbone matters less: swapping the Omni multimodal path for vision-only Qwen2.5-VL-7B costs \(-2.7\) points overall with essentially unchanged E-rate (\(+1.3\) points), suggesting that, at MEMORA-Episodic granularity, audio-aware perception provides a smaller marginal contribution than memory-editor capacity. (iii) Type-conditional behavior is consistent with the encoding interpretation: A1-14B preserves ERecall (\(+1.3\) points) but loses the most on SPref / SRoutine / SHabit (\(-6.4 / -8.3 / -6.8\) points), which are the types that depend most on cross-segment habit/preference consolidation; A2-VL7 instead spreads its loss approximately evenly across all four types (\(-2.2 / -3.1 / -2.7 / -2.8\) points).

12.3.0.3 Cross-backbone confirmation (Qwen3.6-35B-A3B answer agent, full 14 PIDs).

On the same 14-PID held-out cohort and the identical MEMORA-Episodic memory banks, swapping the answer agent from Gemma-4-26B-A4B-it to the larger reasoning model Qwen3.6-35B-A3B (Table 15, Qwen block) preserves the sign and ranking of both effects: memory-processor ablation costs \(-3.3\) points overall versus \(-1.3\) points for perception ablation, so memory-processor capacity remains the larger contributor across answer backbones. The type-conditional pattern is also preserved: A1-14B loses on the cross-segment types (\(-6.6\) on SRoutine, \(-6.2\) on SHabit) while ERecall is essentially unchanged (\(+1.0\)), and A2-VL7 spreads its loss more evenly across types. Two backbone-specific nuances are visible. First, Qwen3.6-35B-A3B is a more committed answerer: its CTRL E-rate is \(30.2\%\) versus \(42.8\%\) for Gemma-4-26B-A4B-it, and the A1-14B E-rate increase is correspondingly smaller (\(+5.0\) points vs \(+19.5\) points), so the under-written memory hurts accuracy through more wrong content choices rather than through more E selections. Second, the absolute \(\Delta\) for A1-14B is smaller in magnitude on the larger reasoning backbone (\(-3.3\) vs \(-5.8\) points), consistent with a stronger answerer extracting more signal from a sparser memory record but not closing the gap. The encoding-level conclusion — memory-processor capacity dominates over perception backbone — therefore replicates across both answer backbones we tested, and the Q3.6-35B-A3B block in Table 15 supersedes the earlier P11+P12 pilot reading.

12.3.0.4 Qualitative side-by-side comparison.

Figure 23 contrasts the three arms on two illustrative EAM-QA items, picked to expose the two ablation-specific failure modes implied by the aggregate accuracy and E-rate numbers in Table 15 (Gemma block). On both items, CTRL (Omni + Qwen3-30B-A3B-Instruct-2507) recovers the gold letter; the memory-ablation arm A1-14B (Omni + Qwen3-14B) selects “E” – consistent with the \(+19.5\) points E-rate inflation we report at the aggregate level; and the perception-ablation arm A2-VL7 (VL + Qwen3-30B-A3B-Instruct-2507) confidently picks a visually plausible but habitually wrong option, consistent with a vision-only backbone that lacks the audio-aware multimodal context that the Omni-encoded memory bank otherwise propagates into the memory-processor records.

13 Implementation details↩︎

This section consolidates the model registry, inference stack, perception preprocessing, and retrieval/agent-loop settings that are shared across benchmark generation (Appendix 8), Embodied Memory evaluation (Appendix [app:embodied-memory]), Planning evaluation (Appendix 10), and the controlled ablations (Appendix 12). Unless stated otherwise, every condition that involves a given role (perception processor, online memory processor, Inferred Knowledge enrichment, retrieval encoder, or answer agent) uses the values reported here, so that cross-condition comparisons are not confounded by undocumented hyperparameter drift.

13.0.0.1 Compute and inference stack.

All open-weight models are served with vLLM 0.19.1 inside a Singularity container so that the dependency graph (CUDA / PyTorch / Transformers / sentence-transformers) is identical across runs. Two GPU classes are used: 80 GB H100 nodes and 80 GB A100 SXM4 nodes; the A100 timing node uses two AMD EPYC 7543 32-core CPUs (64 hardware threads). The perception processors (Qwen2.5-Omni-7B, Qwen2.5-VL-7B-Instruct) are served single-GPU (TP\(=1\), max_model_len \(\in\) ). The online memory processors (Qwen3-30B-A3B-Instruct-2507, Qwen3-14B) are served with TP\(=2\), max_model_len\(=\)​32 768, gpu_memory_utilization\(=\)​0.85, and sampling temperature \(0.6\). All answer agents (Qwen3.6-27B, Qwen3.6-35B-A3B, Gemma-4-26B-A4B-it, Gemma-4-31B-it) use the same TP\(=2\) stack and temperature; the cross-backbone EAM-QA panel uses max_model_len\(=\)​65 536, while Planning and the standard MC5 harness use max_model_len\(=\)​32 768. For Qwen3.x family chat templates we set enable_thinking=false so that the long “thinking” channel does not consume the 65 536-token budget that the ReAct loop reserves for tool returns; Gemma chat templates are used as-released. The Inferred Knowledge enrichment pass is performed by Qwen3.6-35B-A3B served on the same vLLM stack (TP\(=2\), temperature \(0.6\)), so the main-paper pipeline contains no closed-weight model in any role. Enrichment is run once per participant during memory-bank construction and its outputs are then frozen for all evaluation conditions. A closed-weight alternative (gpt-4o-mini) is benchmarked in Appendix 12.1.0.11.

13.0.0.2 Licences, intended use, and release plan.

All artifacts used in this work are publicly released for academic research and we use them consistently with their intended use. The EPIC-KITCHENS-100 dataset [18] is distributed under the CC BY-NC 4.0 licence for non-commercial research on egocentric video understanding, which covers the embodied-memory benchmarking we perform. The Qwen2.5 / Qwen3 / Qwen3.6 model families (perception, memory editor, \(\textcolor{infPurple}{\boldsymbol{Inferred Knowledge}}{}\) enrichment, and answer-agent roles) are released by Alibaba under the Tongyi Qianwen / Apache-2.0 family of open-weight licences for research and downstream development. The Gemma model family (Gemma-4-26B-A4B-it, Gemma-4-31B-it; used as answer agents) is released under the Gemma Terms of Use for permitted academic and non-commercial use. gpt-4o-mini is accessed through OpenAI’s API only as a closed-weight sanity check for the \(\textcolor{infPurple}{\boldsymbol{Inferred Knowledge}}{}\) enrichment slot (Appendix 12.1.0.11); no headline result depends on it. We add no new recordings, identifiers, or personally identifying content beyond what is already present in EPIC-KITCHENS-100; participant identifiers follow the dataset’s pseudonymous labels (P01, P02, …). MEMORA pipeline source code and MEMORA-Bench task lists will be released under permissive research licences (planned: Apache-2.0 for code; CC BY-NC 4.0 for derived task data, inherited from the upstream dataset) following acceptance.

13.0.0.3 Model registry.

Table ¿tbl:tab:impl-models? lists the Hugging Face identifier (or API endpoint) for every model role used in the paper; processor input/output contracts for memory-bank construction appear separately in Table 4. A condition can be uniquely reconstructed as a triple (perception processor, online memory processor, answer agent); the Inferred Knowledge enrichment slot is either Qwen/Qwen3.6-35B-A3B (main-paper, consolidated Inferred Knowledge bank), gpt-4o-mini (closed-weight alternative, used only for the sanity check in Appendix 12), or none (MEMORA-Episodic ablation bank without Inferred Knowledge enrichment).

5pt

13.0.0.4 Perception preprocessing.

Each EPIC-KITCHENS video is split into non-overlapping 10-second segments; frames are decoded at \(2\) FPS and capped at \(24\) frames per segment, which matches the receptive horizon of the Qwen2.5-Omni / Qwen2.5-VL temporal encoder and keeps per-segment context length below max_model_len. For the audio-visual front-end (Qwen2.5-Omni-7B) the native audio stream of the segment is passed alongside the frames through the model’s released audio tower; for the vision-only ablation (Qwen2.5-VL-7B-Instruct) the audio stream is dropped at decode time and only frames are presented. The perception processor outputs a JSON “layered segment” record per 10-second window (action, objects, verbalised_audio when available, and a free-form scene caption), which is then handed to the online memory processor.

13.0.0.5 Retrieval, dedup, and agent loop.

At evaluation time the answer agent sees only an EAM-QA question (or a Planning request) plus a set of typed retrieval tools over the participant’s memory bank. All search tools use the same dense encoder (E5-base-v2) and return the top-\(k{=}10\) records by cosine similarity, with a deduplication pass that collapses any pair of records whose embeddings have cosine similarity \(\geq 0.95\) to the higher-scoring one. The E5 retrieval encoder is kept on GPU during evaluation; the reported memory-tool latency therefore includes GPU embedding time together with CPU-side orchestration, filtering, and vector-search overhead. The Embodied Memory ReAct loop allows at most \(5\) tool-call iterations per question; the Planning ReAct loop allows at most \(8\) (including a forced-answer final iteration), reflecting the longer intermediate-tool-use traces (get_routine_skill \(\to\) search_objects \(\to\) search_activities \(\to\) get_preferences) that Planning queries require. Planning loads the planner once per task-list run through in-process vLLM rather than an OpenAI-compatible HTTP server, so weights stay resident across the full specification batch and tool-call parsing stays inside the evaluation process. Planning latency is measured after one-time model loading on the main Qwen3.6-35B-A3B stack served on A100/vLLM. In a timing run over \(207\) goals, end-to-end retrieval-and-planning takes \(10.40\) s on average, with most time spent in answer-agent generation rather than memory access (Table 16). Since MEMORA forms memory offline before task execution and planning is issued before low-level control, this latency is compatible with the language-level planning interface studied here. At Embodied Memory evaluation time we additionally apply the time-restricted snapshot \(\widetilde{\mathcal{M}}_{t_{\mathrm{ask}}}\) (Section 4): entity states are rolled back to the ask_turn_id of the question, and offline Inferred Knowledge records that post-date \(t_{\mathrm{ask}}\) are withheld so the agent must answer from observations strictly preceding the question.

4pt

References↩︎