1 Introduction↩︎

Large Language Models (LLMs) deployed as autonomous agents have demonstrated remarkable capabilities in interactive environments, multi-step planning, and tool use [1]–[4]. A critical enabler of this progress is memory, the ability to store experiences from solved tasks and retrieve them on future ones, so that an agent can avoid repeating past mistakes, reuse working strategies, and incrementally accumulate domain knowledge rather than starting from scratch on every episode [5]–[9].

Despite rapid progress in what to store (trajectories, reflections, insights, skills, graphs, latent tokens [5], [9]–[13]), most existing memory systems treat memory access as a static pipeline: a single global retriever is called once per step with a hard-coded top-\(k\), a hard-coded graph hop depth, and a hard-coded query template, and uses the same configuration whether memory is empty or contains thousands of trajectories, whether the task is familiar or novel, and whether the agent is progressing or has been executing the same action for five steps. A complementary line of work—most prominently MemGPT [11]—makes memory access adaptive by promoting the LLM itself to the role of memory controller (paginating, formulating queries, and deciding when to recall), at the cost of an additional LLM call per memory operation. MemCon occupies a third design point: a learned but lightweight controller that is adaptive without any extra LLM calls. We show this static-pipeline view is a fundamental bottleneck for agentic memory systems: existing static memory systems either retrieve too aggressively (inflating context and cost while hurting accuracy on simple queries) or too conservatively (missing key reusable plans and reflections), because any single setting is miscalibrated for at least one of the regimes above.

We argue that effective agent memory requires an adaptive control layer sitting on top of the storage backend. Concretely: (i) Early tasks should retrieve less, because there is little useful experience to draw on and indiscriminate retrieval only dilutes the prompt. (ii) Recurring goal types should prefer plan reuse, replaying a distilled, object-generalized template of a prior success, rather than nearest-neighbor retrieval over raw trajectories. (iii) Stuck agents that repeat actions should trigger re-retrieval with an alternative query rather than re-reading the same top-\(k\) that already failed to help. (iv) Long task streams require consolidation and forgetting so that a growing, noisy memory remains useful rather than overwhelming. No single fixed heuristic achieves all four; the right memory operation is a function of task progress, memory state, and the agent’s learning phase.

We introduce MemCon(Memory as a Controlled Process), a framework that reformulates memory access as a sequential decision problem. MemCon casts the choice of memory operation (Retrieve, PlanInject, Re-Retrieve, Consolidate, Forget, NoOp) together with its parameters (top_k, insight_k, graph hop) as actions in a Memory MDP, whose state captures both task progress (goal type, step phase, stuck indicator, locations visited) and memory status (size, plan availability, learning phase). MemCon learns a policy over this MDP online, during deployment, using a lightweight tabular contextual bandit with UCB exploration [14]–[16]. The policy is warm-started from human-readable priors, updated via reverse-discounted credit assignment from binary task success, and persists to disk between tasks. Crucially, MemCon adds zero additional LLM calls: memory control is a millisecond-scale table lookup, not a second LLM invocation.

MemCon is not a new memory store. It is a thin wrapper that intercepts the abstract retrieve and store entry points of any existing memory backend and decides how to call them. The same wrapper applies unchanged to flat vector stores, skill libraries [8], summarisation-based memories [17], latent-token memories [10], and graph-structured memories [9], cleanly separating what is stored (backend) from how it is accessed (controller) and letting any future memory system plug in and inherit adaptive control.

We evaluate MemCon across 6 benchmarks covering both interactive decision-making (ALFWorld [18], PDDL planning, ScienceWorld [19]) and knowledge / web / tool-use QA (TriviaQA [20], WebWalkerQA [21], GAIA [22]); 3 agent frameworks (Lobster, LangGraph [23], Microsoft Agent-Framework); 3 LLM backbones (GPT-4.1-mini, Claude Sonnet-4, DeepSeek-V3.2); and compare against 9 strong memory baselines spanning vector retrieval (MetaGPT [24], MemoryBank [17]), skill libraries (Voyager [8]), trajectory summarization (ChatDev [25]), generative re-ranking (Generative [5], ExperienceBank), insight-based learning (OAgents [26]), graph-based hierarchical memory (G-Memory [9]), and latent-token memory (LatentMem [10]). Across all configurations, MemCon achieves the best or near-best task success—e.g., \(67.9\%\) ALFWorld on GPT-4.1-mini (Lobster), the top score among all 10 memories evaluated in that cell—and consistently delivers 5–\(30\)+ point gains over the no-memory baseline, while simultaneously reducing average token consumption per task by 5–20%.

Contributions.

1. We formalize agent memory management as a Memory MDP and introduce a backend-agnostic wrapper that decouples the memory control policy from the memory storage backend, so any existing or future memory system inherits adaptive control for free (§3.1, §3.3).

2. We propose an online contextual bandit policy with UCB exploration, warm-start priors, and reverse-discounted credit assignment that converges within tens of tasks using zero pretraining and zero extra LLM calls, together with augmented memory operations (§3.2, §[sec:augmented]).

3. We conduct an extensive evaluation of agent memory on six benchmarks and three agent frameworks, showing consistent accuracy gains and token savings, including a component ablation that isolates the learned controller’s contribution (§4).

2 Related Work↩︎

MemCon builds on three lines of work: (i) multi-agent reinforcement learning, (ii) multi-agent LLM systems, and (iii) agentic memory.

2.0.0.1 Reinforcement Learning for Memory and Tools.

Reinforcement learning has long been used to learn task policies from sparse reward, including value-decomposition (VDN [27], QMIX [28]), centralized-critic actor-critic (MADDPG [29], COMA [30]), and on-policy methods (MAPPO [31], IPPO [32]) on cooperative benchmarks such as SMAC [33]; learning to communicate [34] and sequence-modelling views [35], with surveys [36]–[39], broaden the design space. Closer to our setting, several recent works learn memory-management policies for LLM agents using deep RL or LLM-driven controllers [11]; we operate in a much lighter-weight regime, treating memory operations as actions of a single-agent contextual bandit [14], [15], [40] (a special case of tabular Q-learning [16], [41]) with episode-level Monte-Carlo updates. The resulting MDP is small enough to be learnable online without GPUs and without secondary LLM calls.

2.0.0.2 Multi-Agent LLM Systems.

Role-based LLM collaboration (AutoGen [42], CAMEL [43], MetaGPT [24], ChatDev [25], AgentVerse [44]), multi-agent debate [45], [46], and society-of-minds formulations [47], [48] improve reasoning over single-agent baselines, while more recent work optimizes the collaboration graph itself [49], [50]; production frameworks such as LangGraph [23] and Microsoft Agent-Framework supply the execution substrates we evaluate; see [51], [52] for surveys. These systems delegate cross-task learning to an external memory with fixed retrieval parameters. MemCon sits inside that memory module, so it is compatible with single-agent (Lobster), graph (LangGraph), and pipeline (Agent-FW) orchestration unchanged.

2.0.0.3 Agentic Memory.

Retrieval over stored experience is the dominant paradigm. Memory streams, long-term stores, and OS-style paging: Generative Agents [5], MemoryBank [17], MemGPT [11], Mem0 [12], MemLLM [53], Think-in-Memory [54], and memory-assisted prompt editing [55]. Skill- and procedure-centric memories: Voyager [8], MemP [13], Agent Workflow Memory [56], ProcMEM [57], MemSkill [58], HiAgent [59], and LatentMem [10]. Reflection- and rule-based memories: Reflexion [6], ExpeL [7], CLIN [60], OAgents [26], and the graph-structured G-Memory [9], our strongest baseline. These build on RAG [61]–[63]. Most of these systems access memory with fixed top-\(k\), hop, query, and consolidation schedule; the notable exception is MemGPT [11], which makes the access pattern adaptive by promoting the LLM itself to the role of controller, at the cost of additional LLM calls per memory operation. MemCon is orthogonal to both styles: it adds a learned-but-lightweight controller on top of any backend—deciding which operation, with what parameters, and when—without any secondary LLM calls.

3 Method↩︎

This section formalizes each component: the Memory MDP (§3.1), the online policy (§3.2), the backend-agnostic wrapper (§3.3), and two augmented memory operations tailored to long-horizon agentic tasks (§[sec:augmented]). Figure 1 gives a high-level overview of MemCon.

3.1 Problem Formulation: Memory MDP↩︎

Consider an LLM agent solving tasks \(\{\tau_1, \ldots, \tau_N\}\) sequentially. At each task \(\tau_i\), the agent interacts with an environment over \(T_i\) steps with access to a memory system \(\mathcal{M}\) (the backend). Most existing memory backends expose two abstract endpoints, a retrieve call (return up to \(K\) items relevant to a query) and a store call (write the trajectory of the just-finished task), together with optional maintenance hooks for consolidation and eviction. In standard usage these endpoints are invoked with hard-coded hyperparameters (number of items, search depth, query template, consolidation schedule); we instead model the choice of which endpoint to invoke and with what parameters as a sequential decision problem captured by an MDP \(\mathcal{M}_{\text{mem}} = (\mathcal{S}, \mathcal{A}, \mathcal{T}, \mathcal{R}, \gamma)\).

3.1.0.1 State \(\mathcal{S}\).

The state \(s = (s^{\text{task}}, s^{\text{mem}})\) fuses task-progress and memory-status signals: \[\begin{align} s^{\text{task}} &= (\texttt{goal\_type},\; \texttt{step\_phase},\; \texttt{is\_stuck},\; \texttt{objects\_held},\; \texttt{locations}), \\ s^{\text{mem}} &= (\texttt{mem\_size},\; \texttt{plan\_available},\; \texttt{learning\_phase}), \end{align}\] where \(\texttt{step\_phase}\) bins the step count, \(\texttt{is\_stuck}\in\{\text{True},\text{False}\}\) is set when the agent has emitted the same physical action twice in a row, \(\texttt{learning\_phase}\) indicates whether the agent has completed enough tasks to trust learned Q-values, and \(\texttt{plan\_available}\) records whether a distilled success plan exists for the current goal type. We discretize \(s\) into a compact hashable key \(\phi(s)\) for tabular learning; the exact bins, thresholds, and the discretization rule are given in Appendix 8. The discretization yields on the order of a few hundred distinct states per benchmark, enabling rapid online convergence.

3.1.0.2 Actions \(\mathcal{A}\).

Each action \(a = (\texttt{op}, \theta)\) specifies a memory operation together with its parameters: \[\label{eq:action-op} \texttt{op} \in \{\mathrm{\small Retrieve},\; \mathrm{\small PlanInject},\; \mathrm{\small Re-Retrieve},\; \mathrm{\small Consolidate},\; \mathrm{\small Forget},\; \mathrm{\small NoOp}\};\tag{1}\] \[\label{eq:action-theta} \theta = (\texttt{top\_k},\; \texttt{insight\_k},\; \texttt{hop}).\tag{2}\] Retrieve returns the top-\(\texttt{top\_k}\) items from the backend together with up to \(\texttt{insight\_k}\) derived rules (when supported) at search depth \(\texttt{hop}\); Re-Retrieve re-issues a retrieval with an alternative-approach query suffix, used to escape repeated-action loops; PlanInject prepends a generalised success plan when one is available (§[sec:augmented]); Consolidate and Forget call any maintenance hooks the backend exposes (e.g., merging or pruning derived rules) and silently no-op on backends that do not implement them; NoOp skips memory access for the current step. We instantiate \(\mathcal{A}\) as a small finite set of representative \((\texttt{op}, \theta)\) configurations spanning shallow-to-deep retrieval plus the maintenance and plan operations; the exact action set used in our experiments is listed in Appendix 8 (Table 6).

3.1.0.3 Reward \(\mathcal{R}\).

Rewards are observed at task end based on environment feedback: \[\label{eq:reward} r(\tau_i) \;=\; r_{\text{succ}}\cdot\mathbb{1}[\text{success}] \;+\; \lambda\cdot\max\!\left(0,\; 1 - T_i/T_{\max}\right) \;-\; r_{\text{fail}}\cdot\mathbb{1}[\text{failure}],\tag{3}\] where \(r_{\text{succ}}, r_{\text{fail}}, \lambda\) and the horizon \(T_{\max}\) are fixed scalars (values in Appendix 8, Table 5). Successful and more efficient trajectories therefore earn larger credit; failed trajectories earn negative feedback that discourages costly or counterproductive memory usage.

3.1.0.4 Transitions \(\mathcal{T}\).

The environment transition is driven by the LLM agent and is effectively opaque to the controller. We therefore treat \(\mathcal{M}_{\text{mem}}\) as a contextual bandit with episode-level (Monte-Carlo) feedback: even though the underlying problem is sequential, the controller never bootstraps a within-episode value estimate and instead receives one terminal-reward signal per task, which makes the formal regret analysis (Appendix 9) straightforward and bounds sample complexity to tens of tasks.

3.2 Online Policy Learning↩︎

We learn \(\pi: \mathcal{S} \to \mathcal{A}\) online during deployment, with no pretraining and no additional LLM calls.

3.2.0.1 Action selection via UCB.

At each memory decision point we apply the Upper Confidence Bound rule [14], [15], with the exploration bonus computed using the state-specific visit count: \[\label{eq:ucb-rule-main} a_t \;=\; \arg\max_{a\in\mathcal{A}} \left[\; Q(\phi(s_t), a) \;+\; c\sqrt{\frac{\ln N(\phi(s_t))}{N_a(\phi(s_t))}} \;\right],\tag{4}\] where \(c\) is the UCB exploration coefficient (Appendix 8, Table 5), \(N(\phi(s_t)) := \sum_{a\in\mathcal{A}} N_a(\phi(s_t))\) is the total number of decisions ever taken at state key \(\phi(s_t)\), and \(N_a(\phi(s_t))\) is the count of action \(a\) at that state. Unvisited actions receive an \(\infty\) bonus, forcing the first \(|\mathcal{A}|\) visits to each state to span all actions; warm-start priors (below) break ties among such \(\infty\) bonuses to give a sensible initial ordering. The same expression is used both in the implementation (with the per-state count) and in the theoretical analysis (Appendix 9, Eq. 14 ).

3.2.0.2 Warm-start priors.

Before any learning, Q-values are initialized with interpretable priors that encode mild domain knowledge: Retrieve and PlanInject receive positive priors (retrieval is usually helpful, plans are useful when available), Re-Retrieve a smaller positive prior (useful only when stuck), Consolidate a neutral prior, and Forget and NoOp mildly negative priors. Numerical values are listed in Appendix 8 (Table 5). Warm-starting accelerates convergence on the first few tasks, before any real reward has been observed.

3.2.0.3 Credit assignment.

After each task \(\tau_i\), all \((s,a)\) pairs visited during the episode are updated with a reverse-discounted Monte-Carlo return, so that actions taken closer to the terminal outcome receive stronger credit: \[\label{eq:q-update} Q(\phi_j, a_j) \;\leftarrow\; Q(\phi_j, a_j) \;+\; \alpha\left[\gamma^{|\text{ep}|-j-1}\cdot r_i \;-\; Q(\phi_j, a_j)\right],\tag{5}\] where \(\alpha\) is the step size, \(\gamma\in(0,1)\) is the within-episode discount, \(|\text{ep}|\) is the total number of memory decisions executed during episode \(\tau_i\), and \(j\in\{0,1,\dots,|\text{ep}|-1\}\) is the chronological index of the decision being credited (so the last decision receives the undiscounted \(r_i\) and earlier decisions receive geometrically attenuated credit). Numerical values for \(\alpha\) and \(\gamma\) are in Appendix 8. The Q-table is persisted to disk every few updates, so learning carries across runs and task streams.

3.3 Backend-Agnostic Wrapper and Augmented Operations↩︎

MemCon is a thin wrapper around any memory backend \(\mathcal{M}_{\text{inner}}\) that exposes a two-method interface: retrieve (return \(K\) items for a query) and store (write the finished trajectory with its success label); an optional maintain hook unlocks the consolidation and eviction actions, and is silently no-op’d otherwise. On retrieve, the wrapper builds \(s_t\), invokes the policy to pick \((\texttt{op}^*,\theta^*)\), calls \(\mathcal{M}_{\text{inner}}\) with those parameters (unsupported parameters are dropped), and optionally augments the result with one of two domain-agnostic augmented operations (described below); on store, it computes the episode reward, updates Q-values via Eq. 5 , updates the plan index, and delegates persistence to \(\mathcal{M}_{\text{inner}}\). Because the wrapper never reads or mutates the backend’s internal data structures, it is genuinely backend-agnostic and applies unchanged to flat vector stores [17], [24], skill libraries [8], summarisation-based memories [25], latent-token memories [10], and graph-structured memories [9]. The two augmented operations target failure modes in long-horizon interactive tasks: generalised plan injection (a learnable MDP action, PlanInject) extracts the action sequence of a successful task of type \(g\), replaces instance-specific identifiers (e.g.‘‘shelf 3’’ \(\to\) [shelf]) via a regex rewriter, stores the resulting template under \(g\) in a lightweight JSON index, and on future tasks of type \(g\) prepends the template to the backend’s retrieval output (the rewriter degrades to identity on new vocabularies); goal decomposition (a deterministic heuristic, not an MDP action) handles composite tasks requiring the same primitive twice with two objects (e.g., “put two cellphones on the desk”) by injecting “complete all steps for object 1 first, then repeat for object 2” and additionally retrieving the single-object template when one exists. Both augmentations consume only the action transcript and concatenate text to the retrieved context, so they are independent of the backend; the component ablation (§4.3, Table [tab:component-ablation]) shows that the learned controller alone already accounts for the majority of MemCon’s gains, making both augmentations useful but optional.

4pt

4 Experiments↩︎

We design our evaluation to answer four questions: (Q1) Does learned memory control improve over the best fixed-pipeline memory systems, across backends, benchmarks, frameworks, and LLM backbones? (Q2) Does MemCon deliver these gains while using fewer tokens per task, or do the improvements come from simply retrieving more? (Q3) Does MemCon generalize beyond interactive decision-making to QA and web/tool-use settings? (Q4) Is MemCon robust under different LLM backbones, including weaker (GPT-4.1-mini), strong proprietary (Sonnet-4), and open-source (DeepSeek-V3.2) models.

4.1 Setup↩︎

We evaluate on six benchmarks across two regimes, interactive decision-making (ALFWorld [18], PDDL Planning, ScienceWorld [19]) and QA/web/tool-use (TriviaQA [20], WebWalkerQA [21], GAIA [22]), under three agent runners (Lobster, LangGraph [23], Microsoft Agent-Framework) and three LLM backbones (GPT-4.1-mini, Claude Sonnet-4, DeepSeek-V3.2); see Appendix 6 for benchmark sizes, framework descriptions, and the shared task-loader / tool / evaluation protocol. Beyond the no-memory ablation (Empty), we compare against nine memory baselines re-implemented on a shared retrieve/store interface, spanning vector-similarity, summarised-trajectory, LLM-re-ranked, insight-based, latent-token, and graph-structured designs [5], [7]–[10], [17], [24]–[26], [60]; for our reported MemCon numbers the wrapper is plugged into one fixed inner backend (G-Memory [9]) so every comparison shares the same storage layer (the wrapper is backend-agnostic, §3.3). Per-baseline mechanism descriptions and hyperparameter-search details are in Appendix 6; numerical hyperparameter values are in Appendix 8 (Table 5). We report task success rate / answer accuracy as S/A (single deployment run per configuration, no seed averaging); per-task input tokens (Tok) are deferred to Appendix 7.

4.2 Main Results↩︎

Table 1 reports S/A on the GPT-4.1-mini backbone in the main paper; per-backbone S/A tables for Sonnet-4 and DeepSeek-V3.2, together with the average per-task token cost, are deferred to Appendix 7 (Tables 2, 3, and 4). Figure 3 visualises the joint S/A vs.token-cost trade-off, and Figure 2 traces two end-to-end episodes through the controller. We highlight four observations corresponding to Q1–Q4.

(Q1) MemCon is consistently the strongest or near-strongest memory. On GPT-4.1-mini (Table 1), MemCon attains the top S/A on 4 of 6 benchmarks under Lobster, 2 under LangGraph, and 3 under Agent-FW, 9 of 18 cells overall, with concrete examples such as \(67.9\%\) ALFWorld (Lobster, the top score among all 10 memories), \(40.0\%\) PDDL (Agent-FW, \(+3.3\) over the strongest fixed-pipeline baseline in that cell), and \(22.4\)–\(23.0\%\) GAIA across frameworks against \(16.4\)–\(21.2\%\) for the rest. On Sonnet-4 (Table 2), where memory quality becomes decisive because the base model is strong enough to execute any sensible plan, MemCon is top S/A on 15 of 18 framework\(\times\)benchmark cells, including the entire interactive block (e.g., \(68.9\%\) Lobster-PDDL vs.\(67.0\%\) best baseline and \(18.0\%\) no-memory; \(67.1\%\) Agent-FW-SciWorld vs.\(66.0\%\) best baseline and \(17.0\%\) no-memory). The three exceptions are narrow (\(\le\!0.4\) points each). On DeepSeek-V3.2 (Table 3), MemCon is top S/A on all 9 interactive cells (e.g., \(86.7\%\) Lobster-ALFWorld vs.\(84.3\%\) next-best) and 15 of 18 cells overall; the GAIA column is the only one where another baseline occasionally edges MemCon by \(0.3\)–\(0.4\) points. Fixed-pipeline baselines are inconsistent across settings (some are strong on a few splits but catastrophically weak elsewhere), whereas MemCon is the only system that is uniformly top-1 or top-2 across all 54 cells.

(Q2) Gains come with token savings, not extra retrieval. A naive explanation of MemCon’s accuracy gains would be “it retrieves more, hence uses more context.” Figure 3 and Table 4 refute this directly: on GPT-4.1-mini Lobster–ALFWorld, MemCon uses \(39\)K tokens per task vs.\(45\)K for the strongest fixed-pipeline baseline (\(-13\%\)) while lifting S/A from \(59.7\%\) to \(67.9\%\); on Agent-FW–ALFWorld it uses \(37\)K vs.\(43\)K (\(-14\%\)) at \(69.4\%\) S/A; on Sonnet-4 Agent-FW–PDDL it uses \(60\)K vs.\(67\)K (\(-10\%\)) at \(70.7\%\) S/A; and on DeepSeek-V3.2 Lobster-ALFWorld it uses \(57\)K vs.\(67\)K (\(-15\%\)) at \(86.7\%\) S/A. Across all three backbones we see simultaneous 5–20% token reductions and accuracy improvements, consistent with the policy learning to suppress retrieval when memory is sparse or irrelevant and to replace nearest-neighbour retrieval with plan injection when a distilled template already exists (§[sec:augmented]).

(Q3) Generalization to QA / web / tool-use benchmarks. MemCon is not specifically designed for question answering, yet the same controller yields improvements on TriviaQA, WebWalkerQA, and GAIA. On GPT-4.1-mini GAIA, MemCon reaches \(22.4\)–\(23.0\%\) across frameworks vs.\(16.4\)–\(20.6\%\) for the no-memory baseline and \(20.0\)–\(21.2\%\) for the strongest fixed-pipeline alternative. On Sonnet-4 GAIA, MemCon attains \(27.5\)–\(28.9\%\), beating the next-best memory by \(3\)–\(12\) points. The relatively flat TriviaQA gaps (a short-answer benchmark where memory is inherently less important) serve as a sanity check that MemCon does not degrade easy settings while it lifts hard ones.

(Q4) Backbone-robust gains, most pronounced under stronger LLMs. Across the three backbones the ordering is consistent: MemCon is competitive-to-dominant under GPT-4.1-mini, strictly dominant under DeepSeek-V3.2 (top S/A on every interactive cell), and most strongly differentiated under Sonnet-4 (top on \(15/18\)). On Sonnet-4 Agent-FW, ALFWorld improves from an Empty baseline of \(12.7\%\) to \(60.6\%\) with MemCon, PDDL from \(21.0\%\) to \(70.7\%\), and ScienceWorld from \(17.0\%\) to \(67.1\%\). With stronger LLM execution, the binding constraint shifts from reasoning capability to whether the right experience is surfaced at the right time, which is precisely what MemCon’s adaptive controller addresses. Figure 2 illustrates this end-to-end on two qualitatively different queries: one where the controller starts in the cold phase and recovers via Re-Retrieve, and one where a learned plan template lets PlanInject solve the task in a single pass.

Overall, these results support the central claim of the paper: replacing a fixed memory pipeline with a lightweight learned controller yields consistent, framework-/benchmark-/backbone-agnostic accuracy gains, and does so with fewer rather than more tokens.

4.3 Ablation↩︎

We probe two questions: (i) how sensitive is MemCon to each policy hyperparameter, and (ii) how much of MemCon’s gain over the underlying static-pipeline backend comes from the learned controller versus the two augmented memory operations of §[sec:augmented]? We answer (i) with a single-knob sweep (Figures 4–[fig:hp-ablation-bottom]) and (ii) with a four-row component ablation (Table [tab:component-ablation]).

The component ablation (Table [tab:component-ablation]) addresses a confound flagged in earlier reviews: most of the GPT-4.1-mini Lobster-ALFWorld gain over the static-pipeline backend is attributable to the learned UCB controller (\(+5.2\) S/A) rather than to the two augmented operations (\(+1.5\) each). On GAIA, which contains no composite goals of the form targeted by goal decomposition, only the learned controller contributes (\(+1.2\) S/A); the augmented operations add no accuracy because they simply have no opportunity to fire. This confirms that the central methodological contribution (learning when, what, and how much to retrieve) is the dominant source of the empirical gains, and that the augmented operations are useful but optional add-ons whose benefit is restricted to environments with structured composite goals.

5 Conclusion↩︎

We introduced MemCon, a backend-agnostic framework that treats agent memory not as a fixed retrieval pipeline, but as a controlled decision process. By modeling memory operations as actions in a Memory MDP and learning a lightweight online UCB policy, MemCon adaptively decides when to retrieve, reuse plans, re-retrieve, consolidate, or skip memory access without requiring pretraining or additional LLM calls. Across interactive, QA, web, and tool-use benchmarks, MemCon consistently improves task success across multiple agent frameworks and LLM backbones while also reducing token consumption. These results suggest that effective long-term memory for LLM agents depends not only on what is stored, but also on learning how memory should be accessed and managed over time.

6 Experimental Setup Details↩︎

This appendix expands the abbreviated description in §4.1 with full benchmark sizes, framework descriptions, baseline mechanisms, and the hyperparameter-search protocol.

6.1 Benchmarks↩︎

6.1.0.1 Interactive decision-making (long-horizon, multi-step).

ALFWorld [18] contains 134 household tasks across 6 types (put, clean, heat, cool, examine, puttwo); it couples a TextWorld front-end with embodied AI2-THOR back-ends. PDDL Planning consists of 100 classical planning tasks spanning blocksworld, barman, gripper, tyreworld, closely related to PlanBench [64]–[66]. ScienceWorld [19] contains 100 elementary-science experimental tasks (boiling water, melting ice, finding animals by property, etc.).

6.1.0.2 Knowledge / web / tool-use QA.

TriviaQA [20] (200 questions, short-answer open-domain QA); WebWalkerQA [21] (200 multi-hop web traversal questions requiring site navigation and information extraction); GAIA [22] (165 general-AI-assistant questions spanning tool use, web search, and document reading).

6.2 Agent Frameworks↩︎

We use three architecturally distinct agent runners, each sharing identical task loaders, tools, and evaluation protocol so that the only varying factor across baselines is the memory system: Lobster, a single-agent minimalist runner that talks to the OpenAI-compatible chat API directly; LangGraph [23], a graph-structured multi-agent workflow built on LangChain; and Microsoft Agent-Framework, a pipeline-based multi-agent system. All experiments use a 30-step horizon for interactive tasks and the benchmark-default budget for QA, with the same temperature and tool protocol across memory baselines.

6.3 Memory Baselines↩︎

Beyond the no-memory ablation (Empty), we compare against nine memory systems re-implemented on a shared two-method (retrieve/store) interface so that differences are purely algorithmic rather than due to prompt or I/O variation: (1) MetaGPT [24]: pure vector similarity retrieval over stored trajectories, no LLM calls at retrieval time; (2) Voyager [8]: LLM summarises each trajectory before storage, retrieval by cosine similarity over summarised embeddings; (3) Generative [5]: retrieves candidate trajectories and uses an LLM to re-rank them by estimated relevance; (4) ChatDev [25]: periodic LLM-driven phase-based summarisation every 10 steps; no cross-task retrieval; (5) MemoryBank [17]: temporal-decay memory with Ebbinghaus-style exponential forgetting on top of vector retrieval; (6) OAgents [26]: insight-based learning that compares paired successful/failed trials to distil and edit natural-language rules; (7) ExperienceBank: LLM-scored relevance retrieval with generative re-ranking over an explicit experience bank [7], [60]; (8) LatentMem [10]: distils cross-task experience into a small bank of learnable latent tokens, paired with a graph-based retrieval front-end and LatentMem-specific insight prompts; (9) G-Memory [9]: a graph-structured memory combining vector retrieval, a task graph with \(k\)-hop traversal, and scored insight rules with periodic LLM-driven merge.

For our reported MemCon numbers we plug the wrapper into one specific backend (G-Memory) so that every MemCon–baseline comparison is held fixed at the inner storage layer; because the wrapper is backend-agnostic (§3.3), it could equally be composed with any of the other eight, and we expect the qualitative trend (controller adds adaptivity at zero LLM cost) to transfer.

6.4 Hyperparameter Selection Protocol↩︎

MemCon uses the default 9-action set (§3.1), warm-start enabled with the priors of §3.2, and the reward shape of Eq. 3 . The three policy hyperparameters \((\alpha,\gamma,c)\) are selected per backbone via a small grid search on a held-out 30-task subset of ALFWorld (Lobster), independent of the evaluation tasks; warm-start priors, reward constants, and discretisation thresholds are fixed across backbones. All numerical values are listed in Appendix 8 (Table 5). The Q-table and success-plan index are persisted to disk and reloaded across runs; experiments use a single realistic deployment run per configuration (no seed averaging) to reflect the online setting.

7 Per-Backbone S/A Tables and Token Cost↩︎

This appendix reports the full per-backbone S/A tables for the two backbones omitted from the main paper (Claude Sonnet-4 and DeepSeek-V3.2) and provides the average per-task token cost (Tok) accompanying each S/A entry, for all three backbones.

7.1 Claude Sonnet-4 (S/A only)↩︎

4pt

7.2 DeepSeek-V3.2 (S/A only)↩︎

4pt

7.3 Token Cost (Tok per task)↩︎

Table 4 reports the average number of LLM-input tokens consumed per task for the GPT-4.1-mini main results (numbers for the other two backbones are available in the released code; trends are qualitatively identical). The token-cost vs.S/A trade-off is summarized visually in Figure 3 of the main paper.

4pt

8 Hyperparameters and Implementation Details↩︎

This appendix lists the full set of hyperparameters used by MemCon across all experiments. Table 5 consolidates every numeric constant referenced in §3–§4 into one place; Table 6 gives the structural action set \(\mathcal{A}\) which has a different layout. All values are taken directly from the released implementation, and the same defaults are used for every number reported in the main paper unless otherwise noted.

8.0.0.1 Discretised state key.

The MDP state key used by the controller is \[\label{eq:phi} \begin{align} \phi(s) = \langle\,& \texttt{goal\_type},\, \texttt{step\_phase},\, \texttt{is\_stuck},\, \min(\texttt{hold},2),\, \min(\lfloor\texttt{visited}/3\rfloor,4), \\ & \min(\lfloor\texttt{mem\_size}/10\rfloor,5),\, \texttt{plan\_available},\, \texttt{learning\_phase} \,\rangle . \end{align}\tag{6}\] with the bin definitions in Table 5. This coarse discretisation yields on the order of a few hundred distinct state keys per benchmark and enables tabular learning in tens of tasks.

8.0.0.2 Action space.

Table 6 lists the structural action set \(\mathcal{A}\) used in all main-paper experiments (preset "default"). Additional presets ("retrieval_heavy", "plan_first", "compact") are available in the released code; the action-preset ablation in §4.3 (right panel of Figure [fig:hp-ablation-bottom]) shows that the default preset wins on both QA benchmarks.

8.0.0.3 LLM, agent framework, and benchmark settings.

Agent frameworks use their default configurations, modified only to share identical task loaders and evaluators. Interactive benchmarks: ALFWorld (134 tasks, 6 types), PDDL (100 tasks across blocksworld/barman/gripper/tyreworld), ScienceWorld (100 tasks). QA benchmarks: TriviaQA (200 questions), WebWalkerQA (200), GAIA (165). Each reported number is a single realistic deployment run, with the Q-table persisted across tasks; LLM temperature, run protocol, and seed handling follow the values listed in Table 5.

9 Theoretical Justification↩︎

This appendix formalizes and proves the learning guarantees of MemCon. We show that (i) the Memory MDP decomposes into a family of per-state stochastic bandits (Lemma 1); (ii) the UCB rule used in MemCon admits a sub-Gaussian concentration inequality (Lemma 2) yielding an \(O(\log n)\) per-state regret (Theorem 1) and an \(O(|\Phi||\mathcal{A}|\log T)\) global regret (Corollary 1); (iii) the reverse-discounted Q-update is a Robbins–Monro stochastic approximation whose iterates converge almost surely (Theorem 3) and in \(L^2\) (Proposition 4) to the true action value; and (iv) the greedy policy extracted from \(Q\) is asymptotically optimal (Theorem 6). All results are stated and proved from first principles; standard references are [].

9.1 Preliminaries and Notation↩︎

9.2 Decomposition of the Memory MDP↩︎

Lemma 1 reduces the Memory MDP to \(|\Phi|\) independent stochastic bandits and allows us to upper-bound \(\mathcal{R}_T\) state by state.

9.3 Concentration of the Empirical Mean↩︎

Setting \(\varepsilon = c\sqrt{\ln N^{(t)}(\phi) / m}\) with \(c=W/\sqrt{2}\) in 9 gives the familiar UCB confidence radius: \[\label{eq:ucb-radius} \Pr\!\left[\,\left|\hat{\mu}_t(\phi,a) - \mu(\phi,a)\right| \ge c\sqrt{\tfrac{\ln N^{(t)}(\phi)}{m}}\,\right] \;\le\; 2\,\bigl(N^{(t)}(\phi)\bigr)^{-4}.\tag{13}\]

9.4 Regret Bound for UCB-Based Selection↩︎

MemCon selects actions by maximizing the upper-confidence index \[\label{eq:ucb-rule} a_t \;\in\; \arg\max_{a\in\mathcal{A}}\;\underbrace{\left[\; \hat{\mu}_t(\phi_t,a) \;+\; c\sqrt{\tfrac{\ln N^{(t)}(\phi_t)}{N_a^{(t)}(\phi_t)}}\;\right]}_{=:\;U_t(\phi_t,a)}, \qquad c=1.4 \approx W/\sqrt{2}.\tag{14}\]

9.5 Convergence of the Reverse-Discounted Q-Update↩︎

The MemCon update rule on visited \((\phi_j,a_j)\) pairs after episode \(i\) is \[\label{ylhdsowm} Q_{t+1}(\phi_j,a_j) \;=\; Q_t(\phi_j,a_j) \;+\; \alpha_t\bigl[G_j - Q_t(\phi_j,a_j)\bigr], \qquad G_j \;:=\; \gamma^{|\mathrm{ep}|-j-1}\cdot r_i,\tag{23}\] where \(\alpha_t\in(0,1)\) is the step size (constant \(\alpha_t\equiv\alpha=0.15\) in our experiments).

9.6 From Action Values to Policy Optimality↩︎

9.7 Warm-Start and the Finite-Sample Regime↩︎

In practice, the warm-start values encode the domain knowledge that Retrieve and PlanInject usually help while NoOp usually hurts (see §8). By Proposition 7 this shifts effective exploration toward untried or dis-favored arms in the early regime, which matches the \(\lesssim 30\)-episode convergence observed empirically (Remark 2).

10 Prompt Templates↩︎

This appendix lists every prompt string used by MemCon and its evaluation harness. We group prompts into: (C.1) benchmark solver system prompts shared across all memory baselines, (C.2) MemCon-specific prompts injected by the wrapper, and (C.3) baseline-specific memory-write/read prompts, for completeness.

10.1 Benchmark Solver System Prompts↩︎

All memory methods (including MemCon) share the identical solver system prompt per benchmark, so that differences in reported performance isolate the memory component.

10.1.0.1 ALFWorld solver.

10.1.0.2 ScienceWorld solver (excerpt).

10.1.0.3 PDDL solver (Blocksworld example).

Analogous solver system prompts are used for barman, gripper, and tyreworld (see tasks/prompts/pddl_prompt.py in the released code).

10.1.0.4 TriviaQA / WebWalkerQA / GAIA.

QA tasks follow benchmark-standard short-answer or multiple-choice prompts (“Answer: X” format for MCQA, a single concise string for open-answer). We do not modify the benchmark-provided prompts.

10.2 MemCon-Specific Prompt Fragments↩︎

The MemCon wrapper never adds a second LLM call, so it injects only two types of strings into the retrieved context.

10.2.0.1 Generalized plan injection.

Here each step is produced by generalizing a successful trajectory: numbered location and object instances are replaced with categorical placeholders (e.g., "go to shelf 3" \(\to\) "go to [shelf]", "take mug 1 from diningtable 2" \(\to\) "take [mug] from [diningtable]"). The regular-expression rewriter covers the standard ALFWorld/ScienceWorld vocabulary (shelves, cabinets, drawers, containers, common foods, tools, etc.).

10.2.0.2 Goal decomposition (multi-object tasks).

The second block reuses the generalized plan for the simpler single-object variant (put) when one has been learned, providing an additional hint to the LLM agent.

10.3 Baseline Memory-Module Prompts↩︎

For full reproducibility we summarize the memory-write/re-rank prompts used by the baselines we re-implemented on top of the shared two-method memory interface. Full text of these prompts is in the released code.

10.3.0.1 Voyager / MemoryBank trajectory summarizer.

10.3.0.2 Generative / ExperienceBank relevance scorer.

10.3.0.3 ChatDev phase-based summarizer.

10.3.0.4 OAgents rule induction.

10.3.0.5 G-Memory / LatentMem.

Both use a two-stage pipeline: a LatentMem-specific insight extraction prompt (distinct per method but structurally similar to the OAgents rule-induction prompt) plus Chroma-vector retrieval over raw trajectory embeddings. The exact strings are long and exist verbatim in the referenced upstream repositories; we do not reproduce them here but note that they are identical to the ones used in [9] and [10] respectively.

References↩︎