1 Introduction↩︎

Language agents are defined by a loop: observe, reason, act, repeat. Yet the memory they reason over is typically treated as something outside that loop—a database the agent queries once per turn and otherwise leaves alone. This paper asks what happens when memory moves inside the loop: when an agent can read and write an associative store on every step of its reasoning, as cheaply as it accesses its own context window.

We call this regime memory in the loop, echoing—and extending—the familiar “human in the loop.” The obstacle has always been latency. A networked vector store answers in \(50\)–\(200\,\)ms (§2); an agent that consults it at every step pays that cost repeatedly, and recent work shows in-loop retrieval can inflate end-to-end latency by up to \(83\times\) [1]. The field has answered on two fronts. Systems work keeps retrieval in the loop and hides its cost at the serving layer: SearchAgent-X [1] schedules requests by priority and makes retrieval non-stalling. Industry “memory-first” guidance instead moves memory out of the loop, into a layer queried once at the start of a turn and updated at the end [2].

Both answers take the store’s latency as given. We argue it is an assumption, not a law. The latency that makes in-loop retrieval prohibitive is a property of where the store lives, not of the in-loop pattern itself. An in-process store answers in \({\sim}100\,\mu\mathrm{s}\)—three orders of magnitude below the network regime—and at that speed the amplification collapses: the entire per-turn in-loop tax measures \({\sim}1.7\,\)ms (Table 2). More precisely, the per-turn network tax is \(S\times\text{RTT}\), linear in how often the agent touches memory, which is why a networked store forces agents to ration retrieval; an in-process store drives that tax to \({\sim}0\) and removes the rationing. The tradeoff the efficiency literature treats as fundamental dissolves; this substrate-level answer complements the serving-layer one rather than opposing it (§3). And the premise is causal, not correlational: holding a per-turn memory budget fixed and varying only store latency, the task outcome itself flips (§6).

This is more than an optimization. We ground it in the extended-mind thesis [3]: an external resource becomes constitutive of cognition only when it is constantly available, directly accessible without difficulty, and automatically endorsed upon retrieval. A \(100\,\)ms network call is a tool the agent consults; a \(100\,\mu\mathrm{s}\) in-process store is always there—it clears the latency bar the parity principle sets, and in a loop wired to consult it becomes genuine extended working memory rather than an external database (Figure 3). Latency decides whether a store is eligible to be part of the agent’s mind; the loop’s wiring decides whether it actually is (§4).

This is the first step in a broader line of work on memory as a cognitive resource for language agents. It establishes when memory can participate in reasoning: on every step, once retrieval is cheap. What an agent should retain, and how it should organize what it retains, are the natural next questions (§11).

1.0.0.1 Contributions.

(1) We lift retrieval frequency (per-turn vs.per-step) from a serving-layer knob in RAG systems [4], [5] to an agent-level design axis, and show that store latency is what gates it (§3). (2) We reinterpret the parity principle as an engineering criterion—a latency budget—and argue that in-process latency makes external memory eligible as constitutive working memory, which a loop wired to consult it then realizes (§4). (3) We show store latency is causal to task outcome, not merely to wall-clock (§5–§6): under a fixed per-turn memory budget, a scripted loop guard flips from \(0\) to \(10/10\) redundant actions, and a real-LLM guard traces a monotone dose-response—\(0.0\) redundant at in-process speed, \(1.4\)–\(1.6\) at \(+15\,\)ms, \(7.2\) of \(12\) at \(+110\,\)ms where no lookup is affordable (two models, five seeded workloads per rung, exact permutation \(p{=}0.0079\), zero guard errors). (4) We demonstrate the regime end-to-end, against the baselines a reviewer would demand: across all four models of a GPT-5 ladder under a bounded window, recall improves from \(0/5\) (every baseline and window-aware run) to \(3.6\)–\(4.8/5\) with in-loop memory, store ops live at p50 \(80\)–\(165\,\mu\mathrm{s}\) (§6). An instructed restate-every-reply baseline reaches \(5/5\) on this five-fact task, beating the memory tools here; we report that result plainly and show why restatement stops scaling: it pays a per-turn cost that grows with the working set, exactly what the store avoids (§6). Every observed miss traces to the agent’s read policy, never to the store. The remaining bottleneck, network embedding, closes to a measured \({\sim}40\,\mu\mathrm{s}\) complete operation with a small local embedder (§7).

2 Background↩︎

2.0.0.1 Working memory, from cognition to context windows.

The Baddeley–Hitch model—a central executive coordinating limited-capacity buffers, later extended with an episodic buffer [6], [7]—is the standard lens. Applied to language agents, the LLM plays the central executive and the context window plays the buffer; its capacity limit motivates externalization. Growing the window does not close the question, for three size-independent reasons. Rent: every window token is reprocessed on every step, so context is paid for repeatedly, while an external store holds facts at no recurring cost and charges microseconds only for what is fetched. Findability: models reliably miss facts buried in the middle of long contexts [8], whereas associative retrieval goes to the fact by meaning. The cliff: a long-lived agent eventually exceeds any window, and the failure is abrupt—our window\(\times\)facts sweep (§10) locates task failure exactly where the fact span exceeds the window, across three task families. The window is the working surface; it cannot also be the archive. The move has a biological precedent: Ericsson and Kintsch’s long-term working memory [9] shows experts extending working capacity through fast, reliable retrieval structures in long-term memory—and there, as here, the criterion that makes the extension work is retrieval speed.

2.0.0.2 The agent loop and CoALA’s internal actions.

CoALA [10] frames a language agent as modular memory (working memory plus episodic/semantic/procedural long-term memory) acted on by three internal actions: reasoning (update working memory), retrieval (read long-term memory), and learning (write long-term memory). CoALA fixes retrieval as an action but is silent on how often it can fire; we make that the central variable.

2.0.0.3 Two latency regimes.

Retrieval latency clusters into two regimes that differ by orders of magnitude: networked/disk (cloud vector DBs \({\sim}110\,\)ms—a representative cross-region round trip, which we reproduce against a live Qdrant in §6; RAG \(50\)–\(200\,\)ms) and in-process/in-memory (embedded stores report \({\sim}0.25\,\)ms local scans [11]; our live store ops measure p50 \(80\)–\(165\,\mu\mathrm{s}\), §6). Around them sit the network embedding API (\({\sim}200\)–\(400\,\)ms, measured in §6) and the LLM reasoning step itself (\({\sim}1\,\)s). The gap between these regimes is the substance of the argument.

3 The Memory-in-the-Loop Thesis↩︎

3.0.0.1 Definition (retrieval frequency).

Let a turn be one user/environment interaction and a step be one reasoning iteration within a turn (an LLM call). Per-turn retrieval fetches at most once per turn (classic RAG). Per-step retrieval (memory in the loop) may read and write the store at every step (Figure 1).

3.0.0.2 The amplification argument and its inversion.

[1] show that with in-loop retrieval, end-to-end latency scales with per-step retrieval latency and is amplified (up to \(83\times\)) relative to per-turn RAG, because each step blocks on retrieval. Their system, SearchAgent-X, keeps retrieval interleaved and removes the blocking at the serving layer: priority-aware scheduling and non-stall retrieval over high-recall approximate search. We attack the same term one level down. Their measurements are taken in the network/disk regime, where the amplification term \(S\cdot t_\text{store}\) is large because \(t_\text{store}\) is large. Reduce \(t_\text{store}\) from \({\sim}10^2\)–\(10^3\,\)ms to \({\sim}10^{-1}\,\)ms (in-process) and the term becomes negligible with no scheduling machinery at all. Both fixes keep memory inside the loop; they compose. The prescription we reject is the industry “memory-first” pattern that moves memory out of the loop into a service queried once per turn [2].

3.0.0.3 Cost model.

End-to-end latency is approximately \[\text{E2E} \;\approx\; \sum_{\text{steps}} \bigl(t_\text{reason} + f\cdot(t_\text{embed} + t_\text{store})\bigr),\] where \(f\) is the per-step retrieval frequency. In-loop memory is viable iff \(f\cdot(t_\text{embed} + t_\text{store}) \ll t_\text{reason}\). §6 measures \(t_\text{store}\approx 100\,\mu\mathrm{s}\) (in-process) vs.\(t_\text{store}+\text{RTT}\) (network), and confirms \(t_\text{embed}\) (\({\sim}200\)–\(400\,\)ms over the network) is the dominant residual term.

3.0.0.4 Feasibility frontier.

Fix a budget share \(\beta\): memory may add at most a fraction \(\beta\) of end-to-end time. With per-retrieval cost \(c = t_\text{embed} + t_\text{store}\) and per-step reasoning time \(r\), the affordable per-step frequency is \[f_{\max} \;=\; \frac{\beta}{1-\beta}\cdot\frac{r}{c}.\] Measured, at \(\beta{=}0.1\) and \(r{=}1\,\)s: an in-process store with a local embedder (\(c\approx116\,\mu\mathrm{s}\)) affords \(f_{\max}\approx\mathbf{953}\) retrievals per step; the same store behind a network embedder (\(c\approx202\,\)ms) affords \(\mathbf{0.55}\); a cloud vector store plus network embedder (\(c\approx312\,\)ms) affords \(\mathbf{0.36}\). Below \(1\), an agent cannot afford even one lookup per step—rationing is forced arithmetic, not a design taste. Near \(10^3\), per-step access is effectively free (Figure 2).

3.0.0.5 What per-step memory unlocks.

Once a read/write is effectively free relative to an LLM step, capabilities become practical that per-turn retrieval cannot afford: (i) working memory—offload findings and recall the relevant ones at each step, decoupling effective context from window size; (ii) deduplication/novelty—check every observation against what has been seen; (iii) loop/action guards—check before each action whether an equivalent one was already taken; (iv) per-step grounding—verify claims against retrieved support as produced.

4 Memory as Constitutive Cognition: the Parity Principle as an Engineering Criterion↩︎

The engineering contribution here does not depend on any philosophical reading. The latency measurements, the feasibility frontier, and the causal results stand on their own. We invoke the extended-mind thesis not as a metaphor but as motivation: if cognition runs on the interaction between reasoning and continuously available memory, the long separation between language models and external memory looks like a technological limit rather than a cognitive necessity. Set the philosophy aside and the engineering result is unchanged.

The extended-mind thesis [3] holds that an external resource can be a constitutive part of a cognitive process, not merely an input to it. Its criteria are the ones that make Otto’s notebook part of Otto’s memory rather than a reference he consults: the notebook is “a constant in Otto’s life,” its information is “directly available without difficulty,” and upon retrieving information “he automatically endorses it” [3] (a fourth criterion, past conscious endorsement, they float only tentatively).

We read the first two criteria—constancy and direct availability—as a latency budget: they are satisfied to the degree that access is fast enough to be transparent to the ongoing process. For an agent, an in-process store at \({\sim}100\,\mu\mathrm{s}\) is below the threshold of a reasoning step and is effectively transparent; a networked store at \({\sim}100\,\)ms interrupts the step and is consulted as an external service (Figure 3). The third criterion, automatic endorsement, is a property of the loop rather than the store, and we return to it below. This yields the paper’s central claim, stated as an engineering criterion rather than a metaphor:

An external memory is eligible as constitutive working memory for an agent to the extent that a read/write is cheap relative to a reasoning step, and becomes so when the loop is wired to consult it. Latency is what moves a store across the parity threshold—from a database the agent connects to, into memory the agent has.

4.0.0.1 Endorsement is the loop’s job, not the store’s.

Cheap access is necessary, not sufficient. Our own transcripts supply the counterexample: the agents of §6 write to their \(100\,\mu\mathrm{s}\) stores throughout the task (up to \(17\) items) yet read them exactly once per run, through a fixed \(k{=}8\) window—full availability, thin coupling—and every miss in Table 1 is a fact the store held that this single bounded read never surfaced. Clark and Chalmers’ third criterion fails there, and it fails in the loop’s wiring, not in the store’s latency. The loop guard of §5.1 satisfies endorsement by construction: scaffold code consults the store before every action, so retrieval happens and its answer is acted on, on every step. The division of the criteria is precise: constancy and direct availability (criteria 1–2) are latency properties a store either has or lacks, but Clark and Chalmers’ criterion 1 also carries a usage clause—Otto “rarely takes action without consulting” the notebook—which, like automatic endorsement (criterion 3), is a wiring property the loop either implements or leaves to the model’s tool-calling disposition. Latency buys availability; the loop’s wiring buys consultation and endorsement.

4.0.0.2 The threshold is indexed to the step, and that is the claim.

“Cheap relative to a reasoning step” makes the boundary of working memory relative to the cognizer’s cycle time, and we take that horn deliberately. Working memory has always named a role relative to a processing cycle, not an absolute latency band. Run a human through Equation 2: with deliberate acts near \(r{\approx}1\,\)s and access costs of \(50\)–\(200\,\)ms, \(f_{\max}\) at \(\beta{=}0.1\) lands at \({\approx}0.6\)–\(2\)—right at the \(f{=}1\) cliff, which matches the psychological facts: human working-memory access is rationed, capacity-limited, and attention-gated. The in-process store sits three orders of magnitude above the same cliff for an LLM agent. One criterion classifies both cognizers correctly, and it predicts that a \(110\,\)ms store is working memory for a slow enough cognizer—a prediction we accept, since Figure 2 plots exactly this dependence on \(r\). Two checks guard against over-reading the \(\beta{=}0.1\) point. The ordering is \(\beta\)-invariant: \(f_{\max}\) carries the common factor \(\beta/(1-\beta)\) for every stack, so the human-at-the-cliff, in-process-far-above classification holds at any budget share, not only a tenth. And the same criterion places Otto’s own notebook correctly—paper access on the order of seconds sits below the \(f{=}1\) cliff at any \(\beta\), so our latency reading calls the notebook extended long-term memory, not working memory, which is exactly Clark and Chalmers’ own classification of it.

4.0.0.3 Relation to the Library Theorem.

The closest prior treatment of extended cognition for tool-using agents, Mainen’s Library Theorem [12], endorses the parity principle for external stores and, in the same section, rejects the working-memory analogy for context windows: “the mechanisms differ too deeply for that comparison to carry weight.” We concede the mechanism disanalogy in full—no attentional gating, no rehearsal, no phonological loop is claimed here. “Working memory” in this paper names the functional slot in the agent architecture (the sense the field already uses, as in CoALA [10]), not the human mechanism. The two accounts then compose rather than compete: Mainen prices access in page reads and makes organization the gate; we price it in wall-clock and make latency the gate; and his discipline condition—the store must be authoritative, actually consulted rather than bypassed by a model that “already knows”—is the endorsement requirement above.

4.0.0.4 The coupling-constitution objection.

Adams and Aizawa’s classic objection [13] is that no measurement of coupling—speed, reliability, tightness—can show that a resource constitutes part of a cognitive process rather than merely feeding it. We do not claim otherwise. The claim here is functional role parity at the step level: at the grain of description where the field already counts a context window as the agent’s working memory, a store read and written within the same step, at a cost invisible at that grain, occupies the same functional slot. The claim is conditional on that functionalist reading, which the agent-memory literature presupposes whenever it uses the term. A reader who rejects the extended-mind gloss wholesale loses nothing that is measured: every latency number, the feasibility frontier, and the causal results of §6.1 stand on their own as engineering.

Prior work invokes extended cognition descriptively; we use it prescriptively: parity implies a latency target, and meeting it changes the agent’s cognitive boundary, not merely its benchmark scores. The claim is functional, not phenomenal, throughout.

5 Method↩︎

We instantiate in-loop memory as an ephemeral, in-process semantic store and measure its effect on a multi-turn agent under bounded context. The reproduction artifact—Docker build, experiment scripts, and per-run JSON records behind every number in this paper—is available from the author and accompanies the public release.

5.0.0.1 Primitive.

WorkingMemory wraps an in-memory vector store: add, recall(query,k), match(text,threshold), and close. It is allocated, not connected to. The store op is instrumented to record per-call latency, separated from embedding latency.

5.0.0.2 Task and context pressure.

A six-turn trip-planning conversation in which the user states five durable constraints early, then asks the agent to list them all. A windowed session caps visible history to the last \(w\) messages (\(w{=}4\)), so early constraints scroll out—a controlled simulation of finite context. The choice of \(w\) is inside the failure region by design: the window sweep of §10 scans \(W\in\{1,2,3,4,6\}\) against facts required \(D\) and locates the boundary at \(D>W\) (Figure 7). Note the baseline scoring \(0/5\) under this design measures the designed floor—the constraints leave the visible window before the recall question—rather than model capability; the informed baselines below exist to measure what prompting alone recovers.

5.0.0.3 Conditions.

Four conditions, same window, same conversation. Baseline: no memory tools, told nothing about the window. Aware: no tools, told plainly that its visible history is short and earlier messages disappear. Notes: no tools, additionally instructed to end every reply with a notes line restating every constraint so far—the strongest no-tools strategy, restatement made explicit. Memory: the same window plus remember/recall tools backed by an ephemeral WorkingMemory; the agent decides when to call them (real tool-calling, OpenAI Agents SDK). The memory condition’s instructions include recall coaching (store every durable fact, reproduce every recalled item); the informed baselines exist precisely so the tools are compared against informed prompting rather than against ignorance. Full prompts ship in the artifact.

5.0.0.4 Models.

A GPT-5 ladder, small \(\rightarrow\) latest: gpt-5-nano, gpt-5-mini, gpt-5, gpt-5.5.

5.0.0.5 Metrics.

Task success (constraints recalled / 5), reported as mean with min–max range over five repeats per model per condition—independent re-samples of the fixed conversation at the provider’s default sampling; the harness exposes no seed parameter. A second, rubric-based LLM grade (gpt-5-mini) runs alongside the keyword grader; per-operation store latency (p50), pooled across repeats, treating operations as exchangeable; percentiles use the harness’s nearest-rank convention (the sorted value at index \(\lfloor n/2\rfloor\) for the median), so reported medians are the upper of the two central values at even \(n\); embedding latency; wall-clock; token counts. For the causal experiments we report exact permutation \(p\)-values for the pre-designated primary contrasts (two-sided, statistic the difference of arm means, all \(\binom{10}{5}{=}252\) partitions enumerated; §6.1).

5.1 Causal test: the loop-guard task↩︎

The trip task retrieves once per answer, so its outcome measures the value of having memory, not the speed of reaching it: it would score identically against a \(100\,\)ms store. To test the thesis itself—that store latency, everything else held fixed, changes task outcomes—we use a task that wants per-step memory: a loop guard. An agent works through a stream of candidate actions of which a known fraction are semantic duplicates of earlier ones; before executing each action it may ask the store “have I already done this?” (one real match per step). Memory operates under a fixed per-turn latency budget \(B\); each lookup spends its true cost against the budget (lookups only—the guard’s writes after an executed action are never charged, identically in every arm), and when the remaining budget cannot cover a conservative estimate of the next lookup’s cost (its injected delay plus a pad of a few store ops and \(1\,\)ms—the pad is why the \(+30\,\)ms rung of Table 4 affords \(8\)–\(10\) lookups per run, means \(8.4\) and \(9.2\), rather than the naive \(12\)), the step runs unguarded. \(B\) expresses a designer’s cap on memory-added dead time per turn; our operating point \(B{=}100\,\)ms is \(10\%\) of a one-second interactive step, matching \(\beta{=}0.1\) in §3, and coincides with the classic \({\sim}0.1\,\)s threshold below which interface feedback reads as instantaneous to a human [14], [15]. Because any such choice is contestable we sweep \(B\) rather than assert it. The design varies the store’s answer speed and reads off one outcome, redundant actions executed—so the claim is conditional and explicit: under a memory budget below one store round trip, latency alone determines whether the guard exists at all. One distinction deserves emphasis: latency in this design never alters what a lookup returns—the store’s answers are identical, and correct, in every arm. It alters only whether a lookup fits the budget. The causal path from latency to task damage therefore runs entirely through checks that never happen, never through answers that go wrong; the “mistakes” counted in Table 4 are repeated actions, not retrieval errors.

5.1.0.1 Deterministic version (mechanism demonstration).

A scripted agent, no LLM anywhere: \(S{=}20\) candidate actions per turn, half exact duplicates. Guards afforded per turn follow from the budget by construction (\(\lfloor B/\text{op}\rfloor\), capped at \(S\)), executed against a real store so guard correctness is real rather than assumed. We present this as arithmetic made visible, not as an empirical finding: it shows the mechanism, and the LLM version carries the empirical weight. Latency rungs: the measured in-process op plus injected round trips of \(1\), \(5\), \(30\), \(110\), and \(200\,\)ms; every cell is averaged over \(25\) independent workload shuffles, and the full budget grid \(B\in\{25,50,100,200,400\}\,\)ms is reported alongside the \(B{=}100\) operating point.

5.1.0.2 LLM version.

A real model produces the decisions. The agent triages \(24\) security alerts across \(4\) turns (\(6\) per turn); half are template-reworded duplicates of earlier alerts. The model sees only the last \(w{=}4\) exchanges, so it genuinely forgets earlier work; for each alert it answers investigate or skip-as-duplicate, and a code guard checks the store before any investigate executes. (Here \(w\) counts exchanges—eight messages—where the trip task’s \(w\) counts messages; each task’s window is stated in its own unit.) The guard sees only what the system could see in production—never the workload generator’s ground-truth fields. It matches the model’s own extracted action line (“investigate host=\(h\) sig=\(s\)”) through the paper’s local semantic embedder (potion-retrieval-32M, §7), with the match threshold calibrated offline on twenty held-out workload seeds disjoint from the experiment’s (\(F_1{=}1.0\) at threshold \(0.96\) on held-out data). The calibration also documents why the guard does not match raw alert prose: under a static embedder, same-template alerts about different incidents score higher (\(0.94\) max) than true rewordings of the same incident (\(0.58\) median)—template wording dominates the embedding. That measured failure is disclosed rather than hidden, and matching the model’s extracted action line is the design that survives it. On these action lines, true duplicates are near-identical strings (calibration cosine \({\approx}1.0\) against a worst impostor at \(0.918\)), so the threshold in effect performs exact matching on the model’s extracted output; the embedder’s remaining work is rejecting same-format lines about different incidents. Scoring uses the executed set: an execution is redundant only if the same host+signature pair was already executed, a guard match that blocks genuinely new work is counted against the guard (guard_blocked_new), and guard precision/recall are reported with per-lookup records in the artifact. The guard embeds each action line with the in-process local embedder (potion-retrieval-32M, \({\sim}32\,\mu\mathrm{s}\), §7), and the causal result is stated for that stack: a network embedder (\({\sim}200\)–\(400\,\)ms) would exceed the \(100\,\)ms budget in every arm, arm (a) included, which is the feasibility frontier’s point, not a confound—the embedder is held identical across arms, so only the injected store latency varies.

5.1.0.3 Arms.

Same model, same seeded workloads; prompts are identical between arms (a) and (b), and arm (c) differs only by its pinned memory dump. (a) in-process: the guard runs at measured in-process speed, affording every check. (b) delayed ladder: the identical guard pays an injected \(+\{5,15,30,50,110\}\,\)ms per lookup—each affordable lookup genuinely sleeps its delay—so affordability degrades continuously and the outcome traces a dose-response curve rather than a single cliff. (c) per-turn RAG: the industry-default pattern—store contents retrieved once at turn start and pinned in context, updates at turn end, no in-loop guard. A budget sweep (\(B\in\{100,150,250,500\}\,\)ms at the \(110\,\)ms rung) shows what happens when the budget rises past one round trip. Models: gpt-5-nano and gpt-5-mini; five seeded workloads per condition. Primary outcome: redundant investigations executed (of \(12\) duplicates); pre-designated contrasts are arm (a) vs.the \(110\,\)ms rung and arm (a) vs. arm (c), tested by exact permutation.

5.1.0.4 Measurement hosts.

All live agent numbers (Table 1), the causal experiments, and the reproduction artifact come from one host: an Apple M4 (\(10\) cores, \(16\,\)GB) inside Docker. The original development host produced the same qualitative pattern throughout; drift between hosts is reported, not smoothed: the Qdrant loopback gap measured \(7.5\times\) (median) on the original host and \(4.2\times\) on the M4, same direction; the deterministic guard experiment’s in-process op measured \(85\,\mu\mathrm{s}\) in the audit run of record and \(17.5\,\mu\mathrm{s}\) in the current regeneration.

6 Results↩︎

6.0.0.1 The window causes total failure; two mechanisms recover it.

Over five repeats per model per condition, every baseline run scores \(0/5\)—and so does every aware run: telling the model its history is windowed changes nothing without a mechanism to act on it (forty runs, no exceptions). Two mechanisms recover the task (Table 1). The notes baseline—restate every constraint in every reply—solves this five-fact task perfectly: \(5.0\) in all twenty runs, and honesty requires saying it plainly: at this scale, disciplined restatement beats memory tools. Its cost is structural rather than visible here: every live fact is re-emitted in every reply and re-read on every subsequent step, costing \(7\)–\(40\%\) more tokens than baseline for just five facts, rent that grows with the working set and is bounded by what a reply can carry (§2)—the regime where the store’s per-fetch pricing takes over is exactly the regime this small task never reaches. The memory condition reaches \(3.6\)–\(4.8/5\) with a store that never lost a fact; its misses are read-policy failures, decomposed below. A second, rubric-based LLM grade matches the keyword grader value-for-value in all eighty runs. Forgetting is a context problem, not a capability problem: the largest model fails identically to the smallest without a mechanism.

6.0.0.2 Where the misses live: the read policy, never the store.

The store kept all \(244\) writes across the twenty runs and answered every query it was asked (the pre-fix runs, in which an id collision under the network embed destroyed \(47\%\) of writes, are archived in the artifact and analyzed in §7). The misses have one shape: over-writing plus under-reading. The agents re-save the same constraint in fresh wordings (up to five copies of a single constraint) and store incidental non-constraint details besides, growing stores of \(7\)–\(17\) items, then read exactly once per run through the tool’s fixed \(k{=}8\) window; near-duplicate copies of already-covered constraints crowd the eight slots (a typical recall returns the budget twice), and a constraint the store demonstrably holds never reaches the reply. The pattern is one-directional: every run whose store stayed at or under \(10\) items scored \(5/5\), and all eighteen misses come from runs whose stores had outgrown the read window (\(11\)–\(17\) items against \(k{=}8\)), though three larger-store runs still recalled all five—outgrowing \(k\) is necessary for a miss here, not sufficient. This separates the memory substrate, which is reliable, from the model’s tool-use habits, which are not, and it is why we report means with ranges rather than a best run. The store op stays in microseconds live (write p50 \(80\)–\(89\,\mu\mathrm{s}\), \(n{=}44\)–\(81\) ops per model; the single per-run recall pools to a median of \(165\,\mu\mathrm{s}\) over these \(7\)–\(17\)-item stores, \(n{=}20\)), measured between the agent’s tool calls; the clean-room benchmark is the latency measurement of record (\({\sim}60{,}000\) ephemeral create-fill-destroy lifecycles/sec, recall p50 \({\approx}85\,\mu\mathrm{s}\) at \(1{,}000\) items, same host).

6.0.0.3 Reproducing and dissolving the in-loop latency tax.

We isolate the store op (real vxdb recall, embedding excluded) and model a networked store as the same operation plus a round-trip delay (\(\text{network}=\text{store\_work}+\text{RTT}\)), adding only the RTT—charitable to the network store. Let a turn run \(S{=}20\) in-loop retrievals. The retrieval blocking time per turn (Table 2, Figure 4) is \(1.7\,\)ms in-process vs.\(2.20\,\)s at a \(110\,\)ms cloud vector DB—a \({\sim}1{,}300\times\) difference.

6.0.0.4 Live networked baseline (measured).

To validate the model we push the same \(1{,}000\) vectors into vxdb (in-process) and into a real Qdrant container (gRPC over localhost) and measure per-op recall latency directly (Table 3). Even with no network—Qdrant on the same host over loopback—the client-server boundary costs \({\sim}4.2\times\) at the median (\(84\rightarrow352\,\mu\mathrm{s}\)) and \({\sim}4.4\times\) at the p99 tail (\(94\rightarrow410\,\mu\mathrm{s}\)) on this host (the original development host measured \(7.5\times\) and \(27\times\); drift reported in §5.1); the tail matters because an in-loop agent pays it every step. Adding a realistic \(110\,\)ms WAN to the measured Qdrant op reproduces the modeled \(2.20\,\)s/turn almost exactly (\(2.21\,\)s). The loopback number is a hard lower bound on the networked penalty.

6.0.0.5 The honest nuance on “\(83\times\).”

Whether this dead time dominates end-to-end latency depends on how it compares to reasoning. With per-step reasoning time \(r\), the in-loop vs.per-turn amplification is \(\frac{S r + S L}{S r + L}\) for store latency \(L\). When reasoning dominates (\(r\approx 1\,\)s), even a \(110\,\)ms store yields only \({\sim}1.10\times\) amplification; the \(83\times\) of [1] arises specifically when retrieval itself is expensive (their nested LLM-driven search, \(0.6\)–\(4.4\,\)s), i.e.\(L\gtrsim r\). The correct reading is therefore not “memory in the loop gives an \(83\times\) speedup,” but: the network tax is \(S\times\text{RTT}\), so a networked store forces rationing of retrieval, while an in-process store makes per-step, per-observation, per-action memory access feasible at all—a regime off the table at network latency.

6.1 Store latency alone changes the outcome↩︎

Everything above measures time. This section measures task quality, with latency as the only manipulated variable (§5.1).

6.1.0.1 Deterministic loop guard (mechanism).

With the in-process store (op \({\approx}17.5\,\mu\mathrm{s}\) this run), all \(20\) guard checks fit the \(100\,\)ms budget and \(\mathbf{0}\) duplicates execute, averaged over \(25\) workload shuffles. The rungs behave like a dial: \(1\,\)ms still affords all \(20\) guards; \(5\,\)ms affords \(19\) and leaks \(0.8\) [\(0\)–\(1\)]; \(30\,\)ms affords \(3\) and leaks \(9.7\) [\(8\)–\(10\)]; at \(110\,\)ms not one check fits and all \(\mathbf{10}\) execute. The budget grid (Figure 5, right) runs the same arithmetic at every \(B\) from \(25\) to \(400\,\)ms: the cliff moves, the mechanism does not. The other horn of the dilemma is paying instead of rationing: guarding every step costs \(0.4\,\)ms per turn in-process versus \(2.2\,\)s at the cloud store. This panel is floor arithmetic executed against a real store—a mechanism made visible, not an empirical finding; the empirical weight sits in the LLM runs below.

6.1.0.2 Real-LLM loop guard: a dose-response.

The same mechanism holds with a real model producing the actions and a real semantic guard (Table 4, Figure 6). Redundant executions rise monotonically with injected store latency: \(\mathbf{0.0}\) at in-process speed and at \(+5\,\)ms (every workload of both models exactly zero), \(1.4\)–\(1.6\) at \(+15\,\)ms, \(4.0\)–\(4.4\) at \(+30\,\)ms, \(6.2\) at \(+50\,\)ms, and \(\mathbf{7.2}\) of \(12\) at the \(110\,\)ms cloud round trip—where not a single lookup fits the \(100\,\)ms budget, so the guard is priced out entirely and every duplicate that survived the window executes. Exact permutation tests on the pre-designated contrast (in-process vs.\(110\,\)ms): \(p{=}0.0079\) for both models, the minimum attainable at five workloads per arm; the \(30\) and \(50\,\)ms rungs also reach \(0.0079\), the \(15\,\)ms rung \(0.048\). Only the two pre-designated contrasts are confirmatory; the per-rung values are descriptive, and the \(15\,\)ms rung’s \(0.048\) would not survive correction across the twelve tests reported. The guard itself never erred where it ran: precision and recall \(1.0\) wherever defined (a few workloads afford lookups but meet no true duplicate), and zero new investigations falsely blocked anywhere—matching on the model’s own extracted action lines through the real embedder, never on oracle fields. The guard does cognitive work the models cannot: they recognized only \({\sim}5\) of \(12\) disguised duplicates from their windowed context; the store caught the remaining \({\sim}7\) whenever a lookup was affordable.

6.1.0.3 Raising the budget does not buy the guard back.

Sweeping \(B\) at the \(110\,\)ms rung (gpt-5-nano): \(B{=}150\,\)ms affords \(4\) lookups and leaks \(6.2\); \(250\,\)ms affords \(8\) and leaks \(4.4\); even \(500\,\)ms—half a second of memory dead time per turn—affords \({\sim}15\) and still leaks \(1.6\) [\(0\)–\(4\)]. The buyback is sublinear because every affordable lookup costs a full round trip of turn time, while at in-process speed the entire guard costs under a millisecond. The budget is not the constraint; the round trip is.

6.1.0.4 The per-turn baseline is honest competition, and it clarifies the claim.

Arm (c), the industry-default pattern, performs well here—far better than the rationed in-loop guard at cloud latency, and statistically indistinguishable from the in-process guard on this workload’s means (exact permutation \(p{=}1.0\); the whole gap is one seeded workload). Two readings follow. First, rationing is rational at network latency: if lookups cost \(110\,\)ms, falling back to one retrieval per turn beats a guard you cannot afford, which is exactly why the industry converged on it. Second, per-turn coverage has a structural leak: a turn-start memory dump cannot see duplicates that arise within the turn. Both models leak on the same seeded workload (up to \(3\) of \(12\)), consistent with a structural rather than model-specific cause. The in-process guard is the only arm at zero on every workload of both models, and its edge over per-turn widens as agents take more steps per turn relative to their visible window. The claim these results support is precise: latency does not change what the model knows; it changes what the engineer can afford to wire around the model, and that alone moves task outcomes.

6.1.0.5 Cost.

The accuracy matrix consumed \(0.82\)M tokens across its eighty runs; the guard ladder and budget sweep \(1.31\)M across eighty-five (gpt-5-nano/-mini only).

7 The Embedding Bottleneck↩︎

Our measurements relocate the cost. With the store at \({\sim}100\,\mu\mathrm{s}\), the dominant per-operation expense is embedding (\({\sim}200\)–\(400\,\)ms via a network API). “In-loop memory is free” is true of the store, not yet of the operation. The complete in-loop stack therefore requires a local embedder alongside the in-process store. We measured this stack directly: a 32M-parameter static embedder (potion-retrieval-32M) embeds at p50 \(31.7\,\mu\mathrm{s}\), and with the in-process store the complete embed-plus-store operation lands at p50 \(\mathbf{40\,\mu\mathrm{s}}\)—roughly \(5{,}000\times\) below the measured network-embedding path (\(202\,\)ms)—placing the whole in-loop operation four orders of magnitude under a reasoning step. (The frontier of §3 prices the same stack at its conservative end, \(c\approx116\,\mu\mathrm{s}\): it uses the \(1{,}000\)-item recall p50 of \(84\,\mu\mathrm{s}\) plus the local embed, where this benchmark’s store op is measured on a smaller working set; both numbers trace to the same artifacts.) We treat this as a finding: it identifies embedding placement, not vector search, as the remaining obstacle to memory in the loop, and shows the obstacle is already removable. The network embed is not only a time cost; it is a failure window. In early accuracy-matrix runs, the \(200\)–\(500\,\)ms embed call held open a concurrency window in our own tool wrapper in which same-turn writes collided on an id and silently overwrote one another, destroying \(47\%\) of saves until a one-line unique-id fix (the pre-fix runs ship in the artifact). The fix is correct ids, not speed—but a \(32\,\mu\mathrm{s}\) local embed narrows the same window by four orders of magnitude, and the incident is a measured example of what network-latency operations do to in-loop designs. (Reasoning-model wall-clock, roughly one to three minutes per condition in §6, is dominated by model latency, orthogonal to the store.)

8 Related Work↩︎

8.0.0.1 Working memory and cognitive architectures.

Baddeley’s model and its LLM mapping; CoALA’s working/long-term split and reasoning/retrieval/learning actions [7], [10]. These fix retrieval as an action but do not ask how often it can run—the axis we add.

8.0.0.2 Agent memory systems.

MemGPT [16] pages memory in/out of context under memory pressure; Generative Agents [17] retrieve from a memory stream scored by recency, importance, and relevance; Reflexion [18] and Voyager [19] add episodic and procedural memory; surveys consolidate the taxonomy [20], [21]. All optimize what and whether to remember; none of these systems treat store latency as the variable gating the design space. The emerging benchmark wave—MemBench [22], MemoryAgentBench [23]—evaluates what agents remember, at turn granularity; what a store’s speed lets the loop afford is not among its measurements.

8.0.0.3 Retrieval frequency and timing.

The RAG literature already names our variable one level down: [4] catalogue retrieval frequency as a serving-time setting (one-time, every-\(n\)-tokens, every-token), and the Dynamics axis of the agent-memory survey of [5] covers when retrieval is triggered. ProactAgent [24] names the contrast directly—“existing methods typically retrieve memories passively, such as at task initialization or after each step”—and learns when to retrieve. All three treat frequency as a policy to choose or learn; none asks what frequency an agent can afford, which is where store latency enters and where our frontier (§3) sits.

8.0.0.4 Scratchpads and in-session working memory.

ReAct [25] and “Empowering Working Memory” [26] keep working memory in-context; we externalize it without leaving the loop, which only the latency argument makes viable.

8.0.0.5 Efficiency of in-loop retrieval.

[1] quantify in-loop latency amplification and answer it at the serving layer: SearchAgent-X keeps retrieval interleaved and unblocks it with priority-aware scheduling and non-stall retrieval. AgentIR [27] starts from the same observation—per-step memory queries whose cost “sits directly inside the agent’s reasoning loop”—and answers with a cascade retrieval substrate, treating latency as a serving constraint with no task-outcome claim; AMV-L [28] manages tail latency through memory lifecycle management, also serving-side. We answer at the substrate’s location instead (§3); the approaches compose. Industry “memory-first” guidance [2] does prescribe decoupling, with memory as a layer queried once per turn; that is the prescription we reject.

8.0.0.6 Latency and outcomes.

[29] show inference latency flipping task outcomes when an external clock punishes slowness (real-time games, trading)—latency as a race against the world. Our mechanism needs no clock: store latency degrades the agent’s information state, so it repeats work even with unlimited time. Closest on the systems side, [30] characterize agent-memory workloads and relate memory-serving latency to task accuracy across many systems; but every system they measure sits at \({\sim}0.1\,\)s or above, and they vary neither retrieval frequency nor latency by intervention, so the relation they report holds across designs rather than the within-design causal move we isolate. Practitioner writing has reached the intuition—a vendor post benchmarks sub-millisecond retrieval across reasoning steps [31] (its own number measured, competitor latencies simulated, wall-clock only), and Redis’s agent-memory guidance recommends in-memory storage where “response time compounds across multiple reasoning steps” [32], stopping at the network hop—without a frequency axis, a parity criterion, or an outcome experiment.

8.0.0.7 Extended cognition.

[3] and recent AI bridges [33], [34] invoke extended cognition descriptively. Mainen’s Library Theorem [12] is the closest formal predecessor: parity for file-backed agents with organization as the gate (§4 engages its argument against the working-memory framing); [35] operationalises extended cognition for corporate knowledge, with access efficiency inside its knowledge metric but no agents and no latency threshold; a unified review [36] maps externalization of memory, skills, and protocols. We use the parity principle prescriptively, as a latency criterion (§4).

8.0.0.8 What remains unclaimed.

Each ingredient of our claim now exists separately: AgentIR’s in-loop cost arithmetic, Kang et al.’s latency-flips-outcomes under a clock, the Library Theorem’s parity for external stores, industry’s sub-millisecond intuition. To our knowledge, no prior work holds the conjunction—store latency as the gate on retrieval frequency, parity as its criterion—and none demonstrates the causal chain slower store \(\rightarrow\) fewer affordable checks \(\rightarrow\) redundant actions with only the store’s answer speed moved. That conjunction is this paper.

9 Discussion↩︎

9.0.0.1 A decision rule for practitioners.

The cost model of §3 answers the affordability half: per-step access is viable when \(f\cdot c \ll r\), and with an in-process store and local embedder (\(c\approx116\,\mu\mathrm{s}\)) that holds at any realistic frequency. The results of §6.1 answer the task half. Per-turn retrieval remains adequate when each turn needs only facts settled in earlier turns—arm (c) performed respectably for exactly this reason. In-loop access becomes necessary when correctness depends on state that changes within the turn: duplicate work, contradictions, constraints discovered mid-reasoning. The rule of thumb: if a mistake can arise between two retrievals, the retrieval interval is too long—and only a store priced in microseconds lets that interval shrink to a single step.

9.0.0.2 Allocate, don’t connect.

The latency inversion reframes what an agent’s memory is. A networked store is infrastructure: provisioned, shared, connected to, kept alive. An in-process store is a data structure: allocated per agent, per task, even per turn, and discarded when done—the clean-room benchmark’s \({\sim}60{,}000\) create-fill-destroy lifecycles per second is the point, not a stunt. Agent frameworks today expose memory as a service client configured at startup; these results argue for exposing it as a loop primitive in the same category as the scratchpad, so that guard patterns like §6.1’s are library one-liners rather than architecture decisions.

9.0.0.3 Division of labor with the model.

The sharpest lesson of §6.1 is what the store did not do: it did not make the model smarter. The models still failed to recognize seven of twelve disguised duplicates on their own. The system stopped executing them because a code guard could afford to check a reliable store at every step. Reliability was engineered into the scaffold around the model rather than prompted into the model—a design stance that only becomes available when checking costs microseconds.

9.0.0.4 What else free per-step access unlocks.

The loop guard is deliberately the simplest member of a family (§3): novelty checks on every observation, grounding of claims as they are produced, working sets that outlive the window, and shared in-process blackboards for co-located agents. Each is one add or match per step, affordable only above the feasibility frontier. We demonstrate the guard; the rest are measured next steps, not speculation about capability.

10 Limitations↩︎

Two task families (constraint recall, loop-guard dedup) and one window size in the main results; a deterministic window\(\times\)facts sweep across three task families (no model in the loop) locates the failure boundary at fact-span \(>\) window and serves as sensitivity analysis (Figure 7). The trip task is small—five facts, six turns—and the restatement baseline wins at that scale; its breakdown at larger working sets follows from the rent argument (§2) but is argued, not measured here. Constraint grading is keyword-based and deliberately loose, mitigated by a second, rubric-based LLM grade—same provider family, so agreement is corroboration, not independence. The loop guard matches the model’s extracted action line rather than raw alert prose; the offline calibration (§5.1) shows why—a static embedder ranks same-template non-duplicates above true rewordings—and richer matchers over raw prose remain untested here. The budget policy is deliberately naive (guards the earliest affordable steps; no caching, no selective guarding), so the reported leak rates are a floor on what smarter rationing could recover. The networked baseline is measured on a loopback Qdrant (§6), so a WAN/multi-region deployment and an end-to-end agent run against a live cloud store would strengthen it further; the delayed arms inject measured round trips rather than calling one. Accuracy runs are repeats, not seeded replications (the harness exposes no seed); LLM tool-use variance is real and reported as means with ranges, and the memory condition’s misses trace to its read policy (one recall per run through a fixed \(k{=}8\) window), never to store failures; \(k\) is a design constant here, not swept, and larger or repeated reads would likely close part of the gap. The constitutive-cognition claim is functional, not phenomenal (§4).

11 Conclusion↩︎

Memory has stayed outside the agent loop for one reason, and it is not architecture. At network latency, per-step access costs seconds of dead time per turn, so the field engineered around the store: scheduling its calls, rationing them to turn boundaries. We showed the premise is contingent—an in-process store answers three orders of magnitude faster—and, more importantly, that it is causal: holding a per-turn memory budget fixed and moving only the store’s answer speed takes a scripted agent from zero redundant actions to ten of ten, and traces a monotone dose-response in real LLM agents, from \(0.0\) redundant investigations at in-process speed to \(7.2\) of \(12\) at cloud latency. Read through the parity principle as a latency budget, a store that fast becomes eligible to be working memory the agent has rather than a database it consults—and a loop wired to consult it on every step is what realizes the eligibility. The last obstacle, network embedding, falls to a measured \(40\,\mu\mathrm{s}\) complete operation with a small local embedder; the full stack exists today. Retrieval frequency—per-turn versus per-step—is now a design variable rather than a constraint, and the interesting question is no longer whether an agent can afford to check its memory at every step, but what agents will be built by engineers who know that it can.

Three studies follow directly. A long-horizon study: the guard’s edge over per-turn retrieval grows with steps per turn and duplicates within a turn, and measuring that growth curve across task families and model families is the natural next experiment. A confusion-resistance study: everything here uses ephemeral, task-scoped stores; how in-loop access interacts with long-lived, growing memories—where stale and near-duplicate entries compete—is open. And an artifact study: the guard patterns of §6.1 as library one-liners, so that per-step memory is an allocation decision rather than an architecture decision.

This paper settles only the temporal half of the memory question: when memory participates in reasoning. It is a first step toward treating memory as a cognitive resource for language agents rather than auxiliary infrastructure. Two questions follow it. One is the lifecycle of what an agent keeps: how it prioritizes, forgets, and decides what to trust. The other is the organization of what it keeps: structured, symbolic representations suited to individual and collective cognition. Pursued together, they point toward memory as a fundamental computational substrate for intelligent agents.

Acknowledgments and Disclosure of Funding↩︎

Yusuf Khan is the developer of vxdb, the in-process vector store used as the measurement instrument in this work. This work received no external funding.

References↩︎