Beyond Perplexity: A Behavioral Evaluation Framework for Deployment-Memory Claims in LLM Test-Time Training
July 01, 2026
Abstract
Large language model test-time training (TTT) is often evaluated through local proxy metrics: models are updated on recent tokens, retrieved context, target-domain data, or verifiable task attempts, and then judged by perplexity, future-token loss, long-context performance, or reward. These metrics are well matched to claims about stream adaptation, domain adaptation, context compression, and reward-backed test-time improvement. They are weaker evidence, however, for a capability that TTT results are increasingly used to motivate: deployed assistant memory, personalization, or sparse post-deployment learning, which instead requires behavioral evidence such as later recall, paraphrase robustness, retention, locality, conflict handling, and use in downstream actions after the original support context is removed. We introduce a behavioral evaluation framework that calibrates TTT memory claims to the evidence that supports them. It has two components: a claim-calibrated evidence ladder that separates stream/domain adaptation, bridge internalization, and deployment-time behavioral learning; and an evaluation protocol with matched explicit-memory baselines and mutually exclusive failure categories. We validate the framework by auditing recent TTT and memory-adjacent work and by instantiating it as a controlled diagnostic in which, in a sparse nonce-fact setting, one-step LoRA updates lower support and answer loss across three Qwen3 model scales while generated free-form recall stays at zero, exposing a measurable gap between proxy improvement and deployment behavior. The framework gives authors and evaluators a concrete standard for aligning TTT memory claims with the evidence actually reported.
1 Introduction↩︎
Test-time training (TTT) challenges the conventional “train, then deploy” boundary by allowing model states or parameters to change during inference. In large language models, recent work has made this idea technically concrete: models may update from retrieved neighbors [1], learn online hidden-state updates through fast weights [2], perform large-chunk updates for throughput and state capacity [3], or align online updates with next-token prediction [4]. Related work studies targeted context-specific updates [5], meta-learned long-context learning [6], parameter-efficient context memories [7], locally supported parametric memories [8], input-perplexity minimization [9], label-free uncertainty signals [10], unlabeled reinforcement learning [11], and self-directed update data [12], [13].
Across these settings, the evaluation recipe is relatively stable. A model is updated at test time on recently observed tokens, retrieved examples, task attempts, or generated data, and performance is then reported through lower perplexity, better future-token prediction, improved long-context accuracy, or higher reward. These results show real progress on in-sequence adaptation and compact use of recent context, often addressing practical limits of transformer-based systems such as finite context windows or static parameters. They also explain why TTT is attractive for LLM systems: online updates may allow a model to adapt to new evidence rather than relying only on fixed parameters or the current prompt.
The central observation of this paper is that this evidence is not always calibrated to the claims it is used to support. In real-world assistant settings, the relevant question is often not whether an update improves prediction on a nearby continuation. It is whether a deployed model can hear a sparse user preference, acquire a project-specific fact, revise a stale belief, or learn a procedure, and then use that information later after the original support context is removed.
We formalize this as a behavioral evaluation framework for LLM test-time training: claims about memory, personalization, or sparse post-deployment learning are calibrated against behavioral evidence beyond perplexity—later recall, paraphrase robustness, retention, locality, and conflict handling after the original support context is removed—rather than against local likelihood alone.
The fact that perplexity is an imperfect proxy for downstream behavior is not new; similar lessons appear in model editing, retrieval, and memory evaluation. The distinctive issue for LLM TTT is that inference-time parameter or state updates make local likelihood gains especially easy to reinterpret as evidence of learning, memory, or personalization. We call this failure mode evidence migration: evidence that directly supports one evaluation regime is carried into a stronger deployment narrative without the behavioral tests required for that narrative.
The key distinction is between in-sequence adaptation and deployment-time behavioral learning. In in-sequence adaptation, a model observes a token stream, updates its state or parameters on a support chunk, and is evaluated on future tokens from the same stream. In deployment-time behavioral learning, a deployed model receives sparse but high-value interactions—facts, preferences, corrections, or procedures—and must later use them in a personalized and stable way. Local loss reductions can strongly support the first setting while leaving the second only partially tested. Figure 1 illustrates this split with a sparse user-stated preference that should influence a later response.
None
Figure 1: Two evaluation paradigms. Top: deployment-time behavioral learning from a sparse user interaction, illustrated with a user-stated dietary preference and evaluated by whether that preference changes later user-facing behavior. Bottom: perplexity-based evidence, where the model updates on a past support token chunk and achieves lower loss on a future chunk..
This distinction matters because downstream systems are beginning to draw on recent TTT work for memory-like capabilities. Many long-context, label-free adaptation, and reward-backed TTT methods primarily report perplexity, likelihood, uncertainty, throughput, long-context accuracy, or task reward [2]–[6], [9]–[11]. A closer frontier studies parameter-efficient context memories, self-adaptation, and context internalization [7], [8], [14], [15]. These are important steps toward memory-like systems, but deployment-time memory adds a further requirement: sparse user information should remain usable after delay and should be evaluated against retrieval or long-context baselines.
Many existing TTT papers are properly grounded in proxy evaluation and do not themselves claim to solve deployment-time memory. The problem arises when such results are later cited, summarized, or motivated as evidence for stronger claims about memory, personalization, or self-updating assistants. Table 1 summarizes common evidence migration patterns and the behavioral tests needed to support the stronger deployment-time narratives.
| Token-stream TTT | lower future-token loss or long-context task gain | the model learns user facts online | sparse fact recall after context removal, paraphrase, and delay |
| Context compression / internalization | needle recall, context ablation, or parametric-memory evidence | forms persistent assistant memory | cross-session recall, locality, and conflict behavior |
| Self-adaptation from generated data | task improvement under generated updates | personalized continual learning | weak user evidence, user-specific retention, and correction |
| Reward-rich test-time discovery | verifiable reward improvement | broad deployment self-improvement | reward-poor user facts, preferences, and procedures |
2.6pt
We use TTT as an umbrella term for inference-time methods that update model state, especially parameters or fast weights, and treat LLM test-time learning (TTL) as a neighboring adaptation formulation. We use memory, continual learning, and deployment-time behavioral learning for the stronger claim that new information encountered after deployment can be acquired and later used. When we discuss future-token gain, we mean improvement on nearby held-out continuations after a test-time update.
Concretely, this paper makes four contributions:
We identify and name evidence migration: the failure mode in which TTT results grounded in perplexity, reward, or same-stream adaptation are used to support stronger claims about memory, personalization, or post-deployment learning.
We introduce a claim-calibrated evidence ladder, ranging from proxy evidence for stream adaptation, to evidence for context internalization, to behavioral evidence for deployment-time memory, and pair it with an evaluation protocol and matched explicit-memory baselines.
We audit recent TTT and memory-adjacent work through this ladder, distinguishing what current evaluations directly support from what additional tests would be needed for stronger deployment-time claims.
We instantiate and stress-test the framework with a controlled Qwen3/LoRA diagnostic, showing that proxy and answer-likelihood gains can coexist with zero generated recall, and provide claim-specific behavioral templates for factual, preference, correction, procedure, and agent-memory claims.
2 Perplexity is an Incomplete Proxy↩︎
Perplexity remains valuable. It is often the cleanest early signal that a test-time update has changed the model in a nontrivial way. The problem is not that perplexity is uninformative, but that it answers a narrower question than many deployment-time claims require. Lower loss can arise from mechanisms that are useful for stream adaptation but insufficient for post-deployment learning.
Continuous text is a dense-evidence regime. When support and evaluation chunks come from the same passage, they share topic, entities, lexical choices, and local discourse structure. Lower future-token loss may therefore reflect continuation fitting, topic priming, or memorization of nearby regularities, which is precisely what dynamic evaluation [16], [17] was designed to exploit. This is a legitimate stream-adaptation benefit. However, it does not require the model to form a stable, queryable representation of the underlying fact. Dense same-stream evidence is therefore weaker evidence for sparse deployment-time learning.
Associative retrieval differs from semantic acquisition. Long-context TTT can also succeed by building efficient associative state for the current input. Some long-context TTT results are consistent with the view that TTT-style state acts as a compressed associative memory over the current context. For example, TTT-E2E [6] connects TTT-style updates to key-value binding mechanisms and reports that full attention remains substantially stronger on needle-in-a-haystack recall, partly because compressed TTT-style state can discard details needed for exact retrieval. This supports context compression. It is weaker evidence for acquiring a reusable fact, rule, preference, or procedure that should survive paraphrase and delay.
Teacher forcing is easier than behavioral extraction. A model can assign higher probability to the gold continuation of a support fact without later producing the correct answer to a free-form query about that fact. This gap is especially important under open-ended decoding: a small increase in the probability of the right answer tokens may still be insufficient for greedy generation, beam search, or robust answer selection under paraphrase. Lower teacher-forced loss is therefore evidence that the update moved probability mass in a useful direction, but it does not guarantee that the learned information is accessible during open-ended generation. A system can improve language-model loss while still failing to retrieve the new knowledge behaviorally.
Cumulative stream adaptation can be mistaken for sparse-interaction learning. In standard continuous-text TTT protocols, updates often accumulate across a long input stream. This is appropriate when the goal is online adaptation to that stream. It becomes harder to interpret when the same result is used as evidence that a single sparse user interaction can be internalized and reused later. A stream protocol can conflate immediate adaptation to the current support item with cumulative adaptation from all preceding chunks. Deployment-time claims therefore need reset-vs-stream comparisons, or equivalent controls, when the protocol permits them.
Local LM metrics do not test generalization, locality, or conflict. Deployment-time learning requires more than improved prediction of nearby text. The update should persist under paraphrase, delay, mild distribution shift, and conflicting later evidence, while leaving unrelated behavior intact. The model editing literature provides a useful cautionary parallel: ROME [18] and MEMIT [19] made efficacy, generalization, and specificity explicit, while later evaluations found that short-form edit tests can miss specificity failures [20], multi-hop failures [21], long-form failures [22], gradual and catastrophic forgetting under scale [23], broader evaluation mismatches for edited models [24], and deployment-style failures in the wild [25]. The same lesson carries over to TTT memory claims: a locally valid success metric can overstate the practical capability of interest when the claim concerns delayed, user-facing behavior.
Takeaway: Support reconstruction, future-token loss, answer NLL, and, when used, candidate ranking should remain in TTT reports because they are informative mechanism-level signals. For deployment-time TTT, however, they should be labeled as proxy metrics and kept separate from headline behavioral evidence.
3 Deployment-Time Memory as a Behavioral Evaluation Target↩︎
This limitation does not invalidate existing TTT evaluations for stream adaptation, long-context compression, domain adaptation, or reward-backed test-time improvement. The issue is claim calibration: these evaluations do not by themselves establish the stronger deployment-time capabilities implied by memory, personalization, or self-updating assistant narratives.
The regime split matters because memory-style deployment is no longer an abstract desideratum. It is already the target of direct behavioral evaluation. LoCoMo [26] evaluates very long-term conversational memory; LongMemEval [27] benchmarks long-term interactive memory in chat assistants; MemoryAgentBench [28] evaluates memory through incremental multi-turn interactions; MemoryBench [29] targets memory and continual learning in LLM systems; MemoryCD [30] studies lifelong cross-domain personalization; and Mem2ActBench [31] evaluates long-term memory utilization in task-oriented agents. These benchmarks ask whether information from one or a few interactions can be reused after delay, paraphrase, abstention pressure, or conflict. Improving nearby continuation loss after a dense support chunk answers a different question.
Deployment-time learning differs from pretraining and continuous-text adaptation in three ways. It is low redundancy: a fact, preference, correction, or procedure may be stated only once. It is weak and heterogeneous: user evidence often arrives as short, informal utterances rather than clean supervised examples. It is also delayed and behavioral: the model must later answer, revise, abstain, route, or act according to the learned information without damaging unrelated behavior. These properties make behavioral probes necessary: the central question is whether the update changes later behavior in the way required by the claimed deployment use case, not merely whether it improves local likelihood.
The distinction is especially clear in settings with stronger supervision. TTT-Discover [32] shows that test-time training can improve search when the environment supplies verifiable rewards and many attempts can be scored. Personalized assistants face a harder frontier: semantic internalization under weak evidence, as explored by SEAL [12] and Absorber LLM [15], and stressed by direct memory benchmarks such as MemoryBench [29]. Reward-rich discovery and dense stream adaptation are promising, but sparse deployment-time memory imposes a separate behavioral burden.
Matched explicit-memory baselines are therefore central. If the deployed task is to remember a user fact, preference, correction, or procedure, then retrieval, long-context prompting, and memory systems that store and reuse the same evidence are natural comparators. MemoryBank [33] stores long-term user memory, LongMem [34] augments language models with long-term memory, MemGPT [35] manages memory through an operating-system-like architecture, Mem0 [36] targets scalable production-ready long-term memory for agents, Dynamic Cheatsheet [37] maintains an adaptive test-time memory, and MEMORYLLM [14] studies self-updatable language models. These systems retain deployment-time information outside ordinary base-model weights and retrieve, page, curate, or pool it when needed. They need not be framed as competitors that TTT must always beat. Rather, they define the behavioral and systems-level tradeoff that a parametric update must justify.
Takeaway: Deployment-time memory claims require behavioral tests against matched explicit-memory baselines. These comparisons reveal whether parametric TTT adds value under privacy, latency, compression, or context-pressure constraints.
4 Audit↩︎
Claim levels. We audit recent TTT and memory-adjacent work using three claim levels. S-level evidence denotes stream or domain adaptation: the model is updated on a recent stream, target-domain data, retrieved examples, or task attempts, and evaluated by nearby loss, same-stream performance, domain accuracy, long-context performance, or reward. B-level evidence denotes bridge mechanisms such as internalization, parametric memory, context absorption, or self-adaptation: the evaluation suggests that information can be stored, compressed, or transformed inside model state, but does not yet establish sparse delayed user-facing behavior. D-level evidence denotes deployment-time behavioral learning: sparse post-deployment information changes later behavior under recall, paraphrase, delay, locality, conflict, or action-use tests after the original support context is unavailable. Figure 2 summarizes the ladder.
Operationally, we assign S when the primary evidence is same-stream, domain, or task-adaptation performance; B when the evaluation tests memory-like internalization, parametric storage, context absorption, or self-adaptation without fully testing sparse delayed user behavior; and D only when the evaluation directly probes later behavior from sparse post-deployment information after context removal. B-level is intentionally heterogeneous: it covers mechanisms that bridge proxy adaptation and deployment memory, but these mechanisms do not by themselves establish D-level memory.
Audit protocol. We screened more than 40 targeted-search candidates available through April 2026 using bibliography seeding and targeted queries around LLM TTT/TTL, long-context TTT, context memory, parametric memory, personalization, continual learning, and assistant memory. A record was included if its title, abstract, introduction, or motivation explicitly invoked test-time learning or training, memory, persistence, context absorption, self-improvement, personalization, continual learning, or assistant memory. We coded 24 papers and treated the remaining records as background or exclusions, including retrieval-only or external-memory baselines, model-editing and safety background, non-LLM TTT, and architecture-adjacent context memory.
We therefore code claims by calibrating them to evidence rather than by inferring intent, assigning each paper the strongest claim level directly supported by its reported evaluations instead of the level suggested by its broadest motivating statement. Assistant-memory benchmarks are labeled as D targets rather than D-level TTT evidence because they define the behavioral task family but do not themselves show that parametric TTT achieves it. This audit is not a prevalence estimate or systematic review. Appendix 11 gives the search fields, candidate accounting, background and exclusion summary, and boundary-case sensitivity worksheet.
The audit asks what each paper family’s evidence directly supports, and what additional tests would be needed before using it in a deployment-memory claim. Table 2 gives representative examples.
| TTT layers / LaCT / In-Place TTT [2]–[4] | S | Test-time state improves stream, domain, or long-context modeling under dense evidence | Sparse fact or preference behavior after context removal, paraphrase, delay, and locality probes |
| PERK / Locas / Absorber LLM [7], [8], [15] | B | Parameter-efficient memories, local parametric support, or context absorption provide bridge evidence | One-shot facts, preferences, corrections, conflict overwrite, locality, and matched retrieval baselines |
| SEAL / TT-SI / TTT-Discover [12], [13], [32] | B | Self-adaptation, agent improvement, or reward-backed discovery under generated or verifiable update data | Weak user evidence, user-specific retention, correction, and reward-poor deployment settings |
| Assistant-memory benchmarks [27], [29], [31] | D target | Direct behavioral tasks for long-term assistant memory, corrections, personalization, or actions | Matched TTT update budget, explicit-memory baseline, and no-context ablations before using benchmark success to justify parametric TTT claims |
2.5pt
Audit pattern. The main pattern is not that TTT lacks progress; it is that most reported evidence remains below D-level deployment behavior. Dynamic evaluation, TTT-NN, TTT layers, LaCT, In-Place TTT, TTT-E2E, TLM, SyTTA, and TTRL primarily provide S-level evidence: adaptation to recent text, target-domain data, or verifiable tasks. PERK, Locas, SEAL, TT-SI, TTT-Discover, MEMORYLLM, and Absorber LLM move toward B-level evidence by studying parametric memories, self-adaptation, context internalization, or reward-backed discovery. Reasonable reclassifications of S/B boundary cases may change family counts, but they do not change the decision rule. A D-level deployment claim requires behavioral tests showing that a sparse user fact, preference, correction, or procedure changes later behavior after paraphrase, delay, conflict, and removal of the original context.
Takeaway: S- and B-level evidence can motivate deployment-memory hypotheses, but D-level language requires behavioral tests for sparse information after context removal, paraphrase, delay, locality, and conflict.
5 A Controlled Diagnostic Experiment↩︎
We instantiate the framework with a controlled diagnostic experiment that isolates a common inference behind evidence migration: whether improved support loss or answer likelihood is sufficient evidence for deployment recall. A test-time update is applied to a sparse factual support sentence, the support context is then removed, and the model is later queried for the injected fact. Such a cleanly controlled case, in which proxy metrics improve while later behavior does not, demonstrates why proxy evidence and deployment recall should be reported separately.
Diagnostic setup. We use Qwen3 models at three scales, Qwen/Qwen3-1.7B, Qwen/Qwen3-4B, and Qwen/Qwen3-8B, with LoRA adaptation and a minimal online update on each support text. The factual setting
introduces nonce access-code facts and then tests direct, paraphrased, and delayed recall after the support context is removed. We define \(\Delta\mathrm{NLL}=\mathrm{NLL}_{\mathrm{after}}-\mathrm{NLL}_{\mathrm{before}}\),
so negative values indicate improved teacher-forced loss after the update. Full hyperparameters, prompt counts, scoring rules, explicit-memory baselines, conflict-overwrite tests, stress updates, and robustness checks are in Appendices 10 and 12.
Evidence levels. The diagnostic keeps proxy, bridge, and target behavioral evidence separate. Proxy evidence consists of support reconstruction and local \(\Delta\)NLL. Bridge evidence consists of answer \(\Delta\)NLL. This metric tests whether probability mass moves toward the target answer, but not whether the model will produce that answer in open-ended use. Target behavior in the main factual probe is generated success under direct, paraphrased, and delayed prompting after the support context is removed. Appendix 12 adds locality, conflict-overwrite, retrieval, and replacement-memory checks as robustness and comparison evidence. This separation suggests lower loss is informative, but delayed free-form recall must be measured behaviorally.
| Loss improvement \(\Delta\)NLL \(\downarrow\) | Greedy generated success (%) \(\uparrow\) | |||||
|---|---|---|---|---|---|---|
| 2-4(lr)5-7 Model | Support | Direct answer | Paraphrased answer | Direct | Paraphrased | Delayed |
| Qwen3-1.7B | \(-1.151{\pm}0.051\) | \(-0.674{\pm}0.085\) | \(-0.714{\pm}0.071\) | \(0.0\) | \(0.0\) | \(0.0\) |
| Qwen3-4B | \(-1.218{\pm}0.093\) | \(-0.529{\pm}0.081\) | \(-0.531{\pm}0.071\) | \(0.0\) | \(0.0\) | \(0.0\) |
| Qwen3-8B | \(-0.966{\pm}0.026\) | \(-0.504{\pm}0.079\) | \(-0.463{\pm}0.070\) | \(0.0\) | \(0.0\) | \(0.0\) |
4.2pt
Results. Across all three Qwen3 sizes, one-step LoRA updates improve support and answer loss, while generated free-form recall remains at 0.0% for direct, paraphrased, and delayed queries under the fixed greedy decoding and normalized-containment scoring protocol. Table 3 shows this proxy/behavior split directly, support \(\Delta\)NLL improves substantially, and answer \(\Delta\)NLL also improves, yet these gains do not translate into successful greedy recovery of the injected facts.
The zero generated-recall result is not the only signal we report. The same table shows that teacher-forced answer likelihood improves, yet greedy generation still omits the target fact. Appendix 12 gives representative cases where the likelihood signal does not translate into generated recall.
Explicit-memory controls. Matched explicit-memory controls serve as evidence-usability and deployment-comparison checks. The appendix specifies exact-context, BM25-style retrieval, and replacement-memory protocols, but the factual aggregate control reported here is the easy BM25 condition. In the 1.7B seed-replication checks, BM25 hit and memory answering remain at \(48/48\). Harder retrieval checks in Appendix 12 have Hit@1 \(0/48\) under paraphrased-support decoys and \(0/24\) under stale/current conflicts. Thus, the point is not that retrieval trivially solves deployment memory, but that parametric TTT should be compared to explicit memory under matched evidence, query, scoring, and budget constraints.
Stress update and conflict checks. A stronger update can produce generated recall, but it exposes the missing locality dimension. In the Qwen3-1.7B factual setting, the appendix stress check raises direct and delayed recall to 72.9% (\(35/48\)) and paraphrased recall to 54.2% (\(26/48\)), while locality falls from 97.9% (\(141/144\)) to 9.7% (\(14/144\)) (Table 18). Conflict-overwrite checks show a different failure mode: the one-step LoRA update returns neither stale nor corrected code in all \(72/72\) conflicts, while replacement memory is corrected-only in \(68/72\) cases and both-corrected-and-stale in \(4/72\) (Table 17). These checks show why D-level evidence should jointly report recall, paraphrase robustness, delay, locality or interference, conflict behavior, and matched baselines.
Robustness scope. Appendix 12 reports additional checks, the 1.7B seed replications keep direct/paraphrased/delayed recall at \(0/48\), preference/correction and procedure mini-tasks also improve proxy loss while leaving greedy behavior at \(0/24\), and the continuous-text check is reported only as proxy-regime context. We therefore use these checks to qualify the diagnostic, not to claim that the tested one-step LoRA update achieves deployment memory.
Takeaway: The diagnostic separates proxy improvement from deployment recall. Proxy and bridge gains can coexist with zero behavioral recall, and stronger recall can expose severe locality cost. D-level claims, therefore, require generated recall under paraphrase and delay, locality, or conflict reporting, and matched baselines.
6 A Claim-Calibrated Evaluation Protocol↩︎
For TTT results that invoke post-deployment learning, memory, personalization, or self-updating assistants, the framework maps claim language to evidence level. Table 4 gives the decision rule for assigning supported wording from the reported evidence. Stream-adaptation claims can headline stream loss, future-token loss, long-context accuracy, throughput, or reward. Deployment-memory claims should headline later behavior under a deployment-like update episode.
| Improves stream adaptation, target-domain adaptation, or task performance under the stated objective | Remembers user facts, learns preferences, or performs post-deployment continual learning | Reset-vs-stream controls and direct behavioral probes for the claimed deployment target | |
| Provides bridge evidence for compression or internalization under the tested protocol | Robust deployment memory, personalization, or self-updating assistant behavior | Direct generation, paraphrase, delay, and no-support-context evaluation | |
| Immediate write-in under a narrow prompt format | Persistent or robust memory, broad personalization, or safe correction | Paraphrased and delayed recall, locality, conflict categories | |
| Deployment-memory claim under the stated setting and budget | Broad general memory beyond the tested setting, user population, or update mechanism | Broader tasks, stronger baselines, more seeds/models, and stated deployment constraints |
2.3pt
Match probes to claims. D-level evaluation should be claim-specific rather than maximalist. Factual memory requires no-context recall under paraphrase and delay; preference personalization requires later choices or response changes under plausible alternatives; correction claims require mutually exclusive stale/current scoring; procedural claims require later action or task transfer; and agent-memory claims require learned information to affect routing, tool calls, or arguments. Appendix 10 gives operational templates and evidence tiers. Such evaluations can be small: the support episode should introduce a sparse item absent from the evaluation prompt, and the later probe should test the failure modes relevant to the claim. When possible, reset-vs-stream controls should separate one-shot write-in from accumulated stream adaptation.
Keep proxy metrics separate. Support reconstruction, future-token loss, answer NLL, and, when reported, candidate ranking should remain in the report because they explain what the update is doing. But they should be presented as mechanism-level or intermediate signals, not as substitutes for the behavioral target. In particular, answer likelihood and generated answer success should be reported separately: the diagnostic above shows that a model can improve the likelihood of the target answer while still producing the same wrong free-form response.
Use matched baselines. A no-update model tests whether the update helps at all. Exact-context or long-context prompting tests whether the support evidence is sufficient when explicit. Retrieval or memory baselines test whether the same evidence can be stored outside the weights and reused under the same query and scoring rule. A matched baseline should use the same support information, query set, scoring rule, and disclosed budgets, including update tokens, trainable parameters, optimization steps, latency, memory footprint, context length, and retrieval index size when applicable.
Parametric TTT need not beat every explicit-memory baseline on every axis to be useful. Its value may appear under privacy, compression, latency, amortization, offline-operation, or context-pressure constraints. But those constraints should be stated and evaluated directly; otherwise, a memory or personalization claim risks conflating mechanism-level adaptation with deployment-time usefulness.
Takeaway: Claim language should track evidence level. Proxy and bridge metrics explain mechanisms, but memory, personalization, and post-deployment learning claims require behavioral evidence under the claimed use case, with failure categories and matched baselines visible.
7 Discussion↩︎
The framework is deliberately scoped, and several natural objections and boundary cases arise about how far it should be applied. We address them here as design considerations that clarify what the framework does and does not require.
Scope: evaluation against stated objectives. Many TTT papers study stream adaptation, long-context scaling, or task reward under the objective they optimize, without claiming to implement deployment-time memory. Those papers should be evaluated on their stated objectives. The interpretive issue arises when such results are later used to motivate memory, personalization, or self-updating assistants. At that point, S- and B-level evidence needs additional behavioral probes before it supports D-level language.
Perplexity as behavior in dense streams. One response is that perplexity is already behavioral for language models, because next-token prediction is the model’s core behavior. This is reasonable in dense-stream settings, where better prediction may be the desired outcome. Personalized assistants require a different kind of behavior: recalling sparse facts, applying corrections, resolving conflicts, or abstaining when information is missing. In that regime, lower loss is useful evidence about the update mechanism, but deployed memory still has to be shown in later responses.
Relation to external-memory substrates. Assistant memory may often be better implemented explicitly rather than parametrically. Systems such as MemoryBank and MemGPT store, retrieve, page, or curate information outside the base-model weights, and they are aimed directly at long-term interaction and multi-session use [33], [35]. The framework is compatible with this view. If explicit memory is the practical baseline, then parametric TTT should be evaluated against it under matched evidence, query, scoring, and budget constraints.
Context compression as a legitimate target. Methods need not target sparse user facts to be memory-relevant. PERK, Locas, MEMORYLLM, and related systems study adapters, parametric memories, or persistent memory pools that encode context for later use [7], [8], [14]. We treat this as a legitimate B-level objective. The calibration point is narrower: compression or internalization results should be described as such unless they also show deployment behavior under paraphrase, delay, conflict, and locality.
Procedural rather than episodic memory. Test-time learning may be most useful for reusable strategies, code snippets, search heuristics, or reward-backed reasoning rather than user-fact recall. Dynamic Cheatsheet and TTRL are examples of this direction: they use accumulated solutions or reward-style signals to improve future problem solving [11], [37]. The same evidence standard applies, but the behavioral target changes. A procedural-memory claim should be tested through later task performance, transfer, and failure modes for the learned procedure, rather than through factual recall alone.
Stronger update mechanisms. Stronger TTT systems may pass behavioral tests that simple update mechanisms fail. Better objectives, routing, replay, constrained updates, verification, or hybrid parametric–external memory could produce useful deployment-time memory. The framework accommodates this possibility, and specifies what evidence would be needed to establish it: behavioral success under the claimed use case, together with update cost, locality, conflict, and matched baselines.
8 Conclusion↩︎
TTT has made real progress on long-context modeling, stream adaptation, domain adaptation, and reward-backed test-time improvement. Our framework provides a concrete way to keep claims and evidence aligned: under the proposed protocol, deployment-memory wording is supported only when the claimed deployment behavior is directly tested. Lower perplexity, future-token loss, answer likelihood, candidate ranking when explicitly reported, and reward under a stated task are valuable evidence for the objectives they measure, but they should not be treated as sufficient evidence for deployment-time memory.
Showing that a model has acquired a user fact, preference, correction, or procedure for later use requires tests that resemble the claimed use case: direct, paraphrased, and delayed behavior; locality and conflict reporting; disclosed update budgets; mutually exclusive failure categories; and matched retrieval or long-context baselines. The aim is to keep claim and evidence at the same level. If future TTT systems can provide memory, the decisive evidence will appear where memory matters: in later behavior, under realistic constraints, with the interference cost visible.
9 Evidence migration patterns↩︎
This appendix makes the evidence-migration concern concrete without treating the cited papers as overclaiming. The migration risk is interpretive: a result that is well scoped as S- or B-level evidence can become misleading when reused as support for a D-level deployment-memory narrative.
Figure 3: 
.
10 Detailed evaluation agenda and limitations↩︎
10.0.0.1 Purpose of this appendix.
This appendix expands the evaluation protocol in Table 4. The goal is not to require every TTT paper to run every possible memory test. Rather, the evaluation should match the claim. A paper claiming stream adaptation can headline stream loss, future-token loss, long-context accuracy, throughput, or reward. A paper claiming deployment-time memory, personalization, correction, or agent memory should additionally report later behavior under a deployment-like update episode after the original support context is unavailable.
10.0.0.2 Diagnostic role of the pilot.
The diagnostic in Section 5 partially follows this agenda. It specifies exact-context controls, reports BM25-style retrieval and replacement-memory checks, separates answer-likelihood and generated-answer reporting, discloses update-time and token-budget details, and adds a same-family Qwen3 scale axis, a 1.7B three-seed replication, a stress update, mutually exclusive conflict scoring, and small preference/correction and procedure/action probes. Full benchmark construction and representative method comparisons are left for future work.
Diagnostic role: The LoRA/Qwen pilot isolates one evidential question: whether support-loss and answer-likelihood gains are sufficient for generated deployment behavior. The result shows that these proxy and bridge metrics can improve without free-form recall, motivating separate reporting of mechanism-level and behavioral evidence.
10.0.0.3 Minimal D-level evidence by claim type.
D-level evidence is claim-specific. The required behavioral bundle changes with the claim, but in every case the learned information should affect later behavior after the original support context is removed. Table 4 gives minimum evidence shapes for common deployment-memory claims.
Figure 4: 
.
10.0.0.4 Operational behavioral templates.
The same claim-specific idea can be made more operational by specifying the support episode, the no-context query, paraphrase or delay condition, locality or conflict check, and matched baseline. Table 5 gives templates for common deployment-memory settings. These are intended as minimal behavioral shapes, not as a universal benchmark.
Figure 5: 
.
10.0.0.5 Direct and paraphrased retrieval.
A deployment-time update should be tested on direct questions and paraphrases that require extracting the newly introduced fact, preference, correction, or procedure after the support context is removed. Teacher-forced answer likelihood and free generation should be reported separately, because the gap between them is itself informative. A system can move probability mass toward the correct answer under teacher forcing while still failing to produce the answer under open-ended generation.
10.0.0.6 Retention, locality, and conflict.
Updated knowledge should be queried after unrelated intervening turns or tasks. Evaluations should also include locality probes and conflict-resolution tests, because a useful update should not indiscriminately distort unrelated behavior or preserve stale information after a correction. If the claim concerns procedures or agents, the query should require the updated information to select a rule, route, tool, or action rather than merely repeat a string.
10.0.0.7 Matched explicit-memory baselines.
If the intended use case is a personalized assistant or agent, prompt accumulation, long-context inference, retrieval-augmented memory, and other non-parametric memory systems are natural baselines. These baselines should use matched evidence and report context cost, latency, update budget, and failure categories, because a parametric update is most compelling when it improves behavior under constraints where explicit memory is costly or unavailable. Stream adaptation should also be separated from one-shot write-in through reset-vs-stream controls, since cumulative gains across a stream provide different evidence from learning from a single support item.
Figure 6: 
.
In this paper, the easy BM25-style factual condition is a usability control: it verifies that the sparse support facts are answerable when explicit memory retrieves the intended sentence. The harder paraphrase and stale/current retrieval checks are failure-mode probes, showing why stronger explicit-memory baselines should include recency handling, semantic retrieval, reranking, or conflict resolution rather than naive lexical matching alone.
10.0.0.8 Evidence tiers for deployment-memory language.
The strength of the claim should track the strength of the behavioral evidence. Table 7 gives three rough tiers. The tiers are not meant to define a benchmark leaderboard; they are a wording guide for authors and evaluators.
Figure 7: 
.
10.0.0.9 Wording examples.
The decision protocol is meant to be operational. Table 8 gives examples of wording that stays within the evidence level and wording that calls for D-level behavioral evidence.
Figure 8: 
.
10.0.0.10 Claim discipline.
Papers should title and frame their contributions at the level their evidence supports. Continuous-stream adaptation, target-domain TTL, formal test-time discovery, context compression, and deployment-time learning are all legitimate targets. Evidence from one regime should not inherit the behavioral expectations of another. In particular, proxy and bridge metrics should remain in the report, but they should be described as mechanism-level or intermediate evidence unless the later behavior required by the claim is directly tested.
10.0.0.11 Additional limitations.
This diagnostic pass adds a same-family Qwen3 size axis, hard-negative retrieval controls, a 1.7B three-seed replication, conflict scoring, and small preference/correction and procedure/action probes. It leaves several directions open: cross-family model comparisons, a second parametric update mechanism, representative TTT/TTL systems, richer procedure-following tasks, and multi-seed runs for every model size. The confidence intervals are across examples unless explicitly marked as seed replication. Reward-rich settings such as math, code, kernels, and scientific optimization remain an important frontier. These limitations reinforce the central recommendation: specify the evaluation target, disclose the scoring rule, report failure categories, and avoid compressing all evidence into one loss number.
11 Paper-level audit sheet↩︎
This appendix expands the claim-calibrated audit summarized in the main text. The level column records the strongest claim directly supported by the reported evidence as summarized in the paper, while the last column records the shortest behavioral test that would be needed before using the result as D-level deployment-time learning evidence.
11.0.0.1 Audit protocol.
We screened papers available through April 2026 from the paper’s bibliography plus targeted searches using terms such as LLM test-time training, test-time learning, long-context TTT, context memory, self-adapting language models, parametric memory, personalization, continual learning, and assistant memory. A paper enters the audit when its title, abstract, introduction, or motivation explicitly invokes test-time learning/training, memory, persistence, context absorption, self-improvement, personalization, continual learning, or assistant memory. We screened 41 candidate records: 24 are coded below, 5 retrieval-only or external-memory systems are retained as background baselines, 9 model-editing/safety/background papers are cited for context but excluded from claim-level coding, and 3 non-LLM or architecture-adjacent papers are excluded from the audit table. This is single-author coding with sensitivity recoding, intended as a claim-calibration audit rather than a systematic review, prevalence estimate, or inter-coder reliability study. To make disagreements inspectable, we provide paper-level source phrases, boundary-case coding rationales, and sensitivity effects rather than asking readers to trust aggregate counts.
S-level coding means the reported evidence directly supports stream/domain/task adaptation; B-level coding means it supports a bridge mechanism such as internalization, compression, parametric memory, or self-adaptation; D-level coding means it directly tests sparse deployment information after paraphrase, delay, conflict, locality, or action use. Borderline cases are coded to the strongest level directly supported by the reported evaluation, not by the broadest motivation sentence. Eight papers are treated as boundary cases for sensitivity: TTT layers, LaCT, Not Just Context, TTT-E2E, PERK, Locas, MEMORYLLM, and Absorber LLM. Reclassifying these S/B cases changes the counts without changing the paper’s decision rule, because none supplies the full D-level behavioral bundle for sparse user deployment claims.
Figure 9: 
.
Figure 10: 
.
11.0.0.2 Boundary-case coding worksheet.
We provide the following worksheet for inspecting the eight boundary cases used in the audit sensitivity discussion.
Figure 11: 
.
11.0.0.3 Background and exclusion records.
The following records were screened because they are useful comparators, cautionary evaluation background, or boundary cases, but they are not coded as direct LLM TTT evidence in Tables 13–15.
Figure 12: 
.
11.0.0.4 Worked examples.
TTT-NN: S pass, D no. It updates on nearest-neighbor text and reports local language-model improvement, so it supports adaptation to nearby retrieved evidence; deployment-memory use would require delayed no-context recall, locality, and matched memory baselines. PERK/Locas: B pass, D incomplete. They explicitly move toward parameter-efficient or locally supported memories, so their evidence is closer to internalization; D-level assistant-memory claims still need sparse user facts, preferences, and corrections under paraphrase, delay, conflict, and matched external-memory baselines. LongMemEval/MemoryBench: D target behavior yes. They evaluate multi-session or service-time memory behavior, so they instantiate the target capability; a TTT paper borrowing that motivation should report comparable behavior under a stated update budget. This pilot: proxy and bridge evidence yes, D behavior no for the tested one-step LoRA setup; matched retrieval and replacement-memory checks are reported, so the result supports the proxy-insufficiency point: proxy and bridge improvements should not be treated as deployment-memory evidence without matching behavioral success.
Figure 13: 
.
Figure 14: 
.
Figure 15: 
.
12 Diagnostic robustness checks↩︎
12.0.0.1 Pilot experiment details.
Unless otherwise noted, diagnostic runs use H100 GPUs, seed 13, LoRA rank 8, alpha 16, dropout 0.0, learning rate \(5 \times 10^{-4}\), and a 24-token generation budget. The factual probes use 48 prompts per prompt type, the overwrite probes use 24 conflicts, and the locality probes use 144 unrelated prompts. We score generated answers as correct when the normalized first answer line contains the normalized target answer. We define \(\Delta\mathrm{NLL}=\mathrm{NLL}_{\mathrm{after}}-\mathrm{NLL}_{\mathrm{before}}\), so negative deltas indicate improved teacher-forced loss after the update.
The matched external-memory baselines receive the same support evidence as the LoRA update. The exact-context baseline prepends the support sentence directly to the evaluation prompt. The BM25-style retrieval baseline builds a fixed
memory bank whose retrieval unit is the support sentence. For each true fact, we add three hard negatives: the same object with a different prefix, the same prefix with a different object, and an answer-format distractor. We report both top-1 retrieval
hit, counted as correct when the retrieved fact id matches the target id, and end-to-end answer success after prompting with Retrieved memory: <support> followed by the original query. For conflicts, the replacement-memory
baseline discards the stale support, keeps only the correction sentence, and then answers the current-code query.
The supporting checks are deliberately narrow. On 1.7B only, we run a continuous-text proxy-regime check by splitting 32 passages into support, future_1, and future_2, updating on the support chunk, and comparing one-step, stream, and reset evaluation. We rerun the 1.7B factual/conflict core probes with two additional seeds. We also add small 24-item 1.7B preference/correction and procedure/action mini-tasks to test whether the proxy/behavior pattern is limited to nonce access-code strings.
Figure 16: 
.
12.0.0.2 Conflict-overwrite scoring.
Correction claims require mutually exclusive categories. A response counts as successful only when it contains the corrected answer and excludes the stale answer. Responses containing both corrected and stale information are failures, even if a non-exclusive scorer would mark the corrected answer as present.
Figure 17: 
.
The one-step LoRA update produces neither code in all conflict cases, indicating overwrite non-recall rather than successful correction. This is a failure of behavioral access, not merely a failure of conflict arbitration. Replacement memory is usually corrected-only, although the 1.7B model sometimes includes both corrected and stale codes in the same response.
12.0.0.3 Stress update tradeoff.
A stronger update can produce generated recall, but recall alone is not sufficient for deployment-memory claims. Locality probes ask unrelated questions whose answers should remain unchanged after the update.
Figure 18: 
.
The 16-step stress point raises direct and delayed recall to 72.9% and paraphrased recall to 54.2%, but locality falls from 97.9% to 9.7%. We treat this as evidence that stronger updates can move behavior, not as evidence that the update has achieved deployment memory. A D-level claim would need to report both the behavioral gain and the interference cost.
12.0.0.4 Retrieval hardness check.
The original BM25-style factual condition is useful because it shows that the sparse evidence is answerable when explicit memory retrieves the intended sentence. To avoid making lexical retrieval look easier than it is, we add a no-generation lexical-retrieval stress check: one condition paraphrases the true memory snippets and adds same-subject backup-token distractors; the other puts stale and corrected snippets for the same subject in memory and asks for the current code. Table 19 reports retrieval hit rates and oracle top-1 answerability, with model generation left out of this check.
Figure 19: 
.
12.0.0.5 Qualitative diagnostic examples.
The following examples make the failure mode inspectable. They are illustrative rather than a prevalence table, and show cases where answer likelihood improves while free-form behavior still does not change; the rank column is included only as case-level diagnostic context.
Figure 21: 
.
13 Scope and limitations↩︎
This paper presents a behavioral evaluation framework with a calibration audit and a controlled diagnostic experiment. The diagnostic shows that proxy improvement can be insufficient for generated deployment behavior; it does not map the full boundary of parametric deployment-time learning. The audit is structured and reproducible, but its purpose is claim calibration rather than prevalence estimation. These limitations reinforce the recommendation: specify the evaluation target, disclose the scoring rule, and report failure categories instead of compressing all evidence into one loss number.
14 Deployment and Societal Risks↩︎
Deployment-time memory is not only a capability claim but also a governance claim. A system that stores user facts, preferences, corrections, or procedures through parametric updates may make information harder to inspect, delete, scope to a user, or audit than an explicit memory store. Prematurely validating such systems with proxy metrics could encourage deployment before consent, retention, deletion, cross-session isolation, and cross-user leakage are tested. Our behavioral standard therefore has an ethical dimension: D-level evidence should include not only recall and personalization success, but also locality, conflict handling, deletion or forgetting when relevant, and clear disclosure of where information is stored. Explicit-memory baselines are important partly because they often provide clearer audit and deletion surfaces. Parametric TTT methods that claim deployment memory should state what privacy, latency, compression, or offline-operation constraints justify storing information in model state.
Figure 22: 
.