1 Abstract↩︎

LLM-based memory systems are increasingly tasked with maintaining facts that evolve over time. A recurring failure mode is conflict resolution: when the same fact appears in memory with multiple, contradictory values, which value should the agent return? The recently released MemoryAgentBench (MAB; Hu, Wang, & McAuley, 2026) FactConsolidation task (a conflict-resolution-style task under MAB’s Selective Forgetting competency) makes this explicit: facts are numbered, the counterfactual appears with the higher serial, and agents are told that “newer facts have larger serial numbers.” Despite the rule being spelled out, every published memory system substantially underperforms: HippoRAG-v2 reaches 54.0% on single-hop, BM25 reaches 48.0%, Mem0 / Contriever 18.0%, and Zep / Graphiti (a temporal knowledge graph designed for agent memory with bi-temporal edges and validity intervals) scores 7.0% on FC-SH (its lowest column in MAB Table 3, despite reported strengths on other benchmarks like DMR and LongMemEval). The multi-hop variant is nearly unsolved (\(\leq\)​7% across all 22 evaluated systems).

We argue that an underappreciated bottleneck is the assembly step: existing memory-agent baselines often leave conflict resolution to LLM-mediated retrieval, memory update, or answer generation, rather than to explicit version-aware aggregation. A matched-setup comparison (same backbone, same retrieval, same chunking, same TOP_K, same n=100 per cell) shows that replacing the LLM-judgment-based answer pipeline with a candidate-extraction + Python max(serial) pipeline yields +10.8 percentage points on FC-SH (67.2 \(\rightarrow\) 78.0, fact-level chunking, gpt-4o-mini). The gap widens with context length: +8 pp at 6K, +21 pp at 262K. (This is a whole-pipeline comparison: resolver, prompt, output format, and temperature vary jointly; cleanly isolating the resolver’s own contribution is left to future work.) Combined with semantic candidate extraction, this produces three results: 78.0% on FC-SH with gpt-4o-mini, 94.8% on FC-SH with gpt-4o, and 30.2% on FC-MH (gpt-4o-mini, rising to 51.5% with a gpt-4o backbone) via a per-hop deterministic-freshness extension of Self-Ask-style decomposition. At matched-backbone, matched-262K comparisons against MAB Table 3, the gpt-4o-mini pipeline beats the strongest published system (HippoRAG-v2) by +28 pp and the weakest (Zep/Graphiti) by +75 pp; the multi-hop result beats the best published FC-MH result (7%) by +20 pp at 262K. Larger gaps appear when the backbone is also upgraded to gpt-4o (§5.2).

The implication is corrective for the memory-framework subfield: the bottleneck on conflict resolution is assembly (post-retrieval aggregation), not storage (graph, hippocampal, agentic, or typed). For data with explicit version markers, the right primitive is structured candidate extraction followed by deterministic aggregation (Python max over the extracted candidates). A cross-benchmark check on LongMemEval knowledge-update (conversational memory, real timestamps) shows the mechanism ports from max(serial) to max(timestamp) but only ties LLM judgment overall (57.8% vs 64.4%, n=45): deterministic aggregation is the right primitive for current-value conflict resolution, and must be composed with question-type-aware handling (Yes/No, historical, aggregation) for broader memory QA.

2 §1 Introduction↩︎

2.1 §1.1 The puzzle↩︎

LLM-based memory agents are increasingly deployed to maintain information that evolves over time: user preferences that change, policies that supersede earlier versions, factual records that get corrected. A recurring failure mode in this setting is conflict resolution: when the same fact appears in memory with multiple, contradictory values, which value should the agent return?

The convention in most modern memory systems is straightforward: the most recent fact wins. Mem0, Graphiti (Zep), MemGPT, MIRIX, Cognee, HippoRAG-v2, and the various academic systems all explicitly handle “updates” or “supersession” of older information by newer.

Yet all of these systems substantially underperform on this task. MemoryAgentBench (Hu, Wang, & McAuley, 2026), the most comprehensive evaluation of LLM memory systems to date, provides a clean test: the FactConsolidation task constructs a memory of numbered facts where counterfactual versions (with higher serial numbers) appear after the original facts. Agents are explicitly told that “newer facts have larger serial numbers.” Despite this clear specification, every evaluated system substantially underperforms:

• HippoRAG-v2 (the best published RAG): 54.0% on FC-SH.

• GPT-4o long-context (sees the full context at 262K): 60.0% on FC-SH.

• BM25 (MAB’s simplest published RAG baseline, with the freshness rule in its prompt): 48.0% on FC-SH (chunk-512, the MAB default for FactConsolidation).

• MemGPT and Cognee: 28.0% on FC-SH.

• Mem0 and Contriever: 18.0% on FC-SH.

• RAPTOR, GraphRAG, MIRIX: 14.0% on FC-SH.

• Zep / Graphiti (a temporal knowledge graph designed for agent memory with bi-temporal edges and validity intervals): 7.0% on FC-SH, its lowest column in MAB Table 3, despite reported strengths on other benchmarks (DMR, LongMemEval).

• All 22 evaluated systems: \(\leq\)​7% on the multi-hop variant (FC-MH). (This holds for the 22 non-reasoning systems in MAB Table 3. MAB’s own reasoning-model validation (their Table 5) shows o4-mini at 80% on FC-MH at 6K, though it collapses to 14% by 32K; the task is tractable for strong reasoning models at short context, and the long-context / weak-backbone regime is the one our recipe targets.)

A task designed around an explicit rule (“newer is correct”) is being underperformed by every system, including a knowledge graph explicitly designed for temporal memory. The natural question is why.

2.2 §1.2 A matched-setup pipeline comparison↩︎

Our central finding is mechanistic. A matched-setup comparison at fixed chunking shows the size of the gap between an LLM-judgment-based answer pipeline and a candidate-extraction + Python max(serial) pipeline:

The comparison holds fixed: backbone (gpt-4o-mini), retrieval (BM25 TOP_K=10), chunking (fact-level), dataset, and n=100 per cell. The two pipelines differ in three coupled ways: (1) the resolver (LLM-judgment vs Python max(serial)), (2) the prompt and output format (free-text answer following MAB’s published BM25 prompt at temperature 0.7 vs structured JSON candidate extraction at temperature 0.0), and (3) the LLM’s task (decide-and-answer vs extract-candidates).

This comparison therefore measures the whole pipeline-level effect of moving freshness reasoning out of LLM judgment and into structured code, not the resolver step in isolation. Cleanly isolating the resolver’s own contribution — with shared candidate extraction and a harder, multi-candidate regime — is left to future work.

The +10.8 pp pooled-average gap and the widening of the gap at long context (+8 pp at 6K \(\rightarrow\) +21 pp at 262K) point to two LLM failure modes that the extract-then-aggregate pipeline eliminates:

1. Prior-override. When the question’s subject is a real-world entity with a strong training-data prior (e.g., “ice hockey” as the national sport of Finland), and the counterfactual fact assigns a different value (e.g., “pesäpallo”) with a higher serial, the LLM tends to output the prior despite the explicit “newer wins” rule in its prompt.

2. Serial-comparison drift over many candidates. As the candidate pool grows (longer context \(\rightarrow\) more conflicting candidates retrieved), the LLM loses track of which serial is largest. This manifests as a 14-point drop in the LLM-judgment baseline from 75% at 64K to 61% at 262K (§5.4). Note this is drift in the combined extract-and-judge baseline; once structured extraction shrinks the pool to the ~1–3 matching candidates, the freshness pick is reliable: Python max() is exact.

Structured candidate extraction is what eliminates both failure modes: it removes the real-world entity text from the freshness step (no prior-override) and shrinks the pool so serial-tracking no longer drifts. The LLM’s role narrows to semantic candidate extraction, a task it does well, and the final freshness pick is a deterministic Python max(serial) (exact, free, and deterministic).

2.3 §1.3 Three empirical findings↩︎

Building on this matched-setup comparison, three independent results each beat the strongest published baseline by a clear margin (n=100 per cell \(\times\) 4 context lengths each; 95% Wilson CIs):

Total: 2000 evaluations across the master table (§5.1). Two caveats on the headline comparisons:

• The MAB Table 3 single-number reports for each system appear to correspond to its 262K cell (confirmed via Table 10’s context-length ablation, which shows GPT-4o = 60.0 at 262K). Our pooled averages include shorter contexts where some systems would score higher; at 262K specifically the gaps are +28 pp (gpt-4o-mini at 82% vs HippoRAG-v2 at 54%) and +33 pp (gpt-4o at 93% vs GPT-4o long-context at 60%).

• The 6K cells use MAB’s published factconsolidation_*_6k rows (the dataset ships 6K alongside 32K/64K/262K). MAB reports 6K only in its reasoning-model validation (Table 5), not in the main systems comparison (Table 3) or the context-length ablation (Table 10), which use 32K/64K/262K. We include 6K in our master table for completeness but exclude it from apples-to-apples comparisons with the published Table-3 numbers.

2.4 §1.4 Why this matters↩︎

The agent-memory subfield has invested heavily in elaborate storage architectures: knowledge graphs (Graphiti, GraphRAG, Cognee), hippocampal-inspired retrieval (HippoRAG-v2), agentic loops (MemGPT, MIRIX, Self-RAG), typed semantic memory (Memanto), and various hybrid approaches. On the conflict-resolution task, every one of these architectures is matched or exceeded by a much simpler recipe: BM25 retrieval, an LLM that extracts semantically matching candidates, and a Python max() over their serial numbers. Even Zep / Graphiti (a knowledge graph explicitly designed for temporal agent memory) scores 7.0% on FC-SH, lower than any other system in MAB Table 3. (We are careful not to claim Zep is broadly weak: its reported strengths are on DMR and LongMemEval. The result here is that on FC-style conflict resolution, temporal-KG complexity does not help and may hurt.)

This points to a corrective for the memory-framework design space: the bottleneck on conflict resolution is assembly (how facts are identified, ranked, and combined to form an answer), not storage (graph, embedding, agentic, or typed). For tasks with explicit version markers, the right primitive is post-retrieval deterministic aggregation, not LLM judgment.

Our multi-hop finding, that the same architectural principle scales to multi-hop chains via decomposition (CAR pipeline, 30.2% on FC-MH vs published 7%), supports this thesis: multi-hop conflict resolution is tractable when the chain is decomposed properly and each hop’s conflict is resolved deterministically.

2.5 §1.5 Contributions↩︎

1. A matched-setup comparison quantifying the assembly bottleneck. Same backbone, same retrieval, same chunking, same TOP_K, same n, swapping the LLM-judgment answer pipeline for a candidate-extraction + Python max(serial) pipeline: +10.8 pp average gain, growing to +21 pp at the longest context (262K). This is a whole-pipeline effect; cleanly isolating the resolver’s own share is future work.

2. New SOTA on MAB FactConsolidation. Single-hop FC-SH at 78.0% (gpt-4o-mini avg) and 94.8% (gpt-4o avg). Multi-hop FC-MH at 30.2% (gpt-4o-mini) and 51.5% (gpt-4o) via the CAR pipeline. At matched-backbone, matched-262K comparisons against MAB Table 3, the gpt-4o-mini pipeline beats HippoRAG-v2 by +28 pp and Zep by +75 pp on FC-SH; the multi-hop pipeline beats the best published FC-MH result (7%) by +20 pp at 262K.

3. Mechanistic evidence on where the LLM fails. The combined extract-and-judge baseline holds 63–75% at 6K–64K but drops to 61% at 262K; our extract-then-aggregate pipeline does not degrade (71–82% across all lengths; 92–99% with gpt-4o). Conditioning on questions where the fact was provably retrieved (§5.4) localizes the degradation to post-retrieval judgment, not BM25 recall.

4. A retrieval upper bound and method complementarity analysis. 88.5% of FC-SH questions are solved by either pipeline; 11.5% are solved by neither (a lower bound on retrieval ceiling). The methods are partially complementary: 21% of questions are solved only by Python max(serial); 10.5% only by LLM-judgment. (§5.5.)

5. Cross-benchmark generalization and scope. On LongMemEval knowledge-update (conversational memory with real timestamps), the freshness mechanism ports from max(serial) to max(timestamp), but the deterministic pipeline ties LLM judgment overall (57.8% vs 64.4%, n=45; overlapping CIs). The questions it loses are not freshness failures: they are Yes/No-format, historical (“previous”), and aggregation questions for which max() is the wrong operator. This sharpens the thesis: deterministic aggregation is the right primitive for current-value conflict resolution and must be composed with question-type-aware handling for broader memory QA. (§5.7.)

6. Reproducibility. Full open-source code (~50 lines of orchestration beyond standard BM25 retrieval and the candidate-extraction prompt), Langfuse traces for all master-table evaluations, and a single-command replication script.

2.6 §1.6 Scope and honest limits↩︎

• Our primary evaluation is MAB FactConsolidation (2 sub-tasks \(\times\) 4 context lengths each, all using MAB’s published rows). The MQUAKE-derived counterfactual setup may not capture all real-world conflict patterns.

• We add a cross-benchmark generalization check on LongMemEval knowledge-update (§5.7, n=45); it is small (wide CIs) and covers a single question type.

• The principle is scoped to current-value conflict resolution. On LongMemEval knowledge-update, deterministic aggregation ties LLM judgment overall and wins cleanly only on “what is X currently?” questions; Yes/No, historical, and aggregation questions need question-type-aware composition (§5.7).

• Our approach assumes the source data has explicit version markers (serial numbers, timestamps). When this assumption holds, as it typically does in production memory systems, our recipe is the right primitive.

• We tested three backbones (gpt-4o-mini, gpt-4o, and o4-mini). The qualitative finding likely generalizes; quantitative numbers may shift on other model families.

• The matched-setup comparison varies more than the resolver alone. Our LLM-judgment baseline and Headline pipeline differ in prompt format, output format, and temperature (0.7 vs 0.0) in addition to the resolver. The +10.8 pp gap is therefore a pipeline-level effect, not a strictly isolated resolver effect; cleanly isolating the resolver’s own contribution is future work.

• Chunk-4096 is now also matched (a gap in the original draft): the matched LLM-judgment + chunk-4096 cell gives deterministic 80.8% vs LLM-judgment 61.0% (+19.8 pp; §5.3).

• Multi-hop on FC-MH is still substantially harder than single-hop (30.2% vs 78%/94.8%). Our decomposition approach is preliminary.

The remainder of the paper documents the method (§3), experimental setup (§4), main results (§5), and discussion (§6).

3 §2 Related Work↩︎

We organize related work into three threads: (1) memory frameworks evaluated on MemoryAgentBench, (2) multi-hop question decomposition, and (3) temporal / freshness handling in retrieval-augmented systems.

3.1 §2.1 Memory frameworks on MemoryAgentBench↩︎

MemoryAgentBench (Hu, Wang, & McAuley, 2026) is the most comprehensive evaluation of LLM memory systems to date, covering four competencies (Accurate Retrieval, Test-Time Learning, Long-Range Understanding, Selective Forgetting) across 22 systems and 5 backbone models. FactConsolidation is MAB’s conflict-resolution-style task under the Selective Forgetting competency: it introduces counterfactual edits from MQUAKE (Zhong et al., 2023) and tests whether agents can correctly prioritize later-added information over earlier-added.

The benchmark exposes a striking weakness: even the strongest published RAG system (HippoRAG-v2, Gutiérrez et al., 2025) achieves only 54.0% on single-hop FactConsolidation. Industry-standard memory frameworks underperform a deterministic-freshness baseline by 50–87 percentage points: Mem0 (Chhikara et al., 2025) at 18.0%, Graphiti / Zep (Rasmussen et al., 2025) at 7.0%, MemGPT / Letta (Packer et al., 2023) at 28.0%, MIRIX at 14.0%, and Cognee at 28.0%. The hierarchical RAG systems (RAPTOR, GraphRAG, MemoRAG) cluster at 14–21%. The multi-hop variant (FC-MH) is essentially unsolved: every system is at 0–7%. We compare against these published numbers throughout §5.

3.2 §2.2 Multi-hop QA via decomposition↩︎

Our multi-hop pipeline (Chain-Aware Resolution, CAR) is built on the well-established lineage of question decomposition. Self-Ask (Press et al., 2022) showed that prompting an LLM to ask itself sub-questions can improve multi-hop QA. Decomposed Prompting (Khot et al., 2022) and IRCoT (Trivedi et al., 2023) interleave decomposition with retrieval and chain-of-thought reasoning. Iter-RetGen (Shao et al., 2023) alternates retrieval and generation. MultiHop-RAG (Tang & Yang, 2024) provides a benchmark and reference implementation for multi-hop RAG.

A recent paper by Madhwal et al. (2026) titled “Decomposed Prompting Does Not Fix Knowledge Gaps, But Helps Models Say ‘I Don’t Know’ ” is methodologically adjacent: it studies decomposition for multi-hop QA and proposes abstention via cross-regime disagreement, mirroring our finding that the decomposition pipeline naturally produces calibrated “no answer” outputs when the chain breaks.

Our contribution relative to this lineage is not the decomposition itself (we use a standard Self-Ask-style decomposer) but the per-hop deterministic freshness resolution applied at each step of the chain, which existing decomposition methods lack.

3.3 §2.3 Temporal / freshness handling in retrieval↩︎

Several systems address temporal aspects of memory directly:

• temporal-rag (Emmimal, 2025), a small open-source Python library, adds a post-retrieval temporal layer with validity filtering, exponential time decay, and explicit supersedes-chains. It is explicitly single-query and does not handle multi-hop chains. Their README states “Resolving disagreements between two current documents is the LLM’s problem, not the retriever’s”, directly the design choice we contest.

• “Solving Freshness in RAG” (Grofsky, 2025) shows that a simple recency prior reaches 1.00 accuracy on a single-hop freshness task in cybersecurity data, while a heuristic clustering trend detector fails (0.08 F1). Grofsky establishes that deterministic recency works in one setting; we state our delta explicitly. (i) Cross-system evidence on a standard benchmark: on MAB FactConsolidation every published memory system underperforms a deterministic-freshness baseline — industry frameworks by 50–87 pp, and even the strongest published RAG (HippoRAG-v2) by +28 pp at 262K — i.e., the field systematically leaves this on the table, not just that recency works in one case. (ii) The recipe: structured candidate extraction feeding deterministic aggregation, which applies the freshness rule reliably over noisy retrieved context with a weak backbone. (iii) Per-hop resolution inside decomposed multi-hop chains (CAR), absent from Grofsky’s single-hop setting. (iv) A cross-benchmark scoping (LongMemEval) of where the mechanism ports (current-value conflicts) versus where it needs composition (Yes/No, aggregation).

• Memanto (Abtahi et al., 2026) explicitly argues that “knowledge graph complexity is not necessary” for high-fidelity agent memory, achieving SOTA on LongMemEval (89.8%) and LoCoMo (87.1%) with a typed semantic memory + information-theoretic retrieval engine. We extend this “simpler-beats-complex” thesis to FactConsolidation, where Memanto was not evaluated.

• DYNAMICQA + MULAN reproducibility (Dey et al., 2026), MAGIC (Lee et al., 2025), and “QA under Temporal Conflict” (Özer & Yıldız, 2025) all study LLMs’ ability to resolve conflicting facts. They confirm that LLMs struggle to apply explicit in-context freshness rules, particularly when the rule conflicts with training-data priors.

• TruthfulRAG (Liu, Shang, & Zhang, 2025) and Micro-Act (Huo et al., 2025) propose LLM-based conflict-resolution mechanisms over knowledge graphs. Both rely on LLM judgment for the resolution step; both achieve modest results.

3.4 §2.4 Knowledge editing and LLM priors↩︎

A separate body of work documents that LLMs systematically override in-context information that conflicts with their training-data priors. “Uncovering Overfitting in LLM Editing” (Zhang et al., 2024), “Unveiling Divergent Inductive Biases of LLMs on Temporal Data” (Kishore & He, 2024), and “Adaptive Token Biaser” (Bi et al., 2024) all show that knowledge edits propagate beyond intended scope, and that LLMs’ temporal priors are hard to override via prompting alone. Our finding (that free-text LLM-judgment baselines cannot reliably apply the explicit MAB freshness rule when filtering, prior suppression, and answer generation are entangled) is consistent with this literature and motivates our move to deterministic post-processing.

3.5 §2.5 Position of this work↩︎

Our specific contribution lives at the intersection of:

• Multi-hop QA decomposition (Self-Ask lineage): established

• Temporal/freshness conflict resolution (temporal-rag, Memanto, Grofsky 2025): established

• MemoryAgentBench evaluation harness: established

What we add:

1. A matched-setup pipeline comparison quantifying the assembly bottleneck: swapping an LLM-judgment answer pipeline for a candidate-extraction + Python max(serial) pipeline at fixed backbone, retrieval, and chunking gives +10.8 pp average, growing to +21 pp at 262K. (The comparison varies the resolver, prompt format, and temperature jointly; cleanly isolating the resolver’s own contribution is future work.)

2. A specific empirical result on FC-SH (78.0% gpt-4o-mini \(\rightarrow\) 94.8% gpt-4o) and FC-MH (30.2% gpt-4o-mini) that beats every published system at matched-backbone, matched-262K comparison by at least +28 pp on FC-SH and +20 pp on FC-MH (see §5.2 for the full table).

3. A position statement that the bottleneck on conflict resolution is assembly (LLM judgment over retrieved candidates) rather than storage (graph, hippocampal, agentic, typed memory).

We deliberately do not claim novelty of the decomposition mechanism, the freshness idea, or the use of BM25. The novelty is the specific combination + matched-comparison empirical evidence on the FactConsolidation benchmark.

4 §3 Method↩︎

We describe two pipelines: SH-conflict for single-hop conflict resolution, and CAR (Chain-Aware Resolution) for multi-hop. Both share three primitives: fact-level (or chunk-level) retrieval, LLM candidate extraction, and deterministic freshness picking.

4.1 §3.1 The SH-conflict pipeline↩︎

Given a corpus of numbered facts \(\mathcal{C} = \{(s_i, t_i)\}_{i=1}^N\) where \(s_i \in \mathbb{Z}\) is a version serial and \(t_i\) is fact text, and a query \(q\), we compute:

1. Retrieval: \(R = \text{BM25}(\mathcal{C}, q, k=10)\), returning top-\(k\) facts ranked by BM25.

2. Candidate extraction: prompt an LLM with \(q\) and the retrieved \(R\) to produce \(C = \{(s_j, t_j, e_j)\}\), where each \(e_j\) is an extracted answer entity from a fact that semantically matches \(q\). The LLM is explicitly instructed to include all matching items (do not compare serials, do not pick a “best”) and to extract candidates verbatim.

3. Freshness picking: \(\hat{c} = \arg\max_{c \in C} c.s\), then return \(\hat{c}.e\) as the answer. If \(C = \emptyset\), return “no answer”.

The full SH-conflict pipeline is \(\approx\)​50 lines of Python:

The two contrasts with prior approaches:

• Versus chunk-level retrieval (MAB BM25 baseline, Mem0, Cognee, HippoRAG): we index each fact independently, preserving the serial number as the indexing key. Most prior systems chunk at 512 or 4096 tokens, which packs many serials into one chunk and forces the LLM to do intra-chunk freshness reasoning. (MAB uses chunk-512 by default for FactConsolidation; see Hu et al. 2026 Table 15, Appendix E.)

• Versus LLM-judgment freshness (BM25 baseline, all RAG-based memory systems, all agentic memory systems): we move the freshness comparison out of the LLM and into deterministic Python code. The LLM only extracts candidates; it does not pick the winner.

The candidate-extraction prompt is intentionally strict: the LLM must match both subject and predicate verbatim, and must include every matching item (with possibly conflicting answer entities). This pushes the LLM toward conservative behavior: when no candidate matches, the pipeline returns “no answer” rather than hallucinating. Empirically, ~2% of FC-SH questions hit this case (§6.1); we count “no answer” as wrong under SubEM but in a production setting it is a calibrated abstention.

4.2 §3.2 The CAR pipeline: Chain-Aware Resolution for multi-hop↩︎

We call this Chain-Aware Resolution because the method follows an ordered chain of dependent hops and resolves each hop’s conflict before passing its answer to the next hop.

For multi-hop questions of the form “X of Y of Z”, we extend the single-hop pipeline with a Self-Ask-style decomposition step:

1. Decomposition: prompt an LLM with \(q\) to produce a chain of atomic hops \(h_1, \ldots, h_n\) where each \(h_i\) asks about a single relationship. The output uses {hop_k_answer} placeholders to thread the chain.

2. Per-hop execution: for each \(h_i\), substitute placeholder values from the chain, then run the SH-conflict pipeline on the resolved hop query. Take the result as \(\text{hop}_i\_\text{answer}\).

3. Final answer: the last hop’s answer is the answer to \(q\). If any intermediate hop returns “no answer”, subsequent hops that depend on it are skipped; the pipeline returns the last valid answer found, or “no answer” if the chain breaks at hop 1.

Worked example. For “What is the country of citizenship of the spouse of the author of ‘American Pastoral’?”, the decomposition produces:

• \(h_1\): Who is the author of ‘American Pastoral’?

• \(h_2\): Who is the spouse of {hop_1_answer}?

• \(h_3\): What is the country of citizenship of {hop_2_answer}? (the answer to \(q\))

CAR pseudocode:

The decompose prompt uses a Graphiti-style format (system: minimal; user: structured tags + worked example). We use a generic placeholder example (“Alice supervises Bob who manages Carol”) to avoid domain leakage. The HARD_CONSTRAINT in the prompt forbids more than one relationship word per hop, which empirically prevents gpt-4o-mini from compressing 4-hop chains into 2-hop chains.

4.3 §3.3 Why deterministic freshness works↩︎

The candidate-extraction step produces a small set of candidates (typically 1–3, never more than the TOP_K=10 retrieved). Among these, the freshness comparison is a single Python max() over integers, an operation that is exact, fast, and incapable of LLM-style failures.

Concretely, two failure modes of LLM-judgment freshness disappear under the deterministic primitive:

1. Prior-override. When the question’s subject is a real-world entity with a strong training-data prior, and the counterfactual fact assigns a different value with a higher serial, the LLM-judgment baseline frequently outputs the prior despite the explicit “newer wins” rule in its prompt. The deterministic primitive never sees the entity text during comparison, only the integer serial, so the prior cannot override.

2. Serial-comparison drift over many candidates. When the retrieval pool contains many conflicting candidates (typical at 262K context), the LLM-judgment baseline loses track of which serial is largest. This manifests in §5.4 as a 14-point drop from 75% (64K) to 61% (262K). The deterministic primitive does not have this pathology: max() is exact regardless of pool size.

The candidate extraction itself is the LLM’s job: deciding whether a given fact semantically answers the question. LLMs do this well even with weak backbones (gpt-4o-mini). What the baseline does poorly is jointly filtering, resolving freshness, resisting priors, and answering in one free-text step over noisy retrieved context. Structured extraction separates these: it isolates the candidates, leaving the freshness pick to an exact, deterministic Python max().

4.4 §3.4 Implementation↩︎

The complete pipeline code is at scripts/_pipeline.py (shared primitives) and scripts/13_paper_experiment.py (SH-conflict driver) / scripts/14_ablations.py (multi-experiment driver). The full prompts (CANDIDATE_PROMPT, DECOMP_PROMPT, and the MAB-provided BM25 prompt we use as the LLM-judgment baseline) are released verbatim in the code repository (scripts/_pipeline.py).

Total dependencies: openai, rank_bm25, datasets. No vector database, no graph database, no embedding model.

5 §4 Experimental Setup↩︎

5.1 §4.1 Dataset↩︎

We evaluate on MemoryAgentBench FactConsolidation (Hu, Wang, & McAuley, 2026), a conflict-resolution-style task under MAB’s Selective Forgetting competency. The benchmark publishes two sub-tasks (single-hop and multi-hop), and the dataset ships four context lengths (6K, 32K, 64K, 262K tokens). MAB reports 6K only in its reasoning-model validation (Table 5), not in its main systems comparison; we use all four and label 6K as such throughout §5. Across the four lengths this is 8 sub-task rows \(\times\) 100 questions each = 800 test instances per pipeline.

FactConsolidation is constructed from MQUAKE (Zhong et al., 2023): counterfactual rewrites of real-world facts. The original fact and its counterfactual variant are concatenated in order so the counterfactual appears with a higher serial number. Multiple pairs are stacked to reach the target context length. The benchmark task is to answer questions about the subjects in a way that prioritizes the counterfactual (newer serial).

5.2 §4.2 Backbone models↩︎

• gpt-4o-mini (gpt-4o-mini-2024-07-18): primary backbone for the headline experiment and most ablations.

• gpt-4o (gpt-4o-2024-08-06): backbone for Ablation C only.

• o4-mini: reasoning-model backbone for the multi-hop and single-hop diagnostic sweeps (§5.6); not part of the master table.

All MAB-paper baselines we compare against use gpt-4o-mini as their backbone (per MAB §4.1), so our comparison is apples-to-apples on backbone for gpt-4o-mini results.

5.3 §4.3 Pipelines evaluated↩︎

The LLM-judgment baseline uses MAB’s published BM25 answer prompt, while the Headline pipeline uses structured candidate extraction followed by deterministic max(serial). Both use identical fact-level retrieval and TOP_K, but they differ in prompt format, output format, temperature, and resolver. We therefore treat §5.3 as a matched-setup pipeline comparison (not a resolver-only ablation); cleanly isolating the resolver is left to future work.

5.4 §4.4 Hyperparameters↩︎

• TOP_K = 10 (number of retrieved facts per query). Matches MAB.

• Temperature: 0.0 for the deterministic-freshness pipelines; 0.7 for the LLM-judgment baseline (matching MAB’s reported default for the BM25 baseline).

• Chunking strategies:

  • fact-level: parse the context with regex (\d+)\.\s+(.+?)(?=\d+\.\s$), one fact per chunk. Typically yields 500–18,000 chunks depending on context length.

  • chunk-4096: sliding-window chunker, 4096 characters per chunk. ~6 chunks for 6K context, ~250 chunks for 262K context. (Note: MAB’s paper lists chunk-512 as the FactConsolidation default (Table 15, Appendix E), though MAB’s released configs set chunk-4096. We use chunk-4096, which matches both the released configs and what several memory frameworks like Mem0/Cognee/Zep/MIRIX use.)

• BM25: rank_bm25.BM25Okapi with default parameters (k1=1.5, b=0.75).

• Tokenizer for BM25: lowercased alphanumeric tokens via the regex [A-Za-z0-9]+.

5.5 §4.5 Metric↩︎

We use SubEM (substring exact match), the metric used in the MAB paper. For each predicted answer \(\hat{y}\) and ground truth list \(Y\), the answer is correct if any element of \(Y\) appears as a substring (case-insensitive) in \(\hat{y}\). This is the standard short-answer-QA metric.

SubEM caveat. Because SubEM credits substring matches, a verbose answer that incidentally contains the ground-truth string is counted correct. For our pipelines this is a non-issue (outputs are short entities pulled from the extracted candidate), but it inflates long-context oracle baselines somewhat. We report numbers under the same metric as MAB to keep comparisons apples-to-apples.

5.6 §4.6 Compute and reproducibility↩︎

• Total LLM-driven evaluations: 4400 (44 cells \(\times\) 100 questions), the 20-cell master table (§5.1) plus 24 cells of backbone, chunking, and resolver-diagnostic ablations (backbone and chunking analyzed in §5.3 and §5.6; the resolver-diagnostic cells are released in poc_results but their analysis is deferred to future work).

• Total API cost: ~$3 across the full benchmark (dominated by the 4 gpt-4o cells in Ablation C at ~$2; gpt-4o-mini cells contribute ~$0.3 across the headline, LLM-judgment baseline, chunk-4096 ablation, and CAR multi-hop runs).

• Wall-clock: each cell ~5–10 min; all 20 master-table cells run in parallel ~15 min total (the o4-mini multi-hop ablations run longer).

• Reproducibility: full code, prompts, per-question outputs, and Langfuse trace metadata (experiment, competency, dataset, question_index, ground_truth) are available in the project repository (Appendix D). The headline experiment is one shell command: pythonscripts/13_paper_experiment.py--sourcefactconsolidation_sh_<L>.

5.7 §4.7 Baselines↩︎

We compare against the 22 systems reported in Table 3 of MAB v3 (Hu, Wang, & McAuley, 2026), using the numbers as published (we do not re-run the published systems to avoid implementation differences). MAB’s Table 3 reports a single FC-SH and FC-MH number per system; we verified via MAB’s Table 10 (context-length ablation) that the FC columns in Table 3 correspond to the 262K context cell. The systems span four families: long-context FIFO baselines, simple RAG (BM25), embedding RAG (Contriever, Text-Embed-3, Qwen3-Embedding), structure-augmented RAG (RAPTOR, GraphRAG, MemoRAG, HippoRAG-v2, Mem0, Cognee, Zep), and agentic memory (Self-RAG, MemGPT, MIRIX). Detailed comparison is in §5.2.

6 §5 Results↩︎

We organize results into four parts: §5.1 the master table (all 20 cells with 95% Wilson CIs), §5.2 comparison to published systems at the matched (262K) context, §5.3 the matched-setup pipeline comparison, and §5.4 robustness to context length.

6.1 §5.1 Master results table↩︎

All numbers are accuracy (%) on \(n=100\) questions per cell, with 95% Wilson confidence intervals. The AVG column uses the pooled n=400 (all four lengths concatenated). The 6K column uses MAB’s published factconsolidation_*_6k rows (MAB reports 6K only for reasoning models, Table 5); 32K/64K/262K are from MAB’s main comparison. Bold marks the best result in each row; \(\star\) marks the three headline findings.

†6K uses MAB’s published factconsolidation_*_6k rows; MAB reports 6K only for reasoning models (Table 5), not in its Table-3 systems comparison. See §4.1.

Three headline findings:

1. Finding 1 (SH on gpt-4o-mini): Our pipeline (Headline) achieves 78.0% average on FC-SH, beating the best published gpt-4o-mini system (HippoRAG-v2 at 54%) by 24 percentage points. At 262K specifically, the gap is +28 pp (82% vs 54%).

2. Finding 2 (SH on gpt-4o): With gpt-4o backbone, the same pipeline achieves 94.8% average on FC-SH. At 262K specifically, the gap to GPT-4o long-context (60%) is +33 percentage points: same model, different post-processing.

3. Finding 3 (MH): The CAR pipeline (Ablation B) achieves 30.2% average on FC-MH, 23 percentage points above the best published system (Contriever / MemoRAG at 7%).

6.2 §5.2 Comparison to published systems↩︎

Numbers below from MAB v3 Table 3 (Hu, Wang, & McAuley, 2026), all with gpt-4o-mini as backbone except long-context entries which use the named model. Table 3’s single FC number per system corresponds to the 262K context cell (verified against MAB Table 10).

For apples-to-apples comparison, we report our 262K column below (not our pooled average across 4 lengths).

System	FC-SH @262K	FC-MH @262K
Ours (gpt-4o)	93.0	—
Ours (gpt-4o-mini)	82.0	—
Ours (CAR, gpt-4o-mini)	—	27.0
GPT-4o (long-context)	60.0	5.0
HippoRAG-v2 (best published)	54.0	5.0
BM25 (MAB chunk-512 default)	48.0	3.0
Zep / Graphiti (temporal KG)	7.0	3.0

The full 22-system comparison (every MAB Table 3 system, with gaps) is in Appendix E.

At the matched 262K comparison, our pipelines outperform every published MAB Table 3 system on FC-SH and FC-MH. The gap is largest against the most architecturally elaborate systems: Zep / Graphiti (temporal knowledge graph) is 86 percentage points below our gpt-4o result on FC-SH at 262K. Even the simplest published baseline (BM25 at 48%) is 45 percentage points below our gpt-4o result.

6.3 §5.3 Matched-setup pipeline comparison↩︎

We compare two pipelines at fixed backbone (gpt-4o-mini), fixed retrieval (BM25 TOP_K=10), fixed chunking (fact-level), fixed dataset, and matched n=100 per cell:

The +10.8 pp pooled-average gap is statistically reliable (non-overlapping 95% CIs at the marginal level; paired McNemar test \(\chi^2 = 14.6\), \(p < 0.001\); see §5.5), and the widening of the gap at long contexts (+21 pp at 262K vs +8 pp at 6K) supports the candidate-pool-size hypothesis: LLM-judgment degrades as candidate-pool size grows, while the deterministic resolver does not.

Caveat: what this comparison does and does not isolate. The two pipelines differ in three coupled ways: (1) the freshness resolver (LLM judgment vs Python max(serial)), (2) the prompt and output format (free-text answer following MAB’s BM25 prompt at temperature 0.7 vs structured JSON candidate-extraction at temperature 0.0), and (3) the LLM’s task (decide-and-answer vs extract-candidates). The +10.8 pp gap is therefore a whole-pipeline effect. To attribute it strictly to the resolver, one would need an additional cell (same candidate-extraction prompt, same temperature, but LLM-picks-newest instead of Python-max) under shared extraction and a harder, multi-candidate regime; we leave this clean isolation to future work.

A note on chunking. We also vary chunking (fact-level vs chunk-4096) while holding the deterministic-freshness pipeline fixed, to quantify the contribution of chunking strategy alone:

	6K	32K	64K	262K	AVG
chunk-4096 (Ablation A)	87	84	79	73	80.8
fact-level (Headline)	71	78	81	82	78.0
\(\Delta\) from per-fact chunking	-16	-6	+2	+9	-2.8

On average the two chunking strategies are essentially tied (78.0 vs 80.8, well within their 95% CIs), but the curves are opposite: chunk-4096 wins at short contexts; fact-level wins at the longest context (262K). We also ran the matched LLM-judgment + chunk-4096 cell: deterministic max() scores 80.8% vs LLM-judgment’s 61.0%, a +19.8 pp gap (6K +23, 32K +22, 64K +12, 262K +22), larger than the fact-level +10.8, confirming the structured-pipeline advantage persists under coarser chunking.

6.4 §5.4 Robustness to context length↩︎

The LLM-judgment baseline degrades sharply at 262K (61%, down from 75% at 64K, a 14-point drop). Our deterministic pipeline does not: 71–82% across all lengths with gpt-4o-mini, 92–99% with gpt-4o. The gap widens with context length: +8 pp at 6K \(\rightarrow\) +21 pp at 262K (Headline vs LLM-judgment baseline).

This is consistent with the hypothesis that the LLM’s freshness judgment fails when given many candidates with similar surface form. At short contexts, fewer conflicting facts get retrieved and the LLM can track serials. At long contexts, the candidate pool grows (more retrieved facts across more conflicting pairs) and the combined extract-and-judge baseline loses track. Our extract-then-aggregate pipeline does not: extraction isolates the candidates, and the deterministic max() then picks reliably.

Retrieval is not the cause. A natural worry is that the baseline’s 262K collapse reflects worse BM25 top-10 retrieval at long context rather than an LLM-judgment failure. Two checks rule this out. First, both pipelines use the identical top-10 (§4.3), yet the deterministic pipeline does not degrade (81% at 64K, 82% at 262K): a shared retrieval input cannot explain a drop in only one. Second, restricting to the questions where the counterfactual was provably retrieved (those the deterministic pipeline solved, which requires the fact to be in the top-10), the baseline’s accuracy still falls from 78% (64K) to 66% (262K) — even when handed the right fact, the free-text baseline mis-uses it. Of the baseline’s 14-point drop from 64K to 262K, ~10 points are mis-judgments of a retrievable fact and only ~4 (an upper bound, since this bucket also includes the deterministic pipeline’s own strict-extraction over-rejections) are questions neither pipeline solved (§5.5).

The same robustness extends across pipelines:

The Ablation C pipeline (gpt-4o + deterministic freshness + fact chunking) is the most context-robust: only a 7-pp range across 6K–262K, vs the LLM-judgment baseline’s 14-pp drop concentrated at the long extreme.

6.5 §5.5 Retrieval upper bound and method complementarity↩︎

To bound how much of the gpt-4o-mini ceiling is set by retrieval (BM25 TOP_K=10) versus by the freshness pipeline, we compute the union accuracy of the LLM-judgment baseline and the Headline pipeline on the same questions. The union is a lower bound on the retrieval ceiling: if neither pipeline got the question right, the correct answer either was not in TOP_K, or both extraction methods failed to surface it.

Two takeaways. First, 88.5% of questions are solvable by at least one of the two pipelines, which means retrieval is recovering the correct answer fact for the great majority of FC-SH questions; the remaining 11.5% is a lower bound on retrieval failure (questions where neither pipeline succeeds because BM25 TOP_K=10 missed the counterfactual entirely, or the extraction prompt rejected it). This gives a soft ceiling of ~88–95% for any fixed-TOP_K pipeline before retrieval becomes the bottleneck rather than freshness reasoning.

Second, the two pipelines are partially complementary, not strictly ordered. Pooled across n=400:

• Both correct: 227 (56.8%)

• Python max(serial) only correct: 85 (21.3%)

• LLM-judgment only correct: 42 (10.5%)

• Neither correct: 46 (11.5%)

The +10.8 pp headline gap is the net of the +21 pp Python-only wins and the \(-\)​11 pp LLM-judgment-only wins. The LLM-judgment baseline is not strictly dominated by the deterministic pipeline at the question level; it has its own ~10% of correct answers that the extraction-strict pipeline misses (typically when the candidate-extraction prompt over-rejects valid candidates due to predicate-strictness). This nuance is missed by reporting only pooled accuracy.

Paired statistical test. The contingency above (Python-only 85, LLM-only 42) yields McNemar’s \(\chi^2 = (85-42)^2 / (85+42) = 14.6\), \(p < 0.001\) with one degree of freedom, confirming the Headline pipeline’s advantage as statistically reliable on paired questions, not just non-overlapping marginal CIs.

6.6 §5.6 FC-MH results and failure mode analysis↩︎

The CAR pipeline (Ablation B) succeeds at 27–34% on FC-MH across all four context lengths, with 6K and 64K marginally higher (34%, 33%) than 32K and 262K (27%). The variance is consistent with the small per-cell n=100 sample (Wilson 95% CI widths ~17 pp).

Backbone scaling (new). Upgrading the CAR backbone sharply improves multi-hop conflict resolution:

FC-MH	6K	32K	64K	262K	avg
gpt-4o-mini + CAR	34	27	33	27	30.2
o4-mini + CAR	52	42	46	33	43.2
gpt-4o + CAR	57	50	58	41	51.5

gpt-4o+CAR reaches 41% at 262K vs the best published 7% (+34 pp). Notably gpt-4o+CAR (51.5) > o4-mini+CAR (43.2): the rigid per-hop pipeline does not leverage o4-mini’s reasoning. A striking crossover appears against plain reasoning models (MAB Table 5): at 6K, CAR hurts o4-mini (52 vs plain 80: the model already solves the chain end-to-end, and decomposition only adds hop-1 failure surface), but at 32K it rescues it (42 vs plain 14, which has collapsed). Decomposition is what keeps a reasoning model functional in the long-context regime where it otherwise falls apart. On single-hop, an o4-mini backbone reaches 96.5% avg (94–99% across all lengths), edging gpt-4o’s 94.8%: single-hop conflict resolution is effectively solved by a reasoning backbone.

Per-hop analysis. CAR plans an average of 2.56 hops per question across all FC-MH questions, with a maximum of 6 hops. Of the planned hops, 86% are successfully executed (return a non-null answer). The remaining 14% of hop failures (typically hop 1 returning “no answer” due to no candidate match) cascade to whole-pipeline failure, explaining a substantial fraction of the 70% wrong answers; the remainder come from hops that complete but extract the wrong candidate: most commonly counterfactual mis-identification at hop 1 or predicate-mismatch extraction at later hops.

Failure-mode breakdown. From spot-checking ~30 wrong answers across the four FC-SH length runs at 262K:

The most common CAR failure mode is hop-1 hallucination: when the question’s first entity has both a real-world fact and a counterfactual fact, the LLM occasionally picks the entity that does not match the question’s intent, and the chain breaks. Per-hop deterministic freshness correctly picks the newer serial when candidates exist, but cannot recover if the LLM extracted the wrong candidate to begin with.

6.7 §5.7 Cross-benchmark generalization (LongMemEval)↩︎

All results so far are on MAB FactConsolidation, whose freshness signal is an ordinal serial. To test whether the principle generalizes to (a) a different benchmark, (b) conversational rather than synthetic data, and (c) real timestamps rather than ordinal serials, we run the single-hop pipeline on LongMemEval (Wu et al., 2024), filtered to its knowledge-update question type (the freshness-conflict type), using the chat-session timestamp as the freshness signal (max(timestamp) in place of max(serial)). This is n=45 questions (9 per row \(\times\) 5 rows), each row ~1.6M characters of full chat history; backbone gpt-4o-mini.

On LongMemEval knowledge-update the deterministic pipeline does not beat the LLM baseline; the 6.6 pp difference sits well inside the overlapping 95% CIs. The per-question picture is more informative. Pooled across the 45 questions: both correct on 24 (53%), baseline-only on 5 (11%), ours-only on 2 (4%), neither on 14 (31%); the 14 “neither” are retrieval/extraction failures unrelated to freshness.

Critically, the freshness mechanism itself generalizes: max(serial) becomes max(timestamp) and still works on the “what is X currently?” questions that are ~90% of MAB-FC but only ~60% of LongMemEval-KU. The five questions where the baseline wins are not freshness failures (full examples in Appendix F):

1. Yes/No questions (3 of 5). The benchmark expects a “Yes/No” prefix; our extractor returns the matching fact verbatim, so SubEM fails while the free-text baseline synthesizes “Yes, …”. A question-format mismatch, not a freshness error.

2. Historical / relative questions (1 of 5). “What was your previous status before the current one?” Here max(timestamp) returns the current state; the question wants the second-newest. Wrong operator, not wrong mechanism.

3. Aggregation / filtering questions (1 of 5). “How many bass on the earlier trip before 7/22?” This needs filtering by time-of-event, not the latest mention.

This sharpens rather than weakens the thesis. Deterministic post-retrieval aggregation is the right primitive for current-value conflict resolution with explicit version markers, where it dominates (FC-SH +24/+28 pp) and ports cleanly from ordinal serials to real timestamps. For broader conversational memory QA it is one primitive among several: it must be composed with question-type-aware post-processing (a Yes/No wrapper, a k-th-newest selector, an aggregation handler) to cover the other question types. We leave that composable extension to future work (§6.6).

Reproducibility: scripts/15_longmemeval_cross_benchmark.py; result in poc_results/longmemeval_knowledge_update.json; <$0.10, ~2 min.

6.8 §5.8 Cost-quality tradeoff↩︎

The gpt-4o-mini pipeline is essentially free: $0.0001 per query. For applications where 78–80% accuracy is sufficient, this is the optimal choice. The gpt-4o pipeline at $0.005/query provides 94.8%, a 17 pp gain at 50\(\times\) the cost. The decision depends on the application’s accuracy budget.

7 §6 Discussion↩︎

7.1 §6.1 Why does this work?↩︎

The mechanism is straightforward when articulated:

The free-text LLM-judgment baseline is unreliable at applying explicit freshness rules when filtering, prior suppression, and answer generation are entangled in one step. This is documented in the knowledge-editing literature (Zhang et al., 2024; Kishore & He, 2024). When prompted with “facts are indexed by serial numbers; newer serials override older” and shown a counterfactual fact (higher serial) alongside a real-world fact (lower serial), the LLM tends to override the explicit rule with its prior. Worse, this failure scales with candidate-pool size (more conflicting candidates in context degrades accuracy more), which explains the LLM-judgment baseline’s sharp drop at 262K (61%, down from 75% at 64K).

Python max() over extracted candidates has no such pathology. It is exact, fast, and context-independent. The LLM’s role is narrowed to candidate extraction (identifying which retrieved facts semantically answer the question), where it does well.

The candidate extraction is intentionally strict: the LLM is instructed to match both subject and predicate verbatim, and to include every matching item, not pick the “best.” This pushes the LLM toward conservative behavior. When no candidate matches, the pipeline returns “no answer” rather than hallucinating. Empirically, ~2% of FC-SH questions hit this case; we count “no answer” as wrong by the SubEM metric, but in a production setting it is a calibrated abstention.

A precision-recall trade-off, not strict dominance. §5.5 makes the picture richer than “LLM bad, code good.” The deterministic pipeline trades a small amount of recall (the ~10% of questions where the strict extraction prompt over-rejects valid candidates, which the LLM-judgment baseline still answers correctly) for a larger gain in precision (no prior-override, no serial-tracking drift). The net effect is +10.8 pp, but the two pipelines are partially complementary at the question level (21% Python-only, 10.5% LLM-only). A hybrid pipeline that falls back to LLM-judgment when candidate extraction returns empty could plausibly close some of the residual 10.5% gap; we did not implement this hybrid but it is a natural extension.

7.2 §6.2 Implications for memory framework design↩︎

The agent-memory subfield has invested significant engineering in elaborate storage architectures. On this benchmark, architectural elaboration alone does not appear to solve conflict resolution:

• Knowledge graphs (GraphRAG, Cognee, Graphiti): require LLM-driven entity/relation extraction at ingestion, which propagates noise and introduces additional LLM-judgment steps. On FC-SH these score 7–28%.

• Hippocampal retrieval (HippoRAG-v2): the strongest published RAG at 54% on FC-SH. The architectural sophistication (multi-stage retrieval, iterative refinement) does not address the LLM-judgment bottleneck at query time.

• Agentic loops (MemGPT/Letta, MIRIX, Self-RAG): the iteration helps retrieval coverage but adds more LLM steps without making any of them freshness-aware. 14–28% on FC-SH.

• Temporal knowledge graphs (Zep / Graphiti, bi-temporal edges, validity intervals): explicitly designed to handle supersession. Yet Zep scores 7% on FC-SH, its lowest column in MAB Table 3. We emphasize that this is a per-task result on FC-style conflict resolution and does not contradict Zep’s reported strengths on other benchmarks (DMR, LongMemEval). The plausible explanation for the FC result is that Zep uses an LLM at ingestion time to decide whether new facts contradict existing ones (resolve_edge prompt), inheriting the same LLM-judgment problem we eliminate. The lesson is narrow: even temporal-KG infrastructure does not automatically solve conflict resolution if the resolution decisions are still made by LLM prompts.

The right primitive for conflict resolution is deterministic aggregation over candidates that the LLM has identified as semantically relevant. This matches the temporal-rag (Emmimal, 2025) library’s design, though temporal-rag is single-hop only; our CAR pipeline extends to multi-hop.

This view aligns with Memanto (Abtahi et al., 2026), which makes a similar “simpler beats complex” argument on different benchmarks (LongMemEval, LoCoMo) using information-theoretic retrieval + typed memory. We extend the principle to FactConsolidation.

7.3 §6.3 Why does Chain-Aware Resolution work on FC-MH?↩︎

Multi-hop conflict resolution combines two challenges: (1) decomposing the question into atomic hops, and (2) resolving conflicts at each hop. Our CAR pipeline addresses each separately. The decomposition uses a Self-Ask-style prompt with a HARD_CONSTRAINT that forbids more than one relationship word per hop (forcing the LLM to recursively split “X of Y of Z” into multiple hops). The per-hop conflict resolution uses the same SH-conflict primitive.

Existing systems either skip decomposition (BM25, embedding RAG, all memory frameworks) or skip per-hop conflict resolution (Self-Ask, IRCoT, MultiHop-RAG benchmark RAG). CAR does both. The result (30.2% avg vs 7% best published) demonstrates that multi-hop conflict resolution becomes tractable when both components are addressed.

That said, 30.2% is far from solved. Most CAR failures are at hop 1: the LLM occasionally extracts the wrong candidate when both real-world and counterfactual facts appear in retrieval. The error then cascades through subsequent hops. Improving hop-1 reliability is the most promising avenue for further work.

7.4 §6.4 Limitations↩︎

Primarily one benchmark. Our headline results are on MAB FactConsolidation; we add a cross-benchmark generalization check on LongMemEval knowledge-update (§5.7, n=45, wide CIs, one question type). The MQUAKE-derived counterfactuals may not reflect real-world conflict patterns. For example, real production memory data often has timestamps rather than ordinal serial numbers; partial-order updates (e.g., merged document revisions) are not captured.

The 6K column uses MAB’s published 6K rows. MAB ships FactConsolidation at 6K/32K/64K/262K but reports 6K only in its reasoning-model validation (Table 5), not in the main systems comparison (Table 3). We include 6K for completeness but exclude it from apples-to-apples comparisons against the published Table-3 numbers.

Version-marker assumption. Our deterministic max(serial) requires the source data to have a total ordering on facts. FactConsolidation provides this via numbered facts; production memory systems typically have it via timestamps or insertion order. The approach generalizes to any total order but cannot handle partial orders or causal dependencies between updates.

The deterministic pipeline is not strictly dominant. §5.5 shows 10.5% of FC-SH questions are answered correctly by the LLM-judgment baseline but missed by the Headline pipeline (vs 21.3% the other way). The deterministic pipeline wins on net but trades a small amount of recall (strict extraction over-rejection) for a larger gain in precision (no prior-override, no serial-tracking drift). Practitioners adopting this recipe should be aware of the trade-off and may benefit from the hybrid extension described in §6.1.

The matched-setup comparison varies more than the resolver alone. Our LLM-judgment baseline uses MAB’s published BM25 answer prompt at temperature 0.7, while the Headline pipeline uses a candidate-extraction prompt at temperature 0.0 with a deterministic Python resolver. The +10.8 pp gap is therefore a pipeline-level effect (resolver, prompt and output format, temperature, and the extract-then-aggregate task structure vary jointly), not a strictly isolated resolver effect; cleanly isolating the resolver’s own contribution — with shared candidate extraction and a harder, multi-candidate regime than FactConsolidation’s ~2 candidates per question — is future work.

Chunk-4096 is now matched. The matched LLM-judgment + chunk-4096 cell gives deterministic 80.8% vs LLM-judgment 61.0% (+19.8 pp; §5.3): the structured-pipeline advantage persists under coarser chunking.

Failure-mode analysis is qualitative. Our §5.6 failure-mode breakdown is from spot-checking ~30 wrong answers across the four FC-SH length runs at 262K, not a systematic 100-question manual labeling. The categories and proportions should be read as a qualitative pilot, not a precise distribution.

Three backbones explored. The master table uses gpt-4o-mini and gpt-4o; additional diagnostic sweeps include o4-mini (§5.6). Results may differ on other model families (Claude, Llama, open-source models). The qualitative finding (that the free-text baseline is unreliable at entangled in-context freshness reasoning) likely generalizes, but the quantitative numbers may shift.

Multi-hop remains genuinely hard. 30.2% on FC-MH, while a 4–5\(\times\) improvement over published, is still far from solved. The decomposition step has its own LLM-judgment failures (hidden-bridge questions, hop-1 hallucination) that our deterministic freshness does not address.

7.5 §6.5 Recommendations for memory framework designers↩︎

1. Preserve fact-level metadata at indexing time. Whatever chunking strategy you use, ensure the version markers (timestamps, serial numbers, version strings) are available at the granularity at which conflicts can occur.

2. Move freshness/recency comparisons out of LLM prompts and into deterministic code. Do not rely on free-text LLM prompting as the only freshness-resolution mechanism; when explicit metadata exists, deterministic aggregation is exact, cheaper, auditable, and easier to test.

3. Do not assume graph or knowledge-graph infrastructure alone solves conflict resolution. The empirical evidence is that an elaborate temporal KG (Zep / Graphiti) scores 7% on FC-SH despite being designed for temporal memory. Graph infrastructure may be appropriate for other reasons; conflict resolution is not a sufficient reason on its own without an explicit version-aggregation step.

4. Test on FactConsolidation early. It is a clean stress test for conflict resolution that production memory systems should pass. MAB provides it as a Python-installable benchmark.

5. Be honest about which sub-tasks your system handles. A memory framework that gets 18% on FC-SH should not claim to handle “evolving knowledge”: that result is well below a deterministic baseline.

7.6 §6.6 Open questions and future work↩︎

The cleanest open question is whether the same recipe transfers to real-world conflict patterns, e.g., user preference updates in conversational memory, policy supersession in compliance documents, fact corrections in collaborative knowledge bases. The mechanism suggested by this benchmark (free-text LLM judgment fails when freshness resolution is entangled with retrieval noise and prior conflict) may generalize, but requires validation on real-world memory data. Several of the experiments flagged in the first draft are now complete (this version); the rest remain open.

Completed in this version:

1. Matched chunk-4096 + LLM-judgment. Now run (§5.3): deterministic 80.8% vs LLM-judgment 61.0% (+19.8 pp), larger than the fact-level +10.8.

2. Backbone sweep. gpt-4o and o4-mini on CAR multi-hop (§5.6): gpt-4o+CAR 51.5%, o4-mini+CAR 43.2% (vs 30.2% with gpt-4o-mini); o4-mini single-hop 96.5%. A hybrid (LLM-judgment fallback on empty extraction) was a wash (+0.2 pp).

3. Cross-benchmark generalization + timestamps (LongMemEval). §5.7: max(serial) ports to max(timestamp) on conversational memory; the deterministic pipeline ties LLM judgment overall (57.8 vs 64.4, n=45) and wins cleanly on current-value questions, with losses confined to non-freshness question types. This answers the “real timestamps?” question flagged in the first draft.

Remaining future work:

1. A clean isolation of the resolver. The matched comparison (§5.3) varies the resolver, prompt, output format, and task structure jointly. A clean isolation runs candidate extraction once and feeds the identical candidate list to a Python-max resolver and an LLM picker, on a task with more than the ~2 candidates per question that FactConsolidation provides (ideally with timestamp/version-string metadata). We defer this to future work.

2. A composable, question-type-aware pipeline. The LongMemEval losses (§5.7) are Yes/No, historical/relative, and aggregation questions for which max() is the wrong operator. A natural extension routes each question type to its own handler (boolean wrapper, k-th-newest selector, aggregation/counting) atop the deterministic freshness primitive, tested on the full LongMemEval question mix and LoCoMo.

3. Real-world conflict patterns. Whether the recipe transfers to user-preference updates, policy supersession in compliance documents, and fact corrections in collaborative knowledge bases, on production memory data with native timestamps and partial-order updates.

8 §7 Conclusion↩︎

We presented a deterministic recipe for memory conflict resolution that achieves new state-of-the-art on MemoryAgentBench FactConsolidation: 78–94.8% on single-hop (vs HippoRAG-v2’s 54% and GPT-4o long-context’s 60% at 262K) and 30.2% on multi-hop (vs the best published 7%). A matched-setup comparison (same backbone, same retrieval, same chunking, same TOP_K, n=100 per cell) shows +10.8 percentage points from replacing an LLM-judgment-based answer pipeline with a candidate-extraction + Python max(serial) pipeline, with the gap widening to +21 pp at the longest published context (262K), with deterministic max() the simpler, exact, and auditable resolver. Cleanly isolating its accuracy contribution against an LLM picker is left to future work.

The implication is corrective for the memory-framework subfield: the bottleneck on conflict resolution is assembly (how candidates are extracted and aggregated post-retrieval), not storage (graph, hippocampal, agentic, or typed). The right primitive for resolution is deterministic post-retrieval aggregation. Elaborate memory architectures do not help on this task; a short Python function over LLM-extracted candidates matches or exceeds them. A cross-benchmark check on LongMemEval knowledge-update (§5.7) bounds the claim: the freshness mechanism ports to real timestamps but ties LLM judgment overall; deterministic aggregation is the right primitive for current-value conflict resolution and composes with question-type-aware handling for broader memory QA.

This points to a broader design pattern for the next generation of memory systems: narrow the LLM’s job to tasks it is reliably good at (semantic identification, candidate extraction) and delegate comparisons over structured metadata (serials, timestamps, version markers) to deterministic code. We release all code, prompts, and Langfuse traces for reproducibility; the headline experiment runs in 15 minutes for ~$3 in API spend.

9 References↩︎

Abtahi, S. M., Rahnema, R., Patel, H., Patel, N., Fekri, M., & Khani, T. (2026). Memanto: Typed Semantic Memory with Information-Theoretic Retrieval for Long-Horizon Agents. arXiv preprint arXiv:2604.22085.

Bi, B., Liu, S., Wang, Y., Mei, L., Gao, H., Xu, Y., & Cheng, X. (2024). Adaptive Token Biaser: Knowledge Editing via Biasing Key Entities. arXiv preprint arXiv:2406.12468.

Chhikara, P., Khant, D., Aryan, S., Singh, T., & Yadav, D. (2025). Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory. arXiv preprint arXiv:2504.19413.

Dey, R., Ounis, I., McDonald, G., & Moshfeghi, Y. (2026). Temporal Fact Conflicts in LLMs: Reproducibility Insights from Unifying DYNAMICQA and MULAN. arXiv preprint arXiv:2603.15892.

Emmimal. (2025). temporal-rag: A post-retrieval temporal layer for RAG systems. GitHub repository, https://github.com/Emmimal/temporal-rag.

Grofsky, M. (2025). Solving Freshness in RAG: A Simple Recency Prior and the Limits of Heuristic Trend Detection. arXiv preprint arXiv:2509.19376.

Gutiérrez, B. J., Shu, Y., Qi, W., Zhou, S., & Su, Y. (2025). From RAG to Memory: Non-Parametric Continual Learning for Large Language Models. International Conference on Machine Learning (ICML 2025). arXiv:2502.14802.

Hu, Y., Wang, Y., & McAuley, J. (2026). Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions. International Conference on Learning Representations (ICLR). arXiv:2507.05257.

Huo, N., Li, J., Qin, B., Qu, G., Li, X., Li, X., & Ma, C. (2025). Micro-Act: Mitigating Knowledge Conflict in LLM-based RAG via Actionable Self-Reasoning. arXiv preprint arXiv:2506.05278.

Khot, T., Trivedi, H., Finlayson, M., Fu, Y., Richardson, K., Clark, P., & Sabharwal, A. (2022). Decomposed Prompting: A Modular Approach for Solving Complex Tasks. International Conference on Learning Representations (ICLR 2023). arXiv:2210.02406.

Kishore, S., & He, H. (2024). Unveiling Divergent Inductive Biases of LLMs on Temporal Data. arXiv preprint arXiv:2404.01453.

Lee, J., Lee, K., & Kim, T. (2025). MAGIC: A Multi-Hop and Graph-Based Benchmark for Inter-Context Conflicts in Retrieval-Augmented Generation. arXiv preprint arXiv:2507.21544.

Liu, S., Shang, Y., & Zhang, X. (2025). TruthfulRAG: Resolving Factual-level Conflicts in Retrieval-Augmented Generation with Knowledge Graphs. arXiv preprint arXiv:2511.10375.

Madhwal, D., Zhang, L. D., Roth, D., Wolfson, T., & Gupta, V. (2026). Decomposed Prompting Does Not Fix Knowledge Gaps, But Helps Models Say “I Don’t Know”. arXiv preprint arXiv:2602.04853.

Özer, A., & Yıldız, Ç. (2025). Question Answering under Temporal Conflict: Evaluating and Organizing Evolving Knowledge with LLMs. arXiv preprint arXiv:2506.07270.

Packer, C., Wooders, S., Lin, K., Fang, V., Patil, S. G., Stoica, I., & Gonzalez, J. E. (2023). MemGPT: Towards LLMs as Operating Systems. arXiv preprint arXiv:2310.08560.

Press, O., Zhang, M., Min, S., Schmidt, L., Smith, N. A., & Lewis, M. (2022). Measuring and Narrowing the Compositionality Gap in Language Models. Findings of the Association for Computational Linguistics: EMNLP 2023. arXiv:2210.03350.

Rasmussen, P., Paliychuk, P., Beauvais, T., Ryan, J., & Chalef, D. (2025). Zep: A Temporal Knowledge Graph Architecture for Agent Memory. arXiv preprint arXiv:2501.13956.

Shao, Z., Gong, Y., Shen, Y., Huang, M., Duan, N., & Chen, W. (2023). Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergy. Findings of the Association for Computational Linguistics: EMNLP 2023. arXiv:2305.15294.

Tang, Y., & Yang, Y. (2024). MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queries. Conference on Language Modeling (COLM 2024). arXiv:2401.15391.

Trivedi, H., Balasubramanian, N., Khot, T., & Sabharwal, A. (2023). Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions. Annual Meeting of the Association for Computational Linguistics (ACL). arXiv:2212.10509.

Wu, D., Wang, H., Yu, W., Zhang, Y., Chang, K.-W., & Yu, D. (2024). LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory. arXiv preprint arXiv:2410.10813.

Zhang, M., Ye, X., Liu, Q., Ren, P., Wu, S., & Chen, Z. (2024). Uncovering Overfitting in Large Language Model Editing. International Conference on Learning Representations (ICLR 2025). arXiv:2410.07819.

Zhong, Z., Wu, Z., Manning, C. D., Potts, C., & Chen, D. (2023). MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questions. Conference on Empirical Methods in Natural Language Processing (EMNLP). arXiv:2305.14795.

10 Appendix↩︎

10.1 A. Hyperparameters↩︎

10.2 B. Confidence-interval computation↩︎

Wilson 95% CIs computed as \(\text{CI}=\frac{p + z^2/(2n) \pm z\sqrt{p(1-p)/n + z^2/(4n^2)}}{1 + z^2/n}\) with \(z = 1.96\), \(n = 100\) per cell, \(n = 400\) pooled across the four context lengths. Wilson is preferred over Wald at the extremes (e.g., 99% accuracy) because Wald produces intervals that exceed 1.0 there.

10.3 C. Per-cell raw results↩︎

All 20 cells (5 pipelines \(\times\) 4 context lengths) are documented in machine-readable JSON in the supplementary repository at poc_results/. Filenames follow the pattern:

• paper_sh_conflict_factconsolidation_sh_<L>.json (Headline + LLM-judgment)

• ablation_<task>_<chunk>_<model>_factconsolidation_<task>_<L>.json (ablations)

10.4 D. Reproducibility↩︎

Code and data. The full code and per-question JSON results are at https://github.com/cvikasreddy/memory-conflict-resolution (public as of the arXiv posting). All 2000 evaluations in the master table were saved as machine-readable JSON files in poc_results/, including per-question retrieved-fact serials, extracted candidates, chosen serial, and SubEM correctness.

One-command headline replication:

Full benchmark: ~$3 in API spend, 15 min wall-clock with 4 parallel processes.

Langfuse traces: tagged traces available on request (subject to Langfuse organization access).

10.5 E. Full comparison to published systems (MAB Table 3, 262K)↩︎

Numbers from MAB v3 Table 3 (Hu, Wang, & McAuley, 2026), all with gpt-4o-mini backbone except long-context entries (named model); Table 3’s single FC number per system corresponds to the 262K cell (verified against MAB Table 10). “Gap vs ours best” is relative to our gpt-4o FC-SH (93.0) and our gpt-4o-mini CAR FC-MH (27.0). At matched 262K, our pipelines exceed every published system on both FC-SH and FC-MH.

10.6 F. LongMemEval cross-benchmark: failure-mode detail (§5.7)↩︎

The five LongMemEval knowledge-update questions where the LLM baseline beat the deterministic pipeline, none of which are freshness failures:

Category I: Yes/No questions (3 of 5). The benchmark expects a “Yes/No” prefix; our extractor returns the matching fact verbatim:

Q14: "Do you have a spare screwdriver for opening up my laptop?"
  GT:       ['Yes']
  Ours:     "spare screwdriver (I actually have a spare screwdriver...)"  -> SubEM FAILS
  Baseline: "Yes, you have a spare screwdriver..."                        -> SubEM PASSES

Root cause: the candidate-extraction prompt does not synthesize Yes/No; the free-text baseline does. A question-format mismatch with our terse extraction output, not a freshness error.

Category II: Historical / relative questions (1 of 5).

Q10: "What was your previous frequent flyer status on United Airlines
      before getting the current status?"
  GT:       ['Premier Silver']
  Ours:     "Premier Gold"   (max(timestamp) picked the CURRENT status)  -> WRONG
  Baseline: "Premier Silver" (reasoned about 'previous')                 -> CORRECT

Root cause: max(timestamp) is the wrong operator for “the state at t-1”; the pipeline needs a k-th-newest selector.

Category III: Aggregation / filtering across time (1 of 5).

Q19: "How many largemouth bass did I catch on the EARLIER trip to Lake
      Michigan BEFORE the 7/22 trip?"
  GT:       ['7']
  Ours:     "9"  (the most recent mention of bass)                       -> WRONG
  Baseline: "7" (filtered to the 7/10 trip)                              -> CORRECT

Root cause: the question filters by time-of-event, not time-of-mention; max(timestamp) selects the latest mention.

All three are question-type issues addressable by composition (§6.6), not failures of the deterministic freshness mechanism.