July 11, 2026
Agentic LLM coding tools compress long session histories into compaction summaries that subsequent sessions inherit as ground truth. This paper documents a failure mode in Claude Code where partial standard output from timed-out commands (exit code 143) is recorded in compaction summaries as confirmed results, propagating false positives across sessions and model versions without re-verification. The underlying mechanism is a conflation of observation and persistence, where information that appeared in the terminal is treated as equivalent to information written to durable storage. This finding extends the analysis of LLM self-evaluation failures reported in prior work on non-determinism in LLM-as-judge grading [1] by showing that agentic tools exhibit analogous reliability deficits when reporting on their own operational outcomes. The failure has direct implications for any workflow that relies on agentic session continuity for data processing, scientific computation, or multi-step automation.
Keywords: agentic AI, session management, compaction, epistemic failure, false confirmation, data integrity
Agentic LLM tools such as Claude Code [2] and ChatGPT Codex [3] enable users to delegate multi-step programming and data processing tasks to language models operating within persistent terminal environments. These tools maintain session continuity through various mechanisms: Claude Code compresses long conversation histories into compaction summaries that are injected into subsequent sessions as contextual grounding, while Codex operates within sandboxed environments where task state is recorded alongside terminal output.
A core assumption underlying the utility of these tools is that information carried across sessions accurately represents what occurred in prior sessions. When a compaction summary states that a command completed with a particular result, downstream sessions treat that statement as established fact. This assumption is rarely examined empirically, in part because the compaction process itself is opaque to users: the summary is generated automatically, its contents are not independently verifiable against raw session logs, and the model receiving the summary has no mechanism to distinguish summarized observations from confirmed, persisted outcomes.
This paper reports a specific instance in which this assumption failed. During a batch API query session in Claude Code, a long-running script was terminated by the system timeout mechanism (SIGTERM, exit code 143) before its output could be written to disk. The compaction summary nevertheless recorded partial results that had appeared in the terminal prior to termination as if the corresponding iterations had completed successfully. A subsequent session, running on a different model version, inherited these false positives and treated them as confirmed data points. The author’s own interaction with the Claude Chat interface further propagated the error: when presented with the summary’s claims, the chat-based model also treated them as established results without requesting file-level verification.
The finding is not limited to a single anomalous event. Over the same working period, four structurally identical failures were observed across different task domains, all exhibiting the same pattern: information present in the model’s context window was treated as confirmed fact regardless of whether it had been verified against external, persistent state.
This paper contributes the following:
A documented, reproducible failure mode in which session compaction transforms ephemeral terminal output into fabricated confirmed results.
A mechanism analysis identifying the conflation of observation and persistence as the root cause.
A connection to prior work on LLM self-evaluation reliability [1], extending the analysis from evaluation outputs to operational self-reports.
Agentic coding tools operate within extended sessions that can span hours of interaction. As conversations grow, the context window fills, requiring compression mechanisms to preserve relevant information while discarding lower-priority content. Claude Code implements this through session compaction: when the context window approaches capacity, earlier portions of the conversation are summarized into a condensed representation that is then used as the basis for continued interaction [2]. The compacted summary functions as the model’s “memory” of what occurred earlier in the session, and it is the only record available to the model when a new session begins from a prior handoff.
ChatGPT Codex takes a different approach: tasks run in sandboxed environments with persistent file systems, and task state is recorded alongside terminal logs, standard output, and standard error streams. When a user queries the status of a prior task, Codex can inspect both the recorded terminal output and the actual files written to the workspace.
A fundamental distinction in systems that operate on external state is the difference between observing an event and confirming that the event produced a durable outcome. In database systems, this distinction is enforced through commit protocols: a transaction is not considered complete until the commit record is written to persistent storage, regardless of what intermediate states were observed during execution [4]. In software testing, the distinction manifests as the difference between “the test printed PASS” and “the test suite recorded PASS in its results file.”
For agentic LLM tools, this distinction applies at the level of command execution. When a model runs a script that produces output to both the terminal (standard output) and a file, the terminal output is ephemeral: it exists in the session’s context window but vanishes if the process is killed before the file write completes. A reliable session management system must track whether a command’s intended side effects (file writes, database updates, API responses persisted to storage) actually occurred, not merely whether the command produced visible output before termination.
The broader problem of LLM-generated content deviating from ground truth has been extensively studied under the rubric of hallucination. Ji et al.[5] provided a comprehensive survey of hallucination in natural language generation, establishing taxonomies for intrinsic and extrinsic hallucination across multiple generation tasks. Huang et al.[6] extended this analysis to large language models specifically, identifying principles and open questions around factual inconsistency in model outputs. The compaction failure documented here can be understood as a form of operational hallucination: the model’s summary of its own session history deviates from the actual operational record.
On the evaluation side, Zheng et al.[7] introduced the MT-Bench framework for evaluating LLM judges, documenting systematic biases in model self-evaluation. Norman et al.[8] conducted a large-scale evaluation of LLM-as-judge models, reporting on agreement, consistency, and bias across evaluation contexts. Tamba [1] demonstrated that LLM-as-judge evaluation scores are non-deterministic even at temperature zero, a finding that the present work extends from evaluation outputs to operational self-reports.
On the agentic tool side, Yang et al.[9] introduced SWE-bench for evaluating code generation agents, though the benchmark focuses on task completion rather than session-level reliability. Wang et al.[10] examined executable code actions in LLM agents, identifying failure modes in multi-step code generation. Ruan et al.[11] proposed an LM-emulated sandbox for identifying risks of language model agents, demonstrating that tool-using agents can produce unsafe actions that are difficult to detect without systematic evaluation. Shinn et al.[12] introduced Reflexion, a framework for language agents to learn from verbal self-reflection, which implicitly assumes that the agent’s self-reports are accurate, an assumption that the present work challenges.
Zhang et al.[13] surveyed memory mechanisms in LLM-based agents, identifying architectures for short-term and long-term memory management but not examining the epistemic reliability of compressed session summaries. Hitzig et al.[14] analyzed approximately 400,000 Claude Code sessions, finding that users own 70% of planning decisions while Claude handles 80% of execution, a delegation pattern that amplifies the consequences of compaction failures: when users rely on session summaries rather than independently verifying outputs, false confirmations propagate unchecked. To the author’s knowledge, no prior work has specifically examined the conditions under which compaction summaries fabricate confirmed results from interrupted processes.
The failure was observed during a batch API query session conducted in Claude Code on July 11, 2026. The task involved executing a Python script that issued sequential HTTP requests to an external API, processed the responses, and wrote results to a JSON output file. The script was designed to iterate through a predefined list of query configurations (labeled A1 through Z3), executing each query and appending the result to a cumulative output file.
The session proceeded as follows:
The script was initiated within a Claude Code session running model version Opus 4.7.
The first two iterations (A1 and A2) executed successfully: the API returned HTTP 200 responses, and the results appeared in the terminal output.
Before the script could write its cumulative results to the output JSON file, the process was terminated by the system timeout mechanism with exit code 143 (SIGTERM).
Claude Code’s compaction mechanism generated a summary of the session state. This summary recorded A1 and A2 as “completed with specific results,” including extracted field values from the API responses.
A new session was initiated, running on model version Opus 4.6. This session inherited the compaction summary and treated the A1 and A2 entries as confirmed data points.
The author, interacting through a separate Claude Chat session, referenced the summary’s claims and also treated A1 and A2 as established results.
When the actual output file (api_openai_baseline_20260711.json) was inspected, it contained zero successful entries. All 11 query attempts recorded in the file had failed with HTTP 400 errors. The “confirmed results” for A1 and A2
existed only in the ephemeral terminal output of a killed process.
The false confirmation propagated through three distinct layers:
The compaction summary treated terminal output observed before process termination as equivalent to confirmed, persisted results. Exit code 143 was not recorded as a disqualifying condition.
The subsequent session (Opus 4.6) received the summary from the prior session (Opus 4.7) and accepted its claims without file-level verification. No mechanism existed for the downstream model to distinguish between “the summary says X completed” and “X’s results are confirmed in the output file.”
The Claude Chat interface, operating as a separate product surface, also accepted the summary-derived claims as fact when the author referenced them. The chat model did not request or perform independent verification.
Over the same working period (July 10–11, 2026), four structurally identical failures were documented within Claude Code sessions, all exhibiting the pattern of treating context-internal information as externally verified:
A database synchronization process had silently failed for twenty days due to missing INSERT logic. Because no error appeared in the context window, the model reported the system as operational when queried.
A dictionary lookup function was referenced but never executed against its target data structure. The model treated the function’s existence in context as equivalent to its having been applied.
A data repository’s file listing was reported as current despite being weeks out of date, because no “stale” indicator was present in the context.
The A1/A2 false confirmation described above.
All four instances follow the same rule: context-internal equals true, context-external equals nonexistent. Information present in the model’s context window is treated as confirmed; information absent from the context window is treated as though it does not exist, regardless of whether verification would be trivially possible.
The root cause of the compaction failure is the absence of a distinction between two epistemically different states:
Observed: Output appeared in the terminal during script execution.
Persisted: Output was written to a durable storage medium (file, database, etc.)and can be independently verified.
Claude Code’s compaction mechanism treats both states identically. When generating a summary of a session, the mechanism records what appeared in the terminal output without checking whether the corresponding file writes completed. This is analogous to a laboratory notebook that records observations without distinguishing between “the instrument displayed a reading” and “the reading was saved to the data logger.”
When a process is killed by SIGTERM (exit code 143), the partial output that appeared before termination remains in the session’s context window. The compaction mechanism processes this context window content without reference to the process’s exit status. The summary therefore contains entries like “iteration A1 completed with result X” without any qualifier indicating that the process exited abnormally or that the result was never persisted.
A corrected implementation would need to satisfy at least two conditions: (1) the exit code of every executed command must be preserved and associated with the outputs attributed to that command, and (2) a non-zero exit code must trigger either re-verification of claimed outputs against persistent artifacts or explicit annotation of results as unconfirmed.
Once a false positive enters a compaction summary, it is amplified by the inheritance mechanism. The downstream session has no way to evaluate the epistemic quality of individual claims within the inherited summary. Every statement carries equal weight: “the user asked about X” and “iteration A1 completed with result Y” are indistinguishable in their epistemic status within the context window. The downstream model lacks both the metadata to identify which claims require verification and the procedural impulse to verify any claim that arrives via the compaction pathway.
This creates a structural vulnerability: a single false positive in a compaction summary becomes an irrevocable fact for all downstream consumers. The only corrective mechanism available is external verification by the user, which requires the user to independently check output files, a step that defeats the purpose of delegating the task to an agentic tool in the first place.
The compaction failure reported here is structurally related to the LLM-as-judge non-determinism documented in Tamba [1]. That work demonstrated that LLM evaluation scores are unreliable even under ostensibly controlled conditions (temperature set to zero), with position bias and grading variance producing inconsistent safety assessments across repeated evaluations of identical content. The present finding extends this analysis from evaluation outputs to operational self-reports.
In both cases, the underlying problem is that the model cannot accurately evaluate its own outputs. In the evaluation context, the model produces inconsistent scores for the same input depending on presentation order. In the operational context, the model produces a session summary that misrepresents what actually occurred. Both failures reflect a deficit in self-referential reliability: the model’s report about its own behavior does not reliably correspond to the behavior that actually took place.
The connection has a further dimension. The asymmetric skepticism documented in GitHub issue #66273 [15] showed that Claude Code applies differential scrutiny to self-generated content versus external content, consistently over-trusting its own outputs. The compaction failure is the temporal extension of this asymmetry: the model not only over-trusts its current outputs but also over-trusts its own prior session summaries, creating a chain of unverified self-reference that degrades reliability with each link.
The three-layer propagation chain documented in Section 3.3 (compaction, cross-model inheritance, cross-product propagation) represents a partial application of the Tri-Layer Integrated Model [16], which provides a general framework for analyzing recursive classification failures across layered information systems.
Taken together, these findings suggest that agentic LLM tools require what might be called an observer-aware protocol: a set of design constraints that prevent the model from treating its own observations as equivalent to independently verified facts. Such a protocol would require that any claim about external state (file contents, API responses, database records) be verified against the actual state before being recorded in a compaction summary or inherited by a subsequent session.
The most direct implication is that compaction summaries must distinguish between observed and verified outcomes. Any command that exits with a non-zero status code should have its outputs flagged as unconfirmed in the compaction summary. Ideally, compaction would include a verification step in which claimed outputs are checked against persistent artifacts before being recorded as confirmed.
Downstream sessions that inherit compaction summaries should not treat all inherited claims with equal confidence. A tiered confidence model, in which claims derived from verified file contents are distinguished from claims derived from terminal output, would reduce the propagation of false positives. This is functionally equivalent to the distinction between committed and uncommitted transactions in database systems.
When session state crosses model version boundaries (as in the Opus 4.7 to Opus 4.6 handoff observed here), the epistemic risk increases because each model in the chain has an independent opportunity to accept or challenge inherited claims. In practice, no model in the observed chain exercised skepticism toward the inherited summary. Designing explicit verification checkpoints at model-version boundaries would mitigate this risk.
The failure has immediate practical consequences for any user who relies on agentic tools for data processing, API interactions, or scientific computation. Session summaries cannot be trusted as confirmation that operations succeeded. Users must independently verify output files after each significant operation, which introduces a manual verification burden that partially negates the efficiency gains of agentic delegation.
This study has several limitations. First, the primary failure was observed in a single instance with a specific configuration (Claude Code, Opus 4.7/4.6, Windows platform, Python script with API calls). While the mechanism analysis suggests the failure should generalize to any scenario involving process termination before file write completion, broader empirical validation across platforms and task types would strengthen the finding.
Second, the cross-tool comparison with ChatGPT Codex involved fundamentally asymmetric conditions. The Claude Code failure occurred at the end of a multi-hour session with substantial context accumulation across heterogeneous tasks (database repair, repository management, patch development, and API experimentation). The Codex test was conducted in a clean, sandboxed environment with no prior context, no accumulated session history, and no cross-task interference. This asymmetry means the comparison demonstrates that Codex’s file-first verification approach works under light conditions, but does not establish whether Codex would maintain this reliability under equivalent long-session, high-context stress. A heavy Codex user running extended data processing sessions with comparable context accumulation might encounter analogous failures. The comparison should therefore be read as evidence that file-first verification is a viable design principle, not as evidence that Codex is categorically immune to compaction-type failures.
Third, the discovery itself reflects a methodological reality: the failure was identified through extended, intensive use of Claude Code as the author’s primary agentic tool. This participant-observation-like method provides ecological validity that controlled experiments cannot replicate, but it also means the finding is shaped by the specific workflow patterns of a single heavy user. It remains possible that different usage patterns (shorter sessions, fewer concurrent tasks, less context accumulation) would not trigger the failure.
Fourth, the compaction mechanism in Claude Code is not publicly documented in sufficient detail to determine whether the observed behavior reflects a design decision, an implementation oversight, or an emergent property of the summarization process.
Fifth, the corroborating instances (database sync failure, dictionary non-lookup, stale file listing) were observed within the same researcher’s workflow over a 48-hour period, introducing the possibility that specific working patterns or session structures contributed to the failures.
Sixth, an informal, single-session test (n=1) with ChatGPT Codex [3] suggested that a file-first verification approach correctly distinguished completed from interrupted runs under minimal-context conditions. However, the author subsequently exported and deleted all ChatGPT data (approximately 2 GB) due to concerns about its use in model training, and the test artifacts are therefore no longer available for independent verification. This observation is noted for completeness but does not constitute reproducible evidence of architectural differences between tools. Whether Codex would exhibit analogous failures under equivalent long-session, high-context conditions remains an open question for future work.
Session compaction in agentic LLM tools introduces an epistemic vulnerability: the compression of session history can transform ephemeral observations into fabricated confirmations. The specific failure documented here, in which partial terminal output from a timed-out process was recorded as confirmed results and propagated across sessions and model versions, demonstrates that the design of session management mechanisms has direct consequences for data integrity.
The broader pattern, treating context-internal information as externally verified, connects to prior findings on LLM self-evaluation reliability [1] and self-referential asymmetry [15]. In each case, the model’s report about its own behavior diverges from the behavior that actually occurred. Addressing this class of failure requires design-level interventions: verification checkpoints at compaction boundaries, exit-code-aware summarization, and tiered confidence models for inherited session state.
The finding is independently reproducible. The reproduction protocol requires: (1) initiating a long-running script in Claude Code that produces incremental terminal output, (2) allowing the process to be killed by the system timeout (exit code 143) after partial output has appeared, (3) continuing or resuming the session and observing whether the compaction summary records the partial output as confirmed results, and (4) verifying the actual output file to confirm that the claimed results were never persisted.
Code and technical documentation are available at: https://github.com/anthropics/claude-code/issues/76584.