July 14, 2026
This paper defines and proposes TRACE (Typed Reasoning And Commitment Evidence): a typed, versioned schema for recording reasoning traces, one reference procedure for writing records against that schema, and one operating discipline, no durable state change without a record. The paper first states goals and non-goals, then argues in three layers that reasoning is not in the language model: the autoregressive mechanism natively computes association; chain-of-thought and reinforcement learning inherit the mechanism’s limits; and the formal constructs of reasoning theory, from Socratic procedure to Pearl’s ladder, are absent as machinery. The TRACE schema answers the absence with fields and tests: the TraceRecord and its causal specialization, an eight-stage reference writer, a gate-first measurement regime, the TRACE-Bench protocol, and the consumers, memory admission, plan gating, temporal regret, and verdict reuse, whose improved and more auditable decisions are the measure by which the record is judged. A record-consumer contract states the guarantees a record makes to a consumer and the obligations a consumer accepts in return, so the schema is an operational interface rather than a passive document. Two worked examples run in the main text: a music-lessons argument traced from raw sentence to typed verdict, which shows the writer separating association, intervention, and prescription; and a flood search-and-rescue vignette in which a predictive world model reports a confident plan success that its own support and out-of-distribution scores contradict, so the record defers the commitment, requests a bounded observation, revises the claim append-only, and clears a different plan branch. The vignette is illustrative, not an empirical result; a closed-loop evaluation is specified but left to future work, so the paper’s contribution is the schema and its consumer contract, not a performance claim. Appendices carry the full schema, the reference writer’s algorithms and cost model, the contested anatomy, worked illustrations at clinical and policy stakes, the benchmark protocol, the convergence-validity metrics and imported propositions, and usage scenarios for memory admission and plan gating.
This paper defines and proposes TRACE (Typed Reasoning And Commitment Evidence): a typed, versioned schema for recording reasoning traces, one reference procedure for writing records against that schema, and one operating discipline that makes the records consequential. The artifact is the TRACE record; everything else in the paper either writes it, measures it, or consumes it. So that the reader’s expectations are set before any machinery arrives, the paper’s goals and non-goals are stated here, first.
The paper pursues three.
Record. Define one unified schema, concretely a versioned JSON type definition, that captures the typed structure of a reasoning episode: what was claimed, what kind of reasoning the claim required, what evidence and warrants were offered, what was tested, what failed, what is missing, and what verdict followed.
Protocol. Make the same schema the interchange format between an agentic system’s services: writers emit records, consumers act on them, and no service passes reasoning state to another in any other form. The schema is simultaneously a log format and an exchange protocol.
Improvement, through the callers. Improve reasoning quality where it can actually be improved: in the systems that call TRACE and consume its records. SocraSynth [1] and EVINCE [2] elicit against recorded claims; RCA [3] localizes failures to recorded stages; ERM [4] prices recorded errors; RLER [5] revises policies from recorded verdicts; Trivium [6] computes temporal regret over recorded steps; Mnemosyne [7] gates memory and MACI [8] gates plans on recorded outcomes. The record is what makes each of these operations possible, and consumer value is how the record is judged.
Three disclaimers bound the proposal.
TRACE does not itself optimize or improve reasoning. It is infrastructure in service of its callers: it records and adjudicates so that the systems above it can improve reasoning, the way a write-ahead log recovers no transaction by itself.
The eight-stage reference writer is not claimed to be optimal; it is one conforming implementation, and every stage is replaceable without moving the record format.
TRACE is not a universal reasoning-quality metric. Its scores measure compliance with the declared, versioned TRACE policy; no objective scalar of reasoning quality exists, and nothing essential in the design depends on one.
Systems engineering has met this situation twice before. A write-ahead log does not make a database faster or smarter, and a flight recorder does not fly the aircraft. Yet recovery, replication, and audit are impossible without the first, post-incident analysis is impossible without the second, and in both cases the value of the log is proven entirely by what its consumers can do. The TRACE record plays the same role for reasoning in an agentic system. It does not make the underlying model more correct. It makes every durable consequence of the model’s reasoning attributable, contestable, and improvable.
The schema comes with one operating rule, which the companion implementation enforces: no durable state change without a TRACE record. A memory is not committed, a plan is not cleared, and an obligation is not updated unless a TRACE record has been written against the schema and carries a verdict. Section 7.1 makes this closure concrete in a flood search-and-rescue plan: a high prediction is held when its support fails, a drone observation repairs the record, and only the dependent route branch is revised. The companion volume’s implementation chapter closes this loop; Appendices 15 and 16 sketch the two admission consumers.
The paper’s design claim is therefore:
The unit of contribution is the record, not any stage algorithm. Reasoning evaluation should be decomposed into typed, auditable responsibilities that read and write one versioned TRACE record; a claim must be typed before it is tested, and tested at the standard its type demands; and the record is judged not by a quality scalar but by what its consumers, admission control, plan gating, and temporal learning, can do with it.
Two consequences of this claim are worth stating before the machinery arrives. First, the paper’s foundational claim is not that TRACE measures reasoning quality; it is that TRACE records the typed structure of reasoning sufficiently for admission, audit, regret, repair, and downstream learning. Throughout this paper and its companion volume, TRACE names this whole contract: the TRACE schema, the TRACE writer contract, and the TRACE discipline, so that a TRACE, in the ordinary sense of the word, is exactly what the system leaves behind. TRACE still performs evaluation-like functions, but its verdicts are fields of the record, and the stages presented below are one implementation of the writer contract, every one of them replaceable without moving the record format. Second, the scores defined later measure compliance with the declared, versioned TRACE policy, not objective reasoning quality. Section 7 states both points precisely, together with what would falsify the design.
The paper makes five contributions. First, a schema: the TraceRecord and its causal specialization the TraceCausalRecord (Section 3), versioned like an interface, carrying evaluation state, including rung, identification status, bounds, missing evidence, and regret, rather than argument structure alone. Second, a type system for the schema’s fields: four layers of typed evaluation that route claims across formal disciplines (identification theory, proof checking, decision theory, law), where prior argument-mining pipelines and scheme-based systems such as Carneades classify and evaluate within the argumentation layer; an explicit subsumption rule governs which layer adjudicates when they overlap. Third, a reference writer (Algorithms 5 and 6), one conforming implementation of the writer contract, presented for auditability and cost discipline rather than claimed as optimal. Fourth, two complementary worked illustrations: one ordinary sentence followed through every writer stage, and one flood search-and-rescue plan followed from predictive evidence through defer, evidence-seeking repair, append-only revision, and plan-gate consumption. Fifth, a benchmark and ablation plan, TRACE-Bench, which judges the schema on expressiveness, the writer on stage-level and end-to-end quality, and the record on consumer value, and whose baselines include the single-pass writer that serves as the null hypothesis for the staged decomposition.
The paper proceeds as aim, critique, schema, hierarchy, stages, measurement, consumers, illustrations, debate, guarantees, and falsification. Section 2 states the layered critique. Section 3 fixes the terminology, the schema and its key fields, and the writer API (Section 3.3). Section 4 builds the reasoning hierarchy that gives the typing fields their semantics: the CRIT roof (Section 4.1), the four evaluation layers (Section 4.2), the reasoning families with the Toulmin, Walton, and Dung toolbox and the enthymeme-reconstruction algorithm (Section 4.3), and Pearl’s ladder as the causal plug-in (Section 4.4). Section 5 depicts the writer’s stages: the formulation gate (Section 5.1), one claim traced from sentence to verdict (Section 5.2), and the stopping rules (Section 5.4). Section 6 defines the TRACE policy’s quality metrics and hard gates; the benchmark that operationalizes them, TRACE-Bench, is Appendix 13, and the convergence theory and validity metrics of the debate stage are Appendix 14. Section 7 states what the paper claims, including the consumer claim that TRACE is a better admission explainer rather than a better gate. Section 7.1 then gives the operational consumer illustration before the paper states what it does not claim, what would falsify it, and its limits. Worked illustrations at clinical and policy stakes are in Appendix 12.
Reasoning is the capability on which the current era of AI turns. The decisive advances of the last five years were not simply advances in knowledge storage. They were advances in elicited computation: training models to emit intermediate steps or scratchpads [9], prompting them to produce chains of thought [10], [11], sampling and voting over diverse reasoning paths [12], decomposing problems from least to most complex [13], searching over branching thoughts [14], and interleaving reasoning with action [15]. The industry then made reasoning the explicit scaling axis: allocating inference-time compute to deliberation can outperform scaling parameters [16], and reinforcement learning with verifiable rewards produces models that plan, self-check, and revise before answering [17], [18]. Surveys of this trajectory now describe a shift from prompting techniques to “large reasoning models” [19]–[22].
Before the schema can be motivated, the critique it answers must be stated in an organized way, because the problems are of different depths and are too often listed as one undifferentiated complaint. The critique here has three layers, each deeper than the last. Layer one examines the mechanism: what an autoregressive model natively computes. Layer two examines the two current patches, chain-of-thought elicitation and reinforcement learning, and shows that both inherit the mechanism’s limits. Layer three brings in the formal constructs of reasoning theory and asks, construct by construct, where each one lives in a language model; the answer, nowhere, converts the critique into the specification that the TRACE schema answers.
Elicitation, however, changes what the model emits, not what the model computes. Maximum-likelihood training makes association the native operation: an autoregressive model is optimized to continue text as its corpus would, and correlation, not causation, is what corpus statistics carry. When a model’s output appears causal, it is because causally generated text was absorbed associatively, the corpus contains the sentences that causal processes in the world produced, not because the autoregressive mechanism can distinguish a causal relation from an association. Models accordingly deploy causal vocabulary without satisfying causal semantics [23], [24]: the words because, causes, and therefore are high-probability tokens, not inference steps.
The claim should be stated at its defensible strength. Mechanistic interpretability has found nontrivial intermediate structure inside trained transformers, including world-model-like state representations and evidence of forward planning [25], [26], so the layer’s argument is not that no internal structure exists. The argument is narrower and sufficient: the training objective neither requires nor certifies causal semantics, and whatever structure emerges is not exposed as checkable machinery, no typed claim, no identification status, no auditable warrant. A schema is needed precisely because the mechanism offers no interface at which such properties could be verified, whatever is happening inside.
The point has a distributional statement. Autoregressive decoding often favors high-probability continuations under \(P_\theta(\text{continuation}\mid\text{context})\), a distribution shaped by corpus frequency, instruction tuning, and preference optimization rather than by evidential support. Decoding strategies vary, but decoding alone does not distinguish evidential support from high-probability continuation unless evidence is explicitly represented, weighted, and audited. Call the modal basin the region of the model’s conditional distribution containing familiar, high-probability continuations that tend to reinforce one another during generation. Reasoning that remains inside this basin tends to reproduce the familiar answer rather than the best-supported conclusion: the majority view, the standard explanation, the safest hedge.
Conditioning is the steering wheel out of the basin: \(P_\theta(\cdot\mid\text{context},\text{role},\text{contentiousness})\) is a different distribution, and its high-probability continuations may have low probability under the unconditioned distribution. Adversarial role assignment therefore surfaces continuations that are coherent yet unlikely to be produced by an unconditioned pass [1]. Whether an exchange actually left the basin, and whether it returned for the right reasons, is a measurable question, monitored by information-theoretic dials and claim-level audits [2], [19]. The trap also explains why purely self-directed reasoning has no reliable mechanism for escaping it: chain-of-thought, self-evaluation, and self-consistency all sample from closely related conditionals, so their internal agreement may be a property of the basin, not of the world.
Two patches dominate current practice: elicit a chain of thought, and train with reinforcement on outcomes. Both help; neither changes the mechanism, and each fails in the way the mechanism predicts.
A model can produce a plausible chain because the words and steps are strongly associated in its training distribution, not because the chain faithfully caused the answer. The empirical record supports this caution. Chain-of-thought explanations can misreport the factors that actually drive a model’s answer [27]; answers can survive truncation or corruption of the stated chain, showing that the chain is often post hoc [28]; and reinforcement-trained reasoning models reveal exploited hints in their visible chains only a minority of the time [29]. Measured competence is also brittle: accuracy degrades under superficial perturbations of familiar problems [30] and under counterfactual variants of routine tasks [31]. Reported collapses of reasoning models at high problem complexity [32] are themselves disputed [33], and the dispute is revealing: the field lacks agreed instruments for measuring reasoning quality. Read against the families of reasoning developed later in this paper, a chain of thought at its best mimics one family only, abduction: it produces a plausible account after the fact. It performs no validity check, consults no identification condition, and surfaces no defeater, which is why fluency and correctness so often part company.
Outcome reward addresses what answer was accepted while leaving unclear why the answer was correct, why another answer was wrong, or which hidden assumption failed [4], [6]. The reward is also universal: one scalar per episode, blind to the user, the stakes, the claim type, and the evidence standard the claim required, so the same gradient that fixes one context’s error installs another context’s bias. The training signal itself is contested: reinforcement learning with verifiable rewards may mostly re-weight reasoning paths already latent in the base model [34], may expand them under prolonged training [35], and can register “gains” under spurious rewards [36]. In the worst case, the model learns to satisfy the evaluator or the user rather than to repair its own reasoning: models defer to users against their own evidence [37], [38], the route from apparent learning to sycophancy [3]. The layer’s conclusion is blunt: if the mechanism does not support reasoning, patches that elicit more reasoning-like text bear no fruit, and rewarding the text’s outcomes teaches the mechanism to imitate success rather than to reason.
The deepest layer of the critique uses the formal constructs of reasoning theory as instruments. The object of evaluation is the reasoning relation: a typed claim, the reasons offered for it, the hidden warrants that connect premises to conclusion, the evidence standard that applies, the counter-reasons that would defeat it, and the context in which the claim will be used. Centuries of work, from argumentation theory through causal inference, have made each part of that relation a formal construct with checkable conditions. The hammer questions are then direct. Where, in a language model, is elenchus? Where are the argument roles, the warrant, the qualifier, the rebuttal? Where are the argument schemes and their critical questions, the defeaters and the attack graph, the burden of proof, the evidence standard that changes with the claim type? Where is the declared causal graph? Where is \(do(x)\), the identification condition that licenses an interventional claim, and the bounds that apply when it is not met?
Table 1 enumerates the constructs; the answer to every question is the same, nowhere, and each row names the section of this paper where the TRACE schema gives the construct a field and a test.
| Formal construct | What it provides | Section(s) |
|---|---|---|
| Socratic methods (elenchus, hypothesis elimination, maieutic, dialectic) | the testing procedures that refute, eliminate, and draw out claims | Section [sec:sec:ocr-reasoning-vs-debate] |
| Claim types and reasoning families | the standard a claim must be tested against | Section [sec:sec:ocr-reasoning-families] |
| Argument roles (warrant, qualifier, rebuttal) | the anatomy that makes hidden premises visible | Section [sec:sec:ocr-reasoning-families] |
| Schemes and critical questions | pre-packaged cross-examination each argument form owes | Section [sec:sec:ocr-reasoning-families] |
| Defeaters and the attack graph | the accounting of objections and their resolution | Section [sec:sec:ocr-reasoning-families] |
| Burden of proof and evidence standards | who must show what, and to what threshold | Sections [sec:sec:ocr-reasoning-families], [sec:sec:ocr-measurement] |
| Declared causal graph (DAG) | the assumptions an interventional claim rests on | Section [sec:sec:ocr-pearl-map] |
| \(do(x)\) and identification | whether the requested effect is estimable from the declared evidence | Section [sec:sec:ocr-pearl-map] |
| Partial-identification bounds | the honest answer when identification fails: defer with bounds | Section [sec:sec:ocr-pearl-map] |
| Counterfactual machinery | abduction, action, prediction over a structural model | Section [sec:sec:ocr-pearl-map] |
The oldest layer of the tradition makes the same point procedurally. The seven Socratic methods, definition, generalization, induction, elenchus, hypothesis elimination, maieutic, and dialectic, are not opinions about reasoning; they are testing procedures, and each maps onto the formal families: definition and dialectic exercise inductive and deductive reasoning together, generalization and induction are inductive, elenchus and hypothesis elimination are deductive refutation, and maieutic questioning is abductive (Figure 1). The mapping also reveals what the tradition lacks: it supplies testing procedures but no causal type system; that gap waits two millennia for the ladder of Section 4.4. CRIT operationalizes the tradition for machine evaluation (Section 4.1), which is why it sits at the roof of the constructive sections. The hammer question extends accordingly: where, in a language model, is elenchus? A refutation procedure requires a counter-position held and pressed against the claim, and a solitary model sampling from adjacent conditionals holds none.
One more absence completes the critique, and it is social rather than formal. Mercier and Sperber argue that human reasoning evolved not mainly for solitary truth-seeking, but for producing and evaluating arguments in social exchange [39], [40]. The same lesson applies to machine reasoning. A claim must be asked: What supports it? What would defeat it? What evidence is missing? What alternative explanation has not been considered? Evaluation is therefore a division of labor: one side proposes a claim, and another side tests it. A solitary model, sampling from closely related conditionals, contains no second side.
The three layers agree on what is missing, and it is not more reasoning-like text. Every instrument surveyed above elicits reasoning; none of them writes reasoning down in a form that survives the episode. A chain of thought is discarded when the answer is emitted; a debate transcript is prose, not data; a reward is a scalar that remembers nothing about which step earned it. What the field lacks is a shared, typed, versioned schema for recording what was claimed, what kind of reasoning the claim required, what evidence and warrants were offered, what was tested and how, what failed, what is missing, and what verdict followed, with each field backed by the formal construct that Table 1 names. Procedures such as chain-of-thought, retrieval, self-critique, discussion, refutation, and debate remain useful, but only insofar as they improve the typed support relation the record captures. They are instruments; the reasoning relation is the object. The next section defines the record.
| Term | Definition |
|---|---|
| TRACE | Typed Reasoning And Commitment Evidence: the contract as a whole, comprising the TRACE schema, the TRACE writer contract, and the TRACE discipline. |
| TRACE schema | The versioned type definition, frozen as v1.0 before any stage is implemented, specifying the record types, their fields, and the fields’ semantics. |
| TRACE record | One instance conforming to the TRACE schema; the root type is TraceRecord. After this section, “the record” always denotes a TRACE record. |
| TRACE causal record | The causal specialization TraceCausalRecord, adding rung, identification status, bounds, and missing evidence. |
| TRACE writer | Any process that emits a valid TRACE record; the eight-stage procedure of Section [sec:sec:ocr-trace-method] is the reference writer. |
| TRACE policy | The declared, versioned gates, weights, and thresholds under which a record’s score vector and verdict are computed; named by the record’s policy_version field. |
| TRACE discipline |
No durable state change without a TRACE record carrying a verdict. |
| TRACE-Bench | The benchmark and ablation protocol that judges schema expressiveness, writer quality, and consumer value. |
The TRACE schema is a versioned JSON type definition, stated in the vocabulary of Table 2. Version 1.0 defines two types: the root TraceRecord and its causal specialization TraceCausalRecord.
The complete type listings appear in Appendix 9; the fields that carry the paper’s arguments are:
Typing fields: claim_text, claim_type, reasoning_family, argument_scheme. They fix the standard a claim is tested against (Sections 4.1
and 4.3).
Anatomy fields: premises, warrants, role_states, defeaters. They make hidden premises visible, including reconstructed enthymemes and their charity delta (Section 4.3).
Evidence and settlement fields: evidence, source_credibility, criticisms, attack_graph. They carry audited support and the accounting of objections (Section 4.3).
Causal fields, in TraceCausalRecord: causal_rung, graph_fragment, identifiability_status, bounds, missing_evidence (Section 4.4).
Outcome fields: final_status, score_vector, failed_gates, missing, repair, cost (Section 6).
Infrastructure fields: schema_version, writer_id, policy_version, provenance, revision_history, regret_score, consumer_actions (Sections 6 and 7.0.0.1).
Three rules govern the schema. First, outcomes are fields: the verdict, score vector, failed gates, missing items, repair, and cost live inside the record, not beside it; a verdict separated from the record that justifies it cannot exist, and every consumer reads one artifact. Second, versioning: fields are added by version and deprecated by version, never silently mutated; every consumer declares the schema versions it can read; freezing version 1.0 is the first phase of the implementation plan. Third, judgment: a schema cannot be optimal, only adequate, and adequacy is measured on three axes, expressiveness (can the record represent rung collapse, an implicit but essential warrant, a defer that names its missing evidence), cost (reported as CostNorm in the metric vector), and consumer value (the admission, plan-gating, and closed-loop experiments of the next implementation plan). None of these axes requires an objective scalar of reasoning quality.
The TRACE writer is an API. Its signature:
write(record: TraceRecord) -> WriteResult
WriteResult = { record_id, schema_version, policy_version, validation: pass | fail(violations), stored: true | false }
The input is a JSON document declaring conformance to a TRACE schema version. The service validates the document against that version; on pass, the record is parsed and stored in the TRACE repository under the TRACE schema; on fail, the write is
rejected with the violated constraints named, and nothing is stored. Writes are atomic and the repository is append-only: a record is never mutated in place, and corrections enter through revision_history.
Who constructs the record is deliberately outside the API. Any process that emits a valid record is a conforming writer: a caller can construct its own, a single structured model call can emit one whole, and the reference writer, an eight-stage procedure that builds a record from raw text under per-stage contracts, is specified in Appendix 10 with its algorithms, stage table, modality contract, and cost model. TRACE-Bench compares the reference writer against the single-pass writer as the staged decomposition’s null hypothesis (Section [sec:ocr-tracebench]).
One objection should be met head-on rather than left implicit. The reference writer’s model-mediated stages use the very mechanism whose limits Section 2 argues, so the specification may appear circular: the associative generator audits itself. The answer is architectural. The schema converts an unreliable generator into a checkable proposer: every model-mediated emission lands in a typed field that validation, gates, perturbations, and the human-agreement boundary can check, and nothing a stage emits becomes durable except through those checks. The specification does not assume the stages execute reliably; it requires their reliability to be measured at the typing boundary, and the falsification criteria of Section 7 already stake the pipeline’s warrant on that measurement.
The writer produces records; consumers, memory admission, plan gating, temporal regret, and verdict reuse, act on them. Because a record is the only channel between reasoning and durable action, the relationship between a record and a consumer is itself
a contract, and stating it is part of the specification. The write API above, and the consume API of the companion implementation, give the call signatures; this subsection states the invariants those calls must preserve, which a
signature alone does not capture. The contract has two sides.
The record guarantees, to any conforming consumer, four properties. Immutability: a stored record is never mutated in place, so a consumer that has read record version \(v\) may rely on \(v\) remaining exactly as read; corrections arrive as new revisions linked through revision_history, never as silent edits. Version legibility: every record names the schema and policy versions under which
it was written, so a consumer can refuse a record whose versions it does not understand rather than misinterpret it. Self-containment: the verdict, failed gates, missing items, and repair live inside the record, so a consumer never needs the
writer’s private state to act. Provenance: every field is attributable to the stage and inputs that produced it, so a consumer can audit why a verdict holds.
The consumer, in return, accepts four obligations. Version pinning: a consumer declares which schema and policy versions it reads and rejects records outside that set rather than coercing them. Fail closed: an incomplete, invalid, or
unreadable record, or one whose decisive evidence is unavailable, must never be treated as acceptance; the safe default is to withhold the durable action. Authority separation: the record’s verdict is advisory about warrant, not a grant
of authority; the consumer owns the mapping from verdict to durable action under its own declared, versioned policy, and a consumer may hold a technically acceptable claim when its own stakes demand more, or escalate to a human. Action
write-back: a consumer that acts on a record appends its decision to consumer_actions, so the record accumulates its own downstream consequences and the loop from reasoning to action to outcome stays auditable end to end.
Two corollaries follow that the flood vignette (Section 7.1) makes concrete. First, a TRACE verdict and a consumer decision are different objects: defer is a property of the record, while Hold is a plan-gate action the consumer writes back after reading it. Second, the same record may license different actions for different consumers, because authority separation lets a high-stakes consumer withhold what a
low-stakes consumer would commit, without any change to the record. The contract is what makes the schema an operational interface rather than a passive document.
The schema’s typing fields promise that every claim is tested at the standard its type demands. This section builds the hierarchy that redeems the promise: the Socratic roof, the four evaluation layers, the reasoning families with their argumentation toolbox, and Pearl’s ladder as the causal type system. The hierarchy has a cast, and Table 3 introduces it at once: each figure, what he contributed, and which part of the TRACE contract his machinery became. The scholars who contest the lists, Wellman, Govier, Pollock, Gordon, Freeman, and Hitchcock, appear in Appendix 11, where the contested extensions live.
| Figure | Contribution | What TRACE takes | Where |
|---|---|---|---|
| Socrates | the testing procedures: definition, generalization, induction, elenchus, hypothesis elimination, maieutic, dialectic | the questioning stages, operationalized by CRIT | Section [sec:sec:ocr-reasoning-vs-debate] |
| Toulmin [41] | the anatomy of a single argument: claim, grounds, warrant, backing, qualifier, rebuttal | the anatomy fields and the role_states enum | Section [sec:sec:ocr-reasoning-families] |
| Walton [42] | argument schemes, each owing its critical questions | the scheme registry and the stage-3 question batteries | Section [sec:sec:ocr-reasoning-families] |
| Dung [43] | abstract argumentation: attack graphs and acceptability semantics | the attack_graph field and the settlement rule | Section [sec:sec:ocr-reasoning-families] |
| Pearl [44] | the causal hierarchy: association, intervention, counterfactuals, with do-calculus and identification | the causal fields: rung, identifiability status, bounds | Section [sec:sec:ocr-pearl-map] |
Section 1 separated reasoning from the facilities that may elicit or test it. This section uses debate as the clearest stress case for that distinction. Debate is the most explicit adversarial reasoning facility: it assigns roles, creates opposition, elicits counter-reasons, and makes disagreement visible. But the same point applies to chain-of-thought, retrieval, self-critique, discussion, and refutation. None of these facilities defines reasoning. Each is useful only insofar as it improves the support relation linking a claim to its reasons, evidence, warrants, counter-reasons, and context.
Reasoning and debate therefore answer different questions. Reasoning asks whether a conclusion is supported by premises under a valid or acceptable inference pattern. Debate asks whether competing agents or roles can expose missing premises, alternative explanations, contradictions, and overclaims. Debate can improve reasoning, but it does not define reasoning. Before debate can be useful, the system must know what kind of claim is being made and what standard of support that claim requires.
CRIT is the starting point for this volume’s architecture because it supplies a facility-independent evaluator. Given a document \(d\), CRIT identifies the conclusion \(\Omega\), extracts supporting reasons \(R\), evaluates each reason-to-conclusion relation \(r \Rightarrow \Omega\), recursively audits reasons that are themselves claims, seeks rival reasons \(R'\), evaluates counter-reasons, aggregates validity and credibility scores, and then asks for reflective synthesis in context [19]. The public CRIT paper explicitly develops definition, elenchus, dialectic, maieutics, generalization, and counterfactual reasoning as prompt-template methods connected to inductive, deductive, and abductive reasoning [19].
This gives the paper its roof:
Claim first. What is the conclusion, and what kind of conclusion is it?
Reason next. What reasons are offered, and what inference relation connects them to the conclusion?
Evidence and credibility. What evidence supports each reason, and how credible are the sources?
Rival reasons. What counterarguments or alternative explanations have not been considered?
Context. Would the support relation change under a different time, place, population, legal regime, institutional role, or value frame?
Status. Should the claim be accepted, accepted with qualification, revised, deferred, or rejected?
| CRIT method | Evaluator role | Families invoked | Question it puts to the argument |
|---|---|---|---|
| Definition | Claim and term typing | Deductive, definitional | What exactly is claimed, and what terms and scope conditions fix its meaning? |
| Elenchus (cross-examination) | Support testing | Deductive, defeasible, legal | Does each reason support the conclusion, or does it hide a contradiction, missing premise, weak warrant, or unsupported source? |
| Evidence typing | Evidentiary classification | Inductive, statistical, legal | Is the support a theory, an opinion, a statistic, testimony, or a cited claim, and does it have the evidentiary status the claim requires? |
| Source audit (recursive) | Evidentiary tracing | Defeasible, legal, probabilistic | If a reason is itself a claim, what supports that subclaim, and how credible is its source chain? |
| Dialectic | Rival-reason search | Abductive, defeasible, dialectical | What counterarguments, alternative explanations, or omitted evidence would defeat or weaken the claim? |
| Hypothesis elimination | Explanation pruning | Deductive plus abductive | Which candidate explanations fail on contradictions, failed predictions, or missing evidence? |
| Generalization | Instance-to-rule check | Inductive, analogical | Does the move from instances to a broader rule respect sample and similarity conditions? |
| Maieutics | Reflective articulation | Reflective, meta-cognitive | Can the evaluator explain why the score was assigned, and what repair would strengthen the claim? |
| Counterfactual questioning | Context shift and robustness | Causal, abductive, practical | Would the support relation survive a change of time, place, population, action, or background conditions? |
The legal flavor of CRIT is not accidental. Elenchus is cross-examination: it tests whether the reason actually supports the conclusion, whether the support rests on evidence or assertion, whether the source is credible, and whether a reason is itself an unsupported claim. Dialectic introduces adverse positions and counter-reasons. Recursive validation resembles evidentiary tracing: when a premise is a quoted claim, the evaluator follows the source chain. These are legal-evidentiary techniques in computational form, even when the domain is not law. Table 4 assembles the full method set: for each CRIT method, the evaluator role it plays, the reasoning families it invokes, and the question it puts to the argument.
Table 4 also fixes a boundary. Chain-of-thought, retrieval, self-critique, discussion, refutation, SocraSynth, and EVINCE may instantiate some of these operations, and SocraSynth or EVINCE may organize or measure them in debate. But CRIT defines what all of them are trying to improve: the reason-to-claim support relation.
The third column of Table 4 deserves emphasis. CRIT’s Socratic methods are not another taxonomy competing with deduction, induction, and abduction; they are evaluator operations that invoke those families. Definition fixes terms before any inference is scored. Elenchus tests deductive and defeasible support. Evidence typing and source audit carry the inductive and legal-evidentiary load. Dialectic and hypothesis elimination drive abductive comparison. Generalization checks inductive strength, and counterfactual questioning probes causal and practical robustness.
This mapping is the roof under the rest of the paper. It also explains why causal reasoning cannot be treated as the whole of reasoning. A causal claim may require deductive graph reasoning, inductive estimation, abductive diagnosis, and counterfactual evaluation. Those operations play different roles and should not be collapsed into one score.
Table 5 situates the related work by the layer it serves.
The typed-reasoning space divides into four layers. Each layer answers a different evaluator question.
Claim types: factual, definitional, causal, counterfactual, predictive, diagnostic, normative, legal, policy, preference, explanatory.
Reasoning types: deductive, inductive, abductive, analogical, causal, probabilistic, defeasible, practical, deontic, dialectical, narrative, contextual.
Argument schemes: expert opinion, witness testimony, sign, cause–effect, correlation–cause, analogy, precedent, rule, consequences, practical reasoning, values, ignorance, popular opinion, slippery slope, threat, commitment.
Context and value evaluation: audience, stakes, vulnerability, culture, role, institutional norm, fairness, harm, autonomy, reversibility, proportionality.
The second and third layers are easy to conflate, and the distinction matters: a reasoning family names the formal standard of inference, while an argument scheme names a recurring warrant pattern in discourse. A causal claim may instantiate a correlation-to-cause scheme, but the causal family supplies the rung and identification tests. The scheme tells the evaluator which critical questions to ask; the family tells it which formal standard the answers must meet.
The literature base supports these layers but does not by itself provide an agentic evaluation architecture. Argumentation schemes classify defeasible reasoning moves and associate each move with critical questions [42], [45], [46]. Computational argumentation formalizes attacks, defenses, and acceptability among competing arguments [43], [47]–[49], with structured frameworks supplying the internal anatomy of premises, rules, and attacks [50]–[52]. Value-based and practical-reasoning accounts show why the same evidence can support different conclusions under different goals, audiences, and values [53], [54]. Argument mining and argument-quality research provide computational tools for extracting and assessing claims, premises, support, attack, and quality dimensions [55]–[59]. Context-aware reasoning motivates DIKE–ERIS: argument relevance depends on task, role, stakeholder, situation, and institutional setting [60]–[62]. Surveys of reasoning with language models map the modern elicitation and evaluation landscape [21], [22], [63], [64], and causal inference disciplines embedded causal subclaims [44], [65]–[68].
Four further traditions bear directly on the evaluator. First, deductive reasoning has a mature verification stack of its own. Automated theorem proving and proof assistants such as Lean give the deductive family a machine-checkable standard [69]; benchmarks now bridge informal and formal proof [70]–[72]; and trained verifiers with process supervision bring stepwise checking to natural-language reasoning [73], [74]. When the family typing of Section 4.3 says deductive, these are the plug-in checkers. Second, belief revision and truth maintenance supply the formal account of how a TRACE record must change when a premise falls: AGM belief revision [75], truth-maintenance systems [76], [77], and nonmonotonic logics [78], [79] are the classical machinery behind the record’s revision history and RLER’s policy updates. Third, decision theory prices the evaluator’s choices: subjective expected utility [80], applied statistical decision theory [81], and the value of information [82] ground ERM’s regret weights, the stopping rules of Section 5.4, and the defer verdict’s obligation to name the evidence that would most reduce uncertainty. Fourth, the AI-and-law tradition formalizes the legal-evidentiary techniques CRIT borrows: case-based reasoning with precedents [83], dialectical assessment of conflicting legal arguments [84], case comparison incorporating theories and values [85], and computational models of burden of proof [86]. These give the roof’s cross-examination, source audit, and burden allocation their disciplinary home.
One prior artifact deserves direct comparison because it is the closest in kind. The Argument Interchange Format (AIF) is the established schema for representing and exchanging arguments as typed graphs of information and scheme nodes, with an ecosystem of corpora and tools built on it [87], [88]. TRACE shares AIF’s premise, that arguments deserve a typed, machine-readable representation, and departs from it on exactly the dimensions an operational consumer needs: AIF records an argument’s structure, while TRACE additionally records its adjudication. The verdict, score vector, failed gates, missing items, and repair are fields of the record rather than annotations beside it; reasoning families route subclaims to formal plug-ins with rung and identification status for causal claims; the schema is versioned with an append-only revision history; and the record-consumer contract of Section 3.4 states what a downstream system may rely on. AIF is an interchange format for arguments; TRACE is an interchange format for evaluated arguments and the commitments they license.
The design gap is now precise. Existing work gives taxonomies, dialogue semantics, extraction tools, causal methods, and context theories. What the paper supplies is the evaluation-side organization an agentic system needs: which quality standard applies to which reasoning family, how a mixed argument is decomposed and routed, how causal subclaims are mapped to Pearl’s rungs, and how final judgments are scored, audited, and revised.
Once CRIT identifies the claim and reasons, the evaluator must classify the reasoning family. A single passage may contain several families at once. For example, a medical recommendation may contain a diagnostic inference, a causal treatment claim, a probabilistic risk estimate, a practical-action argument, and a value judgment about patient goals. Treating all of them as “causal” or all of them as “debate” loses the type-specific standards. Table 6 defines the ten families this paper uses: the surface signature by which each is recognized in text, the standard it must meet, and its typical failure.
| Family | Surface signature | Primary standard | Typical failure |
|---|---|---|---|
| Deductive | necessity modals (must, cannot), universal quantifiers, definitions | validity, soundness | invalid form; false premise |
| Inductive/ statistical | sample-to-general moves (in most cases, studies show), rates | sample adequacy, reference class, calibration | biased sample; base-rate neglect |
| Abductive | best explains, likely cause, diagnosis verbs | coverage, simplicity, contrast with rivals | just-so story; single-hypothesis fixation |
| Analogical/ case-based | like, just as, precedent invocation | relational correspondence, disanalogy check | surface similarity |
| Causal | rung markers: is associated with (1); if we do (2); would have, had we (3) | rung discipline, identification, assumptions | rung collapse; confounding |
| Probabilistic | credences, odds, likely/unlikely, hedges | coherence, likelihood, calibration | overconfidence; ignored dependence |
| Defeasible | normally, unless, as a rule | critical questions, defeat handling | ignored exceptions; unresolved defeater |
| Practical | should, the right course, means–end verbs | means–end fit, alternatives, side effects, proportionality | suppressed warrant; no alternatives |
| Deontic/legal/ normative | must not, duty, right, rule and precedent citations | rule fit, authority, burden, value transparency | is–ought slide; hidden value premise |
| Dialectical/ contextual | concession and reply markers (granted, however) | burden of proof, issue resolution | consensus mistaken for truth |
Argumentation theory supplies a complementary layer, and it is worth being precise about what each of its three traditions contributes, because the pipeline of this paper uses them for three different jobs. Toulmin supplies the anatomy of a single argument; Walton supplies the tests for a single inference pattern; abstract argumentation supplies the accounting when many arguments attack one another. None of the three replaces CRIT; CRIT is the operator that runs all three.
Toulmin’s model reconstructs any single-step argument into six roles [41]: the claim (what is being asserted), the
grounds (the data offered in support), the warrant (the usually unstated bridge that licenses the step from grounds to claim), the backing (what supports the warrant itself), the qualifier (the claimed strength:
certainly, presumably, some, all), and the rebuttal (the conditions under which the claim would fail). Toulmin’s own example carries all six in one sentence: Harry was born in Bermuda (grounds), so,
presumably (qualifier), Harry is a British subject (claim), since a man born in Bermuda will generally be a British subject (warrant), on account of the relevant statutes (backing), unless both his parents were
aliens (rebuttal). Two properties of the layout matter as much as the roles themselves. First, the reconstruction is not unique: the boundary between grounds and warrant shifts with the analyst, which is why the pipeline records the reconstruction it
chose and why the charity delta measures that choice’s distance from the argument as stated. Second, roles may be empty: a deductively valid argument has no substantive qualifier or rebuttal, and an evaluable record marks such roles absent rather than
inventing content for them. The anatomy matters operationally because most real arguments arrive with only the claim and the grounds visible; the warrant is hidden, the qualifier is inflated, and the rebuttal is missing. Three of CRIT’s core moves are
exactly the recovery of the missing roles: warrant explicitness surfaces the bridge, quantifier consistency compares the qualifier against the grounds, and rival-reason search asks for the rebuttal the author omitted. The Toulmin roles are also where the
TraceRecord of Section 3 gets its fields; Table 13 maps each role to the record field and pipeline component that handles it.
No completeness theorem exists at the level of natural argument, and the literature contests the list in both directions, proposing additions such as the counter-consideration, the undercutting defeater, and presumption with burden of proof, and proposing merges in the other direction [86], [89]–[93]. Two answers make the schema robust to this churn. The first is formal: in the structured-argumentation core used here, one argument step is premises, rule, and conclusion, so its attack surfaces are exactly three, premise, rule, and conclusion, and closure holds by construction [50], [51]; Toulmin’s six roles are a readable projection of that closed core, and every proposed extension already has a home, counter-considerations and undercutters in the attack graph, presumption and burden in external adjudication. The second is architectural: the schema does not adjudicate taxonomy disputes, it versions them. The role vocabulary, the scheme registry, the family taxonomy, and the gate metrics are all versioned under the TRACE policy, so adding a role, subtracting a scheme, or admitting a future theory’s family and its critical-question battery is a version bump, not a refutation; the contract absorbs theoretical progress the way durable interchange standards do, by extension rather than by breakage. Appendix 11 carries the full two-tier development and the extended anatomy table (Table 13).
Number the roles \(R_1\) claim through \(R_6\) rebuttal, with extensions \(R_7\) counter-considerations, \(R_8\)
undercutters, and \(R_9\) presumption and burden. For a given segment, each role is in exactly one of five states, the values of the record’s role_states field: stated, elicited (supplied by
the source in answer to a query), reconstructed (supplied by the evaluator and marked as such), absent, or contradicted (the source’s own statements are mutually inconsistent on this role). Reconstruction is a procedure, not an
art: roles are filled in dependency order; the source is queried when one is available, which is CRIT’s elenchus in interactive form; a missing warrant is abduced as the minimal bridging proposition, and if no plausible bridge exists the step is rejected
as a non sequitur, a hard gate. The procedure emits four scores consumed by Section 6: warrant explicitness, critical-question coverage, quantifier consistency, and the charity delta \(\Delta\), the load-weighted share of reconstructed roles. Elicited roles improve the argument; reconstructed roles improve only the evaluator’s reconstruction of it, and \(\Delta\) preserves the
difference. The full algorithm, with its formulas and its human-audit control on reconstructed warrants, is Appendix 10.
Where Toulmin gives the parts list, Walton, Reed, and Macagno give the catalog of inference patterns: some sixty presumptive argumentation schemes, each paired with the critical questions that probe precisely where that pattern fails [42], [45], [46]. A scheme is a defeasible template: it confers provisional support, not proof, and each critical question names a way the presumption can be defeated. The practical value for an automated evaluator is hard to overstate: once the scheme is identified, the cross-examination comes pre-written (Table 7). The evaluator does not need to invent objections; it needs to route to the right question list and record which questions were answered, which were dodged, and which defeat the argument outright. That record is the scheme-fit and critical-question coverage that Section 6 scores. In the reconstruction algorithm above, Walton schemes supply the question set used in the rebuttal, undercutter, and counter-consideration steps; their answered-over-applicable ratio is the critical-question coverage component of \(\mathrm{SchemeFit}\).
The correlation-to-cause questions of Table 7 shadow the causal family’s own tests: the third-factor question is the confounder audit, the reverse-causation question is edge orientation, and the survives-intervention question is identification itself. The resemblance is the design, not a redundancy: a scheme’s critical questions are the cheap, discourse-level interface to a family’s formal battery, and one rule prevents double charging. Where a question has a formal counterpart, the plug-in adjudicates and the record marks the question answered by it; where none exists, which holds for most of the catalog (partial counterparts are emerging, such as source-credibility models for expert opinion and decision-theoretic checks for practical reasoning), the critical-question answer is itself the verdict, held to the defeasible standard. In one sentence: schemes screen, families adjudicate.
| Scheme | Representative critical questions |
|---|---|
| Expert opinion | Is the source a genuine expert, in this field? What exactly did the expert assert? Is the expert credible and unbiased? Do other experts agree? Is the assertion backed by evidence? |
| Correlation to cause | Is the correlation real or artifactual? Could a third factor produce both variables? Could causation run in reverse? Is there a plausible mechanism? Would the relation survive intervention? |
| Negative consequences | How likely is the harm? How severe? Does the action actually produce it? Are there less costly alternatives that avoid it? |
| Practical reasoning | Does the action achieve the goal? Are there alternative actions? What are the side effects? Do other goals conflict? Is the action feasible? |
| Analogy | Are the two cases similar in the respects that matter? Are there relevant differences? Is there a counter-analogy? |
A real evaluation rarely ends with one argument. The dialectic stage produces counterarguments; counterarguments attract rebuttals; rebuttals attract undercuts. Abstract argumentation [43] answers the bookkeeping question this creates: given a set of arguments and an attack relation among them, which arguments are ultimately acceptable? The core intuition fits in a sentence: an argument stands if every argument attacking it is itself defeated by an argument that stands: innocent until attacked, reinstated when its attacker falls. Different semantics make this circle precise in different ways [48], and structured frameworks such as ASPIC+ and assumption-based argumentation extend the accounting inside arguments, distinguishing attacks on a premise, on the inference rule, and on the conclusion itself [47], [49]–[52]. In the pipeline, this layer is the formal backbone of external adjudication (condition D4 of Section [sec:ocr-debateness]): after debate has populated the attack graph and CRIT has audited each node, acceptability semantics, not the debaters’ consensus, determines which claims survive.
Toulmin tells the evaluator what to extract from one argument; Walton tells it what to ask of one inference pattern; Dung tells it how to settle a field of competing arguments; and CRIT is the evaluator that performs the extraction, asks the questions, and reports the settlement. These traditions do not replace CRIT; they give CRIT the scheme-specific questions needed after the claim has been typed. Section 5.2 shows all three at work on a single sentence.
Table 8 states the mapping this subsection develops: Pearl’s ladder as a causal type system inside the broader roof.
Pearl’s hierarchy separates three causal question types: association, intervention, and counterfactual [44], [94]. In this paper, the hierarchy is not the roof; it is the type system for causal claims inside the roof.
Rung 1: Association. What is observed together? This includes correlations, conditional probabilities, classification, prediction, and observational patterns.
Rung 2: Intervention. What would happen if an action were imposed? This requires a target such as \(do(X=x)\) and assumptions that distinguish doing from seeing.
Rung 3: Counterfactual. What would have happened under an alternative condition, given what actually occurred? This requires an individual or case-conditioned model, not merely a population association.
The mapping to reasoning families is as follows.
| Pearl rung | Primary reasoning families | Formal value | Boundary |
|---|---|---|---|
| Association | Inductive, statistical, probabilistic | Prediction, screening, pattern discovery, priors, hypothesis generation, calibration | Cannot by itself authorize an intervention. |
| Intervention | Causal, deductive-within-model, statistical identification | Estimates the effect of doing, policy choice, treatment selection, manipulation | Requires a causal model and identification assumptions. |
| Counterfactual | Causal, abductive, probabilistic, deductive-within-SCM | Individualized explanation, responsibility, regret, what-if learning, alternative-outcome analysis | Requires observed facts plus a structural model; cannot be obtained from association alone. |
This answers three common confusions.
No. Deduction is an inference relation: if the premises are true and the form is valid, the conclusion must be true. Causality is semantic content about how variables, events, or actions affect one another. A causal graph plus assumptions may allow some causal conclusions to be deduced, but that does not make deduction identical to causality. Deduction tests validity; causal reasoning tests intervention, identification, counterfactual dependence, and assumptions.
Counterfactual reasoning asks about an alternative world conditioned on the actual world. In Pearl’s structural account, it combines observed evidence with a model and an intervention-like modification. Informally, it often requires abduction to infer the case-specific background, action to modify the antecedent, and prediction to compute the consequent [44]. In CRIT terms, counterfactual questioning tests robustness under context shift; in ERM/RLER terms, it is also how the system asks what mistake would have been avoided under a different reasoning policy.
Association is not “mere” reasoning. It is essential for detecting regularities, making predictions, selecting candidate hypotheses, estimating priors, calibrating confidence, and deciding what evidence to collect next. Its limitation is precise: association cannot by itself support claims about what would happen under intervention or what would have happened otherwise. A typed evaluator should preserve association’s value while blocking rung collapse.
For interventional claims, once a causal graph and evidence set are declared, the target effect may be identifiable, partially identifiable, or underidentified. If the available evidence cannot support the requested causal quantity, the correct output is not a confident answer. It is a bounded statement: the claim is underidentified, the available bounds are reported when possible, and the missing evidence needed to tighten the claim is named [95]–[97]. This turns defer from a conversational refusal into a technical outcome.
Proposition 1 (Rung-boundedness of debate). Relative to a declared structural-model class and assumption set, if every unit in the evidence ledger is rung-1 (observational) and the claim under debate is rung-2 or rung-3, then no debate protocol, of any length and among any number of agents, can warrant a verdict tighter than the partial-identification bounds that ledger licenses [95]–[97]. The optimal reachable verdict is the bound, or defer. Debate can add evidence, and it can surface assumptions which, once granted, change the ledger’s rung composition; but exchange alone cannot climb the ladder.
Proposition 1 is the point at which the paper’s two directions meet: Pearl’s hierarchy bounds what debate can achieve; debate determines whether the bound is achieved. The semantic axis sets the ceiling; the procedural axis determines the distance to it.
This section depicts the reference writer’s stages in operation: the formulation gate, one claim traced from sentence to verdict, the gated debate of stage 5, and the rules that stop an evaluation. Figure 2 shows the writer whole: the formulation gate decides whether the text formulates an argument at all, and stages 1 to 7 then construct, test, settle, and score the record. The stage table, the two algorithms, the detail tables, and the cost model are in Appendix 10.
The pipeline of this paper begins with a typed claim. But narratives do not arrive typed; most text is narration, description, or reporting, and only some segments conduct reasoning. The evaluator therefore needs a stage-zero operator, \[\mathcal{R}: \text{text} \longrightarrow \{(\text{segment}, \text{claim}, \text{premises}, \text{markers}, \text{family}, \text{scheme}, \text{confidence})\},\] which segments the narrative, decides which segments reason, and routes each reasoning segment to its family-specific tests. Detection rests on three signal classes, in decreasing order of reliability.
Structural dependence (the criterion). A segment conducts reasoning if and only if it advances a claim whose acceptance is presented as depending on other assertions in the segment; that is, Toulmin roles (claim, grounds, warrant) are assignable [41]. Narration and description fail this test: they assert without staking acceptance on support.
Inferential markers (the flag). Therefore, because, it follows, suggests that, must have, if…then, and concessives (although…nevertheless). Markers are sufficient to flag but not necessary: enthymemes carry no markers, which is why the charity delta belongs partly to detection: the reconstruction cost of an unmarked argument is itself a detection output.
Family signatures (the router). Each reasoning family has a lexical–modal profile that both identifies it and routes it; the surface signature column of Table 6 lists these profiles.
Two properties make this stage more than preprocessing. First, it is measurable: segmentation and typing can be scored against annotated argument corpora with standard extraction metrics [55], [56], giving the success-criteria table of Section 6 a row it otherwise lacks: claim-type accuracy presupposes detection accuracy. Second, the causal rung markers of Table 6 make rung collapse detectable at the surface: a segment whose premises carry rung-1 signatures and whose conclusion carries rung-2 modality (“is associated with…therefore we should administer”) is flagged before any graph is drawn. This cheap test catches a large share of causal overclaims at intake, reserving the full identification machinery of Section 4.4 for claims that survive it.
The preceding sections introduced many instruments: a detection operator, a claim anatomy, reasoning families, argument schemes, causal rungs, debate-ness conditions, and convergence metrics. A reader meeting them for the first time can reasonably ask how they fit together on an actual sentence. This section answers by tracing one claim end to end, the way a compiler text traces one small program from source code to machine code, so that every operator is seen doing its one job on the same artifact. The input is deliberately ordinary:
“Students who take music lessons score higher on math tests, so schools should fund music programs to raise math scores.”
The sentence is a good specimen precisely because it is so agreeable. It compresses an association, a causal leap, and a policy demand into one fluent breath. As we will see at Stage 5, its conclusion sits squarely in the modal basin: it is what an unconditioned language model is most likely to say. The pipeline’s job is to take it apart.
The gate admits the segment and, in admitting it, already finds the story: the premise carries a rung-1 signature (score higher, an observed association) while the conclusion carries rung-2 modality (fund…to raise) wrapped in practical-deontic should, so the surface rung-collapse flag fires before any deeper machinery runs, and the segment is routed to three batteries at once: inductive, causal, and practical.
CRIT fills the TRACE record. Claim \(\Omega\): schools should fund music programs in order to raise math scores. The italicized purpose clause matters, because it makes the policy conditional on a causal mechanism. Grounds: the observed score gap between students who do and do not take lessons. Warrant (hidden, now surfaced): the association exists because music training causes mathematical improvement, and what causes improvement should be funded. Backing: none offered. Qualifier mismatch: the grounds are a population-level tendency; the conclusion is an unqualified prescription. Rebuttal: absent; the sentence considers no alternative explanation and no downside. Already the anatomy shows the argument is an enthymeme with a critical hidden warrant; the charity delta will be large.
The single sentence decomposes into three typed subclaims, each owed a different standard: \(S_1\) (inductive/statistical): music students score higher, a claim checkable against data; standard: sample adequacy, reference class. \(S_2\) (causal, rung 2): funding music lessons would raise scores; standard: identification. \(S_3\) (practical/normative): therefore schools should fund them; standard: means–end fit, alternatives, side effects, plus a value premise about how school resources ought to be allocated. Collapsing the three into one verdict is precisely the error the paper is built to prevent: \(S_1\) may be true, \(S_2\) unidentified, and \(S_3\) unjustified, all at once.
The argument instantiates two Walton schemes in series: correlation to cause (from \(S_1\) to \(S_2\)) and practical reasoning from positive consequences (from \(S_2\) to \(S_3\)). The critical questions arrive pre-written. For correlation to cause: could a third factor produce both variables? (Family income and parental involvement predict both music lessons and math scores.) Could causation run in reverse? (Mathematically stronger students may be steered toward music.) Would the relation survive intervention? (Unknown; that is exactly \(S_2\).) For practical reasoning: are there alternative actions that achieve the goal at lower cost? (Direct math tutoring is an obvious rival.) What are the side effects and what is displaced by the funding? The scheme layer has converted a vague unease into an explicit, finite question list, and the record now shows which questions the original argument answered: none.
The causal subclaim \(S_2\) is typed formally: the grounds report \(P(\text{math} \mid \text{music})\); the conclusion needs \(P(\text{math} \mid do(\text{music}))\). With plausible confounders (household income, parental involvement, school quality) unmeasured in the ledger, the interventional quantity is not identified from the observational association; the honest output is a bounded statement, not a confident effect [96], [97]. The evaluator names what would identify it: randomized assignment of music instruction, a lottery-based natural experiment, or within-family comparisons. It also records that controlled studies of music training have generally failed to find the far-transfer effect on academic outcomes that the argument assumes [98]. Note what just happened: the pipeline did not declare the claim false; it declared it unwarranted at the requested rung given the present ledger, which is a different and more useful thing.
Why is a procedure needed at all? Would a single model not notice all this if simply asked? Often, no, and the maximum-likelihood trap explains why. Music makes children better at math is a culturally beloved claim; text asserting it vastly outnumbers text dissecting it, so the mode of \(P_\theta(\cdot \mid \text{sentence})\) is fluent elaboration, the Mozart-effect genre, rather than critique. A single unconditioned pass tends to stay in that basin, and \(k\) unconditioned passes agree with one another because they sample from near-identical distributions, a point Section 5.3 makes precise: agreement among strongly correlated belief generators is cheap. Role conditioning changes the distribution being sampled. A proponent, a skeptic instructed that its success criterion is finding faults, and a moderator holding contentiousness high force low-probability, coherent continuations into the exchange: the confounder objection, the reverse-causation objection, the tutoring alternative, the null experimental literature. EVINCE’s dials show the signature of a healthy episode: cross-agent divergence rises while the hypothesis set grows, then falls as audited evidence units, not concessions, enter the ledger; the warranted convergence ratio, a metric defined in Appendix 14, stays high because each belief movement co-occurs with a new ledger entry. The debate need not create new knowledge; often it moves probability mass toward stored or retrievable knowledge that an unconditioned pass would not surface.
The exchange leaves a small argument graph. \(A\): the original argument. \(B\): the confounding objection, attacking \(A\)’s warrant. \(C\): the reverse-causation objection, also attacking the warrant. \(D\): the experimental-evidence argument (null far-transfer results), attacking \(S_2\) directly. \(E\): a reply to \(B\) (“some studies control for income”), attacking \(B\); but CRIT’s audit finds \(E\)’s grounds weak (the controls are partial), so \(E\) does not defeat \(B\). Under grounded semantics, \(B\), \(C\), and \(D\) are acceptable; \(A\) is not: every path to reinstating \(A\) requires defeating all three attackers, and none is defeated. The verdict is computed from the graph and the audited ledger, not voted by the debaters; that is external adjudication (D4) in action.
The typed verdict separates the three subclaims. \(S_1\): provisionally accepted if the reported association is supported by the supplied data. \(S_2\): defer, underidentified from the ledger; the named missing evidence is randomized or quasi-experimental assignment; the prior experimental record points against far transfer. \(S_3\): rejected as stated, because a practical conclusion cannot stand on an unidentified causal premise, and its critical questions (alternatives, side effects, value premise) went unanswered. The repaired argument a proponent could honestly make is much more modest: music programs may be worth funding for their intrinsic and demonstrated musical benefits, but not on the promise of raising math scores; if raising math scores is the goal, directly targeted interventions have stronger evidence. The distance between the original and this repair is the charity delta, and it is recorded. The episode also emits its convergence diagnostics: the stop occurred when the log-odds on \(S_2\) crossed the defer threshold (Proposition 2), procedural-invariance reruns with permuted roles reproduce the verdict, and the ledger alone, without the transcript, suffices for a cold judge to re-derive it.
Readers who want the same pipeline at clinical stakes will find the illustration in Appendix 12.
The pipeline’s value in this trace did not come from creating new knowledge. It came from redistributing probability mass so that stored, retrievable, or otherwise available knowledge that an unconditioned pass would not surface gets said, audited, and counted. That is the paper’s argument in miniature: reasoning quality is not a property of fluent output; it is a property of a record that survives inspection. The trace is Algorithms 5 and 6 executing line by line; the record states above are their intermediate values.
The trace ran the causal plug-in because Pearl’s hierarchy gives causal reasoning the most mature formal semantics, but the architecture is family-neutral: every family passes the same gate and stages and contributes its own test battery and outputs. The causal instance is representative, not privileged. The plug-in inventory and the causal module mapping are Tables 11 and 12 in Appendix 10.
Stage 5 is the writer’s only unbounded-cost stage, and it is therefore gated: debate runs only when a hard flag, the stakes, and the budget jointly warrant it. When it runs, its purpose is elicitation under opposition, surfacing the evidence, alternatives, and defeaters that an unconditioned pass leaves inside the modal basin. Exchange carries evaluative force only when four properties hold:
Role separation. Proponent and critic occupy distinct contexts, not merely distinct sentences in one context window.
Objective opposition. The critic’s success criterion is finding faults, not reaching agreement. Sycophancy induced by preference training attacks D2 directly [38].
Epistemic diversity. The parties’ priors and likelihoods are not identical; otherwise agreement is preordained and uninformative.
External adjudication. A standard outside the disputants (a judge, an evidence ledger, a formal checker, a CRIT audit) settles the verdict; the disputants’ own consensus does not.
These four properties define a debate-ness profile on which every reasoning-as-debate method sits, and the empirical record orders itself along it (Table 18, Appendix 14): methods that realize none of the four fail to self-correct, same-model panels that realize only D1 degenerate toward premature agreement, and protocols that enforce D2 and D4 earn formal guarantees. The SocraSynth, EVINCE, and CRIT line of this research program is an engineering program for restoring D2, D3, and D4 to exchanges that would otherwise possess only D1 [1], [2], [19].
Classical agreement results issue the right warning first. Agents with a common prior cannot agree to disagree [99]; iterated communication reaches consensus that need not aggregate all private information [100]; and mechanical opinion averaging converges with no truth content at all [101]. Multi-LLM panels, whether instantiated from one base model or from several vendors trained on overlapping corpora, are strongly correlated belief generators, so convergence among them should be treated as carrying little evidential weight unless it is coupled to new audited evidence. Agreement is cheap. The consequence shapes every metric below: the object that carries evidential weight is not the consensus but the ledger, the set of typed, CRIT-audited evidence units surfaced during the exchange, together with the trajectory by which beliefs moved. Every metric that follows is a ledger or trajectory metric, not an agreement metric. The validity metrics that make these properties measurable, procedural invariance, warranted convergence, evidence-counterfactual sensitivity, the dual-phase signature, effective panel size, and the external-judge gap, are defined in Appendix 14, together with the imported agreement and optimality results; the companion volume computes them.
One boundary closes the stage. Debate is a search procedure: it can expose assumptions, improve evidence coverage, and reduce overclaiming, but it cannot by itself certify truth, and a converged verdict remains a proposal that must pass the admission control appropriate to its domain before any durable state changes. The pipeline improves proposals; it does not make them self-authorizing.
A reasoning controller should not run until agents merely agree. The four conditions below are not ad hoc: they are the qualitative face of a sequential test. Treating each turn’s audited evidence as an increment to the log-odds of the claim, the controller stops when the accumulated log-odds cross thresholds set by the costs of the two errors, thresholds that ERM’s regret asymmetries supply directly (Proposition 2). What follows is the operational statement of that rule; the controller stops when one of the following conditions holds:
Proposition 2 (Optimal stopping). Under the idealization that turns yield conditionally independent evidence increments for a binary claim, the sequential probability ratio test with thresholds \(A=\log\frac{1-\beta}{\alpha}\) and \(B=\log\frac{\beta}{1-\alpha}\), with error rates \(\alpha,\beta\) chosen from the regret asymmetries priced by ERM, minimizes the expected number of turns among all stopping rules with equal error probabilities [102], [103]. The four stopping conditions of Section 5.4 are operational counterparts of this sequential-testing view: qualified proposal and rejection correspond to threshold crossings, diminishing returns to vanishing increments, and defer to a value-of-information judgment that further exchange is not worth its cost.
Qualified proposal: the claim is typed, grounded, audited, calibrated, and has no unresolved high-severity objection.
Rejected claim: the claim fails claim typing, scheme fit, premise support, source credibility, causal rung discipline, identifiability, or contradiction checks.
Defer: the evaluation exposes missing evidence, underidentification, unresolved defeaters, or value-context ambiguity that cannot be resolved with available information.
Diminishing returns: additional exchange produces low evidence gain or repeated arguments without new information.
The defer outcome is essential. A reasoning system must be able to say that the evidence is insufficient, the causal quantity is underidentified, the legal standard is unclear, the value frame is unresolved, or the intervention is not defined well enough to support action. A useful defer decision reports more than unknown: it reports the unresolved claim type, missing variables or assumptions, relevant bounds if available, and the next evidence that would most reduce uncertainty.
The pipeline is evaluable because each stage produces observable outputs. The metrics below (Table 9) are not measurements of objective reasoning quality; they measure record completeness, writer behavior, gate behavior, and consumer usefulness. The objective is not agreement, but improvement of the TRACE record.
| Dimension | Success criterion |
|---|---|
| Claim-type accuracy | The evaluator correctly classifies the claim as factual, causal, practical, legal, diagnostic, normative, or mixed. |
| Scheme fit | The argument scheme is correctly identified and matched to the right critical questions. |
| Premise and evidence support | Premises are evidenced at the standard required by the claim type and context. |
| Source credibility | Testimony, expertise, quotations, and external claims are traced and credibility-rated. |
| Rung discipline | Embedded causal claims correctly distinguish association, intervention, and counterfactuals. |
| Alternative recall | Plausible competing explanations and counterarguments are surfaced. |
| Overclaim control | The conclusion does not exceed the premises, evidence, causal rung, or value frame. |
| Calibration | Confidence after evaluation better matches evidential support. |
| Regret reduction | The expected cost of remaining reasoning error is reduced. |
| Recurrence control | The system makes fewer similar reasoning mistakes in later episodes. |
Success criteria become measurements only when each dimension has a unit, a label source, a scoring rule, and a declared role in the verdict. Table 15 (Appendix 13) supplies these operational definitions; the gate column marks the dimensions that participate in the hard gates of the final status rather than in the optimized residual.
Four boundary notes on the gate column. First, DetectionF1 is marked as a precondition rather than a gate: TypeAcc is meaningful only over the segments detection found, so a weak detector invalidates the rest of the vector before any gate applies. Second, claim typing gates operationally even though TypeAcc does not gate benchmarks: in high-stakes settings, when type confidence falls below a declared threshold the correct action is to defer or route to an external adjudicator, because a mistyped claim makes every downstream verdict malformed. Third, signed quantities such as CalibrationGain and RegretReduction are normalized before aggregation; negative values are reported explicitly and clipped or rescaled according to the task’s declared scoring convention. Fourth, defects are counted once: when a critical question is formally subsumed by a family test, as the correlation-to-cause questions are by the causal battery, the failure is charged to the family metric (here RungCollapseRate or BoundViolationRate), and CQCoverage records the question as answered by the plug-in rather than charging SchemeFit a second time.
Several metrics can be estimated with existing or constructible benchmarks. Rung accuracy and rung-collapse rates can be measured on tasks organized by association, intervention, and counterfactual reasoning; CausalT5K, CLadder, and correlation-to-causation benchmarks provide examples of this style of evaluation [104]–[106]. Calibration improvement can be measured with proper scoring rules, Brier score, or expected calibration error [107]. Clinical examples require expert annotation, but the same structure applies: expert labels define relevant alternatives, missing tests, overclaims, and high-regret errors.
The benchmark and ablation protocol that operationalizes these metrics and gates, TRACE-Bench, is specified in Appendix 13; its baselines include the single-pass writer as the staged decomposition’s null hypothesis and ungated debate, this research program’s earlier evaluation mechanism, as the within-family comparison.
The paper’s claims should be stated at their exact strength, because both overclaiming and underclaiming misdescribe the design.
What is claimed. TRACE is not defended as an optimal eight-stage reasoning evaluator or as a universal reasoning-quality metric. It is defended as a typed, versioned reasoning-log schema and admission-outcome explanation contract for agentic systems: the schema and the stage contracts are the contribution, the pipeline is one implementation for writing the log, and the value is measured by whether downstream consumers, Trivium, MACI, ALAS, SagaLLM [108], and Mnemosyne [7] make better, more auditable, more repairable decisions when they consume the records. TRACE still performs evaluation-like functions; what is disclaimed is only the pretension that its verdicts measure reasoning quality against an objective standard. The scores measure compliance with the declared, versioned TRACE policy, not objective reasoning quality. Within that policy, some conditional verdicts are largely model-independent once their representation is declared: identification against a declared graph is decided by do-calculus, deductive validity by a proof or entailment check, and acceptability by a deterministic semantics over a declared attack graph. The representation boundary, the graph, subclaims, warrants, and attack edges themselves, remains model- or human-mediated and is therefore measured separately by typing, extraction, and human-audit agreement. In addition, the perturbation metrics of Section 14.1, procedural invariance and evidence-counterfactual sensitivity, require neither gold labels nor policy weights. Policy dependence is confined to the aggregate scalar of Equation 1 , which is a reporting device and is never reported without its vector and failed gates.
Two consumer-facing claims complete the set, and both are central to the falsification criteria below. Across all four consumers, the constant contribution is not that the accept or reject decision beats a baseline gate; on many items it may tie. It is that every outcome, admit, reject, and defer alike, arrives with a typed account of why, on what evidence, and what would change it. This claim is uniform, it survives wherever gating accuracy ties, and it is operational in three measurable senses. Faithful: the verdict is reconstructable from the record alone by a cold judge, and it moves under evidence ablation, so the stated reasons are decisive rather than decorative. Actionable: repairs, when applied, flip verdicts at a measurable rate, and defers name evidence that resolves them when supplied. Consequential: systems that learn from records outperform systems that learn from scalar outcomes on downstream task error and error recurrence. The closed-loop experiment of the implementation plan, two identical agents on a multi-week task stream, one writing memory through the gate and one writing freely, measures the third sense directly, and it is the headline test of the schema’s consumer value.
Here a division of labor must be stated to avoid an overclaim. Structural admission, whether the workflow is well formed, its dependencies acyclic, its resource and temporal constraints stated and satisfiable, its preconditions met, belongs to the plan validator and needs no reasoning evaluator; much of it is checkable by solvers and consistency checks. TRACE gates the justificatory residue: the claims a plan rests on that constraint checking cannot reach. Why this branch rather than that one; whether the causal premise behind an action supports the expected effect at the rung the plan requires; whether an assumed precondition is evidence-backed or merely asserted. TRACE audits the plan’s warrants, not its wiring. The measured quantity is accordingly the prevented-error rate on plans that pass structural validation yet fail justification, together with the added latency priced against the cost model of Section 3.3.
The music-lessons trace shows how the writer types and evaluates an argument. A second illustration is needed to show why the resulting record is useful to a consumer. Consider a simulated flood-response mission (Figure 3) with one central Mission Controller, one survey drone, one rescue boat, and one human Incident Commander. The Mission Controller contains the mission state, planner, predictive world model, TRACE gate, and action dispatcher. The drone observes; the boat executes an authorized rescue; and the Flood Environment supplies observations and consequences. Simulation ground truth may contain facts that the Mission Controller has not observed. In particular, the environment contains a debris obstruction in the North Channel, while the Mission Controller legitimately knows only that the route is unverified. Hidden simulation state is not a premise available to the writer.
Four residents are waiting at Riverside Apartments. The planner proposes three structurally valid candidates: dispatch the boat through the short North Channel, send the drone to inspect that channel, or dispatch the boat through a longer observed-open
South Detour. A prediction service reports a plan success value of \(0.92\) for the northern dispatch. The same evidence object records model support of \(0.28\) and an out-of-distribution
score of \(0.82\). These values are deterministic fixtures chosen to expose the record logic; they are not experimental measurements. The distinction they make concrete is decisive: a high reported success value does not
license action when the forecast was produced outside its declared support. Two record mechanisms act together on the northern claim, and the vignette keeps them distinct: a failed hard support gate forbids clearance now, while the
defer status with a named missing field and a bounded repair makes the hold resolvable rather than terminal. A support failure is thus not a bare refusal; it is a refusal that carries the evidence request
that could lift it.
Table 10 follows the records rather than the agents. The predictive route claim, the bounded evidence-seeking action, the revised route claim, and the supported alternative each receive their own TRACE record
because they are different claims with different warrants. The consumer decisions are not TRACE verdicts: Hold and Clear are plan-gate actions written to consumer_actions after the
consumer reads the record.
| Record | Typed claim and TRACE status | Decisive record fields | Consumer consequence |
|---|---|---|---|
| \(R_N^{(1)}\) | Predictive: North Channel is traversable and the boat will arrive before the deadline. defer on a failed support gate. | Reported success \(0.92\); support \(0.28\); OOD \(0.82\); failed hard support gate (OOD above threshold); missing: current route observation; repair: bounded drone verification. | Hold the irreversible northern dispatch. The failed support gate forbids clearance now; the defer is actionable because it names the observation that could move the claim back inside support. |
| \(R_V^{(1)}\) | Practical: the drone can inspect North Channel and return within its admitted envelope. accept or qualify. | Battery, weather, reversible scope, observation target, and any authority requirement. | Clear only the bounded verification action; do not treat it as indirect clearance of the boat dispatch. |
| \(R_N^{(2)}\) | Predictive revision: North Channel is traversable. reject. | New drone observation of the obstruction; contradiction flag; revision_history link to \(R_N^{(1)}\). | Block the northern dispatch while preserving the original forecast and the reason it was held. |
| \(R_S^{(1)}\) | Predictive and practical: South Detour is traversable and the boat can complete the pickup. accept or qualify. | Current route report, capacity, deadline, policy version, and external Incident Commander approval where required. | Clear the structured southern dispatch and cite the exact authorizing record version. |
The vignette complements the music-lessons trace of Section 5.2. The music example runs an argument through the writer and shows why association, intervention, and practical prescription must be typed separately. The
flood example starts after candidate plans exist and shows why the resulting record needs failed_gates, missing, repair, revision_history, and consumer_actions. It also makes the
structural–justificatory division concrete: the planner and validator decide whether a route and asset assignment are well formed; TRACE audits whether the claims licensing that commitment are sufficiently warranted. Appendix 16 gives the same vignette as a plan-gating usage scenario.
This example is a specification illustration, not evidence that TRACE improves rescue performance or that any predictive model is accurate. A future closed-loop study must hold the flood environment, observations, candidate plans, predictive outputs, policies, and scenario seeds fixed while varying the audit substrate. Pre-registered outcomes should include unsupported-commitment rate, false-hold rate, decision latency, record reconstructability, repair success, and localization of invalidated plan branches.
The vignette also clarifies how the paper’s measurements should be read, because the same record fields it exercises, verdicts, gates, evidence, warrants, carry very different evidential weight. The metric vector should accordingly be read in epistemic tiers, from strongest to weakest warrant. Tier one is formally checkable: rung discipline, identification status, bound violations, and deductive validity have ground truth that is a theorem or an oracle model, not an annotator’s opinion. Tier two is checkable without any ground truth: the perturbation metrics ask whether a verdict survives role, order, and seed permutation and whether it moves when decisive evidence is ablated and stays when only rhetoric is paraphrased; a memorized or sycophantic verdict is unlikely to survive a sufficiently rich perturbation suite, because the expected verdict transitions are specified at the level of decisive evidence, warrants, and gates rather than surface labels, which makes this tier the contamination-resistant core of TRACE-Bench, resistant by test design rather than by secrecy. Tier three is moderate-agreement annotation: detection, anatomy, and scheme fit, bounded by inter-annotator agreement and reported with it. Tier four is the contested holistic scalar, which inherits the disagreement of its annotators and is quarantined by the label-source declarations of Table 15 (Appendix 13). The paper’s warrant lives in tiers one and two.
What is not claimed. The following are deliberately left open, and stating them as open is part of the contract:
The eight-stage decomposition is not claimed to be optimal; stages may merge or split as the ablations direct.
The family taxonomy is not claimed to be complete.
The score vector is not claimed to be uniquely correct, and its weights are not claimed to be universal; they are governance decisions, versioned and owned by the governance layer (DIKE and ERIS) of the companion volume.
The model-mediated stages are not claimed to execute reliably; their reliability is measured, not assumed.
The convergence metrics are not claimed to correlate with truth; they measure stability and responsiveness.
The benchmark is not claimed to represent all reasoning.
Falsification. A decomposed framework has a characteristic failure mode: every wrong verdict can be blamed on an upstream mis-typing, so the framework never appears to fail. The defense is to name the judges in advance. The following criteria are pre-registered as the results that would count against the design, and no post hoc reattribution to upstream stages is admissible against them:
If the single-pass writer matches the staged writer on record quality and error attribution at comparable cost, the decomposition is unjustified.
If the no-gates ablation shows no separation in WrongAcceptRate on fluent but unsupported or formally invalid items, the gates are decorative.
If procedural invariance and evidence-counterfactual sensitivity fail to separate systems that differ in evaluation discipline, the consistency metrics are uninformative.
If evaluator-versus-human agreement at the typing boundary falls below its declared threshold, the downstream formal checks are checking the wrong objects and the pipeline’s warrant collapses with it.
If the record-gated agent of the closed-loop experiment does not outperform its ungated twin on downstream task error and error recurrence, the record earns no consumer value and the schema is an archive, not an instrument.
One baseline in this program deserves its own sentence. Ungated debate scored holistically is this research program’s earlier evaluation mechanism, CRIT over SocraSynth and EVINCE exchange; it is retained inside stages 5 and 6 as elicitation and scoring machinery, and retained in TRACE-Bench as a baseline. The benchmark is therefore also the volume auditing its predecessor with its own methods, and the comparison is within-family: what is tested is whether typing and gating improve the instrument that proposed them.
No TRACE record, writer, or debate system can guarantee complete causal truth or wise action. Several limits are unavoidable:
Hidden confounders may remain undiscovered.
Evidence may be absent, corrupted, or unobservable.
A causal quantity may be mathematically underidentified.
Counterfactuals may remain unknowable under the available assumptions.
Agents may share priors and therefore share errors.
Correct causal reasoning does not by itself determine wise action.
Gates verify field consistency, not field truth. A deterministic gate reads what the record says; a writer that fabricates evidence entries or misstates support can populate fields that pass every gate. The schema’s countermeasures are provenance and source-credibility fields that make each entry attributable, and the perturbation suite, which tests that verdicts track decisive evidence; but record faithfulness is not world truth, and the schema does not claim otherwise.
The aim is therefore not certainty. The aim is disciplined improvement: claims should become more explicit, better grounded, more robust to critique, less overconfident, easier to revise, and less likely to repeat the same failure modes over time. The flood vignette adds two further boundaries. Its simulation truth is not an input to the Mission Controller, and technical TRACE clearance does not manufacture emergency-command authority. Both are external constraints on any future implementation.
The paper’s product is an artifact and a discipline. The artifact is the TRACE record: a typed, versioned schema for logging what was claimed, what kind of reasoning the claim required, what evidence and warrants were offered, what was tested, what failed, what is missing, and what verdict followed, with the TRACE causal record as its specialization for the family with the most mature semantics. The discipline is closure: no durable state change without a record.
Around the artifact the paper arranged four things. A type system gives the fields their semantics: CRIT supplies the Socratic roof; the reasoning families and the Toulmin, Walton, and Dung toolbox supply claim types, roles, schemes, and critical
questions; Pearl’s ladder supplies rung, identification, and bounds, with association, intervention, and counterfactuals positioned as one disciplined family inside reasoning rather than reasoning itself. A reference writer, TRACE, fills the fields under
stage contracts, with cost controlled by a cascade and escalation gated by flags, stakes, and budget; the writer is one conforming implementation, and the schema outlives it. A measurement regime holds writer and record to account: hard gates first, scalar
optimization on the residual, the vector never reported without its failed gates, and TRACE-Bench, specified in Appendix 13, whose baselines include the single-pass writer as the decomposition’s null hypothesis and this
research program’s earlier evaluation mechanism, ungated debate scored holistically, as the within-family comparison. And a set of consumers gives the record its value: memory admission, plan gating on a plan’s warrants rather than its wiring, temporal
regret over attributed steps, and verdict reuse. The music-lessons trace shows the writer separating an association, an intervention, and a practical conclusion; the flood-rescue vignette shows a plan consumer using missing,
repair, revision_history, and consumer_actions to hold one branch, request evidence, preserve a contradiction, and authorize another.
The final output for any claim remains a qualified proposal, a rejection, or a defer that names its missing evidence. A fluent consensus can be evidence that reasoning has improved; it is not truth and it is not authority. What the record buys is not correctness but attributability: when the system is wrong, the record says where. Whether that is worth its cost is the falsifiable question the companion volume’s experiments are built to answer.
TRACE schema v1.0 defines two types. The root type:
TraceRecord = { schema_version, writer_id, policy_version, claim_id, claim_text, claim_type, reasoning_family, argument_scheme, premises, warrants, role_states, evidence, source_credibility, counter_reasons, defeaters, context_frame, value_frame, embedded_subclaims, routing_decisions, criticisms, stage_results, attack_graph, audit_result, score_vector, failed_gates, missing, repair, cost, provenance, regret_score, revision_history, consumer_actions, final_status }
For causal claims, the record specializes into the TRACE causal record:
TraceCausalRecord = TraceRecord + { causal_rung, variables, graph_fragment, assumptions, missing_evidence, alternative_explanations, intervention, counterfactual_query, identifiability_status, bounds }
The reference writer constructs a TraceRecord from raw text in eight stages, each holding a declared contract. The decomposition is a reference architecture induced by the information dependencies among the record’s fields: a claim cannot
be tested before it is typed, a scheme’s critical questions cannot be loaded before the scheme is identified, and an attack graph cannot be adjudicated before its edges are audited. Whether the staging earns its cost is settled by the ablations of
Section [sec:ocr-tracebench]. This appendix specifies the pipeline at a glance, the two algorithms split at the record boundary, the modality contract, and the
cost model.
Everything that follows elaborates one eight-stage pipeline, shown in Figure 2 of the main text. Stages read and write one shared artifact, the TRACE record; the two algorithms state the executable contract, split at the record boundary.
The pipeline admits an implementation-level statement. TRACE is intended not as a metaphor for evaluation, but as an executable methodology: each stage has a declared input, output, state update, failure mode, and score contribution. Given an input
passage, the evaluator detects whether the passage contains a reasoning relation, constructs a TraceRecord, decomposes the passage into typed subclaims, routes each subclaim to the appropriate reasoning-family standard, applies scheme-specific
critical questions and formal plug-ins where available, elicits and audits counter-reasons when the escalation gate warrants, and emits a typed verdict with a score vector. This turns the paper’s survey of reasoning theories into an operational pipeline
for writing and auditing TRACE records; the sections that follow develop each line, and Section 5.2 runs the whole algorithm on one sentence. The statement is split at its natural seam: Algorithm 5 turns raw text into a typed record, and Algorithm 6 turns the typed record into a verdict; the record is the only artifact that crosses the boundary.
We call the executable form TRACE: Typed Reasoning Argument Critique and Evaluation. TRACE is not a truth oracle and not a replacement for domain authority. Its output is a structured audit: what was claimed, what kind of reasoning the claim requires, which evidence and warrants support it, which defeaters remain unresolved, what verdict follows, and which error and cost terms were incurred. The contribution of TRACE is to make reasoning evaluation executable without pretending that all reasoning standards reduce to one mechanism: symbolic checks, causal identification, argument-scheme questions, model-mediated critique, retrieval, and human adjudication all become plug-ins behind a shared record contract.
TRACE treats implementation modality as part of the contract. A plug-in may be symbolic, statistical, model-mediated, retrieval-mediated, human-mediated, or hybrid. The requirement is not that every plug-in be purely algorithmic; the requirement is that every plug-in declare its input fields, output fields, failure modes, confidence, cost, and audit trail. What makes the pipeline automatic is the contract at each step: a detector must emit segments and confidence; a router must emit claim types, families, and schemes; each plug-in must emit pass, fail, bound, or defer status; and the adjudicator must write a final status and metric vector back to the record. This is why the record, not the transcript, is the central artifact.
TRACE is affordable because it is a cascade: each step is a filter, and the expensive machinery runs only on the fraction of inputs that survives the filters ahead of it. Writing \(p_a\) for the fraction of segments that are arguments and \(p_e\) for the fraction of arguments whose flags, stakes, and budget trigger the debate escalation in Algorithm 6, the expected cost per segment is \[\mathbb{E}[\text{cost}] \;\approx\; c_0 + p_a\,(c_{\mathrm{intake}} + c_{\mathrm{tests}}) + p_a\, p_e\, c_{\mathrm{debate}},\] where the debate term dominates the constants. Detection is a small classifier over every segment; intake, typing, and scheme selection are one structured call per argument; and the gate keeps \(p_e\) small and stakes-directed. Some family tests, such as proof checking, causal identification, bounds, and graph acceptability, can be implemented algorithmically. Others, such as practical-reasoning assessment, source-credibility judgment, value-context fit, and alternative generation, may be model-mediated, retrieval-mediated, human-mediated, or hybrid. Three economies compound over episodes: stakes-proportional gating ties spend to the expected regret of acting unevaluated; adjudicated claim patterns cache through RLER; and Trivium compiles recurring error classes into cheap formulation-gate flags and stage-3 questions. TRACE therefore does not merely score an isolated argument; it accumulates reusable reasoning-control policy. The cascade controls cost, not model dependence: intake, typing, value-sensitive judgment, and debate remain model-mediated or human-mediated where the contract requires them, and the auditing caveats of the reconstruction algorithm apply to them unchanged.
The immediate design consequence is routing. A causal-sounding claim must be routed to causal standards; a legal-sounding claim to legal and deontic standards; a policy recommendation to practical reasoning, embedded empirical claims, causal subclaims, and value warrants. A model that says \(X\) caused \(Y\) has not yet made an evaluable causal claim unless the evaluator can tell whether the claim is associational, interventional, or counterfactual. A model that says a policy is justified has not yet made an evaluable practical argument unless the evaluator can separate facts, causal effects, alternatives, costs, institutional constraints, and value assumptions.
The family plug-ins and the causal module mapping:
| Family plug-in | Primary tests | Typical outputs |
|---|---|---|
| Deductive | validity, soundness, formal or proof check | valid; invalid; unsound premise |
| Inductive/statistical | sample, reference class, calibration | supported pattern; biased sample; weak generalization |
| Abductive | explanatory coverage, alternatives, simplicity | best explanation; undercompared explanation |
| Causal | rung, identification, assumptions, bounds | identified; bounded; underidentified; defer |
| Practical | goals, means, alternatives, side effects, proportionality | justified action; revise; reject; defer |
| Legal/deontic | rule fit, authority, precedent, burden, value frame | meets standard; fails burden; jurisdiction dependent |
| Component | Role | Measurable Output |
|---|---|---|
| Pearl hierarchy | Type causal claims as association, intervention, or counterfactual | Rung accuracy; rung-collapse error rate |
| CRIT | Evaluate claim, reasons, evidence, credibility, rivals, and context | Invalid-support detection; overclaim reduction; source audit |
| SocraSynth | Elicit competing hypotheses through structured exchange | Hypothesis coverage; alternative-explanation recall |
| EVINCE | Measure evidence-seeking exchange and convergence or drift | Evidence gain; contradiction exposure; calibration shift |
| RCA | Diagnose why a causal judgment failed | Failure-localization accuracy; correction target |
| ERM | Score the epistemic cost of causal error | Regret score; severity-weighted error |
| RLER | Convert failures into revised reasoning policies | Repeated-error reduction; transfer after revision |
| Trivium | Evaluate improvement over time | Long-horizon regret reduction; temporal stability |
Once the roles are numbered, reconstruction stops being an art and becomes a procedure. Number the roles \(R_1\) claim, \(R_2\) grounds, \(R_3\) warrant, \(R_4\) backing, \(R_5\) qualifier, \(R_6\) rebuttal, and the extensions \(R_7\) counter-considerations, \(R_8\) undercutters, and \(R_9\) presumption and burden. For a given segment, each role is in exactly one of five states: stated (present in the text), elicited (supplied by the source in answer to a query), reconstructed (supplied by the evaluator and marked as such), absent (marked empty after elicitation and reconstruction both fail), or contradicted (the source’s own statements are mutually inconsistent on this role). The procedure fills the roles in dependency order, queries the source when one is available, and emits the scores that the metric definitions of Section 6 already provide.
Claim. Extract \(R_1\). If no claim is found, the segment is not an argument; return it to detection.
Grounds. Extract \(R_2\). If absent, query the source: what supports this claim? If elicitation fails, mark absent; an ungrounded claim is scored as assertion, not argument.
Warrant. Test whether \(R_2\) defeasibly licenses \(R_1\) without additions. If not, generate the minimal bridging proposition \(W\) by abduction and present it: your argument assumes \(W\); is that your assumption? Confirmed: elicited. Unconfirmed but plausible: reconstructed. No plausible \(W\): hard gate; reject the step as a non sequitur.
Backing. For each warrant, ask what supports it, and recurse: backing is itself an argument, and the recursive source audit applies to it.
Qualifier. Read the stated modality; compute the strongest qualifier the grounds license; if the stated strength exceeds the licensed strength, raise the quantifier-mismatch flag, an overclaim signal.
Rebuttal. Load the identified scheme’s critical questions; every unanswered exception-type question becomes a candidate rebuttal condition, queried if a source is available.
Extensions. Scan for concessive constructions (\(R_7\)), applicability challenges (\(R_8\)), and default or burden context (\(R_9\)); route \(R_7\) and \(R_8\) to the attack graph and \(R_9\) to external adjudication.
Score. Emit four reconstruction scores: \(\mathrm{WarrantExplicitness}\), the support-weighted fraction of essential warrant roles that are stated or elicited; \(\mathrm{CQCoverage}\), the fraction of applicable critical questions answered; quantifier consistency, a flag comparing the stated qualifier against the strongest the grounds license; and the charity delta \(\Delta\), the support-weighted fraction of essential roles that had to be reconstructed. Weights reflect each role’s support load. The reasonableness of the argument as stated is reported as this vector under the gate-then-optimize rule; elicited roles improve the argument, while reconstructed roles improve only the evaluator’s reconstruction, and \(\Delta\) preserves the difference.
Two remarks close the loop. First, the queries of steps 2, 3, and 6 are CRIT’s elenchus and dialectic in interactive form: the algorithm is the Socratic method run against a schema, which is why its outputs land directly in the TRACE record rather than in a side channel. Second, steps 3 and 5 are themselves evaluator judgments; when the evaluator is a language model, they inherit the reliability questions of any model judgment, and sampled human audit of reconstructed warrants is the corresponding control.
No completeness theorem exists at the level of natural argument, and the literature contests the list in both directions: proposed additions include the counter-consideration of conductive argument [89], [90], the undercutting defeater that attacks the warrant’s applicability rather than the claim [91], and presumption with burden of proof [86], while Freeman would merge the warrant into the grounds altogether [92], [93]. A taxonomy that serious scholars want to both extend and shrink is a useful schema, not a closed enumeration.
Table 13 assembles the resulting extended anatomy: the six canonical roles, the three best-supported extensions, their reading in the formal core, and the pipeline component that handles each. Nothing on the extended list is homeless, and nothing requires the six-role list to be closed: counter-considerations and undercutters are resolved in the attack graph of the acceptability layer, and burden of proof is a parameter of external adjudication.
| Role | Status | Formal-core reading | Handled by |
|---|---|---|---|
| Claim | core | conclusion | record: claim_text, claim_type |
| Grounds | core | premises | record: premises, evidence |
| Warrant | core | inference rule | record: warrants |
| Backing | core | recursion: an argument whose conclusion supports the warrant | recursive backing/source audit |
| Qualifier | core | strength of the rule | quantifier and modality checks |
| Rebuttal | core | exception annotation on the rule | record: defeaters |
| Counter-consideration | extension | attacking argument, conceded and outweighed | attack graph (acceptability layer) |
| Undercutter | extension | attack on the rule’s applicability | attack graph (attack typing) |
| Presumption, burden of proof | extension | default status and proof allocation | external adjudication; legal battery |
This section illustrates the record where mistakes are expensive: stylized clinical records, a worked policy argument, and the typed evaluation that replaces holistic scoring.
The pipeline is easiest to understand in clinical reasoning, where causal mistakes have different consequences at different stages. Diagnosis begins with symptoms and uncertain causal hypotheses. Test selection asks which observation would reduce uncertainty. Treatment selection becomes an interventional and often counterfactual question. Follow-up asks whether the chosen intervention caused benefit, harm, tolerance, resistance, or recurrence. The following examples are stylized records used only to illustrate causal reasoning. They are not clinical advice, and they are not intended to reflect full clinical guideline pathways; they illustrate reasoning-record transitions.
Consider a stylized case in which a patient presents with persistent cough, weight loss, and blood-tinged sputum, and imaging reveals a lung mass. The initial claim is not yet “this patient has lung cancer.” It is a diagnostic causal claim: the observed symptoms and image may be produced by infection, inflammation, metastatic disease, or primary lung cancer. Pearl typing therefore places the early question near association and causal explanation, not yet treatment intervention. SocraSynth generates the differential diagnosis. CRIT rejects invalid jumps such as “a lung mass implies cancer” or “symptom improvement after antibiotics proves infection.” EVINCE then asks what evidence would distinguish the hypotheses: prior imaging, infection markers, biopsy, histology, stage, molecular profile, comorbidities, functional status, and patient preference.
Only after diagnosis and staging does the reasoning problem become primarily interventional: which treatment is expected to improve outcome under this patient’s disease state and constraints? At this point the TRACE causal record changes from a diagnostic record to a treatment record. Candidate interventions may include surgery, radiation, systemic therapy, targeted therapy, immunotherapy, palliative symptom control, or clinical-trial referral. CRIT rejects treatment claims that ignore stage, histology, molecular evidence, toxicity, or the difference between tumor response and cure. EVINCE compares evidence for each intervention. ERM penalizes high-regret errors such as delayed biopsy, omitted molecular testing, unnecessary toxicity, or overconfident recommendation before sufficient staging evidence. If treatment fails or recurrence occurs, RCA localizes the failure: wrong diagnosis, incomplete staging, missing biomarker, toxicity underestimated, resistance not monitored, or goals of care mis-specified. RLER converts the failure into a revised policy, for example: do not recommend disease-control treatment before diagnosis, staging, and treatment-relevant evidence are explicit. Trivium then evaluates whether future cases show fewer missing-test, premature-treatment, or recurrence-monitoring errors.
The example shows why the pipeline must distinguish a valid causal claim from an actionable causal proposal; a second stylized record, early cognitive decline with treatment selection under asymmetric regret, follows the same transitions and is left to the record repository. A claim can be well supported and still fail to justify action if the intervention is infeasible, too risky, poorly aligned with patient goals, or insufficiently evidenced for the decision context. Operational causal reasoning therefore requires more than a correct explanation. It requires knowing which rung the system is on, what evidence is missing, when additional tests are justified, when treatment should be deferred, and how later outcomes should revise future reasoning.
Consider the statement: ads can be deceiving, so ads should not appear in children’s programs. This is not mainly a causal argument. It is a practical-policy argument with embedded empirical, causal, and normative subclaims. A proper decomposition is:
P1. Some ads can deceive. empirical claim P2. Children are vulnerable to deception. empirical, developmental claim P3. Exposure to ads in children’s programs can harm children. causal, interventional claim P4. Removing ads from children’s programs would reduce that harm. causal, policy claim P5. Protecting children from deception outweighs the costs of removing ads. normative, value claim C. Therefore, ads should not appear in children’s programs. normative policy conclusion
The primary scheme is the argument from negative consequences: if action \(A\) permits harmful consequence \(H\), and \(H\) should be avoided, then \(A\) should be restricted or replaced. But practical reasoning is not validated by empirical evidence alone; it also requires a value premise. That is Hume’s gap: facts do not by themselves yield should. The original argument is therefore incomplete, not necessarily false, and the evaluation must say which. Seven dimensions make the evaluation systematic.
The evaluator first identifies the conclusion type (normative policy claim), the embedded subclaims (empirical, causal, value-based), and the primary scheme (negative consequences, practical reasoning). This prevents applying the wrong evaluator: a causal evaluator asks whether \(X\) causes \(Y\); a practical-policy evaluator asks whether, given evidence, consequences, values, alternatives, and affected stakeholders, the action is justified.
Each premise gets its own evidence question: are ads sometimes deceptive (P1); are children less able than adults to recognize persuasive intent (P2); does exposure cause measurable harm (P3); would removal reduce the harm (P4); what costs and competing values would a ban impose (P5)? P2 is well supported: industry self-regulation itself treats children as having limited knowledge, experience, sophistication, and maturity, noting that younger children may not understand persuasive intent or even that they are being advertised to [109].
The original sentence hides its warrant: because children are vulnerable, society has a duty to protect them from deceptive persuasion. That is not a data claim; it is a normative bridge from evidence to policy, and an argument that hides it cannot be evaluated until it is surfaced. Warrant explicitness is low in the original and high after repair, and the score records the difference.
The premise is weak and existential, some ads can deceive; the conclusion is strong and universal, all ads banned from the context. Conclusion strength exceeding premise strength is the practical-argument form of overclaim. The repaired argument needs stronger premises: children cannot reliably distinguish persuasive intent; child-directed advertising predictably exploits this vulnerability; screening only deceptive ads is insufficient or infeasible; therefore broad restriction is justified.
Inside the practical argument sits an interventional claim, P4: \(do(\text{remove ads}) \rightarrow\) lower deception and harm. This subclaim is extracted and routed to the causal plug-in of this paper: Pearl typing marks it interventional; SocraSynth generates the alternatives (labeling, parental controls, advertising literacy, platform rules); CRIT rejects the unsupported causal jump; EVINCE gathers exposure and harm evidence; ERM penalizes overconfident policy when the causal evidence is weak; RLER and Trivium track whether future policy arguments distinguish exposure, deception, harm, and remedy.
A practical argument is evaluated partly by what it considered and rejected. Missing here: could ads be regulated rather than banned; could educational and public-service advertising remain; would removing ads reduce access to free programming; would children migrate to less regulated platforms; would parental controls be more targeted? Pragma-dialectics makes this a general standard: argument quality is judged not only by internal consistency but by whether the discussion reasonably addresses the points at issue, the implicit premises, and the scheme-appropriate criticisms [110].
The value premise P5 is not resolvable by evidence, and its evaluation is reasoning with declared criteria. DIKE asks which fairness, protection, harm-prevention, rights, and regulatory norms apply: a declared, versioned criteria set, the normative analog of \(\mathcal{C}\). ERIS asks whose context is missing: advertisers, parents, children of different ages, low-income families relying on free content, educational broadcasters, different cultures and legal systems. The argument is not fully evaluable without this context. In one jurisdiction a full ban may be proportionate; in another, restricting manipulative formats, requiring disclosure, or banning categories may fit better. An evaluation is complete only when it names the criteria set and version it applied; the governance of those criteria is developed in the volume’s later chapters.
The evaluation practice this implies is a typed extension of CRIT. The original CRIT question, does this argument support the claim, remains necessary, but it is now refined by claim type, scheme, context, and value frame. The upgraded question is:
Given the claim type, argument scheme, context, evidence standard, and value frame, does this argument provide sufficient support for this conclusion?
Table 14 gives the scoring dimensions that operationalize the upgraded question. Each is checkable separately, which is what makes the evaluation stronger than a single holistic score.
| Dimension | Question it answers |
|---|---|
| Argument-type accuracy | Is the claim correctly classified as factual, causal, practical, normative, legal, analogical, or diagnostic? |
| Scheme fit | Is the argument using the right scheme: consequences, expert opinion, analogy, precedent, cause–effect, sign, rule, value, explanation? |
| Premise support | Are the explicit premises acceptable and evidenced? |
| Warrant explicitness | Are the hidden bridges between premise and conclusion surfaced? |
| Inference strength | Does the conclusion follow with the claimed strength? |
| Quantifier consistency | Does the conclusion overgeneralize beyond the premises? |
| Embedded-claim typing | Are embedded empirical, causal, and value claims separated and routed to their evaluators? |
| Counter-consideration recall | Are the major objections and alternatives considered? |
| Context fit | Does the argument adapt to audience, stakes, vulnerability, culture, law, and domain? |
| Value transparency | Are the value assumptions explicit rather than smuggled in? |
| Charity delta | How much reconstruction is required before the argument becomes strong? |
| Final status | Accept, accept with qualification, revise, defer, or reject. |
Applied to the running example, the judgment as stated is: type, practical-policy argument with embedded empirical and causal subclaims; scheme, argument from negative consequences; quality, weak to moderate; reason, the conclusion may be defensible but the argument is underdeveloped, with a hidden normative warrant, a quantifier jump, a missing child-vulnerability premise, missing intervention evidence, and missing counter-considerations. The repaired argument reads:
Because young children have limited ability to recognize persuasive intent, advertising embedded in children’s programs can exploit a vulnerable audience. If such exposure predictably increases deception, unhealthy persuasion, or parent–child pressure, and if less restrictive controls are insufficient, then restricting or banning ads in children’s programs is justified to protect children from unfair persuasion.
The post-repair judgment: much stronger, because claim type, empirical support, causal mechanism, value warrant, and policy scope are now explicit; still open, the age threshold, the definition of an ad, the evidence of harm, the effectiveness of a ban, exceptions for educational and public-service content, and migration to other platforms. The distance between the two judgments is the charity delta, and it is itself a quality signal about the source of the original argument.
The TRACE record of Section 3 generalizes accordingly, with four added fields: the scheme, the scheme’s critical questions with their answers, the warrants, and the embedded subclaims with their routing. The paper’s summary statement for this section is one sentence:
CRIT does not measure whether a sentence sounds reasonable; it types the claim, identifies the argument scheme, reconstructs hidden warrants, evaluates embedded subclaims under the right reasoning discipline, and scores whether the conclusion is justified in context.
The operational metric definitions and their label sources:
| Metric | Unit | Label source | Formula or scoring rule | Gate |
|---|---|---|---|---|
| DetectionF1 | \([0,1]\) | annotated reasoning and non-reasoning segments | segmentation F1 combined with reasoning-segment classification F1 | pre |
| TypeAcc | \([0,1]\) | expert claim-type labels | macro-F1 of predicted against labeled claim types | op. |
| SchemeFit | \([0,1]\) | annotated scheme labels | \(\mathrm{SchemeF1} \times \mathrm{CQCoverage}\), where CQCoverage is answered over applicable critical questions | no |
| EvidenceSupport | \([0,1]\) | evidence-link annotations or expert ordinal score | precision and recall of premise-to-evidence links, or normalized expert ordinal score | no |
| SourceTraceCompleteness | \([0,1]\) | source audit | traced sources over cited or asserted external claims | no |
| SourceReliabilityScore | \([0,1]\) | credibility rubric | normalized expert or rubric credibility score of traced sources | no |
| WarrantExplicitness | \([0,1]\) | warrant annotations | hidden warrants surfaced over essential warrants | no |
| ValueTransparency | \([0,1]\) | value-premise audit | declared value premises over required value premises | no |
| AltRecall | \([0,1]\) | expert or oracle alternative set | recall of surfaced alternatives against the reference set | no |
| CalibrationGain | signed | outcome labels | \(\mathrm{Brier}_{\mathrm{pre}} - \mathrm{Brier}_{\mathrm{post}}\) (or the same difference in expected calibration error); signed, normalized before aggregation as \(\mathrm{clip}((\mathrm{gain}+1)/2, 0, 1)\) | no |
| UnsupportedRate | \([0,1]\) | evidence audit | accepted premises without audited evidence, over accepted premises | yes |
| OverclaimRate | \([0,1]\) | modality and rung audit | conclusions whose quantifier, modality, or causal rung exceeds premise support, over all conclusions | yes |
| RungCollapseRate | \([0,1]\) | rung labels | causal claims concluding at rung 2 or 3 from rung-1 evidence without identification support, over causal claims | yes |
| BoundViolationRate | \([0,1]\) | identification analysis | rung-2/3 verdicts tighter than the identifiable bounds, over rung-2/3 verdicts | yes |
| RegretReduction | cost units | ERM cost matrix | expected error cost before evaluation minus expected error cost after evaluation | no |
| RepeatedErrorRate | \([0,1]\) | RCA error classes | episodes repeating a previously diagnosed error class, over later episodes | no |
| CostNorm | \([0,1]\) | execution logs and budget | \(\min(1, \mathrm{observed\;cost}/\mathrm{budgeted\;cost})\), where observed cost aggregates tokens, turns, wall-clock time, retrieval and model calls, and human-review burden | budget |
An implementation-level methodology needs an implementation-level benchmark. TRACE-Bench is the benchmark protocol implied by the pipeline: it does not ask only whether the final answer is correct. It asks whether the evaluator detected the argument, reconstructed its roles, typed the claim, routed the subclaims to the right standards, asked the right critical questions, invoked the right plug-ins, surfaced the right defeaters, and assigned the right verdict and repair. The benchmark is designed to evaluate both the full TRACE implementation and each stage-level contract separately.
Each TRACE-Bench item is a short passage, usually one to five sentences, paired with a context and optional evidence ledger. Items include non-arguments, single-family arguments, and mixed arguments. The gold record contains: the reasoning segment; claim and subclaims; premises, warrants, backing, qualifier, rebuttal, counter-considerations, undercutters, and burden fields; claim type; reasoning family; argument scheme; applicable critical questions; family-specific checks; evidence status; unresolved defeaters; final status; repair; and score vector. The same item can therefore evaluate both local operators and end-to-end judgment.
| Task | Gold output | Primary metrics |
|---|---|---|
| T0 Detection | reasoning vs. non-reasoning segments; segment boundaries | DetectionF1; false-argument rate; missed-argument rate |
| T1 Anatomy | claim, grounds, warrants, qualifier, rebuttal, undercutters, burden | role F1; WarrantExplicitness; charity delta |
| T2 Typing and routing | claim type, reasoning family, scheme, mixed-subclaim decomposition | TypeAcc; family macro-F1; routing accuracy |
| T3 Scheme questions | applicable Walton critical questions and answers | SchemeFit; CQCoverage; double-count prevention |
| T4 Family plug-ins | validity, sample/reference class, alternatives, Pearl rung, identification, means–end fit, rule/value fit | plug-in accuracy; RungCollapseRate; BoundViolationRate; OverclaimRate |
| T5 Defeaters and alternatives | counter-reasons, missing evidence, undercutters, rebuttals, attack graph | AltRecall; defeater recall; attack-edge F1 |
| T6 Verdict and repair | accept, qualify, revise, defer, reject, or no argument; repaired claim | VerdictAcc; WrongAcceptRate; DeferQuality; RepairUsefulness |
| T7 Cost and learning signal | token, turn, retrieval, tool, and human-review cost; repeated-error labels | CostNorm; RegretReduction; RepeatedErrorRate |
The benchmark should be deliberately mixed. Deductive items test validity and soundness; inductive items test sample and reference-class errors; abductive items test alternative-explanation recall; causal items test association, intervention, counterfactuals, confounding, identification, and bounds; practical-policy items test means–end fit, side effects, alternatives, and value warrants; legal or deontic items test rule fit, authority, precedent, burden, and institutional context. Non-argument passages are necessary because a detector that treats every fluent statement as an argument will look strong on downstream tasks while failing at intake.
A minimal useful TRACE-Bench can be built in three tiers. Tier 1 uses synthetic and templated examples whose gold decomposition is controlled. Tier 2 adapts existing annotated argument-mining and causal-reasoning datasets into the TraceRecord schema. Tier 3 uses expert-written mixed arguments in medicine, policy, law, science, and everyday explanation, because the central challenge is mixed reasoning: a single paragraph may contain empirical, causal, practical, and value claims at once. Annotation should be redundant: at least two annotators fill the record independently, disagreements are adjudicated, and the adjudicated record becomes the gold ledger.
The ablation suite tests whether the architecture matters. The full system is TRACE. Baselines include direct answer, chain-of-thought only, self-consistency, LLM-as-judge holistic scoring, single-pass record emission in which one structured call writes the entire TraceRecord, debate without typed routing, CRIT without family plug-ins, and family plug-ins without Walton critical questions. Two baselines carry special weight. Single-pass record emission is the null hypothesis for the staged writer: it conforms to the schema, so if it matches TRACE on record quality and error attribution at comparable cost, the decomposition is unjustified (Section 7). Debate without typed routing, scored holistically, is the volume’s own earlier evaluation mechanism, so this row is the volume auditing its predecessor with its own methods. The important ablations remove one control surface at a time.
| Ablation | Expected failure mode |
|---|---|
| No detector | Non-arguments are forced into argument schemas; false positives contaminate all downstream scores. |
| No anatomy | Hidden warrants, qualifiers, rebuttals, and burdens disappear; verdicts become fluent but unauditable. |
| No family routing | Deductive, causal, practical, and legal claims receive the wrong standards; type errors propagate. |
| No schemes or critical questions | Critique becomes improvised; defeater recall and counter-consideration coverage fall. |
| No causal plug-in | Correlation-to-cause failures are noticed rhetorically but not adjudicated by rung, identification, or bounds. |
| No evidence ledger | Convergence and confidence cannot be separated from rhetoric, sycophancy, or agreement pressure. |
| No attack graph or external adjudication | Debate consensus is mistaken for acceptability; defeated claims can survive by majority or fluency. |
| No gates; scalar only | A high aggregate score can hide unsupported premises, rung collapse, bound violations, or overclaims. |
| No cost term | Overlong reasoning and redundant debate appear beneficial even when evidence gain has saturated. |
The primary end-to-end measures are WrongAcceptRate and DeferQuality. WrongAcceptRate is the fraction of unsupported, overclaiming, or formally invalid arguments that receive Accept or Qualify. DeferQuality asks whether a defer verdict names the missing evidence or assumption that would actually tighten, flip, or justify the claim when supplied. These two measures capture the practical value of the pipeline: it should not merely sound critical; it should prevent wrong acceptance and make uncertainty actionable.
A useful reporting scalar is record utility:
\[\begin{align} \mathrm{RecordUtility} ={}& \alpha_1 \mathrm{TypeAcc} + \alpha_2 \mathrm{SchemeFit} + \alpha_3 \mathrm{EvidenceSupport} + \alpha_4 \mathrm{AltRecall} \\ &+ \alpha_5 \mathrm{CalibrationGain} + \alpha_6 \mathrm{WarrantExplicitness} \\ &+ \alpha_7 \mathrm{ValueTransparency} + \alpha_8 \mathrm{RegretReduction} \\ &- \beta_1 \mathrm{UnsupportedRate} - \beta_2 \mathrm{OverclaimRate} \\ &- \beta_3 \mathrm{RungCollapseRate} - \beta_4 \mathrm{RepeatedErrorRate} \\ &- \beta_5 \mathrm{CostNorm}. \end{align} \label{eq:ocr-reasoning-quality}\tag{1}\]
The weights should be task dependent. In low-risk explanatory tasks, evidence gain and alternative recall may dominate. In high-stakes action settings, overclaim reduction, regret reduction, and recurrence control should receive higher weight.
The Cost term is not incidental. Reasoning models routinely produce redundant and overlong traces, and surveys of efficient reasoning document substantial token expenditure for little accuracy gain [111], [112]. Cost-aware quality metrics and the stopping rules of Section 5.4 are the evaluator’s countermeasures to overthinking.
Equation 1 should be read as a reporting device, not a training target. Optimized directly, a scalarized score invites Goodhart behavior: alternative recall can be inflated with junk alternatives to offset overclaim penalties, and cost terms can be gamed by early stopping. The operational form is lexicographic: hard gates on the non-negotiables (rung discipline, the unsupported-premise rate, and bound violations, per Proposition 1), followed by scalar optimization on the residual. As a reporting rule, the scalar is never reported without the vector and the list of failed gates: the scalar is a dashboard summary, not the scientific result. The weights \(\alpha_i, \beta_j\) are governance decisions: versioned, audited, and owned by the governance layer (DIKE and ERIS) of the companion volume. All component terms should be normalized to \([0,1]\) before aggregation.
The six validity metrics below make convergence validity measurable; the companion volume computes them. The propositions that follow are the formal spine: imported classical results stated as theorems, their transfer to multi-LLM systems stated as design principles, and the open problem the experiments are designed to inform.
Let an episode consist of turns \(t=1,\dots,T\); let \(b_t\) denote the consensus-relevant belief state after turn \(t\) (for a binary claim, a probability); let \(E_t \subseteq \mathcal{E}\) be the evidence units newly admitted to the ledger at turn \(t\) after CRIT audit at threshold \(\theta\) [19]; and let \(V\) be the final verdict.
Procedural invariance (convergence robustness). Re-run the episode under a perturbation set \(\Pi\) that ought not to matter: role reassignment, speaking order, persona relabeling, sampling seed. Define \(\mathrm{CR} = 1 - \Pr_{\pi \in \Pi}[\,V^{\pi} \neq V\,]\). A consensus that flips with seating order is rhetorical, not epistemic.
Warranted convergence ratio. Decompose belief movement by turn and call a movement warranted when it co-occurs with newly admitted evidence: \[\mathrm{WCR} \;=\; \frac{\sum_{t}\,\lvert b_t - b_{t-1}\rvert\; \mathbf{1}[\,E_t \neq \varnothing\,]}{\sum_{t}\,\lvert b_t - b_{t-1}\rvert}.\] The complement \(1-\mathrm{WCR}\) is a per-episode sycophancy index: concession without evidence, the failure mode documented empirically in multi-agent debate [113], [114].
Evidence-counterfactual sensitivity. Ablate ledger items and re-adjudicate: the verdict should move when decisive evidence is removed and stay fixed when rhetoric-only turns are removed or paraphrased. This is the debate analogue of chain-of-thought faithfulness tests by corruption and truncation [28]. A verdict insensitive to its own evidence is fluent consensus, now detectable.
Dual-phase divergence signature. Let \(D_t\) be a cross-agent divergence (Jensen–Shannon or Wasserstein) over predictive distributions, as tracked by EVINCE [2]. A healthy episode shows an exploration phase in which \(D_t\) rises or holds while the hypothesis set grows, then a contraction phase in which the fall of \(D_t\) correlates with evidence arrivals \(E_t\). Monotone contraction from the first turn with a static hypothesis set is premature collapse: degeneration of thought [115] as a measurable waveform rather than an anecdote.
Effective panel size. With mean pairwise error correlation \(\bar\rho\) among \(n\) agents, estimated on a calibration set of labeled claims, define \(n_{\mathrm{eff}} = n / \bigl(1 + (n-1)\bar\rho\bigr)\). Jury-style aggregation benefits require \(n_{\mathrm{eff}} \gg 1\) [116]; same-model panels have \(\bar\rho \to 1\) and hence \(n_{\mathrm{eff}} \to 1\), a one-line account of why homogeneous debate underperforms and heterogeneous panels are the remedy [117]. Report \(n_{\mathrm{eff}}\), not \(n\).
External-judge gap. Give the ledger, and only the ledger, to an adjudicator that did not participate: a different-family model, a formal checker, or CRIT run cold. Measure the gap between its verdict and the participants’ consensus. A real convergence survives being re-derived from its own evidence by a stranger.
The empirical record of reasoning-as-debate methods, ordered by the four debate-ness conditions of the main text:
| Method | D1 | D2 | D3 | D4 | Characteristic behavior |
|---|---|---|---|---|---|
| Chain-of-thought | – | – | – | – | monologue; gains are computational decomposition, not adversarial testing; chains may be post hoc [27], [28] |
| Self-consistency | – | – | \(\sim\) | vote | ensemble, not debate; variance reduction with fully correlated priors [12] |
| Self-critique | nominal | – | – | – | debate with a mirror; self-correction fails without external feedback [118] |
| Same-model multi-agent debate | ✔ | \(\sim\) | – | – | gains on factuality [119] but degeneration of thought, conformity, and consensus without correction [115], [120], [121] |
| SocraSynth / EVINCE | ✔ | ✔ | engineered | CRIT + information dials | contentiousness restores D2; conditional statistics engineer D3; CRIT and divergence monitors supply D4 [1], [2], [8] |
| Prover–verifier debate | ✔ | ✔ | ✔ | judge | complexity-theoretic verification guarantees [122]–[124] |
Proposition 3 (Aumann agreement). If agents share a common prior and truthfully exchange posteriors that become common knowledge, they cannot agree to disagree, and iterated communication converges in finitely many rounds regardless of the truth of the claim [99], [100].
The transfer of Proposition 3 to multi-LLM debate is a modeling assumption, not a theorem. Agents instantiated from a common base model are strongly correlated belief generators that only approximate the common-prior setting. Under that assumption, the likelihood ratio of the event “the debate converged” with respect to the truth of the claim should be treated as close to uninformative unless the convergence is accompanied by ledger movement, external adjudication, and procedural invariance: consensus alone is not evidence. Evidential weight resides in the ledger and the belief trajectory, not in the agreement.
Proposition 4 (Diversity, not head-count). Under the standard equal-variance, equal-correlation approximation, majority aggregation over \(n\) agents with mean pairwise error correlation \(\bar\rho\) behaves as an independent panel of size \(n_{\mathrm{eff}} = n/(1+(n-1)\bar\rho)\), and jury-theorem guarantees weaken accordingly [116]. The reduction to a single mean correlation is a design approximation, not an exact law of arbitrary error dependence, but the direction of the effect is general. Corollary: a homogeneous multi-agent debate approximates a single agent, and the marginal value of an added agent is the marginal decorrelation it contributes.
Proposition 5 (Verification asymmetry). In the prover–verifier formalization, debate with optimal play permits a polynomial-time judge to decide any problem in PSPACE [122]; doubly-efficient debate makes the honest strategy itself efficient [123], and prover–estimator debate addresses obfuscated arguments [124]. Debate is therefore not a brainstorming heuristic but a verification protocol: it extends the reach of a bounded judge beyond anything solitary deliberation at the judge’s capacity can attain. These guarantees apply to formal debate protocols with optimal play and specified judge models, not to ordinary multi-agent LLM discussion.
Open Problem 1 (Convergence to the ideal aggregator under engineered diversity). Characterize the conditions on role conditioning (contentiousness schedules, persona assignment, corpus separation) under which the consensus of a conditioned multi-LLM debate converges to the ideal aggregator \(b^{*}(\mathcal{E})\); that is, when does simulated prior diversity purchase the aggregation benefits of real* prior diversity? A stylized sufficient condition would require the conditioned agents’ likelihood contributions to be conditionally independent given the claim; whether and when role conditioning achieves this is an empirical question that the validity metrics of Section 14.1 are designed to test.*
Mnemosyne enforces the TRACE discipline at the memory boundary: nothing enters durable memory without a TRACE record carrying a verdict. The gate is one function, admit(record), mapping the record’s final_status and
failed_gates to one of four outcomes: commit, commit with qualifier, quarantine, or reject. Quarantined records persist with their verdicts, retrievable but marked, so a later consumer can distinguish an unproven claim from an absent one.
The music-lessons record of the worked trace (Section 5.2) walks the gate as follows. Subclaim \(S_1\), the observed association, arrives with status Accept under an inductive qualifier and commits with that qualifier attached. Subclaim \(S_2\), the causal claim, arrives as Defer with missing naming
randomized or quasi-experimental assignment; it is quarantined, and the named missing evidence is registered as a trigger. Subclaim \(S_3\), the policy conclusion, arrives as Reject with its
repair proposal; the rejection and the repair are logged, not discarded, because Trivium’s regret accounting consumes them.
Two behaviors then depend on fields no scalar carries. Defer is prospective memory: when evidence matching the trigger arrives, a lottery-based school assignment study, say, the quarantined record is re-adjudicated; if the verdict flips to
Accept, the record commits and the trigger retires, and if it hardens to Reject, the quarantine converts, in either case exercising the actionable sense of the explanation claim (Section 7). Reconsolidation is re-audit: if a committed record’s premise is later defeated, belief revision propagates through revision_history rather than silently overwriting, and every downstream record
whose provenance cites the defeated premise is queued for re-audit. The measured quantities are admission precision and recall against expert judgment, stale-memory correction rate, and defer quality, the fraction of defers whose named
evidence, when supplied, in fact changes the verdict.
MACI enforces the TRACE discipline at the commitment boundary of a multi-agent workflow, under the division of labor stated in Section 7: structural admission, whether the workflow is well formed, its dependencies acyclic, and its resource and temporal constraints satisfiable, belongs to the plan validators. TRACE gates the justificatory residue, the claims a plan rests on that constraint checking cannot reach.
The flood-rescue vignette of Section 7.1 makes the division concrete. The structural validator admits all three candidates: a northern boat dispatch, a bounded drone verification, and a southern boat
dispatch. The boat exists, its capacity is sufficient, the destination and deadline are declared, and each route is syntactically usable by the planner. The validator therefore does not reject the northern branch. TRACE reads the record licensing that
branch and finds a different defect: the reported success value is high, but model support is below policy and the OOD score is above policy. MACI writes Hold to consumer_actions, cites \(R_N^{(1)}\), and leaves the alternative branches available.
The repair field of \(R_N^{(1)}\) names a current route observation. The planner proposes the survey-drone action, and a separate practical record \(R_V^{(1)}\) clears or
qualifies that bounded information-gathering step. When the drone reports the debris obstruction, the repository appends \(R_N^{(2)}\) rather than editing the earlier forecast. The revision_history link
identifies which commitments depended on the failed route premise, so MACI invalidates the northern branch only. The South Detour remains structurally valid; after its own predictive and practical record passes policy and the Incident Commander supplies
external approval, MACI writes Clear for the southern dispatch with the exact authorizing record version.
The measured quantities are the prevented-error rate on plans that pass structural validation yet fail justification, false-hold rate, added latency, repair success, and branch-localization accuracy. The claim defended is still the explanation and control-interface claim, not a rescue-performance claim: every held, qualified, or cleared commitment arrives with why, on what evidence, under which policy, and what later observation changed it. A controlled gated-versus-ungated implementation is future work.
The photorealistic flood search-and-rescue scene (Figure 3) was generated with OpenAI’s GPT-4o image generation. All schematic and lifecycle diagrams are the authors’ own vector figures. The generated image is an illustration only; it depicts no real person, place, or deployment.