1 Introduction↩︎

Large language models increasingly expose their intermediate reasoning, and that reasoning is fluent enough to be persuasive even when it is wrong. In chemistry and materials science the errors take a consequential form: the model states a specific object (a molecular formula, a space group, a formation energy) that is verifiable in principle but confabulated in fact, embedded in coherent prose that offers little signal it is fabricated. These errors concentrate on the long tail: rarely-seen entities are the ones models get wrong, a rarity–frequency dependence documented for factual recall [1], [2] and which we observe across four rarity strata, precisely the regime where model confidence is least trustworthy and an external check is most needed.

Interventions that need no external truth do not fix it: prompting a model to critique and revise its own answer cannot supply a fact it never knew, and such loops often fail to improve or degrade accuracy [3], [4]. What is missing is an external, authoritative signal. Retrieval-augmented generation (RAG) [5] supplies one unconditionally, but pays a coverage and abstention cost and cannot catch a self-contradiction whose entity is not known a priori; LLM-as-judge verification [6] is flexible but unauditable and itself prone to overconfidence. We take a third route: a tiered, deterministic verifier that grounds each extractable claim against authoritative databases (PubChem [7], Materials Project [8], CCCBDB [9]) and physics, issues a hard verdict where an extractor is reliable, and feeds the reference value back into a gated correction loop. The conceptual advance is a check that is deterministic and auditable rather than generative: it returns a hard verdict and, on a flagged claim, the reference value itself as the repair signal—so correction supplies a fact the model never knew rather than resampling the model’s own uncertainty, and every flag traces to a database record or a physical law.

Studying this verifier in a five-condition with recent models, we make five contributions. (i) We decompose gated correction into detection and repair and show the loop is detection-limited: repair succeeds wherever a flag fires, so in-loop detection recall—not repair—differentiates surfaces and is the engineering target. A consistency-triggered second detection stage recovers most of the missed recall where values are extractable, yet the accuracy lift does not follow, because the stage flags without a reference to repair with; detection and repair are both necessary. (ii) Gated correction outperforms a conversational oracle retriever on the intention-to-treat deployment metric at substantially lower retrieval cost, because conversational fact-provision often leaves the reference value unstated in an extractable, subject-bound form; an extraction-enforcing oracle variant is complementary, winning on copyable quantitative claims where gated correction is detection-limited. The gated lift concentrates where baseline error exists and is governed by detection recall, not by model openness or capability tier. (iii) We identify a scope boundary: object-level grounding improves object accuracy and calibration but improves the final, downstream answer only when the verifier’s scope extends to the deliverable itself (the derived quantity), which we demonstrate by extending the tier and restoring the lift. (iv) We show the method ports to two further domains by changing only its orchestration layer, and that the accuracy lift appears exactly where extractable long-tail error exists: none on physical constants, where models are already near ceiling, and a large one on long-tail isotope half-lives and materials formulas, together with a reference-frame-artifact analysis for quantitative properties that recurs across domains. (v) We release the corpus, frozen ground truth, and frozen pipeline with all registration hashes, so every headline number is reproducible under one-shot-holdout discipline.

2 Results↩︎

2.1 Confabulation exists and scales with rarity↩︎

The verification architecture is summarized in Fig. 1. Molecular-formula error at baseline rises monotonically with compound rarity (PubChem-CID-ordered strata): S1 well-known 0%, S2 common 3%, S3 rare 15%, S4 recent 58% (baseline run, pooled over models). Recent drugs (lenacapavir, sotorasib, ensitrelvir) are where models confabulate formulas most, exactly the long-tail objects a database verifier is positioned to catch. The full error taxonomy (by model \(\times\) claim type \(\times\) stratum, with representative examples) is in Table 3 (Fig. 2a).

2.2 Gated grounding closes the gap and beats the oracle on the deployment metric↩︎

On molecular formula the in-loop verifier achieves 82% detection recall (aggregated units throughout; Methods), and gated correction (Mode A) drives error-given-commitment from 22% to 4% (McNemar \(p=7.3\times10^{-22}\)) while flagging only 31.5% of prompt\(\times\)model cells, \(3.2\times\) fewer retrievals than blanket RAG’s 100% (supplementary Table 5). On intention-to-treat (ITT) error (wrong or non-committal under the frozen extractor, the metric a user of the pipeline experiences), gated Mode A 25% (95% CI 21–28) beats conversational oracle RAG 43% (95% CI 40–46): handed the fact in the prompt but not told to restate it, the model frequently answers in prose without emitting an extractable, subject-bound value (42% no-commit under the frozen extractor), while Mode A’s direct correction question holds no-commit at the baseline level of 22% (Fig. 2b). To test whether that no-commit is intrinsic or an artifact of conversational phrasing, we ran a second oracle-RAG variant whose prompt forces explicit restatement of the retrieved value (constrained RAG; Methods). The no-commit is largely recoverable: on molecular formula, 73% of the constrained variant’s frozen-extractor no-commits (and 65% of the conversational variant’s) are cells where the correct formula is present in the text but not bound to the subject in the pattern the frozen extractor requires. The molecular and space-group RAG-variant comparison is therefore confounded by the extractor’s name-binding rather than model behavior; we report those surfaces only as no-commit-under-the-frozen-extractor, not as a model-abstention claim, and note that the extractability limit it exposes is the same axis governing where grounding pays off. Where the deliverable is a single copyable number the confound vanishes and the comparison is decisive: on formation energy, constrained RAG reaches ITT error 1.0% (1200/1212 cells correct; 1201 committed, 11 no-commit), against conversational RAG 25% and gated Mode A 33%. Forcing restatement of a retrieved scalar solves the surface Mode A cannot, which is detection-limited there. Gated correction and constrained retrieval are thus complementary: gated correction wins on cost and on identity claims, extraction-enforcing retrieval wins on copyable quantitative values, both against a retriever with perfect truth access by construction.

A realistic retriever recovers the oracle gain only where entities are name-addressable. The RAG conditions above are an oracle upper bound: the retrieved facts are the same database records that define the ground truth, so coverage is complete and every fact is correct. To bound what a deployed system would see, we replaced the oracle with live entity resolution (molecular names to PubChem; crystal and formation-energy entities to their Materials Project most-stable polymorph) and re-ran RAG-in-prompt on the three surfaces (four models, triplicate, same frozen grader; retriever coverage and per-surface detail in Supplementary Note S5). On molecular formula, where every compound is name-addressable in PubChem, realistic retrieval matches the oracle on error given commitment (0.2% versus 0.3%, baseline 21.7%) and keeps abstention low (4.8% versus the oracle’s 42.0%). On the long-tail crystalline and formation-energy surfaces, where the retriever covers only about a fifth of entities and returns the most-stable polymorph rather than the named phase, it recovers almost none of the oracle gain: error given commitment stays at baseline (formation energy 33.9% versus 31.3%; space group 5.3% versus 5.8%) and abstention rises to 56–59% as the model, handed no usable fact, declines to commit. Imperfect retrieval degrades precisely where the named entity requires phase disambiguation, which is the regime in which the verifier’s claim-specific retrieval, keyed to the named phase rather than to bare composition, still succeeds.

Per-model decomposition. The pooled formula effect is not uniform across models; reported per model (baseline to Mode A error-given-commitment; molecular \(n=141\) units, formation energy \(n=101\) units) in Table 1.

4pt

The formation-energy column tells the complementary story: Qwen3-235B has the largest EF headroom of any model (59% baseline error-given-commitment) yet shows no EF lift (net \(+1\), n.s.), because its in-loop EF detection recall is \(0/57=0\%\). The verifier never catches Qwen3-235B’s formation-energy errors during generation, so there is nothing to repair; headroom without detection yields no lift. The molecular lift is concentrated in the two open-weights models, which carry real baseline formula error (20%, 46%) for the verifier to remove; the two frontier closed models start at 10% and 1% error-given-commitment and show no significant change: a verifier can only remove errors that exist, and the frontier models simply have little formula headroom. On a domain where they do carry long-tail error (isotope half-lives), the same loop lifts both frontier closed models significantly. The method is thus a reliability layer wherever a model is weakest on the long tail, frontier or open.

The axis is detection recall, not model openness. To test whether the open-weights concentration is really about the open/closed distinction or about capability tier, we added two small closed frontier-family models (Claude Haiku 4.5, GPT-5.4 mini; verified live, full corpus, triplicate; registered ex ante). They do not reproduce the open-weights pattern: GPT-5.4 mini shows only a small non-significant molecular lift (33% to 29% err\(\vert\)commit, net \(+3\), \(p=0.38\)), and Claude Haiku 4.5 carries a high baseline molecular error (50%) that the loop does not touch (net 0) because it emits formulas in Unicode subscripts the frozen deterministic extractor cannot parse, so in-loop recall \(\approx 0\). The operative variable is therefore detection recall (whether the committed object is extractable, which interacts with output formatting) \(\times\) baseline error, not open-versus-closed or capability tier per se: the open-weights models of Table 1 simply carry both extractable formulas and real long-tail error.

2.3 An error-reduction identity for gated correction↩︎

Gated correction acts on a fixed population of committed, gradable claims, moving each through a two-outcome tree, so its effect on error-given-commitment decomposes exactly. Fix a surface and the population of claims committed and graded in both the baseline and Mode-A arms. Let \(e_0\) be baseline error-given-commitment; \(r_d\) the in-loop detection recall, the probability that a baseline-wrong claim is flagged and regenerated during Mode-A generation (materials: on flag alone; constants gated rerun: flag conjoined with \(\sigma(as+b)<\theta\)); \(r_p\) the repair success, the probability that a flagged wrong claim becomes correct once the reference value is injected; \(\phi\) the false-positive rate, the probability that a baseline-correct claim is flagged and regenerated; and \(\rho\) the regeneration-break rate, the probability that such a regeneration turns a correct claim wrong. All five are conditional rates estimated directly from the runs; where a Platt do-no-harm gate is active, it is folded into \(r_d\) and \(\phi\).

so the net reduction is \[\Delta = e_0-e_1 = e_0\,r_d r_p - (1-e_0)\,\phi\rho.\]

Proof sketch. A baseline-wrong claim stays wrong unless it is both detected-and-gated and repaired, which occurs with probability \(r_d r_p\); a baseline-correct claim turns wrong only if it is both falsely flagged-and-gated and broken on regeneration, probability \(\phi\rho\). Summing the two contributions over the mass-\(e_0\) and mass-\((1-e_0)\) subpopulations gives 1 by the law of total probability. The complete argument, and the form that expresses the harm term through detector precision, are in Supplementary Note S2.

Equation 1 turns three descriptive findings into exact statements, using only quantities the experiments already report. Detection-limited: repair is high and near-uniform across surfaces (\(r_p\in[0.80,0.97]\)) while recall varies widely (\(r_d\in[0.19,0.94]\)), so the across-surface spread of \(\Delta\) is carried by \(r_d\), not \(r_p\). Headroom-gated and do-no-harm: the gain term scales with \(e_0\), so a near-ceiling surface (\(e_0\!\to\!0\)) can only lose, \(\Delta=-(1-e_0)\phi\rho\le0\), vanishing only when \(\phi\rho=0\); this is why the frontier models incur small net harm on physical constants and why a calibrated gate that lowers \(\phi\) is the natural do-no-harm instrument. Reference-necessity: a detector that flags without injecting a reference replaces \(r_p\) by the model’s unaided resample-correct rate, which for a long-tail identity the model never learned is near zero, so recall can climb with no accuracy lift in the consistency-stage result. Substituting the measured molecular-formula rates (\(e_0=0.22\), \(r_d=0.82\), \(r_p=0.97\)) predicts \(e_1=e_0(1-r_d r_p)=4.5\%\) (harm term negligible) against the observed \(4\%\), so the decomposition quantitatively accounts for the headline lift.

2.4 The mechanism is two-stage: repair is easy, detection is the bottleneck↩︎

Gated correction is detect, then repair, and the two stages behave very differently. Wherever the verifier flags an error, the loop repairs it at 80–97% (molecular formula 97%, formation energy 80%, space group 81%): the model readily accepts an injected reference fact regardless of surface. The lift a surface receives is therefore set almost entirely by in-loop detection recall, the fraction of baseline-wrong prompt\(\times\)model units whose flaggable claim is caught during Mode A generation (formula 82%, formation energy 19%, space group 25%), not by the correction step (Fig. 2c). This in-loop recall is lower than the offline detection recall (where the same verifier scores frozen baseline claim text: formula 0.92, EF 0.31, SG 0.69), because in the loop the model must re-emit the flaggable claim during generation; generation variance, claim-binding context, and gating timing all reduce what the verifier ever sees. The in-loop number is the deployment-relevant one and is used throughout the results; the offline number bounds the extractor. Formation energy, even after we built a dedicated detection tier that raised in-loop recall from a tier-off 7% to 19% (the tier-off state is the rep-0 ablation), gets a small but significant lift (31% to 24%, \(p=7.3\times10^{-4}\)): the residual is precisely the undetected errors, not failed repairs.

A hybrid detection stage raises recall but not accuracy, and localizes why. To test whether adding a detection path to the weakest surface (formation energy) closes the gap, we added a second, consistency-triggered detection stage to Mode A as a two-stage triage (the deterministic tier fires first; the sampling-consistency signal, at its dev-frozen threshold, is invoked only on units the deterministic tier does not flag, not a naive OR; registered and hashed ex ante). Graded with a subject-bound EF grader applied identically to both arms, the stage raises in-loop EF detection recall from 19% (deterministic only) to 94% (Claude Sonnet 5, 79/84) and 95% (Qwen3-235B, 163/171) where values are extractable, but the lift does not follow: err-given-commitment is flat-to-worse (Claude Sonnet 5 28% to 53%, Qwen3-235B 80% to 80%). Two mechanisms explain this and sharpen the paper’s thesis. First, the consistency signal on formation energy has poor precision: it flags 94% of Claude Sonnet 5’s EF cells including 94% of the correct ones (precision 28%), because EF values scatter across samples whether or not the committed value is right, so cross-sample disagreement is not specific to error on this surface. Second, the consistency-triggered path signals unreliability but injects no reference value (unlike the deterministic tier, which carries the MP/PubChem fact), so of the flagged cells that regenerate, only 46% become correct: the model resamples another wrong formation energy it never learned. Detection and repair are both required: a detector that flags without a fact raises recall without raising accuracy. We therefore report the recall gain (the clean, measured quantity) and make no causal accuracy claim. (DeepSeek v4 Pro/GPT-5.5 emit terse, unit-less EF answers the extractor cannot bind in either arm, so their EF cells are ungradeable throughout.)

2.5 Rerank and correction are complementary, indexed by where the truth lives↩︎

Best-of-8 rerank (Mode B) beats gated correction (Mode A) on formation energy (err\(\vert\)commit% vs 24%, ITT 22% vs 33%, \(p=1.1\times10^{-5}\)), the complement of the identity-claim result. For a property the model has seen but samples noisily, drawing eight samples surfaces the correct value from its own distribution; for a long-tail identity it never learned, no number of samples contains the fact and only external correction helps. Consistently, these long-tail identity errors surface as cross-sample disagreement (on rare S3–S4 molecular formulas the sampling-consistency detector catches 86% of errors, mean cross-sample agreement 0.21), but disagreement only signals that the model is unsure, not that any one sample is right; the eight samples scatter across wrong values with no correct one among them, so the disagreement is a reliable error flag that points to no correct answer. Across surfaces, Mode B is best-or-tied on ITT everywhere and wins crystalline (35% vs baseline 49%) and formation energy (22%), while Mode A wins molecular formula on err\(\vert\)commitand cost. Blanket RAG worsens crystalline ITT (56% vs baseline 49%): provided facts raise abstention rather than accuracy on a surface the models rarely commit on. An exploratory post-hoc composite (Mode A on identity claims, Mode B on properties) reaches pooled ITT 22% at 8% err\(\vert\)commit, between the two pure policies (labeled exploratory; no new runs). This formation-energy gain from Mode B carries a compute cost: best-of-8 rerank issues \({\sim}8\times\) the baseline generation calls (\({\sim}\$212\) and \({\sim}92\) min over the full evaluation) against gated Mode A’s \({\sim}1.75\times\) (\({\sim}\$46\), \({\sim}20\) min), the deployment trade quantified per condition (tokens, USD, wall-clock, calls, and retrieval rounds; supplementary Table 5, Supplementary Fig. 4).

2.6 Self-correction without tools does not help and marginally harms↩︎

Self-critique (no tools) improved no surface (molecular formula 22 to 22%, crystalline 6 to 7%) and marginally harmed formation energy (31 to 38% err\(\vert\)commit; ITT 37 to 43%; McNemar \(p=4.5\times10^{-2}\): 55 correct answers broken vs 35 fixed). Consistent with prior evidence that LLMs do not reliably self-correct reasoning without an external signal. A stronger tool-free baseline, Chain-of-Verification [10], in which the model plans and independently answers its own verification questions before revising, gives the same result and closes no surface (Supplementary Note S3). This is the negative control: the lift under tool-grounded conditions is attributable to external truth, not to the extra reasoning turn. We do not lead with this as a headline effect.

2.7 Naive single-frame property verification is dominated by reference-frame artifacts↩︎

A verifier checking a quantitative property against a single reference systematically mislabels correct answers when model and reference use different physical frames. Our MP band-gap ground truth is PBE-DFT, which underestimates experimental gaps by 1–2 eV; models report experimental gaps. A naive verifier flagged 116 band-gap “errors”, of which 107 (92%) were correct experimental gaps; under an accept-either-frame policy, band-gap error falls from 34.6% to 0% over 175 checkable prompts: band gap is not an error surface for these models, it is a frame-mismatch detector. Formation energy is the control: MP (0 K DFT) vs experimental \(\Delta H_f^{\circ}\) (298 K) differ by only 0.11 eV/atom mean, so accept-either-frame rescued just 9 cells (33.7% to 31.3%); formation energy remains a genuine, multi-model error surface. Property verification must register the reference frame ex ante, grade under accept-either-documented-frame where frames diverge, and never “correct” an experimental value to a DFT value. The same frame lesson recurred in the transfer domain for physical constants: the Sackur–Tetrode constant is tabulated at two standard-state pressures, and models reporting the modern 100 kPa value (\(-1.15171\)) were initially graded wrong against a reference using the older 101.325 kPa convention (\(-1.16487\)), a reference version/convention acting as a frame, cured only by pinning the reference standard state in the ground truth. Accept-either-documented-frame thus collapses band gap to a non-error surface while barely moving formation energy, whose frames nearly coincide and which remains a genuine error surface, and leaves frame-insensitive surfaces unchanged (supplementary Table 6).

2.8 Deterministic verifier vs LLM-as-judge vs sampling-based detectors↩︎

We contrast four detectors on the same baseline claims: the deterministic verifier, a frontier reasoning-class LLM-as-judge (no tools, pure parametric), a sampling-consistency baseline (SelfCheckGPT-style [11]) that flags a claim when the object extracted from Mode B’s 8 samples disagrees across samples (flag if modal-object agreement \(\le 0.25\), threshold frozen on a dev half, applied to the holdout half), and a semantic-entropy detector [12] that clusters those same 8 samples by meaning and flags high normalized cluster entropy (threshold \(0.44\) frozen on a dev half). Supplementary Table 7 reports the per-surface precision/recall of all four detectors and shows why the deterministic checker is used: only it returns a reference value to inject.

These offline recalls, scored on frozen baseline claim text, upper-bound the extractor and are systematically higher than the in-loop recalls; the gap (largest for SG, 0.69 to 0.25) reflects generation variance and gating timing, not a different verifier. Each detector has a distinct strength. The deterministic verifier dominates recall where its extractor is reliable (formula 0.92, SG 0.69), and is auditable, improvable, and frozen. The judge is precise where symbolic extraction is hard, but is a fixed black box subject to the same confabulation it is meant to catch. Sampling-consistency is the strongest recall on formation energy (0.67), exactly where the deterministic verifier is weakest (0.31): noisy-sampling errors surface as cross-sample disagreement. The predicted failure mode (consistency should miss confidently-wrong long-tail formulas because all 8 samples agree) does not hold: on rare (S3–S4) molecular formulas consistency recall is 86% (mean agreement 0.21), i.e.models disagree across samples on long-tail objects rather than confabulating one value consistently. Semantic entropy behaves as a refinement of the consistency signal: it tracks consistency on molecular formula (recall 0.93) and improves formation-energy precision (\(0.51\) versus \(0.36\)) by clustering near-duplicate sampled values by meaning rather than by exact string, at slightly lower recall (\(0.57\) versus \(0.67\)). Like sampling-consistency it is detection-only and returns no reference value to inject. The signals are complementary: the deterministic verifier as the auditable backbone, the judge as a fallback where extraction is hard, and the sampling-based detectors as orthogonal error signals.

2.9 Trust-score calibration↩︎

The verifier’s trust score (fraction of checked claims passing) predicts committed-claim correctness on \(n=968\) baseline items; on a held-out half (\(n=484\), base correct rate 0.81) raw trust is overconfident (expected calibration error, ECE, 0.111) [13]. Platt scaling [14] reduces ECE to 0.060 (isotonic 0.088); Brier 0.121 to 0.111. The trust score is discrete (few checkable claims per trace), yielding a coarse reliability curve; ECE is the reportable summary and Platt is recommended for deployment thresholding. Against model verbalized confidence, each generating model asked to self-report P(its committed claim correct), the grounded trust score is better calibrated. On the identical paired population (\(n=958\), items carrying both signals), raw uncalibrated: verbalized confidence ECE 0.139 (overconfident: mean 0.87, accuracy 0.83) vs verifier trust score ECE 0.093. Recalibrating the trust score by Platt on a separate holdout half drops it further to 0.060 (\(n=484\); not directly comparable to the paired set, which is uncalibrated). The models’ own confidence is a poorly-calibrated abstention signal; the external verifier’s grounded trust score is better even before recalibration and substantially better after. Eliciting confidence inline with the answer (a dedicated calibration arm, \(n=1{,}191\)) makes the model more confident (mean 0.91 vs 0.87 post-hoc) without improving calibration (ECE 0.108); both elicitation modes remain worse-calibrated than the grounded trust score (Fig. 3).

2.10 Downstream end-task propagation↩︎

To test whether claim-level grounding translates into end-task accuracy, we registered 52 tasks (frozen and registered ex ante) whose final answer depends on a checkable object: molar-mass computation from formula (26), higher-molar-mass selection (14), and more-stable-phase selection from formation energy (12), weighted toward the rare strata. We ran baseline, gated Mode A, and Mode B (4 models, triplicate, McNemar) and grade the final answer against frozen ground truth. Object errors propagate at 9%, the fraction of baseline task-units whose final answer is wrong (12% for molar mass, 7% for molar comparison, 4% for stability). However, object-only gated grounding did not lift end-task accuracy: ITT accuracy was 83% at baseline, 80% under Mode A, and 85% under Mode B; the Mode-A change is not significant (McNemar net \(-2\) per model, \(p\approx0.6\)) (Table ¿tbl:tab:endtask?). The mechanism is diagnostic: on these tasks the verifier detects objects but (a) fires on incidental SMILES/formula claims the system prompt elicits, (b) fixes only 28% of propagated errors because the final answer depends on downstream arithmetic and comparison the verifier does not check, and (c) adds regeneration noise that breaks 8% of already-correct answers. The (a) flags are largely task-irrelevant: a flag rate of 58%, of which an audit shows 203/205 are category (b) (a real or transient claim flag that does not bear on the answer, e.g.a SMILES for the molecule whose molar mass is asked) and only 2 are factually-incorrect flags, a grounding-scope mismatch rather than an extractor robustness failure.

Closing the scope gap restores the lift. The diagnosis above makes a testable prediction: if end-task accuracy is limited because the verifier grounds the object but not the derived quantity the answer depends on, then extending grounding to that quantity should restore the lift. We added an orchestration-layer derived-quantity tier (registered ex ante) that computes the answer-bearing quantity from the verified object (molar mass as the sum of standard atomic weights over the verified formula, stability from the verified Materials Project formation energies) and injects it when the model’s final answer disagrees. Re-running the same 52 tasks (Mode A + derived tier, 4 models, triplicate): pooled end-task ITT accuracy increased from 83% to 90% (McNemar net \(+14\), \(b=5\)/\(c=19\), \(p=0.0066\)), versus 80% under the null object-only Mode A condition. The lift is concentrated in a single open-weights model: per-model McNemar, Qwen3-235B 73% \(\rightarrow\) 100% (net \(+14\), \(p=1.2\times10^{-4}\)) carries the entire pooled effect, while Claude Sonnet 5 (92 to 94%) and GPT-5.5 (71 to 73%) are not significant (no headroom) and DeepSeek v4 Pro shows a small non-significant harm (96% to 92%, net \(-2\)): regeneration on the derived-quantity prompt breaks 4 already-correct answers while fixing 2 (Table ¿tbl:tab:endtask?). By task type the lift is on molar-mass (86% to 95%); comparison tasks move little. These results identify grounding scope as the relevant boundary: verifying the checkable object improves object accuracy and calibration, whereas extending verification to the answer-bearing derived quantity improved pooled end-task accuracy, with the observed gain concentrated in Qwen3-235B.

4pt

2.11 Cross-domain transfer: a near-ceiling domain and a headroom domain↩︎

To test whether the method template (extract checkable object \(\rightarrow\) tiered identity lookup \(\rightarrow\) gated repair) transfers beyond materials chemistry, we applied it to two further domains chosen to bracket the headroom axis: physical constants (where current models are near-ceiling, testing whether the mechanics port) and isotope half-lives (where a genuine long-tail error mass remains, testing whether the accuracy lift ports). Only the orchestration layer changed in each; the core extractor and verifier configuration were untouched.

Near-ceiling domain: CODATA 2022 recommended values [15]; 35 constants, four rarity strata from famous (\(c\), \(h\), \(e\)) to obscure (Josephson, von Klitzing, Sackur–Tetrode); ground truth frozen and hashed before grading, via a dedicated value-extraction and comparison tier. Grading follows a registered two-part tolerance policy (Methods): exact SI-defining and derived constants by significant figures, measured constants within the larger of the CODATA-2022 standard uncertainty and the 2018–2022 adjustment drift, so neither the adjustment version nor quoted-digit truncation can register as an error.

A pre-registration audit of this domain is itself a finding: our first pass reported a large lift (84% to 99%) that a line-by-line audit traced almost entirely to an orchestration-layer value-parser artifact (full audit in Supplementary Note S1). After hardening the parser and re-grading under the tolerance policy, baseline ITT accuracy is 95.0% and gated Mode A accuracy is 95.7%; McNemar net \(+1\), \(p=1.0\), not significant (Table ¿tbl:tab:transfer?; no model changes significantly). Constants therefore show the method mechanics port cleanly but, being near-ceiling, do not on their own provide an independent accuracy-lift result; the 2018–2022 drift (\(\le 4\times10^{-9}\)) is far below model quoting precision, so zero flags were adjustment-version artifacts. This audit prompted a materials retro-audit for the same failure mode: the materials extractor abstains symmetrically on unparseable formulas (Methods), so the materials lift cannot be parse-manufactured.

Headroom domain isotope half-lives against IAEA nuclear data. To obtain the independent accuracy-lift result the near-ceiling constants cannot provide, we applied the same template to a domain with genuine long-tail error mass: half-lives of 63 rarity-stratified isotopes (famous radioisotopes \(\rightarrow\) superheavy/exotic) graded against the IAEA NDS reference [16] (ground truth and the isotope tier frozen and hashed before grading; 4 models, triplicate). Applying the constants lesson pre-emptively, we hardened the value parser before grading and registered the unit policy ex ante (grade against half-life in seconds, any stated unit accepted). This mattered: a first-pass parser mis-read comma-grouped values (“76,000 years” \(\rightarrow\) 0) and \(N\times10^{M}\) forms, inflating baseline error to 40% and the effect to net \(+50\), a constants-class artifact the mandatory two-channel gate caught before it reached the paper. Baseline ITT accuracy was 81.0%, with error-given-commitment of 11.3%; under Mode A, ITT accuracy increased to 91.0% and err\(\vert\)commitfell to 0.0%; McNemar net \(+30\), \(p=2.3\times10^{-7}\), abstention symmetric (8.7% vs 9.0%), so the lift is not parse-manufactured. The lift follows the same rarity gradient as materials: near-ceiling on famous isotopes (S1 98% to 99%) and large on the long tail (S4 superheavy/exotic 44% to 76%, err\(\vert\)commit% to 0%); see Table ¿tbl:tab:transfer?.

4pt

A suggestive frontier-model lift. Per model, the isotope lift is nominally significant for both frontier-closed models: Claude Sonnet 5 88% to 99% (err\(\vert\)commit% to 0%, net \(+7\), \(p=0.016\)) and GPT-5.5 82% to 93% (8% to 0%, net \(+8\), \(p=0.021\)). These two per-model \(p\)-values do not survive correction for multiple comparisons (Methods), so we treat the frontier-model lift as suggestive rather than confirmatory. Re-estimating this arm at five replicates rather than three leaves the net lifts unchanged and the framing suggestive (Supplementary Note S4). The open-weights lift on the same surface is stronger: Qwen3-235B 70% to 85% (net \(+11\), \(p=0.001\)) survives correction, while DeepSeek v4 Pro is directional (83% to 87%, net \(+4\), \(p=0.29\)). Isotope half-lives are nonetheless the first surface in our study on which grounding lifts a frontier closed model at all: on materials formulas and physical constants the frontier models are at or near ceiling with no headroom, whereas isotope half-lives carry genuine long-tail error mass even for frontier models. The reading is that when a frontier model errs on the long tail and the committed value is extractable, gated correction can reach it, a hypothesis these data support but do not yet establish at corrected significance.

The boundary, stated once. Across three transfer domains the pattern is consistent: the method mechanics port with orchestration-layer changes only, and the accuracy lift appears exactly where the base model carries extractable long-tail error: absent on near-ceiling constants, present and large on long-tail isotope half-lives and materials formulas. Portability is of the mechanism; the magnitude of benefit is set by headroom \(\times\) detection recall (Proposition [prop:identity]).

Compute cost. The interventions differ sharply in cost, which bears on deployment. Gating is the cheap regime: gated Mode A retrieves on only the flagged fraction of prompt\(\times\)model cells—\(3.2\times\) fewer retrievals than blanket RAG on molecular formulas, and \(6.2\times\) and \(8.4\times\) fewer on formation energy and space groups—at a modest generation-cost overhead, whereas best-of-8 rerank (Mode B) is the expensive regime; both magnitudes are quantified per condition above and in supplementary Table 5. The tier ladder keeps per-check cost low: 40.9% of checkable claims are resolved by the Tier-0 syntactic check, 48.7% by database identity lookups, 10.4% by tabulated references, and the DFT tier is invoked on no holdout claim because the tabulated tiers covered every property.

3 Discussion↩︎

The results reframe verifier-in-the-loop grounding as a detection-limited, not correction-limited problem, and place it against its two default alternatives. Against blanket RAG, gated correction is markedly cheaper on molecular formula and better on the intention-to-treat deployment metric, because it does not induce the abstention that conversational fact-provision does. Against LLM-as-judge, it is auditable and frozen where extraction is reliable. The selective-prediction view [17] unifies the metrics: err\(\vert\)commitand abstain trade error against coverage; we report both the ITT and selective views and never collapse them. Rerank and correction remain complementary tools. A deployment caveat follows directly from the near-ceiling models (Table ¿tbl:tab:transfer?): when the base answer is already correct, a flag that fires anyway triggers a regeneration that can break it, so Claude Sonnet 5 and GPT-5.5 show small net harm under gated correction on the constants domain. Decomposing that harm made the fix concrete: every frontier harm cell (Claude Sonnet 5 2, GPT-5.5 3) was a false-positive flag on an already-correct constant, while all open-model lift (DeepSeek v4 Pro \(+7\), Qwen3-235B \(+6\)) came from true-positive flags: the harm and lift populations are separable by flag correctness. A gated rerun on the constants domain (35 prompts, triplicate), using corrected reference values and the dev-frozen Platt-trust threshold, reduced frontier-model harm from five cells to zero while largely retaining the open-weights lift (DeepSeek v4 Pro: \(+7\) unchanged; Qwen3-235B: \(+6\) to \(+4\)). Overall harm fell from seven to two cells, yielding a net gain of \(+11\). This rerun is separate from the aggregated intention-to-treat analysis in Table ¿tbl:tab:transfer?: the counts reported here are raw prompt-by-model cells and are used only to diagnose the mechanism of harm.

The two residual harm cells were DeepSeek v4 Pro outputs for the von Klitzing constant in which the correct value was stated but the response was truncated before a parseable final commitment. More importantly, the frontier-model harm was eliminated by correcting the stale Sackur–Tetrode reference convention, not by the confidence gate, which suppressed no regenerations. On this single-claim surface, the trust score is binary and perfectly correlated with the verifier flag; therefore, validation of confidence-weighted suppression requires a multi-claim setting with non-binary trust scores. The calibrated trust score nevertheless remains a promising do-no-harm signal, given its better calibration than model self-reported confidence.

Our verifier sits between three lines of work↩︎

Retrieval-augmented generation prepends retrieved facts unconditionally; we retrieve only on a detected flag, which we show is both cheaper and, on the deployment metric, more accurate. Self-correction and self-critique [18] revise using the model’s own judgment; our no-tools condition reproduces this negative result and isolates the external signal as the active ingredient. Sampling-based hallucination detection flags low cross-sample agreement; we include it as a third detector and find it complementary, strongest exactly where symbolic extraction is weakest. Related recent uncertainty-based detectors, such as semantic entropy and internal-state probes, work from the same cross-sample or representational signal but, like it, report only that a claim is unreliable and return no reference value to inject; in our detect/repair decomposition they remain detection-only, and a direct comparison against them on the reference-carrying axis is a natural extension. Our two-stage triage relates to three specific lines: detector aggregation (combining complementary signals), which we implement not to maximize F1 but as a causal instrument that raises in-loop recall; two-stage uncertainty triage (a cheap deterministic screen gating a costlier check), which we invert to have the deterministic tier fire first and the consistency signal only on its misses; and stepwise consistency filtering, which we apply to committed objects rather than reasoning steps. Against LLM-as-judge verification, our deterministic tier is auditable and frozen where an extractor is reliable, and we report both side by side rather than assuming either dominates.

Within chemistry and materials specifically, several recent efforts are adjacent but scoped differently. Process-level deterministic benchmarks (ChemCoTBench-style step verification [19], MolecularIQ [20]) verify reasoning steps deterministically but are evaluation-only and molecular-only, with no correction loop, no crystalline surface, and no cost accounting; we add closed-loop correction, the crystalline and property surfaces, and per-retrieval cost. Retrieval/knowledge-graph hallucination work for materials (HalluMat-style detection and mitigation [21]) grounds against retrieved documents or KGs; ours grounds against deterministic physics and structured databases and reports the end-task and cost consequences, not detection alone. Tool-using chemistry agents (ChemCrow [22]) and LLM chemistry benchmarks (ChemBench [23]) document the tool-integration and overconfidence problems and the failure of LLM-judge scoring; we quantify the overconfidence directly (calibration) and avoid LLM-judge labelling entirely. Learned grounded step-rewards (process-reward models [24]) supply soft, learned signals over reasoning steps; we instead issue hard, deterministic verdicts on locally checkable objects, trading coverage for auditability. Finally, verifier-in-the-loop generation is established in fully formalizable domains (Lean/Isabelle proof assistants [25], SMT-backed geometry solvers [26]), where every step is machine-checkable; our contribution is to carry the paradigm into a partially, tiered-verifiable domain where only some claims have authoritative checks, and to show where that partial verifiability does and does not pay off. Uncertainty-gated escalation (deferring or escalating on low-confidence steps) is the applied prior art our gating instantiates against external ground truth. The distinctive contributions are the two-stage (detect/repair) decomposition that localizes the bottleneck to detection, the reference-frame-artifact analysis for quantitative properties, and the object-vs-derived-quantity scope boundary.

Our detect/repair diagnosis has a direct analogue in a different augmentation paradigm. Skill Retrieval Augmentation [27] decomposes agentic skill use into retrieval, incorporation, and application, and finds the binding constraint is not retrieval but incorporation, whether the model recognizes when and which skill to load; our result is the same shape in the grounding paradigm, where repair (application of an injected fact) is easy and in-loop detection (recognizing that a claim needs grounding, and emitting it checkably) is the bottleneck. Both point to need-aware gating and precision calibration as the next lever rather than larger retrieval corpora. Relative to document-grounded factuality checking such as FActScore [28], which decomposes text into atomic claims and scores each by entailment against retrieved passages, our verifier issues hard verdicts against structured databases and physics rather than soft entailment against prose, trading breadth for auditability and a repair signal that carries the reference value. Structured information-extraction components are complementary and partly already inside our pipeline: RDKit canonicalization backs the Tier-0 SMILES check, and name-to-structure parsers such as OPSIN [29] are the natural route to widen extraction coverage, the extractability axis our results identify as limiting. The constrained-RAG variant we add instantiates the extraction-enforcing RAG family, which forces the model to restate a retrieved value rather than answer conversationally. More broadly, the survey of verification-enabled mathematical reasoning [30] frames verifier-in-the-loop workflows for fully formalizable domains; we carry that paradigm into partially verifiable science, and our efficiency findings (gate retrieval, route only checkable claims to external truth) support the case for specialized, tool-integrated, auditable pipelines argued by the domain-specific-superintelligence perspective [31].

3.1 Limitations↩︎

Extraction recall bounds the entire method: a claim the extractor never sees cannot be checked, and in-loop recall (what the loop catches during generation) is well below offline recall. The formation-energy detection tier shares extraction logic with the adjudicator, so the two have correlated blind spots; the independent-GT audit (scoring against PubChem/MP/CCCBDB directly) is the control for this. The verifier removes only errors that (a) exist in the base model and (b) are emitted in a form the extractor can bind. The extractability half of that axis is a real brittleness: the Claude Haiku 4.5 case shows output formatting can silently gate the entire benefit despite abundant headroom, so a deployment would need a formatting-robust extractor. Correction carries regeneration-noise risk; a confidence-weighted gate on the calibrated trust score is the natural do-no-harm instrument, and a gated constants rerun with corrected references eliminates the frontier false-positive-flag harm, but the per-cell gate is a no-op on single-claim surfaces (where trust is binary) and its validation on a multi-claim surface is deferred. The dipole surface is underpowered (\(n=56\)). Results follow a one-shot holdout protocol (no iteration on holdout results), with three replicates per condition; per-replicate variation is in supplementary. On transfer, only the headroom domain yields an accuracy lift; the constants domain establishes portability of the mechanism alone. The hybrid two-stage detection tier raises in-loop recall but its accuracy-lift measurement is pending a reference-injecting variant. As reported above, the consistency stage flags without a reference and the lift does not follow. Converting that recall gain into precision-safe flags through calibrated detector fusion, combining the consistency signal with the calibrated trust score so that only high-confidence disagreements trigger regeneration, is the natural next step but is untested here. In-loop detection recall also depends on the model re-emitting a claim in a checkable position; we did not vary gate timing or anchor-prompt phrasing (earlier/later gates, different re-elicitation anchors), and the sensitivity of recall to that schedule is an open question. Because in-loop recall is gated by output formatting, it should be treated as a deployment-time variable, not a fixed model property (the Claude Haiku 4.5 case).

Outlook. These limitations point to concrete next steps, each aimed at the bottleneck the study isolates—detection. First, the recall recovered by the consistency stage can be converted into precision-safe flags through calibrated detector fusion, combining the consistency signal with the calibrated trust score so that only high-confidence disagreements trigger regeneration; this is the missing reference-injecting variant that would turn the recall gain into an accuracy lift. Second, the extractability ceiling—the sharpest brittleness we find, where output formatting alone can gate the entire benefit—invites a formatting-robust extractor with name-to-structure parsing (for example OPSIN) to widen coverage of the checkable surface. Third, because in-loop recall depends on the model re-emitting a claim in a checkable position, need-aware gating that varies gate timing and re-elicitation anchors is the natural lever for raising it, and its sensitivity is an open, testable question. Finally, the underpowered surfaces (the dipole set, and the suggestive frontier-model isotope lift) warrant confirmatory runs at larger sample. Together these define a program in which detection precision and extraction coverage, not larger retrieval corpora, are the levers that most extend grounded verification.

4 Methods↩︎

Corpus↩︎

528 condition-pinned prompts \(\times\) 4 models, four rarity strata (PubChem-CID-ordered), both molecular and crystalline within each stratum; ground truth frozen and hashed before grading.

Verifier↩︎

Tiered: Tier 0 syntactic (SMILES/formula hygiene) \(\rightarrow\) Tier 1 identity (PubChem formula, named-phase space group) \(\rightarrow\) Tier 1.5 tabulated property (Materials Project formation energy/band gap, CCCBDB dipole) with accept-either-frame policy \(\rightarrow\) Tier 3 DFT (def2-TZVP; last resort). xTB Tier 2 [32] descoped: the property surface is fully covered by Tier 1.5 + frame policy; no corpus claim type requires semi-empirical QM. The extractor and full verifier configuration are frozen and registered ex ante. Every extension beyond this core (detection tiers, transfer domains) was registered ex ante with a content hash; the frozen extractor and verifier configs were never modified.

Conditions↩︎

(1) unguarded baseline, (2) self-critique no-tools (with a stronger tool-free variant, Chain-of-Verification, reported in the supplementary material), (3) RAG-in-prompt (oracle), run in two variants, conversational (facts prepended, answer free-form) and constrained (facts prepended with an instruction to restate the reference value as the final answer), (4) gated Mode A (verifier-guided correction, retrieve only on flag), (5) Mode B best-of-8 rerank. The constrained variant instantiates the extraction-enforcing RAG family and isolates how much of the conversational baseline’s no-commit rate is prompt-shaping rather than intrinsic; both variants draw the same oracle fact block.

Extraction pipeline↩︎

Detection is bounded by what the extractor binds, so we make its operation explicit (the full verify-and-correct flow is summarized in Fig. 1). Each trace is normalized (Markdown/LaTeX stripped, balanced-bracket and valid-atom-start checks) before ASCII regexes extract molecular formulas, SMILES, space-group symbols/numbers, and property values with units. Names are bound to claims by whole-word matching (so “benzene” does not match inside “nitrobenzene”), and values are bound to the nearest named subject. The frozen materials extractor deliberately does not normalize Unicode-subscript formulas (C\(_9\)H\(_8\)O\(_4\)): these become no-commit under identical rules in every arm rather than silent errors. Known failure classes (Unicode subscripts, unit-less bare values, terse final answers with no adjacent subject) are the extractability limit; per-object-type extraction rate and accuracy are in Table 2, and the full extractor is in the released code.

Independent ground truth↩︎

Baseline error scored against PubChem/MP/CCCBDB directly, not by running the verifier, the control for the verifier’s own blind spots. Extraction rate is the ceiling on verifier recall.

Parser-artifact retro-audit↩︎

Motivated ex ante by the constants-domain finding, we tested whether the materials lift could be an extraction artifact of the same kind, the asymmetric bug in which a parser scores a digit-correct baseline wrong and Mode A’s re-elicitation flips it correct, manufacturing lift. The materials extractor (frozen core) behaves differently: it does not normalize Unicode-subscript formulas (C\(_9\)H\(_8\)O\(_4\)), so those become no-commit/abstain (scored as an abstention, not an error) under identical rules in every arm; this is the reported no-commit behavior. The decisive test is therefore whether Mode A converts these subscript-abstains to parseable commitments at a higher rate than baseline, which would inflate the lift. It does not: the subscript-only miss rate is symmetric across arms: baseline 23.8% (134/564, Wilson CI 20.4–27.4%) vs Mode A 24.1% (407/1692, CI 22.1–26.1%), so the abstain behavior cancels and the err\(\vert\)commitlift arises from genuinely changed committed formulas, not parse differences. (The two denominators differ by design: the baseline is a single unguarded condition, \(n=564\); Mode A pools the three tier-active replicates, \(n=1{,}692\).) The artifact concern has two channels, each with its own control. The abstain channel, a correct baseline value left unparsed and scored no-commit, is closed by the symmetry test above. The wrong-parse channel, a correct baseline value misparsed into a different value and scored wrong, is closed by the manual raw-trace audits of every flagged baseline error (the grading-integrity spot checks and the independent-GT audit, which score against PubChem/MP/CCCBDB directly rather than by running the verifier). The materials headline is not parser-manufactured. Limitation: the registered per-cell (a)-genuine / (b)-parse-artifact / (c) classification of a repaired-cell sample requires round-0 committed text, which was not persisted (only the round-0 verdict and final text); the population abstain-symmetry test closes the abstain channel at higher power (\(n=564\) vs 73), and the wrong-parse channel is closed by the manual flag audits.

Statistics↩︎

Aggregated-unit McNemar [33] (majority outcome per prompt\(\times\)model over three reps); prompt-clustered bootstrap 95% CIs; three-outcome accounting throughout. Per-replicate rates in supplementary. We state the dependence structure explicitly. Per-model McNemar tests are the primary inference: within a model, the prompt\(\times\)model units are distinct prompts and independent. Pooled (across-model) tests share the same prompt set across the four models, so their units are not fully independent and their \(p\)-values are reported as descriptive. To confirm the pooled headlines are not artifacts of that dependence, we add a prompt-clustered sensitivity analysis: a cluster bootstrap of the net discordant count that resamples whole prompts (all model rows for a prompt together, 10,000 resamples), reported per headline in supplementary Table 4. Both significant pooled headlines survive: molecular formula net \(+80\) (clustered 95% CI \([62, 99]\)) and formation energy net \(+27\) (CI \([14, 40]\)), each with no resample reaching net \(\leq 0\); crystalline space group is non-significant under clustering as in the primary test (net \(-2\), CI \([-10, 5]\)). No pooled significance required demotion. Across the thirteen McNemar tests that carry a reported \(p\)-value, we control for multiple comparisons with the Holm and Bonferroni procedures at \(\alpha=0.05\) (Bonferroni threshold \(0.05/13=3.8\times10^{-3}\)). The primary headlines survive both corrections: molecular-formula ITT (\(7.3\times10^{-22}\)), isotope overall (\(2.3\times10^{-7}\)), Mode B versus Mode A on formation energy (\(1.1\times10^{-5}\)), the end-task derived-tier per-model Qwen result (\(1.2\times10^{-4}\)), the formation-energy in-loop test (\(7.3\times10^{-4}\)), and the per-model isotope Qwen lift (\(1.0\times10^{-3}\)); the pooled end-task derived-tier headline (\(6.6\times10^{-3}\)) survives Holm but not Bonferroni. The two per-model frontier isotope lifts (Claude Sonnet 5 \(p=0.016\), GPT-5.5 \(p=0.021\)) do not survive either correction, and we present them as suggestive rather than confirmatory. Re-estimating these two tests at five replicates rather than three sharpens them to \(p=0.016\) (Claude Sonnet 5) and \(p=0.008\) (GPT-5.5); GPT-5.5 then crosses the Holm threshold but not Bonferroni, and Claude Sonnet 5 crosses neither, so both remain reported as suggestive.

Transfer domain (physical constants)↩︎

Reference: CODATA 2022, with ground truth frozen and hashed before grading. Grading exploits the exactness structure of the 2019 SI revision: seven defining constants (\(c\), \(h\), \(e\), \(k_{\mathrm{B}}\), \(N_{\mathrm{A}}\), \(\Delta\nu_{\mathrm{Cs}}\), \(K_{\mathrm{cd}}\)) and the quantities exactly derived from them (\(R\), \(F\), \(\sigma\), Josephson and von Klitzing constants, etc.) are exact and graded by significant figures: a truncated but digit-correct value passes, an altered digit fails. Measured constants are graded within the larger of the CODATA-2022 standard uncertainty and the 2018–2022 adjustment drift (\(\le 4\times10^{-9}\) for every measured constant here), so neither adjustment version nor quoted-digit truncation can register as an error. Standard-state-dependent constants (Sackur–Tetrode) are pinned to the modern 100 kPa convention. The value extractor was hardened after the constants-domain audit found the first-pass parser dropped exponents.

Use of AI tools↩︎

Generative AI assistance was used during manuscript finalization for language editing, for drafting and formatting figure-generation code. The large language models studied in this work are the objects of the experiments and are described in Methods; they were not used to generate the scientific insights, the experimental design, or the conclusions. The authors take full responsibility for all content of this manuscript.

Computing resources↩︎

All model generations were obtained through the OpenRouter API. The four primary models were Claude Sonnet 5, GPT-5.5, DeepSeek v4 Pro, and Qwen3-235B; two smaller models, Claude Haiku 4.5 and GPT-5.4 mini, were used only for the small-model detection comparison. Decoding used temperature \(0\) (greedy) for the single-answer conditions and temperature \(0.7\) for sampled generations, including the best-of-8 rerank (\(N=8\)); top-\(p\) was left at the provider default. The maximum-token cap depended on the workflow: \(1{,}200\) tokens for the main arms and holdout harness, \(900\) for the closed-loop correction driver used by the two-stage and isotope experiments, and \(1{,}400\) for the end-task runs. Density-functional theory reference calculations and all database lookups (PubChem, Materials Project, CCCBDB, IAEA NDS) ran on a single workstation CPU. The USD and wall-clock figures in Table S4 are scaled from a measured base generation cost of $0.00418 per trace at 12 concurrent workers. The exact provider-side model-version snapshot identifiers for each model are listed in Supplementary Table 8.

5 Supplementary information↩︎

This supplementary material collects the extended tables, robustness analyses, and notes that support the main-text claims but are not needed to follow the argument: per-object-type extraction and error breakdowns (Tables 2, 3), a parser-artifact audit of the physical-constants domain (Supplementary Note S1), a clustered re-analysis of the headline significance tests (Table 4), the per-condition compute accounting (Table 5, Fig. 4), a reference-frame-policy robustness check (Table 6), a four-detector comparison (Table 7), the model versions and access details (Table 8), a Chain-of-Verification tool-free baseline (Supplementary Note S3), the isotope frontier lift re-estimated at five replicates (Supplementary Note S4), and a realistic non-oracle retrieval bound (Supplementary Note S5). Each item is referenced from the main text at the point it supports; the paragraphs below state what each one shows and its single takeaway.

Extraction and error structure↩︎

The verifier can only correct what its extractor first commits to a parseable claim, so we report the per-object-type extraction rate alongside accuracy. Table 2 gives both: extraction rate (the fraction of cells that commit a parseable value) sets a hard ceiling on verifier recall, which is why the crystalline space-group surface, committed in only 54% of cells, has a low detection recall regardless of how good the adjudicator is.

4pt

Table 3 shows where the 248 genuine baseline errors actually fall. Two patterns determine where correction can help: errors concentrate in the long-tail rarity strata (S3–S4) rather than the well-known compounds, and they concentrate in the weakest model (Qwen3-235B alone accounts for 131 of 248). This is the error structure behind the per-model lift in Table 1: correction lifts accuracy exactly where errors are dense and leaves the near-ceiling frontier models roughly unchanged.

Supplementary Note S1: Physical-constants audit (parser artifact and per-model detail)↩︎

The first-pass constants result (84% to 99%) was traced by a line-by-line audit of every flagged constant: \({\sim}23\) of \({\sim}29\) baseline “errors” were orchestration-layer extraction artifacts: the model stated the correct value in scientific notation (\(1.602176634\times10^{-19}\), \(4\pi\times10^{-7}\), or LaTeX-brace form) but the value parser dropped the exponent, and Mode A “fixed” these by eliciting an ASCII-parseable restatement of the same value. After hardening the parser (Unicode superscripts, LaTeX braces, symbolic-\(\pi\), digit-group spacing) and re-grading the full triplicate under the registered tolerance policy, no model changes significantly: the two frontier models start at 100% baseline and dip slightly under Mode A (Claude Sonnet 5 100% to 97%, GPT-5.5 100% to 94%) from regeneration noise on already-correct answers, while the open-weights models each gain \({\sim}5.7\) points (DeepSeek v4 Pro 91.4% to 97.1%, Qwen3-235B 88.6% to 94.3%); all \(p\ge0.5\). The verifier still detects the genuinely wrong constants that remain (Qwen3-235B dropping the \(\times10^{-8}\) on the Stefan–Boltzmann constant, a missing sign on the electron \(g\)-factor, a \(10^{3}\) unit-scale slip on the Josephson constant). This asymmetric artifact (baseline scored wrong, re-elicitation flipped it to correct) is the failure mode the materials retro-audit checked for and, via the symmetric subscript-abstain test, ruled out for the materials headline (Methods).

Supplementary Note S2: proof of the error-reduction identity↩︎

Setup. Consider the claims committed and graded in both the baseline and Mode-A arms on a fixed surface, with baseline error-given-commitment \(e_0=\Pr(\mathrm{wrong})\). Gated correction applies a fire/hold decision \(D\) per claim (the verifier flag conjoined with the Platt gate \(\sigma(as+b)<\theta\), where \(s\) is the raw verifier score from the main text and \(\sigma\) is the logistic function); a fired claim is regenerated with the reference value injected. Define the conditional rates \(r_d=\Pr(D{=}\mathrm{fire}\mid\mathrm{wrong})\), \(r_p=\Pr(\mathrm{correct\;after\;regen}\mid D{=}\mathrm{fire},\mathrm{wrong})\), \(\phi=\Pr(D{=}\mathrm{fire}\mid\mathrm{correct})\), and \(\rho=\Pr(\mathrm{wrong\;after\;regen}\mid D{=}\mathrm{fire},\mathrm{correct})\). Held claims are unchanged, and any regeneration that leaves the committed population (a regenerated no-commit) is excluded by the committed-in-both-arms restriction, so every claim ends either correct or wrong.

Precision form of the harm term. The fired set has mass \(F=e_0 r_d+(1-e_0)\phi\) and detector precision \(\pi=e_0 r_d/F\), so \((1-e_0)\phi=(1-\pi)F\) and the harm term equals \((1-\pi)F\rho\). Harm therefore grows as precision falls: a high-recall, low-precision detector (the consistency-triggered stage reaches precision \(0.28\) on formation energy) fires largely on already-correct claims, and unless the break rate \(\rho\) is suppressed the gain \(e_0 r_d r_p\) is offset. Two levers reduce harm without lowering recall: injecting a reference, which raises \(r_p\) toward \(1\) and is what the deterministic tier supplies but the consistency signal does not; and gating on a calibrated trust score, which lowers \(\phi\) and hence raises \(\pi\). The identity thus separates the two requirements the experiments establish: detection recall sets the reachable gain, and a reference-carrying, precision-calibrated fire sets how much of it is realized without harm.

The identity is an exact accounting relation, following from the law of total probability applied to the two ways a committed claim can be wrong after correction. As a numerical check on the molecular-formula surface, substituting \(e_0{=}0.22\), \(r_d{=}0.82\), and \(r_p{=}0.97\) with a negligible false-positive break term gives \(e_1=e_0(1-r_d r_p)=4.5\%\), consistent with the observed post-correction error of about \(4\%\).

Robustness of the headline statistics↩︎

The pooled McNemar tests in the main text treat prompt\(\times\)model cells as independent, but the three replicates of each prompt are not. Table 4 re-tests the two significant headlines under a prompt-clustered cluster bootstrap that resamples whole prompts (all four model rows together): both the molecular-formula and formation-energy net effects remain strictly positive across 10,000 resamples (fraction with net \(\leq0\) is 0.000), and crystalline space group stays non-significant, so the headlines are not artifacts of treating replicates as independent.

Compute cost↩︎

Table 5 accounts for the compute of every condition on the same footing: calls per cell and retrieval rounds are measured, USD and wall-clock are scaled from the measured base generation cost. The takeaway is the deployment trade between the two interventions: best-of-8 rerank (Mode B) costs \({\sim}8\times\) baseline generation (\({\sim}\$212\), \({\sim}92\) min) for its formation-energy gain, whereas gated correction (Mode A) costs only \({\sim}1.75\times\) (\({\sim}\$46\), \({\sim}20\) min) because it retrieves only on a flag rather than on every prompt. Fig. 4 visualizes the retrieval side of this saving.

Reference-frame robustness↩︎

A quantitative-property claim can look “wrong” merely because the model and the reference use different conventions (e.g.a computed vs an experimental band gap). Table 6 checks this by re-grading under two policies. Only the band-gap surface is frame-sensitive: its error collapses from 34.6% to 0.0% once either documented frame is accepted (107 of 116 naive flags were correct experimental gaps), which is why band gap is excluded from the property headline. Formation energy, whose frames nearly coincide, barely moves (33.7% to 31.3%) and remains a genuine error surface; the frame-insensitive surfaces are unchanged.

Why a deterministic checker↩︎

Table 7 compares our deterministic tiered checker against three alternatives on the same frozen baseline claims: a frontier reasoning-class LLM-as-judge (no tools), a sampling-consistency baseline (SelfCheckGPT-style), and a semantic-entropy detector. The deterministic checker gives the high recall that matters for catching errors on the molecular-formula surface (0.92) while keeping precision usable (0.51), where the alternatives either trade recall for precision (the LLM judge, 0.71 recall at 0.86 precision) or over-flag (sampling-consistency, 0.85 recall but only 0.40 precision, with many false alarms); no single detector dominates on every surface, but only the deterministic tier returns an external reference value to inject, which is what the correction step requires. The semantic-entropy detector, which clusters the eight samples by meaning before measuring entropy, behaves as a refinement of the sampling-consistency baseline: it matches its recall on molecular formula (0.93) and raises formation-energy precision to 0.51 (from 0.36) at slightly lower recall (0.57 from 0.67), but like it returns no reference value and so cannot drive the correction step.

3pt

Retrieval savings↩︎

Fig. 4 makes the retrieval side of the cost story (Table 5) concrete: because gated correction retrieves only when a claim is flagged, it issues far fewer external lookups than blanket RAG’s one-per-prompt baseline, \(3.2\times\) fewer on molecular formula, \(6.2\times\) on formation energy, and \(8.4\times\) on crystalline space group. The three bars report round-0 Mode A cell flag rates (triplicate mean) on those surfaces; Table 5’s 17 per 100 pools all claim types.

Model versions and reproducibility↩︎

All generations were served through the OpenRouter API. Table 8 records the exact model string for each model so that the runs can be reproduced against the same provider-side snapshots.

Supplementary Note S3: Chain-of-Verification as a stronger tool-free baseline↩︎

Self-critique is a single-pass no-tools control. To test whether a stronger structured self-verification method fares differently, we ran Chain-of-Verification [10]: the model drafts an answer, plans a set of verification questions targeting its own factual claims, answers those questions independently, then revises. No external tool is used at any stage. We ran the four-stage protocol on the three error surfaces (molecular formula, formation energy, crystalline space group) with the four primary models in triplicate, using the same corpus, system prompt, and frozen grader (final-answer rule) as the self-critique arm; 4,427 cells, no API errors. It closes no surface: on the two surfaces whose answer formatting the frozen grader parses without ambiguity, error given commitment is essentially unchanged (molecular 21.7% to 18.8%, crystalline 5.8% to 6.0%), and on formation energy it does not improve on the baseline. A formatting note bounds the formation-energy comparison: about a quarter of Chain-of-Verification answers write the value with a Unicode minus sign, which the frozen extractor’s ASCII regex misreads, inflating the raw error rate; the glyph rate differs across conditions, so we report the raw frozen-grader numbers and do not renormalize the published self-critique value. Under a sensitivity check that maps the Unicode minus to ASCII for all conditions, Chain-of-Verification formation-energy error is 29.5% against a 31.3% baseline, still no improvement. The finding matches self-critique: a stronger tool-free protocol does not reach the gap that tool-grounded correction closes, because the model cannot supply the external reference value it lacks.

Supplementary Note S4: Isotope frontier lift at five replicates↩︎

The two per-model frontier isotope lifts reported in the main text (Claude Sonnet 5 and GPT-5.5) rest on triplicate McNemar tests with \(p=0.016\) and \(p=0.021\), which do not survive correction for multiple comparisons. Because these borderline claims carry the paper’s only frontier-model lift, we re-estimated the isotope arm at five replicates. We generated two additional replicates of both arms (baseline and gated Mode A) over the 63 isotope prompts and four primary models (1,008 cells, no API errors), using the same prompts, ground truth, system prompt, tolerance, and per-model aggregation (correct versus not-correct on model-by-subject units, majority over replicates) as the triplicate analysis. The per-model net lifts are unchanged (Claude Sonnet 5 net \(+7\), GPT-5.5 net \(+8\), Qwen3-235B net \(+11\); DeepSeek v4 Pro rises from \(+4\) to \(+6\)), and the two frontier \(p\)-values sharpen to \(p=0.016\) (Claude Sonnet 5) and \(p=0.008\) (GPT-5.5). Under the same thirteen-test Holm and Bonferroni family, GPT-5.5 then crosses the Holm threshold but not Bonferroni, and Claude Sonnet 5 crosses neither. The added replicates therefore tighten the estimates without changing the conclusion, and we continue to report the frontier lift as suggestive rather than confirmatory.

Supplementary Note S5: Realistic (non-oracle) retrieval↩︎

The RAG conditions in the main text are an oracle upper bound, drawing facts from the same database records that define the ground truth, so coverage is complete and every fact is correct. To bound what a deployed system would see, we built a realistic retriever that resolves each named entity live: molecular names against PubChem (formula and SMILES), and crystalline and formation-energy entities by resolving a formula and querying the Materials Project for the most-stable polymorph’s space group and formation energy. No ground-truth record is consulted during retrieval. Retriever quality varies sharply by surface. Molecular coverage is 141 of 141 entities, all correct, because every compound in the corpus is name-addressable in PubChem, the same source the molecular ground truth draws on. Crystalline coverage is 27 of 127 entities (21%), of which 20 of 27 return a space group within the accepted set; formation- energy coverage is 23 of 101 entities (23%), all within tolerance when returned. The two failure modes on the long tail are a name that does not resolve to a formula, and a formula that resolves to the most-stable polymorph rather than the phase the prompt names (for example rutile returns space group No. 141 where the prompt fixes the No. 136 phase). Prepending these retrieved facts and generating once, realistic retrieval matches the oracle on molecular formula (error given commitment 0.2% versus the oracle’s 0.3%, baseline 21.7%) with low abstention (4.8% versus the oracle’s 42.0%), but recovers almost none of the oracle gain on the long tail: error given commitment holds at baseline (formation energy 33.9% versus 31.3%; space group 5.3% versus 5.8%) and abstention rises to 56.2% and 59.4% as the model, handed no usable fact, declines to commit. The verifier’s claim-specific retrieval, keyed to the named phase rather than to bare composition, is what still succeeds in this regime.

References↩︎