1 Introduction↩︎

Retrieval-augmented question answering (RAG) pipelines increasingly route retrieved passages through a stronger LLM rewriter—a compiler, summarizer, or compressor—before a smaller reader produces the final answer [1]–[3]. On multi-hop benchmarks the rewriter lifts reader F1 by tens of points, and the lift is typically credited to improved evidence quality: better organization, denoising, multi-hop chaining. We ask a more basic question: how much of the lift is caused by the gold answer string being surfaced in the rewritten context, rather than by curation alone? In these same multi-hop settings, the rewriter also surfaces the gold answer string in roughly \(80\%\) of records, so the two explanations—curation and answer-string surfacing—are observationally entangled in the aggregate F1 gains used to justify the pipeline.

The conventional way to break this entanglement is to substring-mask the gold answer in the rewritten context with a sentinel token, most commonly [MASK], and re-run the reader: a collapse to raw-retrieval F1 is taken as evidence of answer-string leakage, while a significantly positive residual is taken as evidence of a non-leakage channel. We show in §4 that this single-sentinel probe is itself unreliable. On 2WikiMultihopQA the [MASK] leaves a \(+4.12\) F1 residual over raw retrieval, but under four alternative sentinels ([REMOVED], a natural-language deletion phrase, a generic word, and a symbol string) on the same paired examples, the residuals instead range from \(-3.33\) to \(-7.81\) F1, and the equivalence criterion is met for only one of the four sentinels; the apparent residual is largely a [MASK]-token artifact. A masking diagnostic that can flip sign with the choice of sentinel cannot, by itself, separate answer-string surfacing from genuine evidence curation.

We therefore replace single-sentinel masking with a controlled intervention audit. For each rewritten context we re-run the reader under four controlled edits: remove the gold answer span, replace a length-matched random non-answer span as a placebo, or insert the gold answer string into rewrites that lack it, either at the prefix or at a midpoint sentence boundary. The remove-minus-placebo contrast on the paired answer-in-compile stratum is a direct, on-distribution estimate of the causal dependence of reader F1 on the gold answer string being present. On the complementary subset, insertion tests whether restoring the gold answer string recovers F1.

Across twelve evaluated (cell, baseline) intervention runs, spanning three reader families (Qwen2.5, Qwen3.5, GLM), two datasets (HotpotQA, 2WikiMultihopQA), and three compiler configurations (MA-only, MB-only, MA\(+\)verify), removing the gold answer drops reader F1 by \(28\) to \(64\) points beyond the length-matched placebo, and prepending the gold to rewrites that lacked it raises F1 by \(+0.7\) to \(+9.7\) points in \(10\) of \(12\) (cell, baseline) combinations.

We make three contributions. First, we present the first controlled answer-presence intervention audit for compile-then-read RAG. The remove / placebo / insert design yields a remove-minus-placebo F1 drop of \(28\) to \(64\) points across twelve (cell, baseline) intervention runs, and reveals that insertion effects depend on position. Second, we give a negative result for single-sentinel masking diagnostics: in a five-sentinel audit on 2WikiMultihopQA, the positive [MASK] residual reverses under all four alternative sentinels. Third, we release a reusable audit kit, including an intervention runner and a sentinel panel, so that future rewriter-gain claims can be tested against a common standard. §2 defines the setup, §3 specifies the audit protocol, §4 reports the results, and §5 discusses what the interventions identify and what they do not.

2 Setup↩︎

2.0.0.1 Pipeline.

A QA question \(q\) is answered from a long retrieved context \(C_q\) by a reader \(\mathrm{Ma}\) in one of four settings: B\(_1\) raw retrieval (reader sees \(C_q\)); B\(_2\) MA-only compile (\(\mathrm{MA}(C_q,q)\)); B\(_3\) MB-only compile, with \(\mathrm{MB}\) a different model family from \(\mathrm{MA}\); B\(_4\) MA compile then \(\mathrm{MB}\)-verify, which may rewrite unsupported sentences.

2.0.0.2 Cells.

The audit is run on four (reader, compiler-family, dataset) cells: S1 (Qwen2.5-7B / Qwen2.5-72B / HotpotQA), S2 (Qwen2.5-7B / Qwen2.5-72B / 2Wiki), S3 (GLM-4.7 / GLM-5 / HotpotQA), S5 (Qwen3.5-35B / Qwen3.5-27B / HotpotQA), with DeepSeek-V3 as \(\mathrm{MB}\) in every cell. Each cell exercises B\(_1\)–B\(_4\) on the same \(1{,}000\)-question subset. The suite labels follow our internal run IDs: S4 is a verifier-variant pilot reported only in the appendix and is not part of the main answer-presence intervention grid. The four cells cover three reader families, two datasets, and two \(\mathrm{MA}\) families, so that no single contrast in §4 is identified by a single (reader, compiler) combination.

2.0.0.3 Datasets and decoding.

HotpotQA distractor split [4] and 2WikiMultihopQA [5] are both multi-hop benchmarks whose distractor pools already contain every gold supporting paragraph for every evaluated query, so compile gain cannot be attributed to compensating for missing retrieved evidence. Records with gold strings shorter than two characters are excluded. Token-level F1 is computed against the original gold [4]. Across cells, the compile output surfaces the gold answer string in roughly \(80\%\) of records; this answer-surfacing rate is the observational entanglement the intervention audit is designed to break. Reader: \(\texttt{temperature}{=}0.01\), \(\texttt{max\_tokens}{=}512\); compilers: \(\texttt{temperature}{=}0.2\), \(\texttt{max\_tokens}{=}2048\).

3 Audit Protocol↩︎

The audit has two layers. The causal-intervention layer (§3.1) is the main test of whether gold-answer presence in the rewritten context causally drives reader F1. The sentinel layer (§3.2) controls a separate concern: the same intervention can mislead if its implementation (e.g.the choice of mask token) leaks exploitable structure to the reader.

3.1 Causal Interventions↩︎

3.1.0.1 Motivation.

Measuring how reader F1 changes when the rewriter is added is observational: in a typical cell, B\(_2\) both re-organises retrieved evidence and surfaces the gold answer string into the rewritten context. To estimate the on-distribution causal effect of answer presence we must edit the rewritten context to add or remove the gold while holding everything else as close to constant as possible, and we must distinguish that edit’s effect from the effect of any edit of the same size.

3.1.0.2 Design.

For each B\(_2\)/B\(_3\)/B\(_4\) compile output \(c\) we apply one of four edits: remove every case-insensitive match of the gold answer in \(c\) with [MASK]; placebo replace a length-matched random non-answer span (deterministic seed \(1729\)) with [MASK]; insert_prepend prepend “Note: <gold>.” to \(c\); insert_mid insert “<gold>.” at the midpoint sentence boundary. The remove/placebo edits target \(\mathtt{ans\_in\_compile}{=}1\); the insert edits target \(\mathtt{ans\_in\_compile}{=}0\).

3.1.0.3 Stratification.

Each record is tagged with two binary flags: \(\mathtt{ans\_in\_b1}\in\{0,1\}\) (the gold appears in raw \(C_q\)) and \(\mathtt{ans\_in\_compile}\in\{0,1\}\). The compile output moves a record into one of four transition buckets, \(\mathtt{(ans\_in\_b1, ans\_in\_compile)} \in \{0{\rightarrow}0, 0{\rightarrow}1, 1{\rightarrow}1, 1{\rightarrow}0\}\). The remove and placebo interventions apply only where \(\mathtt{ans\_in\_compile}=1\); the insert interventions apply only where \(\mathtt{ans\_in\_compile}=0\). Within each (cell, baseline, intervention) we report the F1 mean and a paired \(\Delta\) against the unperturbed compile output, computed by bootstrap (\(1{,}000\) resamples, seed \(42\), \(95\%\) CI).

3.1.0.4 Identification.

Both remove and placebo edit a span of identical word count from the same rewriter output and differ only in whether the deleted content is the gold answer. We therefore read \(\Delta_{\text{causal}}=\Delta_{\text{remove}}-\Delta_{\text{placebo}}\) on the \(1{\rightarrow}1\) stratum (\(\mathtt{ans\_in\_b1}{=}\mathtt{ans\_in\_compile}{=}1\)), where both perturbations apply and the gold is already retrievable from raw context, and interpret it as the average treatment effect of gold-answer presence on reader F1 in \(1{\rightarrow}1\) contexts. Positive \(\Delta\) on the \(0{\rightarrow}0\) insert buckets is the complementary estimate of how much F1 the rewriter would have provided had it surfaced the answer. Because both arms write [MASK] into \(c\) and differ only in whether the masked span is the gold, the common sentinel-token main effect cancels in \(\Delta_{\text{remove}}-\Delta_{\text{placebo}}\), so the sentinel-fragility concern of §3.2 applies to single-sentinel leakage residuals but not to \(\Delta_{\text{causal}}\) itself.

3.2 Sentinel-Fragility Audit↩︎

3.2.0.1 Motivation.

A sentinel the reader can exploit would produce a spurious “non-leakage residual” in the older mask-and-see literature and, to the extent that its effect interacts with the surrounding context rather than acting only as a common additive shift, could leak a second-order term into \(\Delta_{\text{causal}}\) even after the main-effect cancellation of §3.1. We audit this separately on the \(\mathtt{ans\_in\_compile}{=}1\) stratum of cells S1 and S2, following the sentinel-ablation protocol in scripts/preregistration_99c_
[1]sentinel_ablation.md.

3.2.0.2 Design.

Five replacement tokens are applied to the same paired stratum: MASK [MASK] (the conventional choice), REMOVED [REMOVED] (bracketed sentinel without standard placeholder semantics), NATURAL “the answer was removed” (natural-language deletion), WORD thing (generic noun), and SYMBOL ### (symbol string). A length-matched PLACEBO replaces a random non-answer span of equal word count with [MASK].

3.2.0.3 Equivalence criteria.

C2a (sentinel equivalence). A non-[MASK] sentinel passes if its paired \(\Delta\) against [MASK] has CI containing zero, or \(|\Delta|<1.0\) F1, or \(|\Delta|<0.20\times\) the original [MASK]-vs-B\(_2\) effect on the same stratum; the audit passes if \(\geq 3/4\) alternative sentinels pass, otherwise the [MASK] residual is judged sentinel-fragile. C2b (placebo). The B\(_2\)-vs-PLACEBO paired \(\Delta\) has CI containing zero or \(|\Delta|<0.50\times\) the original effect, confirming that any masking-side F1 collapse is answer-specific rather than perturbation-generic.

4 Results↩︎

4.0.0.1 Causal \(\Delta\) on the \(1{\rightarrow}1\) stratum.

Table 1 reports the headline quantity: for each (cell, baseline), the paired remove and placebo \(\Delta\)s on the \(1{\rightarrow}1\) stratum, and their difference \(\Delta_{\text{causal}}=\Delta_{\text{remove}}-\Delta_{\text{placebo}}\). Across the twelve (cell, baseline) intervention runs—three baselines per reader for S1, S2, S3, S5—removing the gold answer drops reader F1 by \(37\) to \(65\) points, the length-matched placebo drops F1 by only \(0\) to \(13\) points (and is mildly positive in S3 and S5), and \(\Delta_{\text{causal}}\) ranges from \(-28.2\) (S1, B\(_2\)) to \(-64.1\) (S3, B\(_2\)) F1. All twelve \(\Delta_{\text{causal}}\) values have the same sign and exceed \(25\) F1 in magnitude. The S5 cell (Qwen3.5-35B reader) also has a mean \(\Delta_{\text{placebo}}\) that is mildly positive (\(+1.9\) to \(+4.2\) F1), which strengthens the contrast: deleting a same-sized non-answer span is not on average harmful in that cell, yet deleting the gold answer collapses F1 by tens of points.

3pt

4.0.0.2 Insertion in the \(0{\rightarrow}0\) bucket.

Prepending “Note: <gold>.” to compile outputs that lacked the gold answer raises reader F1 by a positive \(\Delta\) in \(10/12\) cell–baseline combinations (range \(+0.7\) to \(+9.7\) F1, with S1 and S5 clustered near \(+8\) to \(+10\) in B\(_2\)/B\(_3\) runs and S3 showing a smaller \(+2\) to \(+6\) effect). Inserting the same string at the midpoint sentence boundary instead of the prefix gives a mostly non-positive \(\Delta\) (range \(-13.3\) to \(+5.5\) F1, \(9/12\) negative), so the reader’s use of an injected gold is position-sensitive: prefix-injected gold lifts F1 in the direction that removing the gold lowered it, but mid-context injection does not.

4.0.0.3 Sentinel-fragility (companion).

Table 2 reports the audit on S1 and S2. On S1, all five sentinels collapse post-mask F1 below raw retrieval and \(4/4\) alternatives pass C2a (sentinel-robust). On S2 the [MASK] sentinel reports a \(+4.12\) F1 “non-leakage residual” that the four alternatives invert to between \(-3.33\) and \(-7.81\) F1 (\(1/4\) pass), judging the residual a [MASK]-token artifact. Both cells pass C2b. Because \(\Delta_{\text{causal}}\) in Table 1 is computed against the matched placebo and not against the [MASK]-residual, it does not inherit this sentinel-token exposure.

2.5pt

4.0.0.4 Identity sanity check.

We first rule out rerun instability as an explanation. When the gold answer is absent from the compiled context (\(\mathtt{ans\_in\_compile}{=}0\)), the remove edit changes nothing. Rerunning the reader on these unchanged contexts reproduces the original compile F1, with median per-question \(|\Delta|{=}0.000\) on both datasets. The intervention deltas are therefore not artifacts of a second reader call.

5 Discussion↩︎

Across twelve (cell, baseline) intervention runs, the remove-minus-placebo \(\Delta_{\text{causal}}\) in Table 1 lies in \([-64.1,-28.2]\) F1 with the same sign in every cell; even accepting the upper-end placebo collapse as residual confounding, the gold answer is a necessary input to the bulk of the F1 lift the rewriter delivers on these multi-hop benchmarks. The \(0{\rightarrow}0\) insertion results (prefix positive in \(10/12\) (cell, baseline) combinations, midpoint mostly non-positive: \(9/12\) negative) are the symmetric statement: when the rewriter fails to surface the gold, prefix-injecting it recovers between \(+0.7\) and \(+9.7\) F1, a nontrivial fraction of the \(\sim\!30\) F1 B\(_2\) lift. We do not interpret these as the entire story—rewriters plausibly also de-clutter evidence—but they bound how much of the lift can be credited to “curation quality” without further controlled evidence: most of it cannot. The sentinel-fragility audit (Table 2) rules out a simpler “do the older masking diagnostic bigger” rebuttal: on 2Wiki the [MASK] probe’s positive “non-leakage residual” flips under four alternative sentinels and fails C2a, so the design’s reliance on a remove-vs-placebo paired contrast—rather than on a [MASK]-residual vs raw retrieval—is necessary. For deployment audits, claims of a non-answer-string compile channel should be paired with both layers (sentinel + placebo on the masking side; remove vs.placebo on the \(\mathtt{ans\_in\_compile}{=}1\) stratum); the released audit kit (scripts/p0_intervention.py and the sentinel runner) supplies the standard.

6 Related Work↩︎

6.0.0.1 Leakage and perturbation in QA/RAG.

Train–test overlap and exploitation of contaminated data have been documented and debated in open-domain QA [6]–[8]; in RAG, entity perturbation [9], when-does-retrieval-help analyses [10], [11], and noise-sensitivity audits [12], [13] characterize context-vs-parametric reliance. Counterfactual data [14], contrast sets [15], behavioral testing [16], and distractor-sensitivity probes [17] provide the methodological tradition our placebo condition continues. The missing piece our paper supplies is an in-passage remove-vs-placebo intervention on the rewriter’s output, on the same paired stratum, identifying the causal effect of answer-string surfacing.

6.0.0.2 Sentinel sensitivity and statistical practice.

Prompt- and format-fragility is well-attested [18]–[22]; [23] treats [MASK] as an information-gathering token in pre-training, the only close neighbour to our sentinel-flip result. On the statistical side, we follow standard rigour arguments [24]–[26]; paired bootstrap CIs adapt [27], and the equivalence thresholds follow TOST [28]. We combine these tools into a single audit for one specific RAG failure mode.

Limitations↩︎

The intervention audit covers four (reader, compiler, dataset) cells—three reader families, two datasets, three compiler arrangements—over twelve (cell, baseline) intervention runs. We do not claim \(\Delta_{\text{causal}}\) magnitudes transfer outside this grid. The remove/insert interventions are string-level: aliases, paraphrasings, and entity-mediated cues are not edited, so a rewriter that consistently restates the gold in different words would still pass the remove condition (paraphrastic leakage is not detected). Our scope is restricted to online, per-query rewriting; in offline LLM-curated corpora [29]–[33] the rewriter is query-agnostic and cannot selectively surface a specific gold span, so our magnitudes do not transfer, though we view analogous answer-removal/placebo controls as a reasonable precondition for attributing such gains to curation quality. We do not propose a new rewriter, mitigation, or alias-aware masking scheme; the contribution is diagnostic.

6.0.0.3 Reproducibility.

Exact prompts, decoding parameters, model endpoint identifiers, and the suite-ID to log-file mapping are listed in Appendix 19. The intervention runner and sentinel panel are released as part of the audit kit (scripts/p0_intervention.py and the sentinel-ablation runner).

Supplementary Material↩︎

This appendix collects supplementary experiments around the controlled intervention audit reported in the main text. None is required for the main-text tables, but together they provide the wider evidence base that motivated and constrains it. The sections cover: pipeline-level numbers — full main-suite results (§7), reader-scale attenuation (§8), and cross-rewriter sweeps (§9); mechanism probes — verification-mode ablations (§10), length-controlled regressions on answer preservation (§11), the V\(_5\)-vs-Control length-quartile breakdown (§12), per-question oracle headroom and the selection gap (§13), and the question-blind containment audit (§14); and descriptive / reproducibility material — operator-by-reader pattern (§15), the full answer-mask ablation and dashboard cross-walk (§16), the 2Wiki alias spot-check (§17), prompts (§18), and model endpoints with run-ID and log-file mapping (§19).

6.0.0.4 Suite heterogeneity warning.

The appendix collates experiments that differ from the main-text answer-mask cells along several axes; their absolute numbers are not interchangeable with the main text. The main-text S1/S2 results (Table 2) use the B\(_2\) compile rewriter on a paired \(1{,}000\)-question subset of each dataset with the Qwen2.5-7B reader at \(\texttt{temperature}{=}0.01\). The appendix departs from this baseline in three ways: (i) the scale sweep (§8) and the length / oracle / OLS analyses (§§11–13) substitute the V\(_5\) quote-first rewriter and run on the full \(N{=}7{,}405\) HotpotQA / \(N{=}1{,}005\) Qasper sets; (ii) the cross-rewriter sweep (§9) additionally swaps the rewriter model; (iii) the verification-mode ablations (§10) add a verifier stage. The resulting differences in rewriter setting, sample, and reader produce different absolute F1s (e.g.raw HotpotQA \(=23.0\) on the main-text 1k subset but \(65.13\) on the cross-rewriter full set), so appendix tables should be read for their within-cell deltas, not against main-text numbers. The S1/S2/S3/S5 cells in §7 correspond to the suites used in the main-text intervention audit; the masked-ablation tables in §16 share the S1/S2 answer-mask baseline.

7 Full Main-Suite Pipeline Results↩︎

Table 3 reports the full pipeline F1 across all suites we ran (S1–S5 on HotpotQA/2Wiki, Q1/Q2 on Qasper). The short-paper main text uses S1 (Qwen-7B / HotpotQA), S2 (Qwen-7B / 2Wiki), S3 (GLM-4.7 / HotpotQA), and S5 (Qwen3.5-35B / HotpotQA) for the intervention audit, and S1, S2 for the sentinel-fragility companion. The remaining suites (S4 verifier pilot, Q1, Q2 on Qasper) document that the per-cell rewriter effect can shrink toward zero or flip sign as the reader strengthens or the dataset changes; this is the broader empirical context for the sentinel fragility we report on HotpotQA vs.2Wiki.

3pt

8 Reader-Scale Attenuation↩︎

To isolate a within-family reader effect, we hold the rewriter setting fixed (V\(_5\) quote-first, Qwen2.5-72B rewriter) and sweep Qwen2.5 reader scale from 0.5B to 72B on both HotpotQA and Qasper. Table 4 reports, at each reader scale, the paired F1 delta between the rewriter output and raw retrieval.

6pt

Two consequences for the main-text intervention. First, the attenuation curve in HotpotQA explains why the main-text mask_applied=True stratum compile lift (Table 2, \(+30.45\) F1 for Qwen2.5-7B) is much larger than the equivalent lift for a 72B reader (a near-null \(+1.03\) F1 from the sweep): the audit verdicts are therefore specific to the 7B reader cell, and we do not extrapolate them across reader scales. Second, the Qasper column shows that a uniform compile policy already fails the sign-monotonicity test on a third dataset; the answer-mask intervention is not performed on Qasper because its free-form, abstractive answers do not admit clean substring masking (Limitations).

9 Cross-Rewriter Sweep: Stronger Rewriter Helps but Does Not Close the Reader-Dataset Gap↩︎

One alternative account of the negative Qasper lifts in Appendix 8 is that the rewriter is underpowered. We test this by swapping Qwen2.5-72B-Instruct for Qwen3-235B-A22B-Instruct-2507 under the same question-conditioned Control rewriter setting and the same reader (Qwen2.5-7B). The stronger rewriter improves both datasets: on HotpotQA, Control rewrite F1 moves from \(74.86\) to \(75.50\) (\(+10.37\) over the raw baseline of \(65.13\)); on Qasper, from \(38.50\) to \(40.94\) (\(+2.69\) over \(38.25\)). Rewriter scale is not irrelevant. With the same reader, however, the HotpotQA / Qasper gap persists: HotpotQA remains a high-gain setting and Qasper remains a low-gain one, ruling out the strongest version of a writer-only account in the range we can measure. The mediating factor must lie outside rewriter capacity alone — which is consistent with the dataset-dependent residual reported in the main text under a single rewriter.

10 Verification-Mode Ablations↩︎

The main-text intervention grid covers B\(_2\), B\(_3\), and B\(_4\). For completeness, this section ablates the verification mode applied to the compiled context: Table 5 reports three downstream verification variants — hard rewriting (B\(_4\), the verifier may delete or rewrite sentences from the compile output), soft annotation (B\(_{4s}\), the verifier tags but does not modify), and label-only verification (B\(_{4lr}\), the verifier classifies each compile-output sentence as supported or unsupported but copies it verbatim either way).

3pt

These ablations are reported here only as the broader experimental context within which the answer-mask intervention sits: they do not themselves identify answer surfacing as the operative mechanism. That identification is made by the main-text intervention together with the legibility regression in Appendix 11.

11 Length-Controlled Regression of Per-Question F1 on Answer Surfacing↩︎

As a correlational complement to the main-text intervention, we regress per-question F1 on (i) the rewriter setting (Baseline / Control / V5), (ii) log rewriter-output length, (iii) an answer-in-rewrite indicator (substring presence of any normalised gold alias), and setting\(\times\)length interactions, with cluster-robust SEs on qid. Table 6 and Table 7 report the coefficients for HotpotQA and Qasper across two readers each. The answer-in-rewrite indicator carries a coefficient of \(+0.29\) to \(+0.35\) at \(p<10^{-50}\) in every cell, an order of magnitude larger than any setting-only coefficient (all \(|\hat{\beta}_{\text{setting}}| \le 0.09\)). This is the correlational counterpart to the main-text intervention: when the answer is in the rewrite the reader scores about 30 F1 points higher per question at the same rewrite length, regardless of which rewriter setting produced it.

3pt

3pt

12 Length-Quartile Breakdown of V5 vs.Control on HotpotQA↩︎

The interaction term \(\text{V}_5 \times \log(\text{len})\) in Table 6 is positive and significant in both HotpotQA panels. Table 8 stratifies the V5\(-\)Control delta by length quartile of the joint Control+V5 length distribution, confirming that V\(_5\) underperforms Control on short rewrites and reverses on long ones for both readers.

4pt

13 Per-Question Oracle Headroom and the Selection Gap (Qasper)↩︎

Even on Qasper, where a fixed rewriter setting does not yield reliable aggregate gains, there is substantial per-question variation in which rewrite helps. We quantify this with a per-question oracle over rewrite variants and contrast it with a representative LLM-based router that uses only the question text to select.

4pt

The 2-action oracle leaves \(+8.13\) F1 of reachable headroom over the better single policy on Qwen2.5-7B (\(+7.81\) on Qwen3-8B); the 6-action oracle bounds the reachable variation at roughly \(+20\) F1 over raw. The LLM router (Qwen2.5-72B) recovers \(+0.38\) F1 (\(p{=}0.30\)) over the better single policy and lies \(-7.75\) F1 below the 2-action oracle and \(-19.78\) F1 below the 6-action oracle (\(p{<}0.001\) on both). The selection problem is therefore nontrivial and is not solved by writer scaling: a stronger Qwen3-235B Control rewrite as single policy reaches \(40.94\) F1, which still leaves a \(+5.69\) F1 oracle gap.

14 Question-Blind Containment Audit↩︎

If question-conditioning is the operational mechanism that produces answer surfacing in the compile output, removing the question from the rewriter prompt should cut answer-string containment substantially. Table 10 reports the audit for both compilers and both multi-hop / long-document datasets.

2.5pt

15 Descriptive Operator-by-Reader Pattern↩︎

Table 11 compresses the observed sign and rough magnitude of compile (\(B_2\)) and verifier-edit (\(B_4\)) effects across the reader \(\times\) dataset cells we ran. This is descriptive only; no rule is fit, and we discourage reading any single cell as a deployment recommendation without independent in-domain validation.

3pt

16 Answer-Mask Ablation: Full Table and Cross-Walk↩︎

16.0.0.1 Procedure.

For every \(B_2\) compile record on HotpotQA and 2Wiki we (i) extract the question and the compile-output context from the prompt sent to the Qwen2.5-7B reader; (ii) case-insensitively substring-replace the gold answer string in the context with the literal token [MASK], excluding records with gold length \(<2\) characters; (iii) re-render with the standard reader prompt and re-call Qwen2.5-7B (\(\texttt{temperature}{=}0.01\), \(\texttt{max\_tokens}{=}512\)); and (iv) score F1 against the original gold. We stratify on mask_applied: True when the gold string was present in the compile output (the interventional test), False when masking is the identity (sanity check).

16.0.0.2 Full table.

Table 12 adds the all and mask=False strata to the main-text [MASK] row of Table 2. The mask=False aggregates do not match the original \(B_2\) lift exactly on HotpotQA because reader rerun phrasing varies non-deterministically on yes/no questions; the per-question median absolute difference on the non-yes/no subset is \(<0.0001\) F1 (cf.the identity sanity-check paragraph in the main text).

3pt

17 Alias Spot-Check: Reproducibility Details↩︎

The main-text alias spot-check uses a deliberately permissive heuristic detector. For each 2Wiki mask=True record, after the gold answer string is replaced with [MASK], we test whether any of the following surface forms still appears case-insensitively in the masked compile context:

1. The gold answer with leading “the” stripped (e.g.”the United States” \(\rightarrow\) “united states”).

2. The last token of a multi-word gold answer, lower-cased and stripped of trailing punctuation, with length \(\geq 3\) (e.g.”Ridley Scott” \(\rightarrow\) “scott”).

3. A capital-initial acronym of a multi-word gold answer with length \(\geq 2\), lower-cased (e.g.”National Basketball Association” \(\rightarrow\) “nba”).

4. A 4-digit year extracted from a date-form gold (e.g.”June 12, 1987” \(\rightarrow\) “1987”).

Variants identical to the lower-cased full gold are removed; any variant shorter than 3 characters is discarded. The detector fires on a record if any variant occurs as a case-insensitive substring of the masked context.

17.0.0.1 Result.

On the \(n{=}829\) 2Wiki mask=True stratum, \(96\) records (\(11.6\%\)) are flagged. The remaining \(733\) records have masked-vs-\(B_1\) lift \(+5.20\) F1 (95% CI \([+2.67, +7.70]\), \(2{,}000\)-resample bootstrap, \(\text{seed}{=}42\)). The flagged-subset mean is \(\approx -4.13\) F1, so dropping it mechanically pulls the clean-subset mean above the \(+4.12\) F1 of the full set; we treat the clean-subset value as a robustness check, not an adjusted effect estimate.

17.0.0.2 Permissiveness.

Manual inspection of detector fires shows two recurrent patterns: (a) true alias preservation (“scott” surviving for “Ridley Scott”, “united states” for “the United States”); and (b) generic-last-token false positives (“heart” surviving for “Heart attack”, “school” for “High school”). Pattern (b) means the \(11.6\%\) figure is an upper bound on heuristically-detectable alias leakage, and the clean-subset residual is therefore an underestimate of the surviving non-answer-string component.

17.0.0.3 Reproducibility.

The full alias-spot-check script is at scripts/99d_2wiki_alias_check.py; it reads the released S2_2wiki.jsonl and S3_answer_masked_2wiki.jsonl logs, re-derives the masked context per record, applies the detector, and prints the firing rate plus the clean-subset bootstrap CI used above.

18 Prompts↩︎

18.0.0.1 Compiler (B\(_2\)/V\(_5\)).

The rewriter receives the question and all raw passages:

Identify all facts in the source documents that are relevant to the query. Rewrite those facts into a concise, logically structured summary. Do not fabricate information not present in the source documents.

18.0.0.2 Reader.

The reader prompt is the same for \(B_1\) (raw), \(B_2\) (compile), and the masked \(B_2\) context. Only the Context: field changes:

Context: <evidence>
Question: <question>
Answer concisely:

18.0.0.3 Hard verifier (used in Appendix 10 only).

Compare the summary against the source documents. If the summary contains entities or claims not present in the source documents, output a revised version with unsupported claims removed. If it is fully faithful, output the summary as is.

18.0.0.4 Soft / label-only verifiers (used in Appendix 10 only).

The soft verifier keeps all content but prefixes uncertain or unsupported claims with [UNCERTAIN] or [UNSUPPORTED]. The label-only verifier classifies each sentence as [SUPPORTED] or [UNSUPPORTED] but copies the sentence verbatim.

19 Model Endpoints, Decoding, and Run-ID Mapping↩︎

19.0.0.1 Decoding parameters.

Rewriter calls: \(\texttt{temperature}{=}0.2\), \(\texttt{top\_p}{=}0.95\), \(\texttt{max\_tokens}{=}2048\). Reader calls: \(\texttt{temperature}{=}0.01\), \(\texttt{top\_p}{=}0.95\), \(\texttt{max\_tokens}{=}512\) (default) or \(\texttt{max\_tokens}{=}32\) (strict 1–5-word reader-scale sweep). Failed calls were retried up to 3 times with exponential backoff; examples where any variant failed after all retries were excluded from the paired comparison for that suite. Paired bootstrap intervals use NumPy with \(\texttt{seed}{=}42\), \(1{,}000\) resamples for the main-text intervention audit (\(2{,}000\) in the per-question headroom analysis of Appendix 13, \(10{,}000\) in the long-form mechanism analyses of Appendix 11).

19.0.0.2 Model endpoints.

Table 13 lists the exact API model-field strings used. These are submitted field values, not guaranteed immutable checkpoint names; the table documents the evaluated endpoints but does not remove service-side reproducibility risk. Full-suite runs were produced between April 21 and April 26, 2026 on the same provider platform.

4pt

19.0.0.3 Run-ID mapping.

Table 14 maps the suite IDs used throughout the appendix to the on-disk JSONL log files released alongside the code. Every numeric claim in the paper is derived from these files via the aggregation scripts in scripts/.

2pt

References↩︎