1 Introduction↩︎

Large language models (LLMs) are increasingly deployed as automated proofreaders, code reviewers, and factual checkers, and their evaluation in these roles is often reduced to a single scalar: does the count of errors the model reports roughly match the count in the reference? When that count matches, the system is easily credited with an F1 near \(1.0\) even if its description of which errors occur is poor. This paper is about a concrete consequence of that shortcut, which we call F1 Inflation: the wedge between count-level agreement and span-level agreement on the same outputs. We show that this wedge can be driven wide by a single sentence in the prompt, and that it is large enough to change apparent conclusions drawn from count-only evaluation. The intended takeaway is not that one model family is better than another, but that a protocol which exposes expected counts can make count-based metrics unreliable.

To surface the wedge in a controlled way we need a mechanism that moves counts without changing the underlying text. Numeric anchoring supplies exactly that mechanism. A substantial body of cognitive psychology shows that humans adjust insufficiently away from a salient initial reference value [1], [2], and recent work suggests LLMs exhibit similar behavior on general reasoning tasks [3]. In realistic deployments the anchor is rarely arbitrary: an AI document-review system may preface its query with “this contract has been flagged as containing approximately \(5\) errors,” a sycophancy-adjacent signal [4], [5] that we can inject in a fully controlled way. By pairing each passage with prompts that supply the true count, an inflated count, and a deflated count, we get a clean stress test for evaluation reliability: anything that moves Count-F1 but leaves Span-F1 unchanged is a direct observation of F1 Inflation.

Our research questions are therefore inseparable from the evaluation critique they enable. Does an anchored count inflate Count-F1? Does that inflation survive a span-aware re-scoring? And which model-family behaviors are most sensitive under this protocol? We make four contributions. (1) We formalize and quantify F1 Inflation on a public benchmark, reporting gaps up to \(0.79\) under M2 overlap and \(0.96\) under strict M2 across 4,290 responses. (2) We validate that the inflation is not an artifact of description-based edit extraction: a full ERRANT 3.0.0 replication on a \(100\)-passage corrected-text subset spanning all six models and multi-reference (Annotators 0 \(+\) 1) scoring reproduces the pattern (mean Blind\(\to\)Anchored: Count-F1 \(+0.21\), multi-reference ERRANT \(F_{0.5}\) \(+0.04\); paired bootstrap shows \(\Delta\mathrm{Count}-\Delta\mathrm{ERRANT}>0\) at \(95\%\) CI for \(5/6\) models). (3) We introduce ErrorBench as a stress-test protocol: a 143-passage CoNLL-2014 testbed with five standardized prompt conditions, plus Count Bias (CB) and Anchoring Sensitivity Index (ASI) diagnostics for prompt-induced count distortion. Its role is analogous to a robustness audit for evaluation design, not a model-quality comparison. (4) A multi-model, multi-generational comparison provides model-family descriptive evidence that strong instruction following can coincide with larger anchoring responses and therefore with larger F1 Inflation in this setup. ErrorBench is not intended to compare contemporary models by overall quality; it is intended to diagnose whether an evaluation protocol is vulnerable to prompt-induced count distortion.

2 Related Work↩︎

2.1 LLM Evaluation and Error Detection↩︎

A growing body of work evaluates LLMs on structured proofreading and fact-checking tasks. [6] find that ChatGPT achieves competitive performance on grammatical error correction benchmarks while exhibiting systematic biases toward over-correction. Building on BEA-2019 [7] and CoNLL-2014 [8], recent studies demonstrate that LLMs can match or exceed dedicated GEC systems in low-resource settings but remain sensitive to prompt phrasing [9]. In the scientific domain, LLM-assisted reviewing has attracted attention as a mechanism for targeted review support and peer-review assistance [10], [11]. [10] report pilot experiments on short scientific papers with deliberately inserted errors and find that LLMs perform worse when asked to review holistically but improve when explicitly instructed to find errors. Complementarily, resources such as NLPeer provide broader infrastructure for the computational study of peer review and reviewing assistance [12].

This finding directly motivates our work: if the binary error/no-error framing already modulates behavior, then numerical anchors—which encode a specific expected count—may further distort model outputs in ways not yet studied.

2.2 Prompt Sensitivity and Sycophancy↩︎

It is well established that LLM outputs are sensitive to prompt phrasing, instruction ordering, and demonstration selection [13]–[15]. A particularly relevant phenomenon is sycophancy: the tendency of LLMs to agree with user-stated beliefs even when those beliefs are factually incorrect [4], [5]. [4] show that models frequently validate false user claims across a range of knowledge domains, suggesting that stated information in a prompt can override model knowledge. Similarly, studies of LLM-as-judge evaluation reveal that positional and social biases systematically distort model-generated assessments [16], further motivating the need for anchor-free evaluation protocols. Our work differs in that we study numerical rather than factual anchoring, and focus specifically on count estimation in the error detection task.

2.3 Anchoring Effects in LLMs↩︎

The anchoring effect—first described by [1]—is one of the most robust findings in cognitive psychology. The underlying mechanism, anchoring-and-adjustment [2], posits that people start from a salient reference value and adjust toward their true estimate, but typically stop adjusting too early. When people make numerical estimates, they are disproportionately influenced by this initial reference value. Recent work has begun to investigate whether LLMs exhibit similar phenomena. [3] provide a broad survey of cognitive biases in LLMs and find that anchoring-like behavior arises across diverse reasoning tasks. The Einstellung effect [17]—where prior experience with a solution strategy prevents consideration of alternatives—also has analogs in LLM reasoning. We are not aware of prior work that directly examines numerical anchoring in error detection tasks, where the anchor is the stated count of errors in a passage.

3 Methodology↩︎

3.1 Dataset Construction↩︎

We build ErrorBench from the CoNLL-2014 Shared Task dataset [8], which contains learner English essays annotated with grammatical errors in M2 format. We group consecutive sentences into passages of four sentences each, taking care not to cross document boundaries, then compute passage-level error counts from the M2 annotations. For the main analysis, passage counts and gold edits are derived from Annotator-0 only, because passage construction requires a single consistent reference across all four-sentence windows. We retain passages with 3–7 errors (inclusive) to ensure non-trivial detection tasks while maintaining manageable complexity, sampling 30 passages per error-count bucket (3, 4, 5, 6, 7) for a total of 143 passages (the error-count 7 bucket yielded 23 qualifying passages rather than 30).

Each passage stores raw text, the Annotator-0 error count, M2 error categories, and per-error gold annotations (token span, source text, correction, and sentence context) for downstream matching. These M2 labels are used descriptively on the full benchmark; a \(100\)-passage subset is additionally re-scored with the official ERRANT 3.0.0 pipeline and with Annotator-0 \(+\) Annotator-1 multi-reference max-match scoring in Appendix 14.

3.2 Prompt Conditions↩︎

We design five prompt conditions as our primary independent variable, shown in Table 1. All conditions share a fixed system prompt instructing the model to respond using a structured format (“ERROR N: [description]” followed by “TOTAL ERRORS FOUND: N”) to facilitate automated parsing. Temperature is fixed at 0 across all experiments for determinism.

For the Misleading conditions, we set \(M=N+2\) (Mislead-Over) and \(M=\max(1,N-2)\) (Mislead-Under), where \(N\) is the true error count. This ensures the anchor deviates from truth by exactly 2 in both directions, creating a symmetric experimental design.

3.3 Models↩︎

We evaluate six contemporary LLMs spanning three organizations and two generational tiers. This selection is designed to provide descriptive coverage of model families and instruction-following regimes, not to establish an overall ordering of proprietary systems: GPT-4o and GPT-5.4 [18], [19], Claude Haiku 4.5 and Claude Sonnet 4.6 [20], [21] (abbreviated as Claude H.4.5 and Claude S.4.6 in tables and figures), and Gemini 2.5 Flash and Gemini 3.1 Pro Preview [22]. All models use temperature \(=0\) and max_tokens \(=800\). We perform a single inference per sample-condition-model combination via an OpenAI-compatible API proxy, yielding \(143\times6\times5=4{,}290\) total API calls (4,290 valid).

3.4 Evaluation Metrics↩︎

ErrorBench is a controlled stress-test protocol, so metrics are interpreted as diagnostics of prompt-induced distortion under controlled prompt perturbations. We report four metrics. Count Bias (CB) is the signed difference between the model’s reported error count and the true error count: \(\mathrm{CB}=\mathrm{reported\_N}-\mathrm{true\_N}\). Anchoring Sensitivity Index (ASI) is \(|\mathrm{CB}_{\mathrm{condition}}-\mathrm{CB}_{\mathrm{Blind}}|/\mathrm{true\_N}\). Count-based Approximate F1 is derived from count overlap without span matching, using \(\mathrm{TP}=\min(\hat{n},n)\), \(\mathrm{FP}=\max(0,\hat{n}-n)\), and \(\mathrm{FN}=\max(0,n-\hat{n})\).

For span-aware scoring we adopt the CoNLL-2014 M2 scorer protocol, with one adaptation: because our prompts elicit natural-language error descriptions rather than fully corrected sentences, we extract hypothesis edits \((\mathrm{sent\_idx}, \mathrm{start}, \mathrm{end}, \mathrm{corr})\) deterministically from each description by parsing quoted "src" / "src" \(\to\) "corr" fragments and locating them in the passage via token-level substring match. Descriptions that cannot be localized are counted as false positives. We compute three corpus-level micro \(F_{0.5}\) variants: strict matches on \((\mathrm{sent\_idx}, \mathrm{start}, \mathrm{end}, \mathrm{corr})\), detection matches on \((\mathrm{sent\_idx}, \mathrm{start}, \mathrm{end})\) only, and overlap matches on same \(\mathrm{sent\_idx}\) with non-empty token overlap (standard lenience for boundary disagreement in GEC). Overlap \(F_{0.5}\) is our primary span-aware metric; strict and detection are reported in Appendix 12. For continuity with earlier drafts we also retain a simpler heuristic text-match Span-F1 as a diagnostic secondary proxy. F1 Inflation is defined as \(\Delta F = \mathrm{Count-F1} - \mathrm{M2}~F_{0.5}^{\mathrm{overlap}}\).

For reproducibility, the accompanying source package includes data-preparation, experiment, and analysis scripts together with raw JSONL outputs in the errorbench/ directory. All figures were generated from the errorbench/scripts/make_figures.py script using the passage-level records in errorbench/outputs/results.jsonl.

4 Experiments and Results↩︎

4.1 Main Results↩︎

Table 2 presents Count Bias and Count-F1 across all six models and five conditions. We treat these as descriptive model-family patterns under a controlled stress test, not as evidence about general model quality. Three findings stand out. First, GPT-5.4 follows the Mislead-Over anchor with zero variance (CB \(=+2.000\), SD \(=0\)) in every passage; Claude H.4.5 achieves the tightest Anchored compliance (CB \(\approx 0\), SD \(=0.17\)) among all models. Second, GPT-5.4 shows large unconstrained over-reporting (Blind CB \(=+7.3\)), Claude S.4.6 shows a smaller but still positive Blind bias (\(+2.8\)), and Claude H.4.5 is nearly unbiased under Blind (\(+0.2\)). Third, both Gemini models exhibit a stable undercount bias: Gemini 2.5 (CB \(\approx -2.1\) to \(-2.6\)) and Gemini 3.1 (CB \(\approx -2.7\) to \(-3.0\)) show minimal movement across conditions, suggesting that the Gemini family is comparatively resistant to numeric anchors in this protocol—with even lower ASI for the newer generation.

4.2 Anchoring Sensitivity Index↩︎

Table 3 shows ASI values across conditions and models. GPT-5.4 records the largest ASI observed in this stress test: Mislead-Under ASI \(=1.922\), meaning the model shifts by nearly twice the true error count from its blind baseline when given a deflated anchor. Claude S.4.6 shows the next-largest observed value (1.066). Both substantially exceed their within-family predecessors (GPT-4o: 0.816; Claude H.4.5: 0.463). By contrast, both Gemini models show very low ASI throughout: Gemini 2.5 remains \(\leq 0.178\) and Gemini 3.1 remains \(\leq 0.132\), with the newer generation showing even greater resistance.

4.3 Statistical Significance↩︎

Because all prompt conditions are evaluated on the same passages, we re-estimate condition effects with paired passage-level t-tests and Benjamini-Hochberg FDR correction rather than independent-samples tests. For CB, Anchored and Mislead-Under remain strongly different from Blind for GPT-4o, GPT-5.4, Claude H.4.5, and Claude S.4.6 (all \(q<0.001\)), and Mislead-Over is also significant for all non-Gemini models. The Informed condition is not uniformly null: it produces small but reliable positive CB shifts for GPT-4o (\(\Delta\)CB \(=+0.672\), \(q=0.0024\)) and GPT-5.4 (\(\Delta\)CB \(=+0.483\), \(q=0.0071\)), but remains non-significant for both Claude models and Gemini.

Paired effect sizes (Cohen’s \(d_z=\mathrm{mean}(\Delta\mathrm{CB})/\mathrm{sd}(\Delta\mathrm{CB})\)) reinforce this picture. GPT-5.4 shows large anchoring effects relative to its Blind baseline: Mislead-Under \(d_z=-2.55\) (\(\Delta\)CB \(=-8.57\)), Anchored \(d_z=-2.37\), and Mislead-Over \(d_z=-1.73\)—all well beyond conventional “very large” thresholds. Claude S.4.6 and Claude H.4.5 are moderate-to-large on the Misleading conditions (\(|d_z|\) from \(0.94\) to \(1.22\)), GPT-4o is small-to-medium (\(|d_z|\leq 0.53\)), and both Gemini models remain trivially small throughout (\(|d_z|\leq 0.34\)). The full \(\Delta\)CB and \(d_z\) table by model and condition is reported in Appendix 10.

4.4 Span-level F1 and F1 Inflation↩︎

Table 4 presents Count-F1, M2 overlap \(F_{0.5}\), and F1 Inflation for all six models under key conditions. Count-F1 is inflated by \(0.26\)–\(0.79\) across models in the Anchored condition; the effect is substantially larger than a heuristic substring score would suggest. GPT-5.4 shows the extreme case: Count-F1 improves from \(0.58\) (Blind) to \(0.99\) (Anchored) while M2 \(F_{0.5}\) moves only from \(0.20\) to \(0.20\), yielding an inflation of \(0.79\). Claude H.4.5 goes from Count-F1 \(0.86\) to \(1.00\) while M2 \(F_{0.5}\) moves \(0.43 \to 0.46\)—nearly all the apparent gain is count compliance. The Gemini family, which is largely anchor-resistant in count terms, exhibits flat M2 \(F_{0.5}\) (\(\approx 0.24\)–\(0.34\)) across conditions and a correspondingly flat inflation, consistent with its stable undercount prior. Under strict M2 matching (Appendix 12) inflation reaches up to \(0.96\), because anchor-induced count agreement carries essentially no genuine localization benefit. These results reinforce the main methodological point: count-based gains under anchoring should not be interpreted as evidence of better error localization.

4.5 Case Studies↩︎

We present three qualitative cases drawn from our experimental data; full passage text, gold annotations, and verbatim model outputs are reported in Appendix 11. Case A: fabrication under inflated anchor (GPT-4o). Passage conll_0238 contains 4 ground-truth errors. Under Blind prompting GPT-4o identifies only 1 error; under Mislead-Over (anchor \(=6\)), it reports exactly 6 errors, fabricating 5 additional ones to match the stated count. Case B: suppression under deflated anchor (Claude H.4.5). Passage conll_0210 contains 3 annotated errors. Under Blind prompting Claude H.4.5 reports 11 errors; under Mislead-Under (anchor \(=1\)), it returns only 1 error, suppressing 10 previously identified findings. Case C: resistance to anchoring (Gemini 2.5 Flash). Passage conll_0201 contains 3 errors; the Mislead-Over anchor is 5, yet Gemini reports only 2 errors, consistent with its stable undercount prior and general resistance to numerical anchoring cues.

5 Discussion↩︎

Our results provide evidence consistent with anchoring-like behavior in LLM error detection tasks. The pattern of findings is broadly compatible with [1]’s anchoring-and-adjustment account [2]: models often appear to use the stated error count as a starting point and adjust insufficiently in the face of contradictory textual evidence. A notable pattern in this dataset is that stronger instruction following does not uniformly reduce anchoring sensitivity. GPT-5.4, which shows large unconstrained over-reporting (Blind CB \(=+7.3\)), is also the model most precisely controlled by the Mislead-Over anchor (SD \(=0\)), yielding a large Anchored ASI (\(=1.62\)). Claude S.4.6 similarly records higher ASI values than the smaller Claude H.4.5, despite comparable Blind CB. We interpret this as a capability-alignment tradeoff in the present setup rather than as a universal scaling law.

The six-model comparison reveals both quantitative and qualitative differences as descriptive model-family evidence. Within the GPT and Claude families, newer generations show markedly larger anchoring responses in this protocol: GPT-5.4 Mislead-Under ASI \(=1.922\) vs.GPT-4o \(=0.816\); Claude S.4.6 ASI \(=1.066\) vs.Claude H.4.5 \(=0.463\). The Gemini family displays the opposite pattern: both Gemini 2.5 and Gemini 3.1 show very low ASI (\(\leq 0.178\) and \(\leq 0.132\) respectively), and the newer Gemini 3.1 is even more resistant than its predecessor. Gemini’s low ASI does not indicate superior detection accuracy; instead, it reflects a strong under-reporting prior (Gemini 2.5 CB \(\approx -2.4\); Gemini 3.1 CB \(\approx -2.8\)) that is difficult to move with numeric cues.

Our findings have direct implications for LLM deployment in high-stakes settings. Document review systems, code auditing tools, and medical report checkers that pre-populate error counts may systematically bias downstream LLM analysis. A full ERRANT-based replication on a \(100\)-passage subset across all six models with multi-reference (Annotators 0 \(+\) 1) scoring (Appendix 14) reproduces the core F1-Inflation pattern under the standard edit-based GEC metric, confirming that it is not an artifact of our description-anchored M2 extraction or of single-annotator evaluation.

5.1 Practical Recommendations for LLM Error-Detection Evaluation↩︎

For journal readers interested in deploying or auditing LLM-based proofreading and document-review systems, the results imply four practical safeguards. First, use blind prompts by default: evaluation prompts should not pre-populate the expected number of errors unless the purpose is explicitly to test anchoring vulnerability. Second, treat count metrics as auxiliary: Count-F1 or exact-count accuracy can diagnose coarse calibration, but they should not be reported as the sole evidence of error-detection quality. Third, report span-aware metrics: evaluations should include M2, ERRANT, localization F-score, or an equivalent metric that requires the system to identify where the error occurs, not merely how many errors are expected. Fourth, monitor exact anchor matching in deployed systems: production pipelines should track how often an LLM returns precisely the user- or system-provided count, especially when upstream tools, templates, or human reviewers expose an expected number of issues.

This study has several limitations. Our primary span-aware metric operates on hypothesis edits extracted deterministically from natural-language error descriptions rather than from model-produced corrected text, which is a lower bound on localization quality relative to full ERRANT pipelines; the main passage-level analysis uses only Annotator-0; cross-model contrasts are treated descriptively; and all experiments use temperature \(=0\). The anchor range in our Anchored and Misleading conditions is fixed at the true count and \(\pm 2\) respectively; sweeping multiple anchor magnitudes (e.g.,\(\pm 1, \pm 3, \pm 5\)) is the natural next step. Future work should also replicate findings across different error-detection domains beyond grammatical error correction.

6 Conclusion↩︎

We present a controlled study of how prompt framing, especially numeric anchors, can distort LLM error-detection evaluation. Across six contemporary models and five prompt conditions, anchored prompts consistently shift reported counts for most models, with the largest effects appearing in highly instruction-compliant GPT/Claude systems in our sample and weaker movement in the Gemini family; this is model-family descriptive evidence, not a claim about overall model quality. The central contribution of the paper is methodological: under CoNLL-2014 M2-style scoring on description-anchored hypothesis edits, count-based F1 gains under anchoring do not translate into span-level gains, and F1 Inflation reaches up to \(0.79\) under overlap matching and up to \(0.96\) under strict matching. We therefore position ErrorBench as a stress-test protocol for prompt-induced evaluation distortion rather than an overall model-quality comparison.

Code and Data Availability↩︎

Code, prompts, and analysis scripts are available at TODO-public-repository-url. The public repository should include the errorbench/ directory, raw JSONL outputs, derived summary files, and the scripts used to regenerate all figures.

Acknowledgments↩︎

TODO Add acknowledgments and funding information for the public preprint version.

Limitations↩︎

We highlight several limitations that scope our claims.

6.0.0.1 Description-anchored M2 across all models; ERRANT on a 100-passage subset.

Our primary span-aware metric across the full \(143\)-passage benchmark is CoNLL-2014 M2-style \(F_{0.5}\) (strict / detection / overlap; Appendix 12), computed on hypothesis edits extracted deterministically from the descriptions produced by each model. As a validation step, we re-ran all six models on a stratified \(100\)-passage subset with an extended prompt that also elicits a corrected-text block, and scored the resulting \((\mathrm{source}, \mathrm{correction})\) pairs with the official ERRANT 3.0.0 pipeline under both single-annotator and multi-reference (Annotators 0 \(+\) 1) scoring (Appendix 14). The qualitative pattern—large Count-F1 gains under Anchored that translate into at most small gains in span-level \(F_{0.5}\)—is reproduced on every model. Scaling this corrected-text protocol to the full \(143\)-passage benchmark remains future work. We also retain a legacy heuristic substring Span-F1 (Appendix 13) purely for continuity with the F1-comparison figures.

6.0.0.2 Instruction-compliance floor for the Gemini family.

In the ERRANT replication, Gemini 2.5 and Gemini 3.1 failed to produce a parseable CORRECTED TEXT: block on \(25\)–\(80\%\) of trials (Appendix 14). This depresses their ERRANT rows and means their absolute ERRANT \(F_{0.5}\) values should be read as a compliance-bounded lower bound, not as an accuracy comparison against the GPT and Claude families. Gemini Count-F1 is unaffected (the count is reported reliably in both the description and correction formats).

6.0.0.3 Annotator reference.

Passage construction and Count-F1 ground truth still use Annotator-0 from CoNLL-2014 only; Annotator-1 enters only in the multi-reference ERRANT Appendix 14. Our Anchored and Misleading anchors are derived from Annotator-0 counts, so a fraction of apparent over- or under-counts may be edits accepted by Annotator-1; the multi-reference ERRANT scores in Appendix 14 show that this does not change the Blind\(\to\)Anchored differential.

6.0.0.4 Single inference per cell and fixed decoding.

Each (passage, model, condition) cell corresponds to one API call at temperature \(0\) with max_tokens=\(800\). We do not estimate sampling variance, do not run self-consistency, and do not vary decoding temperature. The headline effects are large relative to any plausible single-sample noise, but the absolute numbers should not be treated as population estimates.

6.0.0.5 Limited anchor range.

The Mislead conditions fix \(|M-N|=2\) in both directions. This is sufficient for the evaluation-reliability claim—we need only one off-by-\(k\) anchor to demonstrate F1 Inflation—but it does not let us recover the full dose–response curve that anchoring-and-adjustment [2] predicts: whether compliance grows linearly, saturates, or reverses at large \(|M-N|\). Varying the anchor magnitude (\(\pm1,\pm3,\pm5\), and progressively more extreme distractors) and measuring the resulting F1-Inflation curve is the natural psychophysical follow-up, and it would multiply the API budget by roughly \(3\times\) in our setup; we flag it explicitly as out of scope for this submission.

6.0.0.6 Dataset and domain scope.

CoNLL-2014 is learner English from a specific population, and we filter to passages with \(3\)–\(7\) Annotator-0 errors. Generalization to other error-detection settings (factual checking, code review, scientific proofreading) is not established here and is left for future work.

6.0.0.7 Model-specific confounds.

For the Gemini family, low ASI coincides with a persistent undercount prior (CB \(\approx -2.4\) to \(-2.8\)). Our data cannot cleanly separate anchor-resistance from a systematic bias toward fewer reported edits. We therefore treat cross-family contrasts as descriptive stress-test evidence rather than as overall ordering claims or causal claims about alignment techniques.

6.0.0.8 API-based contemporary models.

All models were queried through an OpenAI-compatible proxy endpoint between 2025 and 2026. Provider-side changes (system prompts, safety filters, routing) can alter behavior over time, and exact reproducibility depends on the model snapshots listed in Appendix 8.

Ethics Statement↩︎

We use CoNLL-2014, a publicly released shared-task dataset of learner English essays, in accordance with its original terms. It contains no personally identifying information as distributed. Our study does not collect human subject data and does not label individual writers. Model outputs in the Case Studies are produced on publicly released passages and contain no private content. The study reports a failure mode—prompt-induced count distortion—that, if ignored, could inflate reported LLM performance in high-stakes proofreading deployments. We therefore recommend that practitioners report span-aware metrics alongside count-based metrics and avoid prepopulating anchor counts in user-facing pipelines. No deployment of the framework itself is proposed in this paper.

7 Prompt Templates↩︎

All experiments share the system prompt in Figure 7 and vary only the user prompt by condition (Figure 8). The placeholder {text} is the four-sentence passage and \(N\) is the Annotator-0 error count. Prompts are applied verbatim; no additional post-processing is performed before the API call.

8 Model Endpoints and Decoding Settings↩︎

All queries go through an OpenAI-compatible proxy. Endpoints and model identifiers used in the experiment are listed in Table 5. Decoding settings are identical across conditions: temperature \(=0\), max_tokens \(=800\) (max_completion_tokens for GPT-5.4), single sample per cell, with exponential backoff on HTTP 429. The total number of API calls is \(143 \times 6 \times 5 = 4{,}290\).

9 Reproducibility↩︎

Data preparation (prepare_data.py), experiment runner (run_experiment.py), analysis (analyze_results.py), and figure generation (make_figures.py) scripts are provided in the errorbench/ directory alongside the raw per-call JSONL (outputs/results.jsonl) and derived summary CSVs. Passage sampling uses a fixed random seed of \(42\). Four-sentence windows are restricted to within-document boundaries so that no passage spans two CoNLL-2014 source documents.

10 Paired Effect Sizes for Count Bias↩︎

Table 6 reports paired per-passage shifts relative to the Blind baseline. For each (model, condition), we compute \(\Delta\mathrm{CB}_i = \mathrm{CB}_i^{\mathrm{cond}} - \mathrm{CB}_i^{\mathrm{Blind}}\) across all passages \(i\) with both observations, and summarise with mean, standard deviation, and Cohen’s \(d_z = \mathrm{mean}(\Delta\mathrm{CB}) / \mathrm{sd}(\Delta\mathrm{CB})\). Sample sizes are \(n=143\) for all non-Gemini cells; \(n \in [132,135]\) for Gemini 3.1 (a handful of API responses did not parse to a valid count).

11 Case Studies: Full Passages and Model Outputs↩︎

We report the full source passages and verbatim model outputs for the three cases discussed in §4.5. Gold annotations are from CoNLL-2014 Annotator-0.

11.0.0.1 Case A: fabrication under inflated anchor.

Passage conll_0238, \(N=4\). GPT-4o reports \(1\) error under Blind and exactly \(6\) errors under Mislead-Over (anchor \(=6\)), inventing five low-salience or non-errors (spacing around punctuation, an article choice that is standard in the original) to fill the quota.

11.0.0.2 Case B: suppression under deflated anchor.

Passage conll_0210, \(N=3\). Claude H.4.5 reports \(11\) errors under Blind—mostly punctuation concerns outside the Annotator-0 gold set—and collapses to a single report under Mislead-Under (anchor \(=1\)), retaining only the most salient “the their” error.

11.0.0.3 Case C: resistance to anchoring.

Passage conll_0201, \(N=3\). Under Mislead-Over (anchor \(=5\)), Gemini 2.5 Flash reports only \(2\) errors, below both the anchor and the true count—consistent with its persistent undercount prior rather than compliance with the prompted count.

12 Full M2 Scoring Tables↩︎

Table 10 reports the three M2 \(F_{0.5}\) variants introduced in §3.4 for all 30 (model, condition) cells. Strict matches on \((\mathrm{sent\_idx}, \mathrm{start}, \mathrm{end}, \mathrm{corr})\); detection on \((\mathrm{sent\_idx}, \mathrm{start}, \mathrm{end})\); overlap on same sentence with non-empty token overlap. Of the \(21{,}172\) total model error descriptions that parse into our pipeline, \(16{,}552\) (\(78.2\%\)) are deterministically localized to a passage span; the remaining \(21.8\%\) are conservatively counted as false positives. Localization failures are dominated by descriptions without quoted fragments (e.g., “Comma misuse, space before comma”), which cannot be grounded to a specific token span.

3pt

13 Legacy Heuristic Span-F1↩︎

Table 11 reports the heuristic case-insensitive substring-match Span-F1 used in earlier drafts (and still plotted in Figures 4–6 for continuity). This metric counts a gold edit as matched whenever any word from its src/corr fields appears as a substring of any model description, with no span localization. It is substantially more permissive than M2 overlap and is retained only for backward compatibility.

3pt

14 ERRANT Full Replication (Six Models, Multi-Reference)↩︎

To validate that the description-anchored M2 metric used in the main results is not an artifact of hypothesis-edit extraction, we re-ran all six contemporary models on a stratified \(100\)-passage subset (\(20\) passages per Annotator-0 error-count bucket in \(\{3,\dots,7\}\), seed \(=42\)) with an extended prompt that requests both the original ERROR N: description list and a CORRECTED TEXT: block containing the full passage with all errors fixed. This produces \(2{,}999/3{,}000\) usable outputs (6 models \(\times\) 5 conditions \(\times\) 100 passages; one Gemini 3.1 call returned no content and is dropped). Each corrected passage is tokenized and sentence-aligned to the source, and edits are extracted and scored with the standard ERRANT 3.0.0 pipeline against gold edits produced through the same ERRANT tokenizer. For multi-reference scoring we use the official CoNLL-2014 Annotator-0 and Annotator-1 gold edits from the shared-task M2 release [8] and take the sentence-level max-match over the two references. All metrics are corpus-level micro-averages.

14.0.0.1 Compliance caveat.

GPT-4o, GPT-5.4, Claude H.4.5, and Claude S.4.6 produce a parseable CORRECTED TEXT: block on \(\geq 499/500\) trials. Gemini 2.5 and Gemini 3.1 fail on \(25\)–\(80\%\) of trials depending on condition (column noC in Table 12). Empty outputs contribute zero predicted edits and therefore depress ERRANT \(F_{0.5}\) of those rows; the Gemini rows should be read with this compliance floor in mind. No Gemini row is used to support the core F1-Inflation claim.

14.0.0.2 Result.

The qualitative pattern from Table 4 is reproduced across every model. The Anchored condition drives Count-F1 to \(\geq 0.93\) for all six models (mean \(0.975\), vs.\(0.763\) in Blind, \(\Delta = +0.212\)), while Annotator-0 ERRANT \(F_{0.5}\) under Anchored is within \(\pm 0.10\) of the corresponding Blind baseline in five of six comparisons (Gemini 3.1 is the exception, and its Anchored gain is driven by a \(24\)-point reduction in the noC compliance gap rather than by improved span quality). Averaged over the six models, Blind\(\to\)Anchored lifts Count-F1 by \(+0.212\) while lifting ERRANT \(F_{0.5}\) by only \(+0.047\) (Annotator-0) or \(+0.038\) (multi-reference)—a \(4\)–\(5\times\) gap that mirrors the description-anchored M2 result on the full \(143\)-passage benchmark. Adding Annotator-1 raises absolute ERRANT \(F_{0.5}\) by \(0.12\)–\(0.16\) on average but leaves the Blind\(\to\)Anchored differential essentially unchanged, confirming that the F1-Inflation signal is not driven by single-annotator idiosyncrasies.

14.0.0.3 Statistical significance (paired bootstrap).

For each model we resample the 100 passages with replacement (\(1{,}000\) bootstrap iterations, seed \(=42\), paired across Blind and Anchored) and recompute Count-F1 and multi-reference ERRANT \(F_{0.5}\). Figure 9 plots the mean and \(95\%\) CIs for \(\Delta\)Count-F1 and \(\Delta\)ERRANT \(F_{0.5}\) (Anchored \(-\) Blind) per model. Every \(\Delta\)Count-F1 CI is strictly positive; four of six \(\Delta\)ERRANT CIs cover or cross zero (Claude H.4.5, Claude S.4.6, GPT-4o, Gemini 2.5). The F1-Inflation contrast \(\Delta\mathrm{Count}\text{-F1}-\Delta\mathrm{ERRANT}_{\mathrm{M}}\) is strictly positive at \(95\%\) for five of six models: GPT-5.4 \(+0.359\;[+0.304,+0.416]\), GPT-4o \(+0.236\;[+0.113,+0.404]\), Claude S.4.6 \(+0.138\;[+0.083,+0.187]\), Gemini 2.5 \(+0.136\;[+0.069,+0.210]\), Claude H.4.5 \(+0.118\;[+0.071,+0.163]\). Gemini 3.1 is the only borderline case (\(+0.074\;[-0.015,+0.154]\)), and as noted above its ERRANT row is compliance-confounded.

3pt

14.0.0.4 Context for absolute magnitudes.

Our passage-level multi-reference ERRANT \(F_{0.5}\) values of \(0.40\)–\(0.57\) are broadly consistent with previously reported LLM-based GEC performance on CoNLL-2014 [6], while supervised GEC systems on the sentence-level CoNLL-2014 test set report ERRANT \(F_{0.5}\) in the \(0.55\)–\(0.69\) range [9]. The absolute numbers are not the point: the central observation of this paper is that Count-F1 is driven to \(\geq 0.93\) for every model under Anchored prompting while multi-reference ERRANT \(F_{0.5}\) stays within \(\pm 0.06\) of the model’s own Blind baseline in five of six cases, so Count-F1 cannot be trusted as a proxy for span-level capability under anchor-bearing prompts, regardless of where that model sits on any absolute GEC evaluation table. This is why ErrorBench should be used as a stress-test protocol for evaluation design rather than as an overall model-quality comparison.

References↩︎