1 Introduction↩︎

AlphaFold [1] and the AlphaFold Protein Structure Database (AFDB) [2], [3] have changed the bottleneck in structural biology. For a large fraction of catalogued proteins, a predicted 3D structure is now available, and AFDB provides open access to more than \(200\) million predicted structures. The remaining challenge is not only to obtain structures, but to ask precise questions of them: which region is low-confidence, how far apart are two residues, whether a window is both high-confidence and contact-rich, or whether two domains have high predicted aligned error. These are not merely textual questions. They are measurements over a molecular object.

This creates a mismatch for evaluating large language models (LLMs). If a model is asked, “How many Angstroms separate the alpha-carbons of residues 207 and 220 in the AlphaFold structure?”, a fluent answer is not enough; the answer should be correct only if it matches the value computed from the predicted coordinates. String-overlap metrics and LLM judges can assess whether a response resembles a reference, but they cannot by themselves verify that a number, residue set, region, or structural label was derived from the protein structure.

We treat protein structural QA as denotational semantics: a natural-language question corresponds to a hidden typed program, and its meaning is the value obtained by executing that program on a protein structure. For example, the question above denotes distance(residue(207), residue(220)); running this program on the AlphaFold prediction for human protein A6ND36 yields the gold answer (\(16.25\) Å). Under this view, evaluation becomes deterministic: a model is correct only when its prediction matches the structure-derived denotation.

We introduce ProtStructQA, a large-scale executable benchmark for residue-level structural QA over AlphaFold-predicted proteins. ProtStructQA releases \(382.2\)K questions: a \(330\)K active benchmark across seven structural families and a \(52.2\)K hard-negative robustness pool. The benchmark covers \(10\)K proteins from human, mouse, fruit fly, and chicken, and asks about pLDDT confidence, \(C_\alpha\) distances, PAE, solvent exposure, secondary structure, contacts, topology, and held-out compositions of these primitives. Each question is generated from a hidden DSL program and evaluated by deterministic execution rather than text overlap.

This setup lets us ask a controlled inference-time question: where is the structural denotation computed? Direct prompting compiles in one step; chain-of-thought plans in language; grammar-constrained executable voting samples and executes candidate programs; ReAct-style tool use externalizes structural access through multi-turn tool calls. A shared executor isolates whether failures arise from language planning, program syntax, structural computation, or answer formatting.

Across Qwen3 models from \(0.6\)B to \(8\)B, we observe a capability-dependent denotation threshold between Qwen3-\(1.7\)B and Qwen3-\(4\)B, replicated on Gemma-3-\(1\)B and Gemma-3-\(12\)B. Below the threshold, small models often cannot produce executable denotations, and multi-turn tool use dominates. Above the threshold, chain-of-thought flips from mostly harmful to strongly beneficial, and free-form CoT or executed CoT wins nearly all evaluation cells; the transition is mechanistically a shift from unparseable to valid structural denotations.

Our contributions are:

• Executable structural QA. We introduce ProtStructQA, a large-scale benchmark in which each natural-language protein question has a hidden typed DSL program and a gold answer obtained by deterministic execution on an AlphaFold-predicted structure.

• Typed denotational semantics for protein questions. We formalize structural QA as compilation from language to a measurement algebra over residues, regions, residue pairs, confidence, PAE, solvent exposure, secondary structure, and contacts.

• Controlled inference-time evaluation. We compare direct prompting, chain-of-thought, grammar-constrained executable voting, executable voting with CoT, and ReAct-style tool use without fine-tuning, across Qwen3 (\(0.6\)B–\(8\)B) and a Gemma-3 cross-family replication (\(1\)B and \(12\)B).

• The denotation threshold. We show a scale-dependent crossover: sub-threshold models benefit most from tool-mediated denotation, while supra-threshold models use chain-of-thought as an internal compiler from language to structural measurement.

2 Related Work↩︎

2.0.0.1 Protein representation and foundation-model benchmarks.

TAPE [4], PEER [5], and ProteinGym [6] collectively evaluate protein representation, function/localization/interaction prediction, and mutation-effect tasks. These suites are essential for measuring protein foundation models, but they are not designed to evaluate natural-language questions whose answers must be computed from residue-level 3D structural measurements. ProtStructQA is complementary: it evaluates whether a language model can map a surface question to an executable operation over an AlphaFold-predicted structure.

2.0.0.2 Protein-language QA and chat systems.

Recent work has begun to connect proteins and natural language through QA datasets, instruction tuning, protein-to-text generation, and multimodal protein chat. PQA/Pika [7] introduces a curated protein QA benchmark for free-form enquiry, contrasting with earlier PDB-QA datasets of predefined questions over PDB entries; ProtT3 [8] generates text descriptions from protein-sequence inputs; Mol-Instructions [9] and InstructProtein [10] align biomolecular knowledge with natural-language supervision; and ProtChatGPT [11], ProteinGPT [12], and Prot2Chat [13] let users query protein sequence and/or structure through LLM interfaces. These efforts evaluate generation with lexical, semantic, or LLM-judge metrics, or task-specific QA scoring. ProtStructQA differs in its evaluation target: each question has a hidden residue-level DSL program and a deterministic answer produced by execution over predicted 3D coordinates.

2.0.0.3 Executable and compositional reasoning benchmarks.

ProtStructQA inherits from a tradition of compositional reasoning benchmarks: SCAN [14] (grammar-based compositional generalization), CFQ [15] (executable SPARQL queries), CLEVR [16] (functional programs over synthetic visual scenes), and GQA [17] (scene-graph question generation with functional programs). ProtStructQA imports this denotational idea into a scientific domain: the “scene” is an AlphaFold-predicted protein, the objects are residues and regions, and the answer is a structural measurement rather than a visual attribute or database fact.

2.0.0.4 Structured decoding, programs, and agents.

Grammar-constrained decoding enforces output structure at the token level [18]–[21], and JSONSchemaBench [22] benchmarks syntactic validity at scale. Program-aided methods such as Program-of-Thought [23] and PAL [24] separate reasoning from computation; self-consistency [25] aggregates multiple reasoning traces. ReAct [26] interleaves reasoning with tool use. ProtStructQA uses these methods not merely to test format compliance, but to ask when valid structure becomes correct scientific denotation: a syntactically valid DSL program is useful only when its execution matches the structural quantity requested by the question.

2.0.0.5 Positioning.

In summary, ProtStructQA is not simply another protein QA dataset or another constrained-decoding benchmark. It combines protein structural data, typed executable semantics, and controlled inference-time reasoning to evaluate whether language models can compile natural-language questions into measurements over predicted molecular structure. This intersection is the paper’s contribution.

3 The ProtStructQA Benchmark↩︎

ProtStructQA is built around four objects: a predicted structural world, a natural-language question, a hidden typed denotation, and a deterministic oracle. The world is an AlphaFold-predicted protein structure; the question is a paraphrase of a hidden template; the denotation is a DSL program; and the oracle is the executor that computes the answer from coordinates, confidence scores, PAE, solvent exposure, and secondary-structure annotations. Figure 2 summarizes the construction pipeline. In plain terms, each question is generated from a hidden program, and the correct answer is the value of running that program on the protein’s predicted structure.

3.1 Protein Panel↩︎

We sample \(10{,}000\) AlphaFold-predicted protein structures across four UniProt reference proteomes [27]: \(4{,}000\) human, \(2{,}500\) mouse, \(1{,}500\) fruit fly, and \(2{,}000\) chicken (Table 2, App. 10). Human serves as the in-distribution anchor, while the other species define species-stratified cross-proteome shifts: mouse provides a close mammalian comparison, chicken a non-mammalian vertebrate comparison, and fruit fly an invertebrate comparison. The species selection is intended to create controlled proteome-level distribution shifts, not to prove broad phylogenetic generalization. Sampling is length-stratified within species (final-panel lengths range from \(16\) to \(2{,}321\) amino acids), and predicted structures come from AFDB v6 [3], produced by AlphaFold [1] and released under CC BY 4.0.

For each protein, we extract the \(C_\alpha\) trace, per-residue pLDDT, the PAE matrix, secondary structure assigned by DSSP and collapsed to helix/strand/coil [28], and relative solvent-accessible surface area (SASA). These features define the executor’s structural state, and the benchmark therefore evaluates answers relative to AlphaFold-predicted structures and the per-residue annotations derived from them, not experimental structural ground truth. Figure 1 shows two example questions; the per-species protein counts (Table 2) and the panel’s length, pLDDT, and secondary-structure composition (Figure 4) are reported in App. 10.

3.2 Formal denotational setup↩︎

Informally, each question is paired with a hidden program; running the program on a protein returns a number, a yes/no, a residue, a region, a residue set, a residue-pair set, or a secondary-structure label. The formal setup below names the pieces of this pipeline so later sections can reference them precisely.

Let protein \(p\) have structural state \[S_p = (X_p,\, c_p,\, A_p,\, \sigma_p,\, \rho_p),\] where \(X_p \in \mathbb{R}^{n_p \times 3}\) are \(C_\alpha\) coordinates, \(c_p \in [0, 100]^{n_p}\) are pLDDT scores, \(A_p \in \mathbb{R}_{\ge 0}^{n_p \times n_p}\) is the PAE matrix, \(\sigma_p \in \{\mathtt{H},\mathtt{E},\mathtt{C}\}^{n_p}\) gives per-residue secondary structure, and \(\rho_p \in [0, 1]^{n_p}\) is relative solvent accessibility. Each benchmark example is a tuple \[e_i = (p_i,\, q_i,\, z_i,\, y_i,\, \tau_i),\] where \(q_i\) is the natural-language question, \(z_i \in \mathcal{G}\) is the hidden DSL program (§3.3), \(\tau_i\) is the answer type, and \[y_i \;=\; E(z_i,\, S_{p_i})\] is the gold answer obtained by deterministic execution. Multiple paraphrases of the same template-parameter assignment map to the same hidden program and the same denotation, while semantically similar questions can have different programs. A prediction \(\hat{y}\) is correct only if it matches \(y_i\) under the type-specific metric defined in §5.

3.3 DSL for structural reasoning↩︎

The DSL is intentionally small: it is not a full language for structural biology, but a closed measurement algebra over residues, regions, residue pairs, and proteins, covering common structural operations while keeping every question executable and type-checkable. The DSL operates over typed values Residue, Region, ResidueSet, PairSet, Bool, Int, Float, and SecStruct; typed execution rejects ill-formed programs such as averaging a residue pair or comparing a secondary-structure label to a distance threshold. Regions are \(1\)-indexed inclusive (range(s, e) covers \(e{-}s{+}1\) residues). Primitives cover per-residue properties (pLDDT, DSSP label, relative SASA, neighbor count), pairwise properties (\(C_\alpha\) distance and PAE, with optional sequence-separation constraints via min_sep), and region- or protein-level aggregates (mean pLDDT, contact density, radius of gyration, helix counts). They compose through comparison and Boolean logic with exists, count, filter, argmin, argmax, sliding windows, and all-pairs quantification, producing both atomic measurements and held-out compositions (e.g., whether any \(40\)-residue window is simultaneously high-confidence and contact-rich). Several primitives correspond to closed-form structural quantities: \[\begin{align} \operatorname{distance}(i, j) &= \lVert X_i - X_j \rVert_2, \\ \operatorname{mean\_plddt}(R) &= \tfrac{1}{|R|}\sum_{i \in R} c_i, \\ \operatorname{mean\_pae}(R_1, R_2) &= \tfrac{1}{|R_1||R_2|}\!\!\sum_{i \in R_1, j \in R_2}\!\! A_{ij}, \\ \operatorname{contact\_density}(R) &= \tfrac{1}{\binom{|R|}{2}} \!\!\sum_{\substack{i, j \in R \\ i < j}}\!\! \mathbf{1}\bigl\{\lVert X_i - X_j \rVert_2 < 8\,\text{\AA}\bigr\}. \end{align}\] The full \(30\)-primitive vocabulary, the formal grammar, and the program pattern of every one of the \(31\) templates are listed in App. 16.

3.4 Question Templates↩︎

Questions span \(7\) families and \(31\) templates (per-family and per-template counts in App. 15). Families A–F cover atomic structural operations (confidence, distance, PAE, solvent exposure/packing, secondary structure, contacts/topology).

Each template is rendered through multiple natural-language paraphrases (median \(29\) per A–F template, \(5\)–\(6\) per G template) spanning terse, interrogative, imperative, and structural-biology wordings. A paraphrase changes the wording but not the denotation: for a fixed template and parameter assignment, all paraphrases map to the same hidden program and answer. The paraphrase pool was generated by an LLM (Claude Opus 4.7 [29]) and then author-verified to remove paraphrases that revealed the answer and to make sure every paraphrase kept the same slot names as its template; consistency between paraphrase and program is then validated on a 3,500-question stratified sample (§3.7). Family G is not new biology; it is new composition. This is the closest analogue to compositional splits in SCAN, CFQ, CLEVR, and GQA, grounded in protein structural measurements (App. 10, Figure 5).

3.5 Hard Negatives↩︎

Hard negatives are semantic stress tests in structural space. They preserve much of the linguistic surface while changing the denotation or crossing a decision boundary, so that template priors alone are insufficient. They test whether a model binds the question to the measured structural property, rather than exploiting wording-only heuristics.

We construct hard negatives by perturbing template parameters (residue indices, region endpoints, or numeric thresholds) so that the resulting question is close to an original but has a different executed answer (two construction types HN1/HN2 with per-class breakdown in App. 12). We release a \(52.2\)K hard-negative robustness pool and evaluate a stratified \(4{,}357\)-question hard-negative split.

3.6 Generalization axes↩︎

ProtStructQA evaluates four generalization axes (Table 5). The in-distribution split (\(12{,}000\) questions) tests new proteins and parameter values under seen template families. The compositional split (\(6{,}000\) questions, Family G) tests held-out compositions of known operators. The cross-species split (\(10{,}000\) questions) tests species-stratified cross-proteome shift on mouse, fly, and chicken proteins. The hard-negative split (\(4{,}357\) questions, §3.5) tests semantic robustness under near-surface-preserving structural perturbations.

Within the human in-distribution pool, we partition proteins \(80\%\) training / \(10\%\) development / \(10\%\) test, with no protein appearing in more than one split. Family G is held out of the exemplar pool. We do not fine-tune model weights on any split. The paper evaluates models on a \(32{,}357\)-question stratified subsample of the four evaluation tracks (§[sec:benchmark:subsample]). We use “cross-species” and “cross-proteome” descriptively. The split measures robustness to proteome-level distribution shift across our four selected species, not general biological extrapolation to arbitrary species, protein families, or experimental conditions.

3.7 Validation↩︎

The main risk in executable benchmark construction is that errors in program generation, parsing, or structural preprocessing become false gold labels. We validate ProtStructQA as a trust stack: schema and type gates, deterministic execution, paraphrase-program consistency, cross-tool agreement, and distributional checks on subsampled splits.

First, every generated question passes a six-step sanity gate (file format, single-split membership, cross-species questions on non-human proteins, both HN classes labeled correctly, gold program re-execution, and a question-only baseline confirming that the wording alone is insufficient). Repeated execution of the same program on the same structural state always produces the same answer.

Second, we verify that paraphrases recover the intended program parameters: from each of the seven families we sample \(500\) questions (\(3{,}500\) total) and confirm that every numeric literal in the gold program appears in the natural-language question and that the question contains at least one family-specific keyword. All \(3{,}500\) pass.

Third, we recompute representative structural primitives with independent toolchains: BioPython [30] for distances, pLDDT, and Shrake–Rupley SASA; raw AFDB JSON for PAE; biotite P-SEA [31], [32] for secondary structure. All agree on evaluated templates (App. 13).

Finally, for the three subsampled evaluation splits, stratified sampling preserves relevant marginals: KL divergence is at most \(0.0037\) for every tracked category (family, species, template, answer type), so subsample trends transfer to the full pool within statistical noise (App. 14).

4 Baseline Suite↩︎

We evaluate five inference-time baselines that differ in where the structural denotation is computed. Direct prompting asks the model to compile the question into an answer in one step. Chain-of-thought lets the model plan in natural language before emitting an answer. Executable voting samples grammar-valid DSL programs, executes them, and aggregates the executed answers under a type-aware loss. EV+CoT combines language planning with executable voting. ReAct-style tool use externalizes structural computation through a multi-turn tool interface. All are inference-time only (no fine-tuning) and share the same DSL output schema; they differ only in decoding strategy (§5). Iterative refinement and deliberate-search methods [33]–[35] are out of scope.

• Standard [36] (direct compiler): few-shot direct prompting at \(T{=}0\); one DSL emission per question, compiling \(q\) into a typed program in a single forward pass.

• CoT [37] (internal compiler with language planning): prepends a 4-step task-decomposition checklist (answer-type \(\to\) primitive \(\to\) scope \(\to\) thresholds); same sampling and exemplars as Standard. Vanilla zero-shot [38] and PAL-style variants [23], [24] are evaluated as prompt-robustness checks (App. 17).

• EV [23]–[25] (grammar-constrained executable voting): the program-form analogue of self-consistency. We sample \(k{=}3\) grammar-constrained programs \(\hat{z}_1, \ldots, \hat{z}_K\) from \(p_\theta(\cdot \mid q,\, z \in \mathcal{G})\) at \(T{=}0.7\), top-\(p{=}0.95\) via LLGuidance [39], execute each to obtain candidate denotations \(\hat{y}_k = E(\hat{z}_k,\, S_p)\), and aggregate by type-specific minimum-risk decoding [40]: \[\hat{y} \;=\; \arg\min_{u \in \{\hat{y}_1, \ldots, \hat{y}_K\}} \sum_{k=1}^{K} \ell_\tau(u,\, \hat{y}_k),\] where the type-specific loss is \[{!}{ \ell_\tau(u, v) = \begin{cases} \mathbf{1}[u \ne v] & \tau \in \{\text{Bool}, \text{Int}, \text{SecStruct}\}, \\ |u - v| & \tau = \text{Float}, \\ 1 - \mathrm{IoU}(u, v) & \tau \in \{\text{Region}, \text{ResidueSet}, \text{PairSet}\}. \end{cases} }\] These reduce to majority vote, sample median, and IoU-medoid respectively. Thus EV is not only self-consistency over text; it is voting in the executor’s answer space.

• EV+CoT (language planning combined with executable voting): EV with the CoT prefix applied per sample, the self-consistency \(+\) CoT combination [25] on executed programs rather than free-form text.

• ReAct [26] (tool-mediated denotation prosthesis): multi-turn agent interleaving reasoning, tool actions, and observations (<think>, <act>, <obs>, <answer> delimiters). Five DSL tools (point-residue lookup, residue-pair distance, mean PAE over a region, region summary stats, and raw DSL execution), four primer examples, \(\mathrm{\small max\_turns}{=}8\). Tool calls externalize structural access; the model need not internally compile the full question into a single expression. The final <answer> is free-form text (analyzed in §6.4).

5 Experimental Setup↩︎

Models and inference. We evaluate Qwen3-0.6B, Qwen3-1.7B, Qwen3-4B, and Qwen3-8B [41] via vLLM [42] v0.16.0 on a single H100-80GB in bfloat16. All baselines receive the same compact protein summary (length, mean pLDDT, helix and strand counts, secondary-structure bands, pLDDT bands) and four train-pool few-shot exemplars per question (families A–F, target template excluded); this design isolates inference-time behavior rather than supervised adaptation to ProtStructQA. EV and EV+CoT: temperature 0.7, top-\(p\) 0.95, \(k{=}3\), max_tokens=192, grammar enforcement via LLGuidance [39]. Standard and CoT: temperature 0, max_tokens=384. ReAct: temperature 0 (greedy, following [26]), max_tokens=384/turn, \(\mathrm{\small max\_turns}=8\). Overall GPU usage for the experiments is approximately \(150\) hours on H100.

Metrics. A prediction is scored under a type-specific tolerance, \[{!}{ M_\tau(\hat{y}, y) = \begin{cases} \mathbf{1}\bigl[|\hat{y} - y| \le \max(0.5,\;0.05 \cdot \max(|y|,|\hat{y}|))\bigr] & \tau = \text{Float}, \\ \mathbf{1}\bigl[|\hat{y} - y| \le \max(2, 0.10|y|)\bigr] & \tau = \text{Int}, \\ \mathbf{1}[\hat{y} = y] & \tau \in \{\text{Bool}, \text{SecStruct}, \text{Region}\}, \\ \mathbf{1}\bigl[\mathrm{IoU}(\hat{y}, y) \ge 0.9\bigr] & \tau \in \{\text{ResidueSet}, \text{PairSet}\}, \end{cases} }\] applied in each Float quantity’s native unit (Å for distance/PAE, pLDDT points for confidence, unitless for SASA fraction and contact density). Marginal accuracy is the mean, \(\operatorname{Acc} = \tfrac{1}{N}\sum_{i=1}^{N} M_{\tau_i}(\hat{y}_i, y_i)\). Following [43] we report bootstrap \(95\%\) CIs (1,000 resamples [44]; App. 19) on marginal accuracies and McNemar’s test [45] on matched-question pairs, with Bonferroni correction tied to the number of planned comparisons; per-cell statistical significance for Qwen3 is reported in App. 18 (Tables 9 and 10), and prompt-form robustness in App. 17. EV and EV+CoT run at three seeds; Standard, CoT, and ReAct are reported single-seed (seed 0).

6 Results: The Denotation Threshold↩︎

6.1 A Denotation Threshold Between 1.7B and 4B↩︎

Table 1 reports the full ablation: 5 methods, 4 splits, 4 Qwen3 scales. On the eight sub-threshold cells (0.6B and 1.7B), ReAct reaches the highest compositional accuracy (\(28.7\)–\(31.1\) pp above the next-best method) and dominates on every other split. At 4B and above the picture inverts: free-form CoT wins six of eight cells outright, and grammar-constrained EV+CoT wins the remaining two (both at 8B on in-distribution and cross-species). The threshold sits between 1.7B and 4B.

The species-stratified cross-proteome shift (mammal, avian, invertebrate) does not flip any winner relative to in-distribution, and hard-negative changes non-agentic accuracy by less than three points uniformly (some splits up, some down). The denotation threshold therefore reflects model capability, not a quirk of the human-only test pool.

3pt

6.2 The Threshold is a Parseability Transition↩︎

The denotation threshold (Figure 3) is explained by a change in failure mode. Below threshold, small models fail before semantics: they emit malformed programs, invalid answer types, or free-form text that cannot be parsed into the required denotation. Above threshold, parseability improves sharply, and remaining errors are more often semantic or aggregation errors.

The threshold becomes mechanistic when we decompose accuracy as \[P(\mathrm{correct}) \;=\; P(\mathrm{valid}) \cdot P(\mathrm{correct}\mid\mathrm{valid}).\] \(P(\mathrm{valid})\) is the rate at which the model produces a parseable typed output, and \(P(\mathrm{correct}\mid\mathrm{valid})\) the rate at which a parseable output matches the gold. Below the threshold, models fail mainly on \(P(\mathrm{valid})\); above, the remaining headroom is in \(P(\mathrm{correct}\mid\mathrm{valid})\), which is what chain-of-thought improves.

Per-cell parse-failure rates (App. 20) make this concrete: between Qwen3-1.7B and Qwen3-4B, Standard and CoT parse-failure both drop sharply, and the pattern replicates between Gemma-3-1B and Gemma-3-12B. At Qwen3-8B compositional, grammar-constrained EV still has higher parse-failure than free-form CoT.

The result is therefore not “CoT always helps” or “grammar always helps.” Each method changes a different term in the decomposition: tools and grammar improve \(P(\mathrm{valid})\); CoT improves semantic planning only after the model can maintain a valid output channel; and constrained sampling can hurt long compositional programs when syntactic validity narrows the generation path too aggressively.

6.3 Grammar is Selectively Valuable↩︎

Family-level results (Table 3, App. 11) show that grammar and execution remain valuable even above the threshold, but not uniformly. Family C (PAE region-pair aggregation) and Family E (secondary-structure labels) continue to favor EV+CoT over free-form CoT at 8B because the correct program must preserve exact region boundaries, pair orientation, or discrete structural labels; deterministic execution prevents the arithmetic and formatting errors that free-form CoT can introduce.

In contrast, families that reduce to scalar lookup or arithmetic over AlphaFold-derived quantities (Families A (confidence), B (distance), D (solvent exposure), F (contact topology)) favor free-form CoT at 4B and 8B. Once a model can identify the correct primitive and scope, strict program generation becomes less necessary and may introduce sampling failure.

The compositional split is the clearest stress test. CoT succeeds when the model can reason through the operator composition in text and emit the final value, while EV+CoT can fail because generating a long nested grammar-valid program is itself difficult. Failure under EV+CoT in this regime does not mean the DSL cannot express the answer; it means the model struggles to generate the correct executable expression under constraints.

6.4 Agents as Denotation Prosthesis for Sub-Threshold Models↩︎

ReAct dominates at 0.6B and 1.7B because it supplies a denotation prosthesis: instead of requiring the model to internally compile the full question into an executable expression, tools expose local structural operations and raw DSL execution. For sub-threshold models, this externalises the part of the task they are least able to perform. At 4B and 8B, ReAct loses this advantage: once models can internally compile most questions, multi-turn tool use introduces over-deliberation, formatting errors, and lost turn budget.

ReAct’s compositional accuracy is highest at the sub-threshold scales, drops at \(4\)B, and partially recovers at \(8\)B. The 8B agent runs significantly longer and over-constructs candidate programs that fail to execute, so additional turns reduce rather than improve accuracy. We describe this non-monotonic trajectory cautiously: the evidence supports a minimum at 4B and a mechanism involving over-deliberation, but not a precise functional form across model scale. The 8B failure is sub-family-localized, concentrated on G1 and G3 (full decomposition in App. 10).

6.5 Replication on Gemma-3↩︎

The same flip replicates on Gemma-3-1B/12B [46] (ReAct wins at \(1\)B, CoT wins at \(12\)B), confirming the threshold is not Qwen3-specific (App. 21, Table 13).

7 Conclusion↩︎

Scientific QA should not stop at plausible answers. In structural biology, the meaning of a question is a measurement over a molecular object. ProtStructQA makes that measurement executable by pairing natural-language questions with hidden typed programs and deterministic answers over AlphaFold-predicted structures. This executable view reveals a denotation threshold: small models need tools to reach the structure, while larger models can use chain-of-thought as an internal compiler from words to structural measurements. By reframing protein QA as compilation from language to denotation, ProtStructQA provides both a benchmark and a diagnostic lens for scientific language grounding in the AlphaFold era. ProtStructQA highlights executable grounding as a useful direction for evaluating scientific QA beyond plausible answer generation.

8 Limitations↩︎

ProtStructQA tests structural reasoning under generated natural-language questions, not the full distribution of questions asked by practicing structural biologists. Although templates have multiple paraphrases and family-specific lexical variation, the language remains controlled. Families A–F share template identity between train and test (different proteins and parameter values), so the in-distribution split measures parameter-OOD, not template-OOD; Family G provides the only template-OOD test. Broader human-authored language generalization is left to future work.

Gold answers are correct relative to AlphaFold-predicted structures and the per-residue annotations derived from them, not relative to experimental crystal structures or dynamic biology. This is intentional: the benchmark evaluates whether models can query a specified predicted structural object, and low-confidence regions are kept to preserve a realistic uncertainty signal. ProtStructQA results should not be interpreted as validating AlphaFold predictions or as experimental claims about the underlying proteins.

The denotation threshold is observed across Qwen3-\(0.6\)B to Qwen3-\(8\)B and replicated on Gemma-3-\(1\)B and Gemma-3-\(12\)B (§6.5). We do not claim that the parameter-count location of the threshold is universal across model families, training recipes, or larger scales. The cross-family replication is limited to two Gemma scales and three baselines (Standard, CoT, ReAct), so broader cross-family evaluation, including more model families, more intermediate sizes, and the full five-baseline suite, is an important next step. Fine-tuning is out of scope.

The DSL covers single-chain monomeric structural measurements over coordinates, confidence, PAE, solvent exposure, secondary structure, and contacts. It does not model dynamics, ligand binding, allostery, multimeric assemblies, evolutionary covariation, or experimental uncertainty beyond AlphaFold-derived confidence quantities.

The non-monotonic ReAct-on-compositional trajectory in §6.4 rests on three Qwen3 data points; resolving the curve shape would require intermediate scales. We therefore describe the pattern as “non-monotonic with a minimum at \(4\)B” and interpret the mechanism (tool use helps sub-threshold models, but can introduce over-deliberation and formatting failures above threshold) rather than claiming a universal scaling law.

9 Ethics Statement↩︎

ProtStructQA is constructed from publicly released AlphaFold Database predictions [1], [3] (CC BY 4.0) and UniProt [27] reference proteomes, with attribution and licensing aligned to the upstream resources. The benchmark contains no human subjects or personally identifying information. Code is released under the MIT license; the derived dataset is released under CC BY 4.0, matching the upstream UniProt and AFDB licenses.

The benchmark evaluates question answering over existing predicted structures and does not provide design objectives for optimizing pathogenicity, toxicity, immune evasion, or experimental protocols. Protein-analysis tools can nevertheless be dual-use in broader settings, so we avoid framing the benchmark as a protein-design capability and preserve standard biosecurity caveats.

Because answers are relative to predicted structures, users should not treat benchmark outputs as experimental biological facts. The intended use is evaluation of structural QA and language-to-denotation reasoning, not wet-lab decision making.

10 Qualitative Evidence for the Denotation Threshold↩︎

3pt

10.0.0.1 ReAct 8B sub-family failure decomposition.

At Qwen3-8B, ReAct fails almost entirely on the compositional sub-families G1 (set-valued, ResidueSet, \(\approx\)​13% accuracy) and G3 (nested-quantifier Boolean, \(\approx\)​0% accuracy), while reaching near-ceiling on the simpler Boolean sub-family G2 (\(\approx\)​100%). The two failure modes differ. G1 failures are formatting-driven: inspection of failing rollouts shows the agent identifies the correct residues during the tool loop but commits its final answer as free-form prose rather than a typed ResidueSet literal, so the deterministic executor never sees the answer (Example 4 below). G3 failures are not a formatting issue (the answer type is already Boolean); rather, the agent rarely completes the nested existential-with-distance check correctly within the available turn budget. Together these effects explain the 36-point 8B compositional gap between ReAct and EV+CoT.

We present four concrete examples drawn directly from the per-question output logs. The first three illustrate failure modes that aggregate into the denotation-threshold behavior reported in Section 6; the fourth illustrates the ReAct agent’s answer-formatting failure mode on Family-G1 questions (Section 6.4).

10.0.0.2 Example 1 (1.7B): Grammar+vote rescues a small model from ambiguous-question misinterpretation.

Question: “Average AlphaFold pLDDT, 75 to 130?”
Family A, type Float, gold: \(80.86\)

Failure mode (Standard/CoT): validity. The emitted program cannot be parsed. The phrase “75 to 130” is naturally read as a residue range (range(75, 130)). The free-form 1.7B model misinterprets “75” as a pLDDT threshold:

• Standard 1.7B emits the malformed program mean_plddt(range(1,187) where plddt(r) >= 75 and plddt(r) <= 130) and a free-form “Answer: 82.6”; neither parses, so pred=None. \(\times\)

• CoT 1.7B makes the same misinterpretation (mean_plddt(all_residues) where plddt(r) >= 75 and plddt(r) <= 130). \(\times\)

• EV 1.7B samples three grammar-constrained programs; the consensus is mean_plddt(range(75, 130)) which executes to \(80.86\). \(✔\)

Grammar constraints prevent the small model from emitting syntactically near-valid but semantically wrong programs, and the \(k{=}3\) vote breaks ties when one of the three samples is correct.

10.0.0.3 Example 2 (8B compositional): Free-form CoT handles longer nested logic that grammar-constrained sampling fails to generate reliably.

Question: “Is at least one 40-residue stretch high-confidence (mean pLDDT \(>70\)) AND contact-rich (contact density \(>0.3\))?”
Family G, type Bool, gold: False

Failure mode (EV+CoT): validity. Grammar-constrained sampling cannot emit a parseable nested program. This is a Family G compositional question requiring (i) a sliding window over 40-residue stretches, (ii) two predicate evaluations per window, and (iii) an existential aggregator.

• CoT 8B reasons step-by-step in free text and commits to False. \(✔\)

• EV+CoT 8B fails to generate the required longer nested DSL expression under grammar-constrained sampling; all \(k{=}3\) samples emit unparseable or partial programs and pred=None. \(\times\)

This pattern, where grammar-constrained sampling fails on longer nested expressions that free-form CoT simulates in text, accounts for the \(\sim\)​11.5pp gap by which CoT beats EV+CoT on the compositional split at 8B.

10.0.0.4 Example 3 (8B Family C): Grammar wins on region-pair PAE aggregation even at large scale.

Question: “What is the average AlphaFold relative-position uncertainty between the 333–369 and 380–433 regions?”
Family C, type Float, gold: \(5.77\)

Failure mode (CoT): denotation. The free-form trace identifies the right primitive but computes the aggregation incorrectly. This is a single-step PAE aggregation, but 8B free-form CoT mis-arithmetics the result.

• CoT 8B writes a multi-paragraph derivation and outputs \(2.99\) (a wrong arithmetic mean). \(\times\)

• EV+CoT 8B samples mean_pae(range(333,369), range(380,433)), executes it deterministically, and returns \(5.77\). \(✔\)

This explains why even at 8B, Family C (PAE) is dominated by EV+CoT by 22.7pp (Table 3): aggregations over contiguous regions are simple to express in DSL but error-prone for the model to compute in free-form text.

10.0.0.5 Example 4 (1.7B ReAct on G1): Agent reasoning succeeds, answer formatting fails.

Question (Family G1, type ResidueSet): “Which residues are simultaneously buried (rel SASA \(<\) 0.2) AND poorly predicted (pLDDT \(<\) 70)?”
Gold: [188] (single-residue ResidueSet)

Failure mode (ReAct): answer-format. The agent’s tool observations identify the correct residue, but the final <answer> commit is in free-form prose rather than a typed ResidueSet literal. The 1.7B ReAct agent runs five turns: it iteratively calls run_dsl and inspect_residue, narrowing to a candidate region beginning at residue 188. But on the final turn it emits the free-form sentence:

“residues 188–293 have a mean pLDDT of 66.20…all residues in this region are simultaneously buried (rel SASA \(<\) 0.2) AND poorly predicted (pLDDT \(<\) 70).”

The parser cannot extract a ResidueSet literal from this string, so pred=None and the answer is scored wrong. The agent’s reasoning identifies residue 188 (the gold answer) at the start of its candidate region, but the final <answer> commit is in natural language rather than the expected [N,…] typed format. This illustrates a structural limitation of multi-turn agents on structured-output benchmarks: without the deterministic executor in the loop on the final commit (as in EV / EV+CoT), agent-generated answers can fail type-formatting checks even when the underlying reasoning is partially correct.

10.0.0.6 Summary of failure modes.

Across the per-question logs we observe four dominant patterns: (1) at small scale (1.7B), free-form text misinterprets ambiguous question fragments and fails to commit a parseable program; grammar + vote rescues it; (2) on compositional questions at any scale, grammar-constrained sampling becomes less reliable on longer nested expressions (an underlying generation difficulty, not a DSL expressivity limit), while free-form CoT can reason in text and emit only the final scalar; (3) on region-pair aggregations, free-form models produce internally inconsistent arithmetic, while the executor delivers a correct answer deterministically; (4) multi-turn ReAct agents reason correctly through the tool loop but fail to format the final <answer> when the expected output type is structurally ambiguous (e.g., Family G1 ResidueSet answers). The denotation-threshold finding is the aggregate of the first three patterns weighted by family composition; the fourth characterizes a distinct, complementary weakness of agentic methods.

11 Per-Family Accuracy at 8B↩︎

Even at 8B where CoT wins overall, family-level inversion persists on the structured-output facets. Families C (PAE) and E (secondary structure) favor the grammar+execution branch (EV+CoT) by \(22.7\) and \(6.4\) pp respectively over CoT, while Families A (pLDDT aggregates), B (distance comparisons), D (SASA aggregates), and F (contact topology) favor CoT, whose answers reduce to small arithmetic comparisons the free-form chain can simulate directly. The per-family pattern shows that grammar constraints are not globally beneficial or globally harmful: their value depends on whether exact typed structure prevents the dominant error mode for that family.

12 Hard-Negative Sub-class Breakdown↩︎

The hard-negative split combines two construction types. HN1 (structural counterfactual, \(n{=}4{,}138\), \(95\%\)) keeps the template and protein fixed and resamples template parameters until the gold answer changes. HN2 (threshold flip, \(n{=}219\), \(5\%\)) keeps the protein, template, and residues fixed and changes only a numeric threshold. Table 4 reports the per-class accuracy for all five baselines at each scale.

HN2 is uniformly harder than HN1: all 20 (model, baseline) cells satisfy HN1\({>}\)HN2, with mean gap \(+10.3\) pp. The widest gaps occur for ReAct at supra-threshold scales (4B: \(+18.9\) pp; 8B: \(+22.9\) pp), indicating that the agent’s tool-call routine over-relies on the specific threshold value seen in the query. EV+CoT at 4B exhibits the narrowest gap (\(+2.9\) pp): grammar enforcement closes the threshold-sensitivity gap once the base model is fluent in the DSL. The HN2 sample size is small (\(n{=}219\) per (model, baseline); some per-template strata contain only \(\approx 44\) examples), so per-cell standard errors are roughly \(\pm 7\) pp at \(95\%\) confidence.

3.5pt

13 Gold-Answer Correctness Validation↩︎

We validate gold correctness at two levels, complementing the 6-gate sanity validator (§3.7): (i) paraphrase fidelity between the natural-language question and the gold DSL program, and (ii) cross-tool agreement between our pipeline’s per-primitive values and an independent reference toolchain. Paraphrase fidelity is run on \(n{=}500\) per family (\(n{=}3{,}500\) total); cross-tool agreement uses per-template subsamples detailed below.

13.0.0.1 Paraphrase fidelity.

For every sampled question, we verify (a) that every numeric literal in the gold DSL program (residue indices, range endpoints, thresholds) also appears in the paraphrase, and (b) that the paraphrase mentions at least one family-specific concept term from a hand-curated lexicon (e.g., “pLDDT” for A; “distance” for B; “PAE” for C; “SASA” or “neighbors” for D; “helix” or “DSSP” for E; “contact” or “radius of gyration” for F). On \(3{,}500\) sampled questions (\(500\) per family), \(3{,}500/3{,}500\) (\(100\%\)) pass both checks. The check is designed to catch obvious paraphrase/program mismatches (numeric drift, missing referents, wrong family concept) rather than to prove full semantic equivalence between every paraphrase and its canonical program.

13.0.0.2 Cross-tool agreement.

For each family that maps to a standard biophysical primitive, we re-compute the underlying quantity with an independent reference and compare to the value our pipeline stores. Implementations differ in code path; in some cases (e.g., SASA) the algorithm itself differs.

• Family A (pLDDT). Reference: BioPython PDBParser reading the B-factor column from the AlphaFold PDB. Our pipeline uses a hand-written fixed-column parser. Agreement: \(500/500\) questions match the BioPython recomputation; max abs.difference \(1.5 \times 10^{-5}\) pLDDT, mean \(3.5 \times 10^{-6}\) (float-precision round-trip).

• Family B (distance). Reference: BioPython CA-atom coordinates piped through numpy.linalg.norm. Our DSL operates on the cached ca_xyz array. Agreement: \(249/249\) (template B1, distance float); max abs.difference \(6.0 \times 10^{-6}\) Å.

• Family C (PAE). Reference: AlphaFold per-pair PAE JSON loaded with stdlib json. AFDB delivers PAE as integer values, which our pipeline stores as a uint8 matrix (lossless re-encoding); agreement is measured after dequantization to float. Agreement: \(133/133\) (template C1, mean PAE over inter-region block); max abs.difference \(9.4 \times 10^{-7}\) (float-precision noise).

• Family D (SASA / neighbor counts). SASA is validated against BioPython Bio.PDB.SASA.ShrakeRupley (pure-Python Shrake–Rupley) versus our pipeline’s FreeSASA C library [47] (different algorithm): per-residue Spearman \(\rho = 0.995\) (mean over \(104\) proteins, \(\rho_{\min} = 0.987\)); buried/exposed classification at threshold-far residues: \(96/96\) match. Neighbor counts are deterministic \(C_\alpha\)-distance computations and are independently recomputed from coordinates in the validation script.

• Family E (DSSP H/E/C). Our extractor uses pydssp (pure-Python DSSP [28], H-bond based). Reference: biotite’s native P-SEA secondary-structure annotation [31], which classifies SS from \(C_\alpha\)-trace geometric criteria (inter-residue distances and dihedral angles) rather than backbone H-bonds (algorithmically independent). Across \(273\) proteins, \(94.2\%\) per-residue agreement on helix-vs-non-helix and \(79.6\%\) 3-state agreement (DSSP and P-SEA disagree at \(\alpha\)-helix capping boundaries by construction). Gold-answer match: \(273/273\) on templates E1, E2, E3.

• Family F (contacts and topology). Contact-density templates reduce to \(C_\alpha\)-coordinate distance computations and inherit Family B agreement; the remaining geometric aggregates (e.g., radius of gyration, contact count) are deterministic coordinate functions independently recomputed in the validation script and matched within float precision.

• Family G (compositional) inherits the primitive-level checks of A–F (each compositional gold answer is produced by composing already-validated A–F primitives).

14 Stratified Evaluation Subsampling Protocol↩︎

All four paper-eval splits are released. The three subsamples (cross-species, compositional, hard-negative) are stratified by template\(\times\)species (cross-species and compositional) and by template (hard-negative). They are rigorously train-disjoint, and the distribution shift is bounded by KL\(\,\leq 0.0037\) on every (family, species, template, answer-type) marginal (see manifest in the release), so subsample trends transfer to the full pool within statistical noise. Split sizes and pool sizes are listed in §[sec:benchmark:subsample].

6pt

15 Family and Template Question Distribution↩︎

Table 6 summarises the full \(382{,}200\)-question release at the family level: each family contributes between \(3\) and \(6\) templates and roughly \(30\)K–\(60\)K questions across the active benchmark, with the \(52.2\)K hard-negative robustness pool counted on its own row.

2.5pt

Table 7 breaks this down at the template level, restricted to the four evaluation pools (in-distribution test, full compositional pool, full cross-species pool, full hard-negative pool; \(274{,}200\) questions in total). Templates B1–B4 (Distance) and G1–G3 (Compositional) carry the largest mass (10K–13K each), while the secondary-structure templates E3–E6 are smaller (3.2K–3.4K) because they target rare topology patterns. Counts are determined by the per-protein sampling budget at question-generation time (§3).

4pt

16 DSL Vocabulary and Complete Template Catalogue↩︎

This appendix is intended to make the abstract description in §3 concrete. We list (i) every operator in the DSL in plain English, and (ii) the exact program pattern of every one of the 31 templates the benchmark uses.

16.0.0.1 What is the DSL?

The DSL is a small formal language that lets us write a short program describing a question. A program takes a single protein as input and returns a single answer. The grammar is fixed, and the executor runs each program deterministically against the protein’s extracted features (pLDDT, distances, PAE, SASA, secondary structure).

16.0.0.2 What is a template?

A template is a parameterised question pattern. It bundles four things: a canonical DSL program with named placeholder slots, the expected answer type, the rules that decide which parameters are legal for a given protein, and a list of natural-language paraphrases that re-word the same question (\(26\)–\(30\) for A–F, \(5\)–\(6\) for G). The \(31\) templates are author-designed. The paraphrase pool was generated by an LLM (Claude Opus 4.7) and then author-verified to remove paraphrases that revealed the answer and to make sure every paraphrase kept the same slot names as its template. Every other step of the pipeline (protein selection, parameter sampling, gold-program slot-filling, DSL execution, paraphrase choice, and split assignment) is deterministic given the released random seeds.

16.0.0.3 How a question is built.

For every protein we pick a template, sample legal parameters, fill the template’s canonical DSL program with those parameters to obtain the gold program, execute it to obtain the gold answer, and fill one randomly chosen paraphrase to obtain the natural-language question text. At evaluation time, the model sees the question text together with a compact protein summary. For the non-agent baselines, the model must produce a valid DSL program. For ReAct, the model can call DSL tools and then return a final typed answer. Figure 6 walks through this end-to-end for one concrete question.

Plain-English DSL vocabulary↩︎

16.0.0.4 Per-residue values (one number per residue).

• plddt(r): AlphaFold’s confidence for residue \(r\) (range \(0\)–\(100\); higher is more confident).

• ss(r): secondary structure of residue \(r\) as a DSSP letter ("H" helix, "E" strand, "C" coil).

• rel_sasa(r): relative solvent-accessible surface area; values near \(0\) mean buried, near \(1\) mean exposed.

• n_neighbors(r): how many other residues sit within \(8\) Å of residue \(r\) (a packing measure).

16.0.0.5 Per-pair values (one number per pair of residues).

• distance(r1, r2): \(C_\alpha\)–\(C_\alpha\) distance in Å.

• pae(r1, r2): predicted aligned error in Å for the relative position of two residues (lower is more reliable).

16.0.0.6 Region-level aggregates (one number per contiguous span).

• mean_plddt(reg), min_plddt(reg), max_plddt(reg): summary pLDDT over a span.

• mean_rel_sasa(reg): average accessibility in a span.

• contact_density(reg): fraction of in-span residue pairs that are in contact (\(<\) \(8\) Å).

• radius_of_gyration(reg): how compact the span is.

• mean_pae(reg1, reg2), max_pae(reg1, reg2), count_high_pae(reg1, reg2,\(\tau\)): inter-region PAE summaries.

16.0.0.7 Whole-protein scalars.

n_helices(), n_strands(), length(longest_run("H")).

16.0.0.8 Constructors (how to refer to residues or spans).

residue(i): a single residue. range(s, e): the contiguous span from \(s\) to \(e\). first(k) / last(k) – the first / last \(k\) residues. sliding_window(k): the set of all length-\(k\) contiguous spans. all_residues: the set of every residue. all_pairs(min_sep=k): the set of ordered residue pairs \((i, j)\) with sequence separation \(|i - j| > k\).

16.0.0.9 Comprehensions and operators (how to combine the above).

count / filter / exists / forall \(x\) in \(S\) where \(P\): iterate \(x\) over \(S\) and either count, collect, or test the predicate \(P\). argmin / argmax \(x\) in \(S\) by \(E\) – return the element of \(S\) that minimizes / maximizes the expression \(E\). Comparison operators (\(<\), \(\le\), \(==\), \(\ne\), \(>\), \(\ge\)) and Boolean operators (and, or, not) compose predicates.

Complete catalogue of the 31 templates↩︎

Table 8 lists every template, the canonical DSL program it produces (with named slots), the answer type, and one example paraphrase. Slot names match the variables the parameter sampler fills in. In template A2, the program slot w is a local alias for the parameter window, and term1 is the terminus ("N" or "C") sampled per question; the program shown in the table is the term1="N" branch. Family A–F templates are seen during training as few-shot exemplars; Family G is held out, so the compositional split tests genuinely unseen template types. The hard-negative split is generated by perturbing parameters within the existing \(31\) templates.

4pt

17 Prompt-Form Sensitivity at Sub-Threshold↩︎

To check whether the sub-threshold CoT result depends on our specific prompt design, we evaluate two alternative CoT formulations at Qwen3-1.7B alongside our task-decomposition variant: vanilla zero-shot CoT [38] (“Think step by step before answering”), and PAL-style CoT [23], [24], where reasoning is carried by inline “#”-comment annotations within the few-shot exemplars rather than by a prompt prefix. Pooled accuracy across all \(32{,}357\) paper-eval questions at \(1.7\)B:

• No CoT (Standard): \(23.79\%\)

• Vanilla zero-shot CoT (Kojima): \(18.62\%\) (worst)

• Task-decomposition CoT (our published variant): \(22.54\%\)

• PAL-style CoT (exemplar comments): \(24.47\%\) (best)

The three CoT variants span only \(5.9\)pp from worst to best; even the best CoT prompt (PAL, \(24.5\%\)) sits \(6.5\)pp below EV (\(31.0\%\)) and \(25.8\)pp below ReAct (\(50.3\%\)). Instruction-driven CoT (task-decomposition and Kojima) does not improve over no-CoT direct prompting at \(1.7\)B; PAL-style CoT delivers only a marginal lift over no-CoT (\(\Delta{=}+0.68\)pp pooled; \(p\!<\!10^{-5}\) on test_iid by McNemar). Prompt-engineering perturbations cannot close the gap to the structural methods that EV and ReAct realize; the sub-threshold capability ceiling is robust to prompt-form choice.

18 Statistical Significance Detail↩︎

EV and EV+CoT use three seeds; per-cell seed noise is much smaller than the threshold-defining accuracy gaps. We report per-cell McNemar’s paired-test results for all \(80\) planned pairwise comparisons (5 method-pairs \(\times\) 4 splits \(\times\) 4 model scales) with Bonferroni correction at \(\alpha = 0.05/80 = 6.25 \times 10^{-4}\). Because McNemar’s test requires matched per-question predictions while Table 1 reports EV and EV+CoT as three-seed means, the paired statistics in Tables 9 and 10 use the seed-0 EV and EV+CoT runs. Some \(\Delta\) values therefore differ from the three-seed means in Table 1 by less than one accuracy point. Table 9 covers the sub-threshold scales (Qwen3-0.6B and 1.7B) and Table 10 covers the supra-threshold scales (Qwen3-4B and 8B).

34 of 40 sub-threshold comparisons are significant; the 6 ties cluster where small models fail similarly or where EV and EV+CoT are close at 1.7B. 31 of 40 supra-threshold comparisons are significant; the 9 ties cluster at CoT vs EV+CoT and Standard vs EV at \(\ge 4\)B, where grammar adds little once the base model is capable enough. Columns in both tables follow the same convention: “A-only”/“B-only” = questions correct in exactly one method of the pair; \(\Delta\) accuracy \(=\) (B\(-\)A), so a positive \(\Delta\) favors method B and a negative \(\Delta\) favors method A; the final row per (model, split) cell compares ReAct to the strongest non-agentic baseline.

4pt

4pt

19 Bootstrap Confidence Intervals↩︎

Table 11 reports the bootstrap \(95\%\) CI half-width for each (model, method, split) cell in Table 1, computed by resampling question indices \(1{,}000\) times (EV/EV+CoT pool across the three seeds). Half-widths are typically below \(1\) pp on in-distribution and cross-species cells, and reach \(1.0\)–\(1.5\) pp on the smaller hard-negative split (\(n{=}4{,}357\)) and on ReAct compositional cells where the agentic decoding has higher intrinsic variance.

4pt

20 Parse-Failure Rate Decomposition↩︎

To support the mechanism claims in §6.2, we measure the per-question parse-failure rate per (model, baseline) cell on the in-distribution and compositional splits. A parse failure is a question where the inference pipeline could not produce a usable answer (\(\texttt{pred\_answer} =\) None): for Standard and CoT this means the free-form output yielded no extractable DSL program; for EV and EV+CoT it means all \(k{=}3\) grammar-constrained samples failed to execute; for ReAct it means the agent never committed a parseable typed <answer> literal. Table 12 reports the overall rate by (model, baseline, split).

Two patterns support the §6.2 mechanism claim. (1) Between \(1.7\)B and \(4\)B on the in-distribution split, free-form parse-failure drops by \(40.9\) pp for Standard (\(64.8\% {\to} 23.9\%\)) and \(54.9\) pp for CoT (\(68.4\% {\to} 13.4\%\)); the capability threshold is in part a parseability threshold. (2) At \(8\)B on the compositional split, free-form CoT reaches a \(7.4\%\) parse-failure floor while grammar-constrained EV remains at \(29.9\%\) and EV+CoT at \(23.3\%\), consistent with grammar-constrained sampling becoming less reliable on longer nested expressions.

3pt

21 Cross-Family Replication: Gemma-3 Detail↩︎

This appendix accompanies §6.5 with the full Gemma-3 per-cell accuracy.

6pt

References↩︎