1 Introduction↩︎

Recent advances in large language models have increasingly emphasized reasoning as a key driver of performance on complex tasks [1]–[3]. Techniques such as chain-of-thought prompting [4], reasoning-specific training [5], [6], and test-time compute scaling [7] have shown substantial gains on benchmarks such as GPQA [8], SciBench [9], and MMLU-STEM [10].

The problem is that these benchmarks conflate two distinct capabilities: retrieving the correct domain-specific knowledge, and applying the appropriate reasoning procedure over that knowledge. When a model fails a chemistry problem, it is unclear whether the failure reflects an inability to recall the relevant formula or an inability to execute the required reasoning steps. Without disentangling these factors, a basic question goes unanswered: do reasoning mechanisms improve reasoning itself, or do they primarily improve knowledge utilization?

We introduce IsoSci, a benchmark of isomorphic cross-domain science problem pairs designed to answer this question directly. Each problem is paired with a structurally identical counterpart from a different scientific domain: both require the same sequence of logical and computational steps, but depend on entirely different domain knowledge. If a model succeeds on one problem but fails on its isomorphic counterpart, the gap must be attributed to missing knowledge, not to reasoning ability.

Using IsoSci, we evaluate five model pairs across four model families, covering both traditional reasoning-vs-standard comparisons and toggle-based comparisons (same model, reasoning on vs.off). Across 8,408 evaluations spanning four scientific domains, we find that 91.3% of reasoning-mode gains are knowledge-dependent rather than structure-invariant (63/69 gains across all five pairs; Wilson 95% CI [82.3%, 96.0%]), for short-horizon procedural science problems. Enabling reasoning has minimal effect on overall accuracy for high-capability models (below 5pp across domains), and a reasoning-specialized model (o3-mini) that outperforms its standard counterpart on GPQA (\(+\)​19.2pp) underperforms on IsoSci (\(-\)​24.7pp), showing that benchmark choice determines conclusions about reasoning model utility.

These findings suggest that reasoning mechanisms function primarily as extended knowledge retrieval on short-horizon science tasks, increasing the probability that relevant domain facts are surfaced during generation rather than improving logical procedure execution.

1.0.0.1 Contributions.

(1) A construction methodology for isomorphic cross-domain science problem pairs that hold reasoning structure constant while varying domain knowledge, applicable at any scale or domain. (2) The \(p_{\text{know}}\) metric (Eq. 5 ), which decomposes reasoning-mode gains into knowledge-dependent and structure-invariant components. (3) IsoSci, a 144-pair benchmark spanning four scientific domains under CC-BY-4.0, with empirical findings on knowledge dependence of reasoning gains, toggle effects, and benchmark-dependent model comparisons.

2 Related Work↩︎

2.0.0.1 Reasoning in large language models.

Methods for eliciting multi-step behavior include chain-of-thought prompting [4], [11]–[13] and test-time compute scaling [14]. Evaluations of these methods typically report end-task accuracy improvements, treating different mechanisms as interchangeable, without analyzing how they alter the balance between intermediate computation, search, and reliance on memorized patterns. For scientific reasoning, benchmarks such as MMLU-STEM [10], SciBench [9], and GPQA [8] cover undergraduate to graduate-level science questions across multiple formats and difficulty levels, and have been widely used to track progress across model generations. Their evaluations are primarily aggregate, however, offering limited insight into the sources of model success or failure.

2.0.0.2 Disentangling reasoning and knowledge.

Isolating reasoning ability from knowledge in LLMs remains an open problem [15]–[17]. Chain-of-thought analyses suggest intermediate steps function more as structured memory retrieval than logical inference [4], [18], [19], and benchmark performance is known to be sensitive to knowledge coverage [20]. The closest concurrent work is [21], who train a PubMedBERT classifier to label biomedical QA items as reasoning-heavy or knowledge-heavy, finding that only 32.8% require multi-step reasoning and that models consistently underperform on that subset. IsoSci differs in three respects: we construct matched pairs with structurally identical solution procedures by design rather than classifying existing items post-hoc; our metric \(p_{\text{know}}\) operates at the pair level and can isolate whether a gain transfers across domains, which stratum-level accuracy cannot.

2.0.0.3 Benchmark design and controlled evaluation.

Recent work improves benchmark quality through adversarial filtering [8], domain stratification [10], and tolerance-based grading [9], addressing memorization and grading fidelity [21], [22]. These designs remain aggregate and do not control for solution procedure across items. IsoSci extends this line by enforcing structural equivalence through isomorphic cross-domain pairs, enabling comparisons where reasoning demands are held fixed and performance can be decomposed into knowledge-dependent and structure-invariant components.

3 The IsoSci Benchmark and Evaluation Protocol↩︎

This section formalizes the IsoSci benchmark and the \(p_{\text{know}}\) decoupling metric. The benchmark holds the reasoning structure of a problem constant across a cross-domain pair while varying the domain knowledge required, so that an accuracy gap between the two members attributes to knowledge rather than to reasoning. The 144-pair release is one instantiation of the methodology, which extends to any scientific domain or scale.

3.1 Notation and Preliminaries↩︎

Let \(\mathcal{D} = \{\text{phys}, \text{chem}, \text{bio}, \text{earth}\}\) denote the four scientific domains (physics, chemistry, biology, earth science). Let \(\mathcal{S} = \{s_1, \dots, s_5\}\) denote the five structure types listed below. Let \(\mathcal{X}\) denote the space of natural-language problem statements and \(\mathcal{Y}\) the space of admissible answers (multiple-choice letters, numerical values, or short text strings). A problem is a tuple \(q = (x_q, a_q, d_q, s_q) \in \mathcal{X} \times \mathcal{Y} \times \mathcal{D} \times \mathcal{S}\) with text \(x_q\), gold answer \(a_q\), domain \(d_q\), and structure \(s_q\); let \(\mathcal{Q}\) denote the set of all such problems. For \(q \in \mathcal{Q}\), let \(K(q)\) denote the set of domain-specific knowledge atoms required to solve \(q\) (formulas, physical or chemical constants, named domain entities). Let \(\mathcal{M} \subset \mathcal{D} \times \mathcal{D}\) denote the set of cross-domain mappings considered, with \(|\mathcal{M}| = 6\) covering each unordered pair of distinct domains.

Let \(\mathcal{F}\) denote the set of LLM configurations under evaluation; each \(f \in \mathcal{F}\) is a (stochastic) mapping from a prompt to a generated string in \(\mathcal{Y}^{*}\). We define the evaluation function \[E : \mathcal{F} \times \mathcal{Q} \to \{0, 1\}, \qquad E(f, q) \;=\; \mathbf{1}\!\left\{ \operatorname{extract}\!\bigl(f(\operatorname{prompt}(x_q))\bigr) \equiv a_q \right\}, \label{eq:eval}\tag{1}\] where \(\operatorname{prompt}(\cdot)\) wraps \(x_q\) in the zero-shot chain-of-thought template (Section 4.3), \(\operatorname{extract}(\cdot)\) applies the cascade of Section 4.3, and \(\equiv\) is exact match for letters or strings and \(\pm 2\%\) relative tolerance for numerical answers. We write \(\Pi \subset \mathcal{F} \times \mathcal{F}\) for the set of evaluated model pairs; each \((R, S) \in \Pi\) has \(R\) in the reasoning configuration and \(S\) in the standard configuration.

3.2 Formal Definition of Isomorphic Pairs↩︎

3.2.0.1 Structure types (\(\mathcal{S}\)).

IsoSci restricts attention to five short-horizon (3 to 5 reasoning steps) structure types for which the bijection \(\phi\) in Definition 1 is tractable to verify:

1. Formula recall and substitution: recall a domain law, substitute given values, compute (e.g., ideal gas law, Beer-Lambert law).

2. Unit conversion chain: multi-step unit tracking across a sequence of conversions.

3. Conservation law application: identify and apply a conservation principle (energy, mass, charge, momentum).

4. Proportional reasoning: use ratio or scaling relationships to recover an unknown quantity.

5. Two-step causal chain: qualitative reasoning where cause \(A\) implies effect \(B\) implies effect \(C\), with no numerical computation.

Table 1 shows a representative pair with structure \(s =\) formula_recall_and_substitute and three solution steps under non-overlapping knowledge sets.

3.3 Dataset Construction↩︎

Construction proceeds in three stages, summarized as \(\mathcal{Q}^{\text{seed}} \xrightarrow{\text{generate}} \mathcal{Q}^{\text{cand}} \xrightarrow{\text{verify}} \mathcal{Q}^{\text{pass}} \xrightarrow{\text{balance}} \mathrm{\small IsoSci}{}\).

3.3.0.1 Stage 1: Seed collection.

The seed pool \(\mathcal{Q}^{\text{seed}}\) aggregates 2,315 items from GPQA Diamond [8] (\(n = 198\)), SciBench [9] (\(n = 585\)), and MMLU-STEM [10] (\(n = 1{,}532\)). Earth-science seeds, absent from these sources, are generated synthetically with claude-sonnet-4-5 (96 problems; prompt in Appendix 8.4), giving 2,411 problems. Token-overlap deduplication (\(\operatorname{Jaccard} > 0.40\)) yields \(|\mathcal{Q}^{\text{seed}}| = 2{,}190\) unique problems (physics 867, chemistry 639, biology 588, earth 96).

3.3.0.2 Stage 2: Isomorphic partner generation.

For each mapping \((d, d') \in \mathcal{M}\), we sample up to \(n_{\text{seed}} = 25\) seeds from \(\{q \in \mathcal{Q}^{\text{seed}} : d_q = d\}\) and prompt claude-sonnet-4-5 to generate \(n_{\text{cand}} = 3\) candidate target problems \(q'\) per seed satisfying conditions (i) to (iv) of Definition 1 under the source structure \(s_q\) (prompt in Appendix 8.2). This yields \(|\mathcal{Q}^{\text{cand}}| = 429\) candidate pairs across \(\mathcal{M}\).

3.3.0.3 Stage 3: Automated verification.

A panel of three judges, \(\mathcal{J} = \{\text{claude-sonnet-4-5}, \text{GPT-4o-mini}, \text{DeepSeek-V3}\}\), scores each candidate on four criteria \(\mathcal{C} = \{c_{\text{logic}}, c_{\text{indep}}, c_{\text{diff}}, c_{\text{self}}\}\) corresponding to logical equivalence, domain independence, difficulty parity, and self-containment. Let \(r_j(c, q, q') \in \{1,2,3,4,5\}\) denote the rating from judge \(j \in \mathcal{J}\) on criterion \(c \in \mathcal{C}\). A pair \((q, q')\) is accepted into \(\mathcal{Q}^{\text{pass}}\) if and only if \[\min_{j \in \mathcal{J}} \;\min_{c \in \mathcal{C}} \;r_j(c, q, q') \;\geq\; 3.5. \label{eq:accept}\tag{2}\] Of 429 candidates, 217 satisfy 2 (50.6% pass rate). After balancing across \(\mathcal{M}\) to a target of 22 to 25 pairs per mapping, the released benchmark is \(\mathrm{\small IsoSci}{} = \mathcal{Q}^{\text{pass}}_{\text{bal}}\) with \(|\mathrm{\small IsoSci}{}| = 144\) pairs (288 problems). Distribution and overall acceptance rates are reported in Table 2.

3.3.0.4 Robustness of the verification rule.

Because claude-sonnet-4-5 contributes both to candidate generation in Stage 2 and to the judge panel \(\mathcal{J}\), a confound is possible: shared blind spots could inflate the accepted set \(\mathcal{Q}^{\text{pass}}\). Recomputing 2 over \(\mathcal{J} \setminus \{\text{claude-sonnet-4-5}\}\) rejects only \(1/144\) released pairs, retaining 99.3%; no pair accepted by the two-judge panel is rejected by the three-judge panel.

3.3.0.5 Human expert audit.

Two PhD-level annotators independently rated a stratified sample of 50 candidates (25 LLM-accepted, 25 LLM-rejected) on \(\mathcal{C}\). Inter-annotator agreement was substantial (\(\kappa = 0.714\), exact agreement 84%), comparable to LLM-human agreement (\(\kappa \in \{0.686, 0.648\}\)). Against human consensus on the 42 pairs with full annotator agreement, the LLM ensemble attained precision \(0.941\), recall \(0.842\), and \(F_1 = 0.889\). The single false positive involved formula overlap across domains; the three false negatives indicate a conservative bias that reduces dataset size without contaminating it.

3.4 Comparison with Existing Science Benchmarks↩︎

Table 3 positions IsoSci relative to existing benchmarks. The defining property, isolating reasoning from knowledge via isomorphic pairs, is absent from all prior work. Recent benchmarks emphasizing multimodality (PhysUniBench [23]), symbolic correctness (QuantumBench [24]), or research-grade derivation (FrontierMath [25]) target complementary dimensions of scientific reasoning. IsoSci provides a controlled diagnostic specifically for the knowledge-versus-reasoning attribution question, which the above benchmarks do not address by design.

4 Experimental Setup↩︎

4.1 Model Configurations↩︎

We evaluate \(|\Pi| = 5\) paired configurations \(\Pi = \{(R_i, S_i)\}_{i=1}^{5}\) in two tiers (Table 4). Three main pairs are run on all four benchmarks; two supplementary pairs are run on IsoSci alone because of API latency and budget constraints. A pair is traditional if \(R_i\) and \(S_i\) are distinct trained models from the same family, and toggle if \(R_i\) and \(S_i\) are the same model with the provider’s reasoning flag set to True and False respectively. To avoid potential contamination, we exclude all Anthropic-family models from \(\Pi\) because claude-sonnet-4-5 was used in Stages 1 to 3.

The pair o3-mini versus GPT-4o-mini follows the canonical reasoning-versus-standard comparison [26]; because traditional pairs differ in pretraining and RLHF objectives in addition to reasoning training, causal claims about the reasoning mechanism are most cleanly supported by the toggle pairs. Qwen3-32B [27] and Gemini 2.0 Flash [28] expose a reasoning toggle via the reasoning.enabled parameter, holding all model weights and decoding parameters constant; including a mid-capability (Qwen3-32B) and a high-capability model (Gemini 2.0 Flash, 91.9% MMLU-STEM) tests whether the toggle effect is capability-dependent. Supplementary pairs DeepSeek-R1 versus DeepSeek-V3 [29] and QwQ-32B versus Qwen2.5-72B extend coverage of traditional pairs.

4.2 Evaluation Benchmarks↩︎

Main pairs are evaluated on \(\mathcal{B} = \{\mathrm{\small IsoSci}{}, \text{GPQA}, \text{MMLU-STEM}, \text{SciBench}\}\) with item counts 288, 198, 750, and 585 respectively (a total of 1,821 items). IsoSci mixes formats inherited from its seed sources: 103 pairs (71.5%) are 4-way multiple choice (MMLU-STEM seeds), 38 pairs (26.4%) are free-response numerical (SciBench seeds), and 3 pairs (2.1%) are short answer. Supplementary pairs are evaluated on IsoSci alone. The total evaluation volume is \(10{,}926\) API calls, of which \(8{,}408\) (\(76.9\%\)) returned valid responses; exclusion analysis appears in Appendix 12.

4.3 Evaluation Protocol↩︎

4.3.0.1 Inference.

For each \(f \in \mathcal{F}\) and \(q \in \bigcup \mathcal{B}\) we sample \(y = f(\operatorname{prompt}(x_q))\) once at temperature \(0\) with maximum output 8,192 tokens. The prompt template is the zero-shot chain-of-thought instruction:

For multiple-choice items (GPQA, MMLU-STEM, and the MCQ subset of IsoSci), the four choices are appended to \(x_q\) in a deterministically shuffled order (seed equals item index).

4.3.0.2 Answer extraction.

The map \(\operatorname{extract} : \mathcal{Y}^{*} \to \mathcal{Y} \cup \{\bot\}\) applies a five-pattern cascade and returns the first match: (1) the substring after **Final Answer:**; (2) the contents of \boxed{...}; (3) the numeric content of the last display equation $$...$$; (4) the substring after Therefore or The answer is; (5) the last numeric expression in \(y\). For multiple-choice items, an additional pass matches written-out answer text against the choice options. Full algorithm appears in Appendix 9.

4.3.0.3 Grading.

For IsoSci, the equality test \(\equiv\) in 1 is conditioned on the format inherited from the seed: letter match for MCQ items, \(\pm 2\%\) relative tolerance for free-response numerical items, and exact string match for short-answer items. GPQA and MMLU-STEM use letter match; SciBench uses \(\pm 2\%\) tolerance throughout. The \(\pm 2\%\) rule may be inappropriate for logarithmic quantities (pH, p\(K_a\)); manual review of 50 SciBench chemistry responses identified 3 cases (\(<\)​0.6pp impact) where rounding convention differences caused false negatives.

4.3.0.4 Aggregation.

Domain-stratified deltas (Table 7) pool all records for a given domain across \(\mathcal{B}\) and weight each record equally; this means benchmarks contributing more domain-relevant items contribute more to the aggregate (for example, MMLU-STEM contributes 250 biology items versus GPQA’s 19). Per-benchmark breakdowns appear in Appendix 7. IsoSci-specific quantities (\(p_{\text{know}}\), \(\Delta_{\text{acc}}\) in Table 6) use IsoSci alone.

4.4 Reasoning-Knowledge Decoupling Metric↩︎

For a pair \((R, S) \in \Pi\) and a problem \(q \in \mathcal{Q}\), define the gain indicator \[G_{R,S}(q) \;=\; \mathbf{1}\!\left\{ E(R, q) = 1 \;\wedge\; E(S, q) = 0 \right\}. \label{eq:gain}\tag{3}\] For each isomorphic pair \((q, q') \in \mathrm{\small IsoSci}{}\) and each \((R, S) \in \Pi\), classify the pair by its joint gain pattern under 3 and aggregate over IsoSci: \[\begin{align} k_s &= \sum_{(q, q') \in \mathrm{\small IsoSci}{}} G_{R,S}(q)\,\bigl(1 - G_{R,S}(q')\bigr), \nonumber \\ k_t &= \sum_{(q, q') \in \mathrm{\small IsoSci}{}} \bigl(1 - G_{R,S}(q)\bigr)\,G_{R,S}(q'), \nonumber \\ k_b &= \sum_{(q, q') \in \mathrm{\small IsoSci}{}} G_{R,S}(q)\,G_{R,S}(q'). \label{eq:counts} \end{align}\tag{4}\] A reasoning-mode gain on a pair is knowledge-dependent if it occurs on exactly one member of the pair (it contributes to \(k_s\) or \(k_t\)): the reasoning configuration improves on one domain but not on the structurally identical other, indicating that the gain reflects domain-specific knowledge rather than shared reasoning structure. A gain is structure-invariant if it occurs on both members (it contributes to \(k_b\)). The knowledge-dependence ratio is \[p_{\text{know}} \;=\; \frac{k_s + k_t}{k_s + k_t + k_b}, \label{eq:pkno}\tag{5}\] computed over the \(n_{\text{gain}} = k_s + k_t + k_b\) pairs on which any gain exists. We report Wilson 95% confidence intervals on \(p_{\text{know}}\) (Table 6). Because \(k_b\) requires improvement on both pair members, \(p_{\text{know}}\) is a conservative upper bound on knowledge-dependence: asymmetrically expressed reasoning gains, those improving the harder member only, contribute to \(k_s + k_t\) even when they reflect structural improvement.

4.5 Assumptions↩︎

The methodology depends on the following assumptions, each anchored empirically where possible.

1. Verifiable bijection. For every released pair \((q, q') \in \mathrm{\small IsoSci}{}\), the bijection \(\phi\) in Definition 1 exists and the solution procedure is preserved under \(\phi\). Anchor: the acceptance rule 2 filters pairs that fail this property, attaining \(F_1 = 0.889\) against human consensus on the audited subset.

2. Disjoint knowledge. \(K(q) \cap K(q') = \emptyset\) for every released pair. Anchor: criterion \(c_{\text{indep}}\) in 2 enforces this; the dominant LLM-judge failure mode is formula overlap across domains, identified in 1 of 50 audited cases.

3. Toggle isolation. For toggle pairs, \(R\) and \(S\) correspond to the same model weights with all decoding parameters held constant except the provider-specific reasoning flag, so any \(E(R, q) - E(S, q)\) difference attributes to the reasoning mechanism rather than to model identity.

4. Pair-level independence. Pair outcomes \((G_{R,S}(q), G_{R,S}(q'))\) are independent across distinct pairs in \(\mathrm{\small IsoSci}{}\), supporting Wilson confidence intervals on \(p_{\text{know}}\) and bootstrap confidence intervals on \(\Delta_{\text{acc}}\).

5. Coverage. The seed pool \(\mathcal{Q}^{\text{seed}}\) is representative of short-horizon, information-complete, procedural scientific problems at the undergraduate to early-graduate level. long-horizon derivation, hypothesis generation, and open-ended synthesis are out of scope.

4.6 Scope and Limitations↩︎

The methodology covers short-horizon (3 to 5 step) procedural problems across four scientific domains and the five structure types of \(\mathcal{S}\) (Section 3.2). Long-horizon derivations, research-grade symbolic computation, hypothesis generation, and open-ended synthesis are out of scope. Generation and verification rely on LLMs, with a residual false-positive rate of approximately 6% against human consensus; convergence of \(p_{\text{know}}\) across five model pairs from four families provides empirical evidence that this does not change the direction of the finding. For toggle pairs, the reasoning flag alters generation behavior but not model weights, so toggle comparisons test inference-time reasoning rather than reasoning-specialized training. Asymmetric truncation (approximately 23% of API calls; Appendix 12) imposes a conservative bias: a robustness check on the valid-response subset shifts the pooled \(p_{\text{know}}\) estimate by 0.9pp (Appendix 14). Additional limitations are discussed in Section 6.0.0.1.

5 Results↩︎

Table 5 reports accuracy for all six model configurations across all four benchmarks.

4pt

5.1 Reasoning-Knowledge Decoupling on IsoSci↩︎

Table 6 presents the core IsoSci result: the attribution of reasoning-mode gains to knowledge-dependent versus structure-invariant improvements. The \(n_{\text{gain}}\) column reports the number of pairs on which any gain exists per model pair; \(p_{\text{know}}\) is computed exclusively over this subset, and Wilson 95% CIs are reported. The wide CIs (e.g., [67.6, 100.0] for Gemini and DeepSeek-R1) reflect small \(n_{\text{gain}}\) values, a direct consequence of near-zero or negative overall deltas for several pairs.

Under the symmetric definition (Eq. 5 ), which counts gains on either pair member as knowledge-dependent, the three main model pairs yield a pooled ratio of \(41/43 = 95.3\%\) (Wilson 95% CI [84.5%, 98.7%]), with per-pair values of 100%, 89.5%, and 100%. Two supplementary pairs evaluated on IsoSci only — DeepSeek-R1 vs.DeepSeek-V3 and QwQ-32B vs.Qwen2.5-72B — are consistent with this pattern. Including all five pairs, the pooled \(p_{\text{know}} = 91.3\%\) (63/69 gains, CI [82.3%, 96.0%]), with tighter CIs than the three-pair estimate. QwQ-32B shows the most structure-invariant gains (\(k_b = 4\)) and equal source and target deltas (\(+\)​4.2pp on both), suggesting partial cross-domain transfer; its \(p_{\text{know}} = 77.8\%\) is the lowest of the five pairs but remains well above chance. Appendix 14 reports a robustness check on the valid-response subset; the pooled three-pair estimate shifts by only 0.9pp (94.4%, CI [81.9%, 98.5%]), confirming the finding is not driven by asymmetric truncation. A source/target label-swap permutation test (Appendix 15) finds no significant directional imbalance for the main pairs (\(p = 0.065\)–\(0.195\)), though we note a marginal result for QwQ-32B (\(p = 0.048\)).

Across all five model pairs, knowledge-dependent gains dominate structure-invariant gains, with \(p_{\text{know}}\) ranging from 77.8% to 100% and a pooled estimate of 91.3%. The pattern is consistent: when the reasoning configuration improves on one member of a pair, it typically does not improve on the isomorphic counterpart despite an identical solution procedure. The convergence across five model pairs spanning four model families (OpenAI, Google, Qwen, DeepSeek) and both comparison types mitigates concerns about LLM-judge reliability in the construction pipeline and reduces the risk that the finding is model-specific.

5.1.0.1 Interpretation.

The IsoSci design provides a direct test: if reasoning improvement were structural (better at multi-step inference regardless of domain), gains would manifest equally on both members of an isomorphic pair. The data rejects this for the problem types covered by IsoSci. Extended reasoning appears to help models retrieve and apply domain facts more thoroughly, but does not improve the logical procedure applied once those facts are retrieved. Whether this finding extends to long-horizon or open-ended scientific reasoning remains an open question.

5.2 Reasoning Toggles Provide Minimal Benefit on Science↩︎

Table 7 reports accuracy deltas (\(\Delta_{\text{acc}}\) = reasoning \(-\) standard) stratified by domain across all benchmarks. For the two toggle pairs—where architectural confounds are eliminated—gains are 0–4pp at most. For Gemini (91.9% MMLU-STEM), the toggle makes essentially no difference across all four domains, with all 95% CIs including zero. Qwen3 shows small positive gains that barely exclude zero (physics: \(+\)​1.1 to \(+\)​4.7pp). McNemar’s test confirms this: for both toggle pairs the discordant counts are small and nearly equal (Qwen3: \(b=21\), \(c=20\); Gemini: \(b=8\), \(c=9\)), yielding McNemar statistic \(= 0.0\) (\(p = 1.0\)) — no evidence of a systematic toggle effect in either direction.

4pt

5.3 Benchmark Choice Determines Conclusions About Reasoning Models↩︎

This benchmark-dependent reversal has a direct implication: conclusions about reasoning model utility on science depend entirely on which benchmark is used. GPQA Diamond, with its emphasis on conceptual depth and multi-step derivation, favors reasoning-specialized models. IsoSci, with its structured formula-substitution problems, reveals that the same model is less capable of efficient knowledge retrieval. Neither benchmark alone provides a complete picture; together they diagnose where and how a model’s science capability breaks down.

6 Conclusion↩︎

We introduce IsoSci, a benchmark of isomorphic cross-domain science problems designed to disentangle reasoning from domain knowledge in LLM evaluation. The core contributions are a construction methodology for isomorphic problem pairs applicable at any scale or domain, and the \(p_{\text{know}}\) metric that decomposes reasoning-mode gains into knowledge-dependent and structure-invariant components. Across five model pairs and four model families, most reasoning-mode gains are knowledge-dependent, and enabling reasoning yields only marginal improvements on short-horizon procedural science tasks. The reversal for o3-mini, strong on GPQA Diamond and weak on IsoSci, shows that no single benchmark provides a complete picture of scientific capability. We hope the isomorphic-pair methodology and \(p_{\text{know}}\) metric support more precise evaluation of scientific reasoning and motivate larger instantiations covering longer-horizon problem types.

6.0.0.1 Limitations.

(1) Dataset scope: With 144 pairs, IsoSci is smaller than prior benchmarks and does not support fine-grained sub-domain analysis; however, it suffices for the controlled pairwise comparisons underlying our claims. (2) Model coverage: We evaluate a limited set of models; however, results are consistent across traditional and toggle-based comparisons, suggesting the observed patterns are not model-specific. (3) Grading noise: Automated grading (±2% tolerance with pattern-based extraction) may introduce minor errors, mainly from formatting or unit mismatches; manual checks indicate this affects \(<3\%\) of cases and does not change overall conclusions. (4) Toggle interpretation: The reasoning.enabled flag changes generation behavior but not model weights; it remains the cleanest available method for isolating reasoning at inference time.

7 Full Accuracy Results with Confidence Intervals↩︎

Table 8 reports complete accuracy results for all six model configurations across all four benchmarks and four scientific domains, with 95% bootstrap confidence intervals (1,000 samples).

3pt

8 Evaluation Prompts↩︎

8.1 Zero-Shot Chain-of-Thought Evaluation Prompt↩︎

All models receive the following system-level instruction prepended to each question:

For GPQA Diamond (MCQ), the question is formatted as:

Answer choices are shuffled using a deterministic seed equal to the question index, ensuring reproducibility. The correct answer letter varies per question.

8.2 Isomorphic Partner Generation Prompt↩︎

The following prompt was used to generate isomorphic partner problems (Stage 2):

8.3 LLM Judge Verification Prompt↩︎

The following prompt was used for automated pair verification (Stage 3):

8.4 Earth-Science Seed Problem Generation Prompt↩︎

The following prompt was used to generate the 96 synthetic earth-science seed problems in Stage 1 (Section 3.3). The same prompt template was used for any domain where benchmark coverage was insufficient; in practice only earth science required synthetic generation. Temperature was set to 0.7 to encourage topical diversity.

9 Answer Extraction Algorithm↩︎

Algorithm 3 describes the answer extraction procedure applied to all model responses.

10 Example Dataset Pairs↩︎

We present representative examples from IsoSci, illustrating the structure of isomorphic problem pairs across domains. Each pair consists of a source and target problem with identical reasoning structure but distinct domain knowledge.

11 Dataset Release and Reproducibility↩︎

11.0.0.1 Dataset and Croissant metadata.

IsoSci is released on HuggingFace at https://huggingface.co/datasets/isosci/isosci (anonymized for review). The repository includes train (80 pairs) and test (64 pairs) splits stratified by domain mapping, provided for downstream studies that require held-out evaluation sets, such as fine-tuning or few-shot prompting experiments. All results in this paper use the full 144-pair set without train/test separation. In addition to full pair metadata, and a Croissant-compliant croissant.json metadata file at the repository root. The Croissant file was validated locally using the mlcroissant package prior to submission.

11.0.0.2 Evaluation code.

The full pipeline is released at https://anonymous.4open.science/r/isosci-603C/. The pipeline is implemented in Python 3.9+ and requires only standard scientific libraries plus the requests package for API calls. All random seeds are fixed for reproducibility.

11.0.0.3 Computational cost.

Table 9 reports approximate API costs for replicating our evaluation.

11.0.0.4 Model versions.

Table 10 lists exact model identifiers used in this study, accessed via the OpenRouter API.

11.0.0.5 Toggle implementation.

For both Qwen3-32B and Gemini 2.0 Flash, we pass "reasoning": {"enabled": true/false} as a top-level field in the OpenRouter API request body. When enabled=false, the model generates a direct response without a visible <think>...</think> block; when enabled=true, the model prefixes its response with an explicit reasoning chain before the final answer. Temperature is 0 in both conditions and no other parameters are modified. Manual inspection of 20 randomly sampled response pairs confirmed that the toggle controls the presence of a visible <think> block but that both conditions generate comparably long final responses on structured scientific problems. We note that this toggle suppresses visible chain-of-thought generation but does not modify model weights; it is possible that models internally perform multi-step reasoning in standard mode without surfacing it, in which case our comparisons measure the effect of visible extended reasoning.

12 API Exclusion Analysis↩︎

Of 10,926 API calls (\(6\) configurations \(\times\) \((288 + 198 + 585 + 750)\) items), 8,408 (76.9%) returned valid responses used in analysis. The remaining 2,518 (23.1%) were excluded due to output token limit truncation (\(\approx\)​18%, concentrated in reasoning-on configurations on free-response benchmarks), API timeout (\(\approx\)​3%), and format errors (\(\approx\)​2%). Reasoning-on configurations show higher exclusion rates on SciBench (28% vs.% for standard mode), consistent with longer reasoning chains hitting the 8,192 token limit. This asymmetric exclusion could in principle suppress observed reasoning-mode gains; we note it as a conservative bias — if anything it understates reasoning-mode accuracy, making the null finding for toggle pairs more rather than less credible. Evaluations affected by an earlier 2,048 token limit were rerun after the limit was increased.

13 Statistical Tests↩︎

13.0.0.1 McNemar’s test.

We used McNemar’s test with continuity correction to assess whether reasoning and standard configurations produce significantly different correct/incorrect patterns on paired IsoSci items (\(n=288\) per comparison). The test statistic is \((|b-c|-1)^2/(b+c)\) where \(b\) = items correct under reasoning only and \(c\) = items correct under standard only. Results are reported in Table 11.

For o3-mini vs.GPT-4o-mini, the test confirms a highly significant difference (\(b=16\), \(c=87\), stat\(=47.57\), \(p<0.001\)): the standard model (GPT-4o-mini) outperforms on substantially more items than the reasoning model does.

For the two toggle pairs, the discordant counts are small and nearly equal: Qwen3-32B (\(b=21\), \(c=20\)) and Gemini 2.0 Flash (\(b=8\), \(c=9\)). The continuity correction yields a statistic of 0.00 in both cases (\(p=1.0\)), indicating no evidence of a systematic toggle effect. We note that the accuracies are not literally identical — Qwen3-32B shows 37.2% vs.% and Gemini shows 61.8% vs.% — but the discordant pairs are balanced in both directions, meaning gains and losses from enabling reasoning cancel out almost exactly across the 288 items.

13.0.0.2 Bootstrap confidence intervals.

All reported CIs use 1,000 bootstrap samples with the percentile method. The random seed is fixed at 42 for all bootstrap calculations.

13.0.0.3 Binomial test on source/target gain asymmetry.

As a complement to \(p_{\text{know}}\), we test whether source-only gains (\(k_s\)) and target-only gains (\(k_t\)) occur with equal probability under the null hypothesis \(H_0: \Pr[\text{source-only}] = 0.5\). This tests directional asymmetry in where the knowledge bottleneck falls, which is distinct from the primary \(p_{\text{know}}\) finding.

Across the three main model pairs (\(k_s = 24\), \(k_t = 17\), \(n = 41\) asymmetric gains), the binomial test yields \(p = 0.349\): no significant directional asymmetry within the main pairs. Pooled across all five model pairs (\(k_s = 41\), \(k_t = 22\), \(n = 63\)), the test yields \(p = 0.023\), indicating that source-only gains outnumber target-only gains at conventional significance. This asymmetry is driven by the supplementary pairs (DeepSeek-R1 and QwQ-32B) and should be interpreted with caution: it suggests that source problems may be slightly harder or more knowledge-discriminating than target problems on average, but it does not affect the primary finding that knowledge-dependent gains (\(k_s + k_t\)) dominate structure-invariant gains (\(k_b\)) across all five pairs.

14 Robustness Check: \(p_{\text{know}}\) on Valid-Response Subset↩︎

A potential concern is that asymmetric response truncation could bias \(p_{\text{know}}\) if truncated responses are systematically correct or incorrect. We recompute \(p_{\text{know}}\) under the symmetric definition (Eq. 5 ) restricted to items where both configurations produced valid, non-truncated responses (non-empty response, non-empty extracted answer, no API error).

Table 12 reports results. Qwen3-32B shows the largest exclusion rate (88 invalid reasoning responses, 86 invalid standard responses out of 288), leaving 194 items (67.4%). The restricted accuracy for Qwen3-32B rises from 37% to 54%, confirming that invalid responses are concentrated on harder items.

Despite this, the pooled restricted \(p_{\text{know}} = 94.4\%\) (34/36 gains, Wilson 95% CI [81.9%, 98.5%]) is within 3pp of the full-set estimate of 95.3% (43 main pairs, CI [84.5%, 98.7%]), and the CIs overlap substantially. The finding is robust to exclusion of invalid responses.

The Qwen3-32B restricted \(n_{\text{gain}} = 16\) is smaller than the full-set value of 19, as expected: restricting to valid responses removes some pairs where a gain existed, shrinking the denominator. The \(p_{\text{know}}\) estimate changes from 89.5% to 87.5%, well within the overlapping confidence intervals. The direction of the finding is unchanged across all three model pairs and both the full-set and restricted analyses.

15 Source/Target Label Permutation Test↩︎

A potential concern is that the asymmetry between source-only gains (\(k_s\)) and target-only gains (\(k_t\)) reflects a systematic difficulty imbalance between source and target problems rather than domain knowledge asymmetry. To test this, we performed a label-swap permutation test: for each pair, we randomly swapped the source and target labels with probability 0.5, recomputed \(|k_s - k_t|\) on the shuffled data, and repeated for 1,000 iterations. The empirical \(p\)-value is the fraction of permutations yielding \(|k_s - k_t| \geq\) the observed value.

Results are reported in Table 13. The observed source/target imbalance is not statistically significant at any conventional threshold (\(p = 0.065\)–\(0.195\) per model pair). We cannot rule out that some portion of the \(k_s\)/\(k_t\) asymmetry reflects difficulty differences between source and target problems rather than directional knowledge asymmetry.

Crucially, this test addresses a secondary question about the direction of knowledge-dependent gains, not the primary finding. The main claim rests on \(p_{\text{know}} = (k_s + k_t)/(k_s + k_t + k_b) = 95.3\%\), which measures whether knowledge-dependent gains (\(k_s + k_t = 41\)) dominate structure-invariant gains (\(k_b = 2\)). This ratio is unaffected by the source/target label assignment: swapping labels converts \(k_s\) gains into \(k_t\) gains and vice versa, but leaves \(k_s + k_t\) unchanged. The permutation test therefore has no bearing on the primary finding.

References↩︎