Simplicity Paradox: Debunking myths about prompting and datasets for LLM evaluation
May 07, 2026
Abstract
Probing the capabilities of Large Language Models (LLMs) and building robust solutions for Multiple-Choice Question Answering (MCQA) remain central challenges in natural language understanding. Furthermore, the rapid proliferation of LLMs has created the implicit assumption that more sophisticated prompting techniques yield better performance. Several studies claim better performance with more sophisticated prompting techniques, but do not provide a comprehensive evaluation. We address this gap through a comprehensive empirical study of 8 prompting techniques across 10 multiple-choice question answering (MCQA) datasets, encompassing 27 model configurations and roughly 4,300 unique questions evaluated more than 430,000 times. Our findings reveal a striking paradox that baseline prompting consistently outperforms complex reasoning techniques on various benchmarks. Only minimal expert and inductive role framing (CoT-Expert and CoT-Inductive) yields a small but statistically significant \(\sim\)3 percentage-point (pp) gain over baseline whereas every other elaborate technique we tested matches or under-performs it, often by large margins (up to 31 pp for Self-Analogical). We further investigate three critical phenomena: (1) the unexpected victory of Qwen3-30B-A3B-Thinking-2507 in Elo ratings, (2) the performance-efficiency trade-offs across model variants with different thinking budgets, revealing model-dependent optimal configurations, and (3) the substantial variation in dataset difficulty, with 60% of benchmarks below 70% accuracy and a 47.5 pp spread from easiest to hardest, indicating considerable room for model improvement. These results suggest that the LLM evaluation community may be overcomplicating prompt engineering and that substantial performance gaps remain across diverse benchmarks, offering opportunities for genuine model improvements rather than prompt optimization.
0pt
1 Introduction↩︎
The field of LLM evaluation has witnessed an explosion of prompting techniques, each claiming to unlock better reasoning. From Chain-of-Thought (CoT) [1] to self-consistency [2] and a growing family of plan-and-solve, analogical, and self-generated variants, the community has invested considerable effort in developing increasingly sophisticated prompting strategies. This has fostered an implicit assumption that complex prompting is necessary for strong performance.
But what if this assumption is wrong? This question arose while building ReasonLab, a framework designed to systematically evaluate prompting techniques across models and datasets. As results accumulated, a pattern emerged that contradicted both our expectations and much of the existing literature, simple baseline prompting was consistently matching or beating its more elaborate counterparts. To investigate rigorously, we ran 430,738 evaluations spanning 8 prompting techniques, 10 multiple-choice question answering (MCQA) benchmarks, and 27 model–configuration pairs (covering 13 distinct models with multiple thinking-budget settings where applicable). Repetition of \(\sim\)4,300 unique questions enables variance estimation and statistical testing that single-pass studies cannot support.
We focus on MCQA because it remains the dominant evaluation primitive in the LLM literature and underlies most public leaderboards, while also being the setting in which prompting techniques have been most aggressively promoted. Although this scoping limits direct claims about open-ended or agentic settings, it enables a controlled, well-powered comparison that prior empirical studies of prompting have not provided at this scale.
1.1 Key Findings↩︎
Our analysis surfaces four findings that challenge conventional wisdom about LLM evaluation.
1. Prompt complexity does not help non-reasoning models. On a matched comparison of 100 (dataset, model) cells, where every technique was evaluated on the same 10 nonreasoning configurations (§5), the baseline reaches 52.08% mean accuracy, finishing third behind CoT-Expert (55.71%) and CoT-Inductive (55.40%). Crucially, the two techniques that do edge out baseline are not multi-step reasoning frameworks but minimal role-framing variants, and the gap is small (\(\sim\)3 pp). Every elaborate scaffold we tested be it Chain-of-Thought, Plan-and-Solve, Self-Generate or Self-Analogical matched or under-performed baseline, with the largest deficit (Self-Analogical) partly attributable to format-collapse failures on smaller models (Appendix [app:selfanalog]). The headline is therefore narrower than “baseline is best”, hence, within the regime of models without native reasoning, prompt complexity is not what helps.
2. The Qwen surprise. Among models with publicly disclosed parameter counts, Qwen3-30B-A3B-Thinking-2507 narrowly tops the Elo leaderboard at 1657, ahead of GPT-OSS-120B (rank 4, 4\(\times\) larger) and Llama-3.1-405B (rank 18, 13\(\times\) larger). The top eight configurations cluster within 14.3 Elo points and Qwen3-30B-A3B-Thinking-2507’s lead over the second-ranked Claude Sonnet 4 (low) is only 3.9 Elo (a 50.6% expected head-to-head win rate), so we do not claim a single dominant model. The defensible takeaway is sharper that parameter count is a poor predictor of MCQA performance at the frontier.
3. Reasoning budgets show a steep step and a flat plateau. We configure reasoning effort with explicit token caps mapped to vendor labels: nothink, low = 1,024 reasoning tokens, and medium = 8,192 reasoning
tokens. The largest returns come from enabling reasoning at all: switching from nothink to medium adds +22.5 pp for GPT-5 (2025-08-07) and +19.7 pp for GPT-5-4, but only +3.6 pp for Claude Sonnet 4, the value of explicit
reasoning is itself model-dependent. Within the on-state, an 8\(\times\) token increase (low \(\rightarrow\) medium) adds only +1.0 to +4.1 pp at the cost of +350 to +880
reasoning tokens per question on average.
4. Benchmarks are not saturated. Despite widespread claims to the contrary, 6 of our 10 datasets sit below 70% mean accuracy across all evaluated models, and 4 of 10 remain below 70% even for the best model on each dataset. The spread from easiest (OpenBookQA, 83.4% mean) to hardest (SuperGPQA, 35.8% mean) is 47.5 pp, indicating substantial headroom for genuine model improvements rather than further prompt optimization.
2 Related Work↩︎
Prompting Techniques for LLMs The prompting literature has grown rapidly since Chain-of-Thought (CoT) prompting [1] demonstrated that eliciting step-by-step reasoning can improve performance on arithmetic and commonsense tasks, spawning Zero-shot CoT [3], Self-Consistency [2], Plan-and-Solve [4], Self-Ask [5], Tree-of-Thoughts [6], and analogical prompting [7]. The recent Prompt Report [8] catalogues over 50 distinct techniques. A parallel critical strand has questioned how robust the reported gains actually are: prompt formatting can swing accuracy by tens of points [9]–[11], CoT primarily helps on math and symbolic tasks [12], and self-verification frequently fails to improve answers [13]. Sprague et al. [12] is the closest prior work, providing a critical task-level meta-analysis of when CoT actually helps; ours extends the critical line by also covering reasoning-budget tradeoffs and pairwise (Elo) ranking under a unified protocol across 10 datasets and 27 model configurations.
Evaluation Frameworks and Benchmarks LLM evaluation infrastructure has matured along two axes. Broad benchmark suites such as HELM [14], BIG-bench [15], MMLU [16], and the lm-evaluation-harness [17] provide wide task coverage and standardized scoring, but treat the prompt as fixed and focus on cross-model comparison. Closer to our setting, PromptBench [18] and the Prompt Report [8] systematically vary prompts, but neither pairs technique-level comparison with reasoning-budget analysis or pairwise (Elo-style) ranking under a unified protocol.
Table 1 situates ReasonLab in this landscape: an evaluation harness, not a benchmark, that additionally logs reasoning- and output-token counts so accuracy can be reported jointly with cost.
| Framework | Multi-Technique | Multi-Dataset | Token Analysis | Pairwise Ranking |
|---|---|---|---|---|
| HELM [14] | ||||
| BIG-bench [15] | ||||
| lm-eval-harness [17] | ||||
| PromptBench [18] | ||||
| Prompt Report [8] | Partial | |||
| ReasonLab (Ours) |
Reasoning Models and Test-Time Compute Recent reasoning-tuned models such as OpenAI’s o1/o3 family, GPT-5, Claude with extended thinking, DeepSeek-R1 [19], and Qwen3 [20] expose a configurable thinking budget at inference time. A growing body of work examines how additional test-time compute trades off against accuracy [21], [22], and finds that returns are often non-monotonic and model-dependent. Our third finding contributes to this discussion with a controlled comparison across thinking budgets within the same model families.
MCQA as an Evaluation Format Multiple-choice question answering remains the dominant primitive for LLM leaderboards but carries known artifacts. Robinson and Wingate [23] characterize how models exploit option likelihoods, and Zheng et al. [24] document strong answer-position bias. Pezeshkpour and Hruschka [25] show that simply permuting option order can change predictions on a substantial fraction of items. We adopt MCQA because it underpins the benchmarks our findings speak to most directly, and we mitigate format-level artifacts through fixed answer-extraction logic and repeated sampling.
Dataset Saturation The saturation claims in [26], [27] have been challenged by [28] and by harder benchmarks such as GPQA [29] and SuperGPQA [30]. Results from our work in §[sec:difficulty] adds direct evidence from our 10-dataset suite.
3 Methodology↩︎
ReasonLab exposes five interchangeable components of the MCQA pipeline, (1) question presentation, (2) context construction, (3) reasoning trigger, (4) answer generation, and (5) answer extraction so that any prompting technique in the literature can be expressed as a substitution at one of these points (Figure 1). This lets us compare techniques head-to-head under a single protocol rather than inheriting each technique’s original protocol.
3.1 Datasets↩︎
We evaluate on 10 MCQA datasets spanning physics, medicine, mathematics, IoT/time-series, science, and multi-domain graduate knowledge: FailureSensorIQ [31], CURE-Bench [32], CorrectBench [33], PhysicsQA [34], AIME 2025 QA [35], MedQA [36], MMLU-Pro [37], OpenBookQA [38], SuperGPQA [30], and Time-MQA [39]. The design space and the skew of the answer-keys are visualized in Figure 2. And per-dataset record counts, average question lengths, option counts, and full per-letter answer-key distributions are reported in Appendix Table 5.
The 10 datasets cover a wide range along three axes that matter for MCQA evaluation, primarily, question length (55–753 tokens on average), number of answer options (2–10), and domain breadth (single-domain such as PhysicsQA to broad multi-domain suites such as MMLU-Pro and SuperGPQA). Six of the ten datasets exhibit answer-key skew above the uniform \(1/k\) baseline, which we control for through fixed answer-extraction logic.
3.2 Models↩︎
We evaluated 27 model configurations spanning eight families. Where a vendor exposes a reasoning-effort toggle, we map the labels to explicit reasoning-token caps and run the available settings: nothink (no reasoning), low = 1,024 reasoning tokens, and medium = 8,192 reasoning tokens. These caps disentangle the contribution of test-time reasoning from the raw model capability.
Two design choices shape this set. First, we prioritize coverage of the major frontier families (GPT-5, Claude Sonnet 4, Gemini 2.5 Pro, Llama, Qwen3, Mistral, Granite, GPT-OSS) over depth within any single family. Second, we keep a parameter range from
8B to 405B disclosed parameters so that the Qwen-surprise analysis (§[sec:qwensurprise]) is not driven by a single point.
Technique–model application policy. A central methodological choice is to apply the seven non-baseline techniques only to the 10 configurations without a vendor-exposed reasoning toggle- GPT-OSS-20B, GPT-OSS-120B, Llama-3.2-11B-Vision, Llama-3.3-70B, Llama-4-Maverick-17B-128e, Mistral-Large-2512, Mistral-Medium-2505, Mistral-Small-3.1-24B-2503, Granite-3.3-8B, and Granite-4-h-small. The remaining 17 reasoning-mode configurations are evaluated under baseline prompting only (full configuration list in Appendix [app:models], Table 4). Stacking external CoT triggers on top of native reasoning would conflate the elicited prompt with the model’s internal trace, this isolation costs us technique data on the reasoning-capable configurations, which we view as the right trade for a clean comparison.
3.3 Prompting techniques↩︎
ReasonLab implements 8 prompting techniques (Table 2) spanning four families: direct prompting, reasoning triggers, prompt-side robustness, and self-generated context. The two CoT variants (Expert, Inductive) are minor framings of CoT-Standard but yield qualitatively different results (§4).
| Technique | Trigger T⃝ / mechanism |
|---|---|
| Baseline | Direct prompt; answer letter only. |
| CoT-Standard [1] | “Let me think step by step.” T⃝ |
| CoT-Expert | “…as a reliability engineer.” T⃝ |
| CoT-Inductive | “…step-by-step inductive reasoning, given the domain-specific nature.” T⃝ |
| Self-Plan-and-Solve [4] | “Gather information, devise a plan, answer step by step.” T⃝ |
| Prompt-Repeat [40] | Direct prompt repeated multiple times. |
| Self-Generate [41] | Generates contextual document, then conditions on it. |
| Self-Analogical [7] | Generates related MCQA exemplars, then conditions on them. |
4pt
3.4 Raw vs.matched comparison↩︎
The application policy creates two natural levels of analysis, and we report both throughout the paper. Raw comparison. For each technique, we aggregate accuracy across every
(dataset, model) cell on which it was run. This uses all available data, but is unbalanced: techniques applied to a stronger or weaker subset of models will have inflated or deflated raw means.
Matched comparison. We restricted ourselves to the \(10 \times 10 = 100\) (dataset, model) cells common to all 8 techniques, all 10 datasets crossed with the 10 baseline-only configurations enumerated in
§3.2. This is a fully balanced comparison where every technique is evaluated on exactly the same questions on exactly the same models. We use the matched comparison as our headline result because it is the only
comparison that licenses paired statistical tests across techniques. Reported \(p\)-values come from paired \(t\)-tests on the 100 matched cells, with each cell’s accuracy as the unit of
observation.
3.5 Evaluation metrics↩︎
We report accuracy per cell (technique, model, data set) and token cost as separate input, output, and reasoning-token counts (cost–accuracy frontiers in §[sec:thinking]). All experiments use temperature=0, identical system prompts, and a single rule-based parser that takes the first letter token after a fixed
answer marker, with a regex fallback over the option set.
Elo ratings treat each (model, question) outcome as a pairwise match against every other model on the same question, therefore, a model wins if correct while the opponent is not, ties if both are correct or both wrong, and loses otherwise.
We use the standard Elo update (\(K\!=\!32\), initial rating \(1500\)) and shuffle match order across 100 random seeds, reporting the median per configuration with seed spread as a
confidence interval. This yields 85K–101K matches per configuration. The complete details of the implementation are given in Appendix [app:implementation].
4 Results↩︎
4.1 Overall Performance: The Baseline Paradox↩︎
| Matched (n=100) | Raw | Paired vs baseline | |||||
|---|---|---|---|---|---|---|---|
| 2-3 (lr)4-5 (lr)6-8 Technique | Mean | 95% CI | Mean | \(n\) | \(\Delta\) (pp) | \(p\) | W/T/L |
| CoTExpert | 55.71 | \(\pm\)3.96 | 55.71 | 100 | -3.63 | \(<\!\!10^{-6}\) | 28/5/67 |
| CoTInductive | 55.40 | \(\pm\)3.93 | 55.40 | 100 | -3.31 | \(<\!\!10^{-6}\) | 30/4/66 |
| baseline | 52.08 | \(\pm\)3.89 | 60.79 | 244 | — | — | — |
| PromptRepeat | 51.55 | \(\pm\)4.10 | 52.89 | 138 | +0.54 | 0.205 | 48/5/47 |
| SelfGenerateMethod | 50.91 | \(\pm\)3.71 | 50.91 | 100 | +1.17 | 0.021 | 63/4/33 |
| SelfPlanSolve | 42.82 | \(\pm\)5.37 | 42.82 | 100 | +9.26 | 0.002 | 36/4/60 |
| CoT | 42.63 | \(\pm\)5.36 | 41.34 | 104 | +9.45 | 0.002 | 37/3/60 |
| SelfAnalogic | 21.38 | \(\pm\)2.02 | 21.38 | 100 | +30.70 | \(<\!\!10^{-6}\) | 97/0/3 |
Table 3 reports both the raw and the matched comparison. The raw means show baseline at 60.79% well ahead of every other technique, but the raw set is not balanced across model coverage because baseline includes the strongest configurations in our suite (GPT-5, Claude Sonnet 4.6, Qwen3-30B-A3B-Thinking-2507), and the other techniques do not. To control for this, we restrict to the \(10\!\times\!10\!=\!100\) matched cells where every technique was evaluated against the same 10 model configurations. Figure 3 (a) shows the mean accuracies resulting with 95% confidence intervals.
The picture that emerges is more interesting than “baseline beats everything.” On the matched set, which is restricted to the 10 configurations without a vendor-exposed reasoning toggle (§5), baseline ranks third at 52.08%, behind CoT-Expert (55.71%, \(\Delta=-3.63\) pp, \(p<10^{-6}\)) and CoT-Inductive (55.40%, \(\Delta=-3.31\) pp, \(p<10^{-6}\)). Both are single-line primer, a role frame (“…as a reliability engineer”) and an inductive-reasoning trigger, not multi-step reasoning frameworks. Prompt-Repeat is statistically indistinguishable from baseline (+0.54 pp, \(p=0.21\)). The genuinely complex scaffolds underperform, namely, Self-Generate by 1.17 pp, Plan-and-Solve by 9.26 pp, Chain-of-Thought by 9.45 pp, and Self-Analogical by 30.70 pp. Self-Analogical’s 21.38% sits at or below the random-chance floor for 4-option questions, and a failure-mode analysis (Appendix [app:selfanalog]) attributes 10.5% of its outputs to format-collapse on smaller models (notably gpt-oss-20b-medium, which produces no parseable answer on 6 of 10 datasets); the remaining error mass reflects genuine technique failure rather than parser artifacts. Figure 3 (b) shows the per-cell distribution of these deltas, with paired W/T/L counts.
The per-dataset structure of this result, summarized in Figure 4, is instructive. CoT-Expert wins outright on 6 of 10 datasets, CoT-Inductive wins 3, and Self-Generate wins 1 (AIME 2025, the hardest math benchmark in our suite). Baseline does not win a single dataset outright. Yet across cells, baseline’s standard deviation (19.83) is the lowest among the non-trivial techniques (CoT and Plan-and-Solve are at 27.4), making it the most consistent prompt: “everywhere good, nowhere best.” The combination of elaborate reasoning scaffolds underperforming a one-line prompt, yet baseline itself failing to win any dataset is the Simplicity Paradox at the heart of this paper.
4.2 The Qwen Surprise↩︎
Figure 5 reports Elo ratings computed via pairwise per-question comparisons across all datasets and techniques (\(K\!=\!32\), initial rating 1500, \(\sim\)85K–101K matches per configuration); the top-15 numerical ranking is provided in Appendix Table 6. Qwen3-30B-A3B-Thinking-2507 occupies rank 1 with an Elo of 1657, ahead of Claude Sonnet 4 (low) at 1653, GPT-5-4 (medium) at 1653, and GPT-OSS-120B (medium) at 1652. The top eight configurations, drawn from the Qwen, Claude, GPT, and GPT-OSS families, cluster within 14.3 Elo points of each other.
We caution against treating Qwen3-30B-A3B-Thinking-2507’s rank as decisive on its own. Its 3.9 Elo lead over the second-placed configuration corresponds to an expected head-to-head win rate of just 50.6%. The defensible reading is comparative rather than ordinal among models with publicly disclosed parameter counts, Qwen3-30B-A3B-Thinking-2507 (30B total / approximately 3B active) is competitive with GPT-OSS-120B (rank 4, 4\(\times\) larger) and exceeds Llama-3.1-405B (rank 18, 13\(\times\) larger).
4.3 Model Variant Analysis: Does More Thinking Help?↩︎
Several recent model families expose a reasoning-effort control that we map to explicit token budgets in our experiments- nothink (no reasoning), low = 1,024 reasoning tokens,
and medium = 8,192 reasoning tokens. We analyse this control on the four base models that have multiple variants on a comparable dataset slice, namely, Claude Sonnet 4, Gemini 2.5 Pro, GPT-5 (2025-08-07), and GPT-5-4. For each model
we compute mean accuracy and mean reasoning-token usage on the intersection of datasets where every variant of that model was evaluated, then compare the within-model deltas. Figure 6 reports the resulting accuracy–cost
curves alongside paired variant deltas.
Two regimes emerge clearly. Enabling reasoning is the dominant lever, switching from nothink to medium adds +22.5 pp for GPT-5 (2025-08-07, 75.79% vs.%, paired across 10 datasets) and
+19.7 pp for GPT-5-4 (76.96% vs.%). Claude Sonnet 4, however, gains only +3.6 pp from the same switch (71.97% vs.%), suggesting Claude relies less on visible reasoning to score well on MCQA than the GPT-5 family does and its
accuracy–budget curve is non-monotonic where the low setting (1,024 tokens) actually under-performs nothink by 0.5 pp, with the gain only appearing at medium suggesting that small reasoning budgets may even disrupt whatever
reasoning Claude is doing implicitly during baseline generation. For other models returns within the on-state are much smaller and much more expensive, for instance, an 8\(\times\) token-budget increase from low
(1,024 tokens) to medium (8,192 tokens) adds only +1.0 pp for GPT-5-4, +1.6 pp for GPT-5 (08-07), +2.6 pp for Gemini 2.5 Pro, and +4.1 pp for Claude Sonnet 4, while consuming an additional 350–880 reasoning tokens per question on average (most
models do not exhaust the 8,192-token cap). The literature’s framing of “test-time compute” as a single dial is therefore too coarse because enabling reasoning and increasing the budget within the on-state are different interventions with very different
return profiles, and the magnitude of the enable-reasoning step is itself model-dependent.
4.4 Dataset Difficulty: Substantial Headroom Remains↩︎
Figure 7 reports the distribution of baseline-prompted accuracy on each of our 10 datasets, taken across all available model configurations. Two summary statistics are worth highlighting. First, the spread from easiest to hardest dataset is large as OpenBookQA achieves a mean accuracy of 83.35% across models, while SuperGPQA reaches only 35.84%, a 47.5 pp gap. Second, headroom remains substantial, i.e. 6/10 datasets have a mean accuracy below 70%, and 4/10 are still below 70% even for the best-performing model on each dataset (Time-MQA, CorrectBenchQA, FailureSensorIQ, SuperGPQA).
This pattern is informative. The hardest benchmarks in our suite, namely, SuperGPQA (graduate-level multi-domain), AIME 2025 (Olympiad mathematics), and FailureSensorIQ (industrial sensor reasoning) are precisely the ones where no prompting technique has rescued performance. SuperGPQA in particular tops out at 63.8% even for the strongest model we evaluated, and elaborate reasoning prompts give no measurable advantage on it. Reports of MCQA “saturation” appear premature, at least within the difficulty regime our benchmarks span; the field has more headroom than the discourse suggests, and it sits in genuine reasoning, specialised knowledge, and temporal understanding rather than in prompt engineering.
We defer extended discussion of why elaborate prompting fails, what the Qwen surprise implies for scaling, and how to rethink test-time “thinking” to Appendix [app:discussion].
5 Limitations↩︎
Matched comparison covers only non-reasoning configurations. Our matched 100-cell comparison evaluates the 7 non-baseline prompting techniques on the 10 model configurations without a vendor-exposed reasoning toggle. The 17 reasoning-capable configurations (GPT-5 family, Claude Sonnet 4 / 4.6, Gemini 2.5 Pro, Qwen3) were evaluated under baseline only, because stacking external CoT triggers on top of native reasoning would conflate the elicited prompt with the model’s internal trace. As a consequence, we cannot rule out, the possibility that elaborate prompting helps reasoning-capable models in their nothink mode or alongside their native reasoning. Designing a clean protocol that separates “model is reasoning” from “prompt elicits reasoning” is open future work.
Technique implementation choices. Each technique is implemented following its original paper’s public template; we did not tune exemplar count, prompt wording, or output format per model. Self-Analogical in particular was originally validated on math reasoning under different exemplar settings, and its performance on our broader domain mix may not reflect the technique’s ceiling under a tuned implementation (Appendix [app:selfanalog]).
MCQA scope. We focus on multiple-choice question answering because it is the dominant evaluation primitive for LLM leaderboards. Our findings do not extend directly to open-ended generation, agentic tasks, or settings where reasoning traces are scored in addition to final answers.
Decoding non-determinism. We use temperature\(=\)0 with a single sample per cell. For vendor-hosted reasoning models this is not strictly deterministic; we hold all controllable parameters constant and rely on the matched comparison and the large per-technique cell counts (10 datasets \(\times\) 10 configurations \(\times\) \(\sim\)430 questions) to absorb residual variation, but we do not characterise it directly.
6 Conclusion↩︎
Across \(\sim\)430,000 model–question evaluations spanning 8 prompting techniques, 10 datasets, and 27 model configurations, four scoped claims survive controlled comparison. First, on the 10 non-reasoning configurations where elaborate prompting can be cleanly compared to baseline, no multi-step scaffold reliably beats a one-line direct prompt; whether this extends to reasoning-capable models is open (§5). Second, parameter count is a poor predictor of Elo rank at the frontier. Third, the dominant lever in test-time compute is enabling reasoning at all, not scaling its budget. Fourth, the 47.5 pp spread between easiest and hardest of our benchmarks argues for redirecting effort from prompt scaffolding toward base-model capability, architecture, and harder evaluations.
7 Self-Analogical failure-mode analysis↩︎
Self-Analogical’s 21.38% mean accuracy on the matched set sits at or below the random-chance floor for 4-option questions, raising a natural concern about whether the result reflects technique failure or implementation artefact. We audited all 43,480 Self-Analogical model–question outputs across the 10 matched-set configurations to characterise failure modes.
Format-collapse rate. 10.5% of outputs (4,575 of 43,480) return no parseable answer after three retries and are scored as incorrect. This rate is concentrated on smaller models: gpt-oss-20b-medium produces no parseable
response on 6 of 10 datasets (a 100% format-collapse rate on those datasets), while granite-4-h-small and gpt-oss-120b-medium stay below 7% format-collapse on every dataset.
Performance on parseable outputs. Restricted to outputs that the shared parser successfully extracts an answer letter from, Self-Analogical still scores well below baseline (the gap narrows but remains \(>\)25 pp). This rules out parser failure as the dominant explanation for the headline number; the technique produces confident wrong answers on the majority of questions where it produces an answer at all, often anchoring on its self-generated exemplars rather than the question itself.
Implementation note. Our prompt template follows the public release of Yasunaga et al. [7]; we did not tune exemplar count or template wording per model. The original paper validated the technique primarily on math reasoning, where retrievable analogies are denser than in our broader domain mix (medicine, IoT, time-series, graduate-level science). The 21.38% number should therefore be read as a lower bound on what a tuned implementation could achieve, not as a definitive verdict on analogical prompting in general.
8 Model configurations↩︎
| Family | Configuration | Thinking budget | In matched set |
|---|---|---|---|
| GPT-5 | gpt-5-2025-08-07 | nothink / low / medium | only nothink |
| GPT-5 | gpt-5-4 (preview) | nothink / low / medium | only nothink |
| GPT-OSS | gpt-oss-20b | medium | ✔ |
| GPT-OSS | gpt-oss-120b | medium | ✔ |
| Claude | Claude Sonnet 4 | nothink / low / medium | only nothink |
| Claude | Claude Sonnet 4.6 | low / medium | — |
| Gemini | Gemini 2.5 Pro | low / medium | — |
| Llama | Llama-3.2-11B-Vision | nothink | ✔ |
| Llama | Llama-3.3-70B | nothink | ✔ |
| Llama | Llama-3.1-405B | nothink | — |
| Llama | Llama-4-Maverick-17B-128e (fp8) | nothink | ✔ |
| Qwen3 | Qwen3-8B | medium | — |
| Qwen3 | Qwen3-30B-A3B-Thinking-2507 | medium | — |
| Mistral | Mistral-Large-2512 | nothink | ✔ |
| Mistral | Mistral-Medium-2505 | nothink | ✔ |
| Mistral | Mistral-Small-3.1-24B-2503 | nothink | ✔ |
| Granite | Granite-3.3-8B | nothink | ✔ |
| Granite | Granite-4-h-small | nothink | ✔ |
6pt
9 Implementation details↩︎
All experiments use identical system prompts and decoding parameters across techniques. We use temperature=0 and a single sample per (technique, model, dataset, question) cell; we acknowledge that temperature-zero decoding is not strictly
deterministic for vendor-hosted reasoning models, but we hold all controllable parameters constant and rely on the matched comparison and large per-technique cell counts to absorb residual non-determinism. Answer extraction uses one shared parser across
all techniques; the same regex and fallback logic apply identically to Baseline and to every technique’s final-answer field.
10 Dataset statistics↩︎
| Name | Rec. | ||||
| len | |||||
| (min–max) | Domain | Answer distribution | |||
| FailureSensorIQ [31] | 500 | 130 | 4.2 (2–5) | IoT | A:26.4% B:25.2% C:19.8% D:16.2% E:12.4% |
| CURE-Bench [32] | 779 | 260 | 4.4 (4–5) | Medical | A:23.7% B:24.4% C:24.6% D:20.5% E:6.7% |
| CorrectBench [33] | 494 | 279 | 4.9 (4–40) | Multiple | A:16.0% B:19.8% C:14.6% D:15.6% E:10.3% |
| PhysicsQA [34] | 45 | 225 | 4.1 (3–6) | Physics | A:13.3% B:35.6% C:33.3% D:17.8% |
| AIME 2025 QA [35] | 30 | 527 | 4.3 (3–6) | Mathematics | A:40.0% B:20.0% C:10.0% D:10.0% E:20.0% |
| MedQA [36] | 500 | 753 | 5.0 (5–5) | Medical | A:22.0% B:20.6% C:23.2% D:18.8% E:15.4% |
| MMLU-Pro [37] | 500 | 262 | 9.6 (3–10) | Multiple | A–J \(\in\) [8.6%, 12.0%] |
| OpenBookQA [38] | 500 | 55 | 4.0 (4–4) | Multiple | A:27.6% B:25.2% C:26.4% D:20.8% |
| SuperGPQA [30] | 500 | 324 | 9.7 (4–10) | Graduate-level | A–J \(\in\) [8.0%, 11.8%] |
| Time-MQA [39] | 500 | 211 | 3.4 (2–4) | Multiple | A:26.2% B:39.8% C:27.4% D:6.6% |
4pt
11 Elo rankings: full top-15 table↩︎
| Rank | Model | Elo | Matches | Params |
|---|---|---|---|---|
| 1 | Qwen3-30B-A3B-Thinking-2507 | 1657 | 101,161 | 30B (3B active) |
| 2 | claude-sonnet-4-low | 1653 | 85,581 | — |
| 3 | gpt-5-4-medium | 1653 | 101,161 | — |
| 4 | gpt-oss-120b-medium | 1652 | 101,161 | 117B (5.1B active) |
| 5 | claude-sonnet-4-medium | 1651 | 85,581 | — |
| 6 | claude-sonnet-4-nothink | 1650 | 85,581 | — |
| 7 | gpt-5-2025-08-07-medium | 1649 | 101,161 | — |
| 8 | gpt-5-2025-08-07-nothink | 1643 | 101,161 | — |
| 9 | claude-sonnet-4-6-low | 1616 | 28,580 | — |
| 10 | claude-sonnet-4-6-high | 1607 | 76,436 | — |
| 11 | claude-sonnet-4-6-medium | 1604 | 28,580 | — |
| 12 | mistral_medium_2505-nothink | 1552 | 101,161 | — |
| 13 | llama-3-3-70b-nothink | 1551 | 101,161 | 70B |
| 14 | granite_4_h_small-nothink | 1547 | 101,161 | 32B |
| 15 | gemini-2.5-pro-low | 1544 | 85,581 | — |
| “—” indicates the model developer has not publicly disclosed parameter count. | ||||
12 Discussion↩︎
12.1 Why Does Elaborate Prompting Fail?↩︎
The matched comparison demands explanation. Why do the four most heavily promoted multi-step reasoning techniques, Chain-of-Thought, Plan-and-Solve, Self-Analogical, and Self-Generate, consistently match or under-perform a one-line direct prompt, while two minimal role primers (CoT-Expert, CoT-Inductive) yield small but reliable gains? We see four complementary factors at play, and notably the same factors explain both halves of the asymmetry.
The first is error propagation. Multi-step reasoning chains create opportunities for cascading error, a wrong assumption in step 2 of a 5-step chain corrupts steps 3 through 5, and there is no recovery mechanism. Direct prompting sidesteps this by providing a single path from question to answer. Role primers (“as a reliability engineer”, “step-by-step inductive reasoning”) condition the model’s distribution without forcing it to externalise a long chain, capturing some of the priming benefit of CoT without inheriting its compounding-error cost.
The second factor relates to how models are trained. Modern instruction-tuned LLMs undergo extensive fine-tuning on direct question-answering, optimised for the pattern “here is a question; give the answer.” Adding explicit multi-step reasoning instructions may push the model away from the distribution it was tuned for. Role-priming, by contrast, is closer in form to the system messages models routinely see during instruction tuning, which may be why minimal expert framing transfers better than full reasoning scaffolds.
Third, prompt overhead matters. Every token spent on elaborate instructions is a token not available for question content. Complex techniques consume multiples more input tokens than baseline, and longer prompts also leave more room for adversarial or off-task interpretation by the model. Role primers cost only a single short clause and so escape this trade-off.
Finally, many prompting techniques suffer from overfitting to the tasks they were originally validated on. Chain-of-Thought, for instance, was developed and evaluated primarily on arithmetic reasoning; Self-Analogical was developed on tasks where useful analogies were retrievable. Our datasets span a wider domain mix, medicine, IoT, time-series, graduate-level science, etc. and the high variance of these techniques across our datasets (CoT and Plan-and-Solve have \(\sigma \approx 27\) pp on the matched cells, versus baseline’s 19.8) is consistent with the hypothesis that their benefits are task-specific rather than general.
12.2 The Qwen Surprise: Implications↩︎
Qwen3-30B-A3B-Thinking-2507’s narrow first-place finish in our Elo analysis should not be over-interpreted, but neither should it be dismissed, the broader leaderboard structure tells a real story. Eight model configurations from four families cluster within 14 Elo points at the top, and several of those configurations are 4–13\(\times\) larger than Qwen3-30B-A3B-Thinking-2507 on disclosed parameter counts. Whether or not Qwen3-30B-A3B-Thinking-2507 is truly the single best configuration in our suite, the weaker claim, that a 30B model is competitive with disclosed-size frontier configurations on MCQA is supported by the data and is the more durable finding.
What might explain this? Several research directions emerge that do not depend on which model holds rank 1. Architecture matters, Qwen3-30B-A3B-Thinking-2507 uses a sparsely-activated mixture-of-experts design that may improve effective knowledge retrieval per active parameter, and investigating these architectural differences could inform next-generation model design. Training data quality matters, rather than scaling dataset size alone, focusing on diversity, curation, and relevance to evaluation domains may yield better parameter efficiency. Fine-tuning strategy matters, Qwen’s instruction-tuning and RLHF procedures may be particularly well-aligned with multiple-choice formats, suggesting that alignment recipes are an under-explored axis of model improvement. And we need benchmarks that probe different aspects of model capability to avoid over-fitting to specific patterns. Our results suggest that parameter count alone is an insufficient proxy for capability on MCQA tasks at the frontier and that whatever scaling laws still hold do not hold on this axis.
12.3 Rethinking "Thinking"↩︎
Our variant analysis points to an uncomfortable truth that returns to additional reasoning tokens are sharply non-linear. Going from nothink to medium adds 19.7–22.5 pp for GPT-5 family models, but the 8\(\times\)-larger token budget from low (1,024) to medium (8,192) adds only 1–4 pp at the cost of several hundred additional reasoning tokens per question. The framing of “test-time compute” as a single dial that can be turned up for proportional gains is therefore misleading, the nothink \(\rightarrow\) low step is one large jump, followed by a long shallow plateau within the on-state.
Even the enable-reasoning step is model-dependent. Claude Sonnet 4 gains only +3.6 pp from nothink to medium, far less than the GPT-5 family’s \(\sim\)+20 pp. This suggests that for some models, a substantial portion of the “reasoning” is already happening implicitly during baseline generation, and explicit chain-of-thought primarily exposes it for inspection rather than producing it. If true, this matters for evaluation because comparing models by their reasoning variants alone may favour models whose reasoning is more visible without telling us anything about their underlying capability.
Three directions look more promising than budget scaling. First, structured reasoning with explicit verification could replace free-form generation, addressing the cascading-error failure mode that explains why elaborate prompting techniques under-perform baseline (§[sec:baseline]). Second, adaptive reasoning budgets driven by question difficulty would avoid the fixed allocations whose returns we show to be small. Third, the field needs metrics that assess reasoning quality beyond final-answer correctness, since our results suggest token count and answer accuracy are loosely coupled within the on-state.
References↩︎
Corresponding authors↩︎