1 Introduction↩︎

As Large Language Models (LLMs) are developed towards increased robustness in reasoning, reliable evaluation of their problem-solving capabilities is becoming instrumental. Several benchmarking datasets have been proposed to facilitate principled assessment of LLMs, including GSM-Base - a dataset of 8.5k mathematical problems at grade school level [2].

However, as new models are released, they might be exposed to the publicly available benchmarking datasets, causing data contamination and possibly inflating the reported performance. Evaluating data contamination effects was the motivation behind the GSM-Symbolic study [1]. The researchers extracted 100 questions from the GSM8K benchmark and, using carefully constructed templates to vary names and numbers involved, prepared 50 new variants of each of these questions, creating a dataset of 5000 examples¹. Having tested 25 models of varying architectures and sizes on the dataset, the authors observed a consistent performance decline on the new variants compared to the original 100 questions and hypothesised that this result demonstrates limitations in logical reasoning capabilities of LLMs.

In this work, we revisit the GSM-Symbolic study to quantitatively assess the extent of the reported deterioration and provide more insights concerning possible failure mechanisms. Our re-evaluation reveals that the per-model accuracy deltas often fail to reach statistical significance (\(p < 0.05\)) when subjected to rigorous hypothesis testing. Furthermore, focusing on the models which do exhibit significant performance changes, we propose alternative prompt formats to help isolate potential contributing factors: reliance on the exact strings of words from the GSM8K questions (as assumed by [1]), difficulties in object attribute tracking, and limitations of arithmetic nature.

2 Related Work↩︎

Despite long-standing calls for rigorous hypothesis testing in Natural Language Processing (NLP) evaluation [3], [4], the absence of statistical significance testing remains a persistent methodological concern. [5] observed a declining trend in the use of statistical validation of evaluations of LLM-generated text over the 2022-2024 period. [6]’s reevaluation of several advanced prompting techniques revealed a general lack of statistically significant differences when compared to the respective baselines. The authors suggest that the field might be approaching a replication crisis, mirroring similar historical occurrences in psychology [7], [8] and medicine [9], [10].

First described by [11], the language-as-fixed-effect fallacy is a type of statistical error known in psychological and linguistic research, referring to treating specific language materials as fixed rather than random effects. By ignoring the inherent, idiosyncratic variance between individual linguistic items (e.g. particular words or sentences), researchers risk making broad claims about language processing that are actually false positives driven by the specific sample used. To mitigate this fallacy, the field has shifted to mixed-effect modelling of the experimental results [12], e.g. through Generalised Linear Mixed Models (GLMM) [13]. GLMMs extend generalised linear models (e.g. logistic regression) by incorporating both fixed effects - population-level factors of theoretical interest - and random effects that capture variability across items or subjects, enabling statistically valid inference over heterogeneous, non-independent data. More recently, [14] and [15] argued for adopting mixed models in the fields of Natural Language Processing and Generation.

Reliable evaluation of logical and mathematical reasoning capabilities of LLMs has been the motivation behind multiple benchmarks, such as MATH [16], BIG Bench HARD [17], FrontierMath [18], or the subject of the current study, GSM8K [2] with several descendants,

including GSM-Symbolic [1], GSM-Plus [19], GSM-Identity [20], and GSM-Ranges [21]. However, systematic evaluation of LLMs poses a number of challenges, rendering the interpretation and generalisability of the results more nuanced. As demonstrated by [22], LLMs display high sensitivity to prompt formatting, with surface-level modifications linked to potentially vast changes in performance independently of difficulty of the problem to be solved. The effect seems to be independent of model family or scale [22], [23], with changes as trivial as letter case or separator swap drastically changing models’ outputs [24]. Due to impracticality of manual analysis, benchmarks frequently expect strict formatting of the answers. However, literature suggests that formatting constraints may induce a cognitive load collapse (particularly in smaller models), leading to not just incorrect answers, but a complete failure to handle the problems [25], [26]. Furthermore, number processing in LLMs is sensitive to tokenisation mechanisms used [27], [28]. Models struggle to generalise their arithmetic capabilities to larger numbers, expressing which requires more digits [29], [30] - an issue resulting in recent argumentation for single-token number embeddings [31], [32]. Approaches such as Program of Thought [33] and Program-Aided Language Models [34] attempt to circumvent the large-number limitations through instructing models to generate Python code from which the final numerical answer is calculated using an external interpreter. However, as demonstrated by [21] in their high-number-range adaptation of the GSM8K benchmark, offloading the calculations to an external tool might not be sufficient as higher values frequently deteriorate the underlying reasoning processes of a model.

3 Methodology↩︎

This section details the experimental setup underlying our re-evaluation. We begin by defining the datasets, metrics, and models used (Sections 3.1–3.2), before describing the prompt formats designed to probe candidate failure modes (Section 3.3). We then introduce a large number effect metric to separate arithmetic difficulty from reasoning failures per se (Section 3.4), and describe the statistical framework — based on GLMMs with per-question random effects — used to assess significance of the observed effects (Section 3.5). Finally, Section 3.6 presents the research questions and experimental design. The code developed for all experiments and evaluations performed in the scope of this project is available at https://github.com/the-mysh/gsm-symbolic-benchmarking.

3.1 Definitions↩︎

3.1.0.1 Datasets

Our experiments focus on two datasets from the GSM-Symbolic study [1]: GSM-Variants (labelled as main or Symbolic in the study), i.e. the 5000 template-generated questions; and GSM-Base (GSM8K 100 in [1]), i.e. the corresponding 100 originals chosen from [2]. A question from the GSM-Base dataset and its 50 variants from GSM-Variants share the same integer template ID. We refer to the phenomenon of decreased accuracy on GSM-Variants questions (w.r.t. to performance GSM-Base problems) as the variant effect.

3.1.0.2 Metrics

Across the paper, we express average accuracy as a percentage of correctly answered questions. Accuracy deltas (differences in accuracy between two datasets) are expressed in percentage points (pp). We use the term variant performance delta to describe raw change in accuracy, in percentage points, from GSM-Base to GSM-Variants datasets: \(\Delta_{var} = \overline{A}_{GSM-Base} - \overline{A}_{GSM-Variants}\). The accuracies (\(\overline{A}\)) in this context are averaged across the entire respective datasets.

3.2 Models↩︎

The GSM-Symbolic study evaluated the performance of 25 LLMs. For economical and practical reasons, we exclude the four proprietary models (all developed by OpenAI) from our analysis. In addition, we omit Phi-3-small-128k-instruct (Microsoft) due to technical difficulties. Otherwise, we follow the original list, presenting 20 open-weight models for our experiments. The models and limitations can be consulted in Appendix 8.

3.3 Prompt Formats and Failure Modes↩︎

Throughout the work and in accordance with the GSM-Symbolic study, we use 8-shot prompting with the standard GSM8K shots [35], where a shot is a single mathematical problem with a step-by-step solution. First, we reproduce the GSM-Symbolic results using the original prompt format, which we refer to as GSM prompt. In later experiments (see Section 3.6), we modify the solutions to the eight example problems alongside with the prompt introduction line. The exact templates of all prompts used in our experiments, described below, are included in Appendix [apx:full-prompts].

To test whether the models indeed rely on the exact phrasing of the original GSM prompt, we propose a simple rewording of the original answers. We remove the explicit instruction to follow a stepwise approach, but include step introduction phrases (First, calculate..., Next, ..., etc.) where their relevance is justified by the problem structure. We refer to this prompt format as simple natural language (NL) prompt.

Solving grade school problems exemplified by GSM8K requires robust variable binding² and internal state tracking. To isolate the fragility of these capabilities as the possible factors contributing to the variant effect, we propose a structured NL prompt. In addition to the modified phrasing of the simple NL prompt, we include explicit sections with headers: Given, To find, and Solution, detailing all the relevant quantities in the first section and re-stating the calculation target in the second. By introducing these more rigid formatting constraints, we aim to provide a supportive scaffolding, partially offloading the cognitive capabilities to the introduced structure. On the other hand, as discussed before, strict formatting requirements could be a factor negatively influencing performance of smaller models [25], [26].

Finally, we adapt the example answers into two types of code prompts, where the model is asked to provide a valid Python function accepting no arguments, performing the necessary calculations using variables defined inside the function, and returning the calculated result. With these prompts, we aim to investigate arithmetic processing as the potential source of the variant effect. Analogously to the NL prompts described above, we propose two code prompts: a structured code prompt following the same three-part structure as the structured NL prompt, and a simple code prompt, which maximally limits any additions over the standard short explanations of steps taken to solve the problems.

Answers from experiments using the GSM, simple NL, and structured NL prompts are extracted using standard pattern matching: we first attempt to match the target response line (The final answer is) is followed by a number; if this fails, we extract the last number in the response, following GSM-Symbolic. For the code prompts, we attempt to execute the output function via a Python interpreter after safety checks, collecting error type information for non-executing outputs (Appendix 11).

3.4 The Effect of Large Numbers↩︎

We believe it is imperative to separate purely arithmetic difficulties from limitations of mathematical reasoning understood as a process of constructing logically consistent solution procedures, grounded in context and common sense. To this end, we evaluate whether larger numbers in problems’ texts are a predictor of altered performance. We construct a large number effect metric, denoted as \(\gamma\) reflecting the total digit load of a question. For each question, we extract all numerically expressed integers (i.e. we ignore cases such as ‘three’, ‘twice’, ‘half’, and all numerically expressed fractions). We then compute logarithms of base 10 for each number (negative numbers, if any, are taken at their absolute value; for 0, we take the value of 1) and sum the obtained values for a given question. For a given question, \[\gamma = \sum_{n \in \mathcal{N}} log(n)\] where \(\mathcal{N}\) - set of all numerically expressed integers in the question text. This metric captures two effects: firstly, the logarithm assigns unit growth for each added digit, accounting for the number-splitting effect in tokenisation; secondly, larger numbers are expected to appear less in training data and therefore errors in arithmetic operations involving these numbers can be deemed more likely.

We test for this large number effect on combined data from both GSM-Variants and GSM-Base datasets, separately for each LLM. We assess two aspects: (1) whether the large number effect is present (i.e. whether larger numbers are associated with decreased performance); (2) whether, controlling for the number magnitude, variant effect remains significant. The details of the test setup are discussed in Section 3.5. When fitting the statistical model, we use centre \(\gamma\) to zero mean; we denote the centred metric as \(\gamma_c\).

3.5 Statistical Significance Testing↩︎

We model the influence of the different factors on response accuracy using a GLMM. We use a binomial family with logit link, appropriate for the binary correct/incorrect outcome. The choice of the model (GLMM) allows accounting for the non-independence of repeated observations - results for the same question template should not be treated as statistically independent.

We define two instances of the model. With GLMM 1, we model the binary outcome \(Correct\) - whether a given question was solved correctly by a given LLM - as a function of question type (\(Variant\): 0 for GSM-Base, 1 for GSM-Variants) as a fixed effect, with a random intercept (baseline difficulty) per template (\(Id\)). The random intercept accounts for the fact that GSM-Base questions and their GSM-Variants share the same underlying template, making their responses non-independent; it allows the baseline difficulty to vary freely across templates rather than assuming all questions are equally hard. The fixed effect of \(Variant\) then estimates whether belonging to the variant set is associated with a reliable change in accuracy, over and above this template-level variability. In Wilkinson notation [37], [38], GLMM 1 is specified as follows: \[Correct \sim Variant + (1 | Id)\]

Next, we extend GLMM 1 by including the centred large number effect metric \(\gamma_c\) (see Section 3.4) - as a second fixed effect. This yields GLMM 2: \[Correct \sim Variant + \gamma_c + (1 | Id)\] where the estimate for \(\gamma_c\) tests whether numeric load is independently associated with accuracy, and the estimate for \(Variant\) - whether the variant effect persists after controlling for it - since questions from GSM-Variants may differ from their GSM-Base counterparts not only in surface form but in the numerical values they employ.

We fit both models separately for each LLM (see Section 3.2) and prompt format (see Section 3.3). We perform the fitting using a Python library pymer4 [39], wrapping over R’s LME4 [38]. We report the obtained fixed-effect estimates as odds ratios (OR). OR > 1 indicates increased odds of a correct response relative to baseline, OR < 1 decreased odds. In figures, we report log ORs for visual symmetry around zero. The P values corresponding to the estimates serve as an indication of statistical significance of each of the effects.

Note: to evaluate the global significance of the variant and large number effects across all models, we apply the Holm-Bonferroni correction across the evaluated LLMs, separately per effect. We select this over the standard Bonferroni procedure as it offers superior statistical power while maintaining strict control over the family-wise error rate, ensuring that our critique does not unfairly penalize marginal but real effects. We nonetheless report and discuss uncorrected p-values alongside corrected ones, as our primary interest lies in model-specific failure profiles rather than family-wise conclusions.

3.6 Research Questions and Experiments↩︎

We consider three principal research questions.

RQ1. Does the variant effect reported by the GSM-Symbolic study exhibit statistical significance under GLMM?

RQ2. What are the possible failure modes responsible for the observed performance decline?

RQ3. Are models sensitive to larger numbers in the range used in the GSM-Base and GSM-Variants datasets of GSM-Symbolic?

To answer RQ1, we reproduce the GSM-Symbolic experiments (Ex1; model limitations in Section 3.2), fitting GLMM 1 (Section 3.5) per LLM and comparing P values against \(\alpha = 0.05\) to evaluate statistical significance of the variant effect.

For RQ2, models significant under Ex1 are evaluated on the four alternative prompt formats (Ex2.A–D: simple NL, structured NL, simple code, and structured code prompts - see Section 3.3), using GLMM 1 per LLM and prompt format to assess whether the variant effect vanishes.

For RQ3, we reuse raw model evaluation data from Ex2.A to Ex2.D in corresponding experiments Ex3.A to Ex3.D. We fit GLMM 2 (see Section 3.5) to combined GSM-Variants and GSM-Base results (per LLM and prompt format) to evaluate the statistical significance of the large number effect, as well as the residual significance of large-number-effect-corrected variant effect.

4 Results↩︎

In this section, we present the results of our experiments: evaluating statistical significance of results presented in [1] and investigating possible underlying causes of the reported performance decline.

4.1 Reproduction of GSM-Symbolic↩︎

Results of Ex1 are illustrated in Figure 1. Full details are included in Tables 2 and 3 in Appendix 10. Out of 20 models subjected to GLMM evaluation, 10 show statistically significant variant effect when considered individually. For gemma-2-2b, we observe a degenerate result, for which we provide additional analysis in Appendix 10.3.

4.2 Alternative Prompt Formats↩︎

Detailed results for the four alternative prompt formats (Ex2.A - Ex2.D) are shown in Appendix 10.2. For four of the LLMs, the variant effect is rendered statistically insignificant on all four alternative prompt formats. These models are: Phi-3.5-mini-instruct, gemma-2-9b, Mathstral-7B-v0.1, and Mistral-7B-Instruct-v0.1. In further analysis, we focus on the remaining six models, for which we observe a significant variant effect with at least one of the new prompt formats: gemma-2b, gemma-2-2b, gemma-7b-it, phi-2, Meta-Llama-3-8B-Instruct, and Meta-Llama-3-8B-Instruct. The dynamics of the variant effect for these models are shown in a Figure 2. Note: for gemma-2-2b, we again observe degenerate GLMM estimates with the structured NL prompt and structured code prompt.

4.3 The Large Number Effect↩︎

Figure 7 in Appendix 10.4 presents the distributions of numerically expressed integers extracted from questions of the GSM-Base and GSM-Variants dataset of the GSM-Symbolic benchmark. To formally confirm the visible systematic shift towards larger numbers in GSM-Variants, we conduct a two-sample Kolmogorov-Smirnov test on the unrolled occurrences in both datasets. The test confirms a highly statistically significant difference between the underlying distributions (\(D = 0.1201\), \(p < 0.001\)). This result contradicts the cursory assessment made in one of the ablation studies of the GSM-Symbolic paper (Ablation: The Impact of Arithmetic Accuracy, Appendix A.6), where the authors stated: “overall, the range of numbers does not increase significantly” [1].

The results of experiments E3.A to Ex3.D, i.e. of our evaluation of the large number effect on model performance, are shown in Table 14 and Figure 8 in Appendix 10.4. As demonstrated by these results, large numbers are a predictor of decreased performance on pooled GSM-Base and GSM-Variants questions for 6 of the models with the canonical GSM prompt, 8 models with the simple NL prompt, and 7 with the structured NL one. Either of the code prompts removes the significance of the large number effect, except in the case of gemma-2-2b model on the structured code prompt.

The test for residual significance of the variant effect, accounting for the large number effect, yields mixed results - as shown in Table 15 in Appendix 10.4. In just over 50% of cases, the significance of performance delta disappears under the correction. This is the case for Phi-3.5-mini-instruct, Mathstral-7B-v0.1, Meta-Llama-3-8B-Instruct, gemma-7b-it, and Mistral-7B-Instruct-v0.1 (5/10 cases) with GSM prompt; phi-2 (1/2) with simple NL prompt; Meta-Llama-3-8B-Instruct and gemma-2-2b (2/3) with the structured NL prompt; gemma-7b-it and Meta-Llama-3-8B (2/2) with the simple code prompt; and none (0/1) with the structured code prompt. In the remaining 8 instances, the residual variant effect persists despite the large number effect correction.

5 Discussion↩︎

We discuss the implications of our findings concerning the statistical standards of LLM evaluation and model-specific failure profiles.

5.1 The Importance of Statistics↩︎

[1] argue that the observed drop in performance on template versions GSM8K problems indicates lack of genuine reasoning capabilities in the affected models and by extension, in LLMs of the time. Our results suggest that this conclusion might be premature. The variant effect is strikingly less prevalent a phenomenon than implied in the GSM-Symbolic study. On the contrary, we demonstrate that only 10 of the 20 models we re-evaluate display statistically significant variant effect.

Furthermore, our analysis based on the models which individually (pre-correction) exhibit significant variant effect suggests that the influence of number distribution shift should not be ignored when interpreting results of GSM-Symbolic. While sensitivity to large numbers is indubitably an issue to be considered, failures due to arithmetic capabilities should not, in our opinion, be classified as reasoning errors.

5.2 The Failed Estimate↩︎

Appendix 10.3, contains an analysis for the degenerate fits observed across three prompt formats (GSM prompt, structured NL prompt, and structured code prompt) for gemma-2-2b. The model shows a consistent small-to-moderate variant effect (log OR \(\approx\) 0.3 across formats) but standard errors cannot be reliably computed likely due to numerical issues in Hessian inversion, themselves a consequence of the sparse binary data structure. We therefore exclude this model from further analysis.

5.3 Case Study of Failure Modes↩︎

While the variant effect may be indicative of data contamination in the affected models, our results suggest that the underlying failure mechanisms vary. Our analysis reveals several likely phenomena - behaviourally-motivated hypotheses subject to further verification through mechanistic interpretability techniques - described below.

5.3.0.1 Fragility of variable binding

For gemma-2b, the variant effect remains significant under simple rewording of the example solutions (simple NL prompt), but vanishes when the structured NL prompt is used. As discussed, the supportive scaffolding introduced by this prompt likely offloads the difficult (for the model) tasks of associating values with objects and tracking the resulting internal state to the external scratch pad of the generated response.

5.3.0.2 Limited arithmetic capabilities

For phi-2, variant effect remains significant across the NL prompt adaptations, but disappears entirely with the code prompts, where the otherwise significant large number effect also vanishes. For Phi-3.5-mini-instruct and Mistral-7B-Instruct-v0.1, for which the variant effect vanishes with all proposed prompt adaptations, the drop in accuracy on the GSM prompt can be fully explained by sensitivity to large numbers. For these models, arithmetic capabilities - and not lack of reasoning - seem to be the primary factor contributing to the variant effect.

5.3.0.3 Reliance on learnt patterns

Models gemma-2-9b and Mathstral-7B-v0.1 also exhibit the variant effect only with the original GSM prompt, but unlike the aforementioned Phi-3.5-mini and Mistral-7B-Instruct, are fully or partly immune to the number effect - possibly owing to the larger capacity (Gemma with 9B parameters) or training specific for mathematical reasoning (Mathstral). For these two models, we do not find an explanation for the original variant effect other than replication of patterns learnt from training data.

5.3.0.4 Dual-task interference

For Meta-Llama-3-8B, NL perturbations (simple NL prompt, structured NL prompt) are associated with a strong decrease in accuracy and, as shown in Fig. 9, a large number of empty answers. This behaviour can be interpreted in terms of dual-task inference, mirroring a phenomenon known in psychology [40]. Attempting to follow out-of-distribution formatting requirements while simultaneously solving a mathematical problem, the model fails at both tasks. Similar effects on a smaller scale seem to occur for two other models, in each case under different conditions. For Meta-Llama-3-8B-Instruct, the variant effect resurfaces on the structured NL prompt; on the other hand, with the structured code prompt, offloading the arithmetic operations to an external tool appears sufficient to counter-balance the effect of length or strict formatting. For gemma-7b-it, the picture is reverted, but describing a similar underlying mechanism; following the requirements of code syntax exhausts the working memory, leading to significant variant effect on simple code prompt. However, supportive scaffolding introduced by the structured NL prompt appears to restore the model’s relative performance.

5.3.0.5

Overall, our results demonstrate the diversity of possible root causes for failures on reasoning benchmarks and highlight the inappropriateness of blanket statements when analysing the outcomes.

6 Conclusions↩︎

This work presents a re-evaluation of the GSM-Symbolic benchmark, and its primary message is methodological: claims about the reasoning capabilities - or incapabilities - of LLMs demand statistical rigour. The widespread absence of significance testing in LLM evaluation is not merely an academic concern; it shapes research agendas, informs policy, and influences public understanding of AI systems.

Our results challenge the central claim of [1]. Applying GLMM with appropriate random effects for item-level variability, we find that the variant effect is statistically significant for only 10 of 20 re-evaluated models. For the models showing significance pre-correction, we demonstrate that a systematic shift towards larger integers in the main GSM-Symbolic dataset constitutes a previously overlooked confound, partially or fully explaining the decline in several cases.

Crucially, for models whose variant effect proves robust to prompt perturbations, no single failure mode prevails. Instead, we observe a diverse landscape: limitations in arithmetic capabilities, fragile variable binding, and dual-task interference - the compounding of out-of-distribution entity processing with rigid structural constraints. Only three of the LLMs seem to purely rely on patterns learnt from their training data. The heterogeneity of the results underscores the importance of careful, case-based investigation, possibly supported by mechanistic understanding, in evaluating or speculating about cognitive capabilities of LLMs. We hope this work contributes to a more epistemically careful discourse around LLM evaluation, one that treats statistical validation not as an optional refinement, but as a prerequisite for credible claims.

Limitations↩︎

While our re-evaluation provides a statistically rigorous perspective on the GSM-Symbolic benchmark, our methodology has several constraints that invite further investigation.

6.0.0.1 Simplification of the large number effect metric

The metric we introduce to quantify arithmetic difficulty (\(\gamma\), see Section 3.4) acts as a proxy for total number load. However, it only captures the magnitude of integers present in questions. It does not account for the presence of fractions, decimals, the complexity of specific arithmetic operations (e.g., carrying over in addition, division by primes), or the nuances of model-specific tokenization boundaries, all of which are known to impact arithmetic performance. Our chosen heuristics offers a balance between capturing two major expected confounds and the overall clarity of presentation (i.e. a single, expressive metric for the discussed class of numerical issues). Further work could devise individual metrics isolating the possible contributing effects.

6.0.0.2 Granularity of prompt perturbations

For models such as Phi-3.5-mini-instruct, gemma-2-9b, Mathstral-7B-v0.1, and Mistral-7B-Instruct-v0.1, the variant effect vanishes under all tested prompt variations. While we interpret this as a heavy reliance on superficial pattern matching, we cannot rule out extreme sensitivity to formatting. Further studies utilizing small-scale perturbations are required to isolate pattern matching from general format fragility.

6.0.0.3 Behavioural abstraction of cognitive mechanisms

Our hypotheses regarding working memory exhaustion, dual-task interference, and the fragility of variable binding are inferred from black-box behavioural evaluations. While these cognitive aspects align with the observed performance deltas across structured versus programmatic prompts, validating the exact root causes of these failures would require white-box interpretability techniques, such as activation patching or circuit tracing, to map how specific attention heads route out-of-distribution entities.

6.0.0.4 Scope of evaluated models

For practical and economic reasons, our evaluation is restricted to open-weight models, all under 30 billion parameters. We excluded frontier proprietary models. Moreover, with reproduction of the GSM-Symbolic study as our primary motivation, we do not include more recent LLMs. Consequently, our findings regarding format sensitivity and the statistical insignificance of the variant effect may not fully generalize to the state-of-the-art models that typically define upper-bound reasoning capabilities. However, the methodological critique, emphasising the necessity of statistical significance testing with appropriate tools and of controlling for dataset distribution shifts, applies universally to all models evaluated on this benchmark.

Ethical Considerations↩︎

6.0.0.5 Epistemological responsibility and AI discourse

This work challenges definitive claims regarding the presence or absence of “genuine reasoning” capabilities in Large Language Models. We emphasise the ethical necessity of rigorous statistical validation before making broad assertions about AI capabilities. Prematurely concluding that models lack reasoning abilities - or conversely, overestimating their abilities based on contaminated benchmarks - can misinform public understanding, guide flawed policy decisions, and skew future safety research. Our findings advocate for a more nuanced, statistically grounded discourse regarding model evaluation.

6.0.0.6 Use of cognitive metaphors

Throughout this paper, we employ terminology derived from cognitive science and psychology, such as “working memory”, “cognitive load”, and “dual-task interference”. We explicitly state that these terms are used as functional metaphors to describe mechanistic and architectural phenomena within LLMs, such as variable binding and internal state tracking. Their use is intended to provide an intuitive framework for understanding failure modes, and should not be interpreted as an assertion that these models possess human-like sentience, biological cognition, or consciousness.

6.0.0.7 Computational and environmental cost

We acknowledge the environmental impact associated with our experimental methodology. Re-evaluating the GSM-Symbolic benchmark required running inference across 20 distinct LLMs on thousands of prompt variations. While this incurs a notable compute and carbon footprint, we believe this cost is justified by the critical need to validate high-impact benchmark results and prevent a potential replication crisis within the Natural Language Processing community.

6.0.0.8 Use of AI assistance

In this work, we used the assistance of AI chat bots (Claude, Gemini) to identify the relevant sources cited across the article and in particular those mentioned in the Related Work section, as well as to help with concise phrasing of the text. In addition, coding agents (GitHub Copilot) were employed to help construct model evaluation, data analysis, and visualisation scripts.

Appendix↩︎

7 Artifacts↩︎

7.1 Software and Infrastructure↩︎

Model inference and evaluation pipelines were implemented in Python utilizing the PyTorch framework [41] and the Hugging Face transformers library [42], alongside datasets [43] for benchmark data management. Data manipulation and preprocessing were conducted using pandas [44] and numpy [45].

All statistical hypothesis testing, including the formulation of GLMMs, was executed using pymer4 [39] - which interfaces with the R lme4 package [38] - as well as scipy [46] and statsmodels [47]. Visualizations were generated relying on matplotlib [48] and seaborn [49].

7.2 Computational Experiments↩︎

The evaluations were performed on NVIDIA A40 and NVIDIA V100 GPUs. The total computational budget required to run the full evaluation pipeline across all 20 models and prompt variations is estimated at approximately 1,000 GPU hours. The evaluated open-weight models range in size from 2 billion to 27 billion parameters, with exact model versions and sizes specified in Appendix 8.

7.3 Artifact Licences and Terms of Use↩︎

The artifacts utilized in this study are publicly available for research purposes. The GSM8K dataset [2] is distributed under the MIT licence and the GSM-Symbolic benchmark [1] - under CC-BY-NC-ND 4.0. The evaluated Large Language Models are subject to their respective open-weight licenses: models from the Mistral and Mathstral families are released under the Apache 2.0 License; the Microsoft Phi models are released under the MIT License; the Google Gemma models are governed by the Gemma Terms of Use; and the Meta-Llama-3 models are governed by the custom Meta Llama 3 Community License. All third-party software packages and libraries used for inference and statistical analysis (e.g., PyTorch, Hugging Face transformers, pymer4) are open-source and utilized in accordance with their respective permissive licenses (primarily MIT, BSD, and Apache 2.0). All models, datasets, and software were used strictly for research and evaluation purposes. The custom evaluation code and data analysis scripts developed for this study will be released under the MIT License upon publication to facilitate reproducibility.

8 Model Specifications↩︎

In our experiments, we use models downloaded from the Hugging Face repository [42]. Table 1 lists the models we evaluate together with the corresponding checkpoints (commit ids) for reproducibility. When reporting the results throughout the paper, we omit model family names for brevity.

Models from the GSM-Symbolic paper which were omitted in the current study:

• microsoft/Phi-3-small-128k-instruct

• gpt-4o-mini

• gpt-4o

• o1-mini

• o1-preview

We did not evaluate closed models (GPT-4o, o1) because of technical and economical constraints. For microsoft/Phi-3-small-128k-instruct, configuring Flash Attention on our hardware proved impassable despite a considerable time investment.

9 Prompt Formats↩︎

In this section, we present the full prompt formats used in our experiments. See Section 3.3 for more details.

GSM Prompt↩︎

To facilitate comparison with the original prompt format from GSM8K and GSM-Symbolic studies, we reproduce here the full 8-shot GSM prompt, as specified in EleutherAI LM-Harness framework [35].

As an expert problem solver, solve step by step the following mathematical questions.

Q: There are 15 trees in the grove. Grove workers will plant trees in the grove today. After they are done, there will be 21 trees. How many trees did the grove workers plant today?
A: Let's think step by step. There are 15 trees originally. Then there were 21 trees after some more were planted. So there must have been 21 - 15 = 6. The final answer is 6.

Q: If there are 3 cars in the parking lot and 2 more cars arrive, how many cars are in the parking lot?
A: Let's think step by step. There are originally 3 cars. 2 more cars arrive. 3 + 2 = 5. The final answer is 5.

Q: Leah had 32 chocolates and her sister had 42. If they ate 35, how many pieces do they have left in total?
A: Let's think step by step. Originally, Leah had 32 chocolates. Her sister had 42. So in total they had 32 + 42 = 74. After eating 35, they had 74 - 35 = 39. The final answer is 39.

Q: Jason had 20 lollipops. He gave Denny some lollipops. Now Jason has 12 lollipops. How many lollipops did Jason give to Denny?
A: Let's think step by step. Jason started with 20 lollipops. Then he had 12 after giving some to Denny. So he gave Denny 20 - 12 = 8. The final answer is 8.

Q: Shawn has five toys. For Christmas, he got two toys each from his mom and dad. How many toys does he have now?
A: Let's think step by step. Shawn started with 5 toys. If he got 2 toys each from his mom and dad, then that is 4 more toys. 5 + 4 = 9. The final answer is 9.

Q: There were nine computers in the server room. Five more computers were installed each day, from monday to thursday. How many computers are now in the server room?
A: Let's think step by step. There were originally 9 computers. For each of 4 days, 5 more computers were added. So 5 * 4 = 20 computers were added. 9 + 20 is 29. The final answer is 29.

Q: Michael had 58 golf balls. On tuesday, he lost 23 golf balls. On wednesday, he lost 2 more. How many golf balls did he have at the end of wednesday?
A: Let's think step by step. Michael started with 58 golf balls. After losing 23 on tuesday, he had 58 - 23 = 35. After losing 2 more, he had 35 - 2 = 33 golf balls. The final answer is 33.

Q: Olivia has $23. She bought five bagels for $3 each. How much money does she have left?
A: Let's think step by step. Olivia had 23 dollars. 5 bagels for 3 dollars each will be 5 x 3 = 15 dollars. So she has 23 - 15 dollars left. 23 - 15 is 8. The final answer is 8.

Q: <Question here>
A: Let's think step by step. 

Simple NL Prompt↩︎

As an expert problem solver, solve the following mathematical questions.


Q: There are 15 trees in the grove. Grove workers will plant trees in the grove today. After they are done, there will be 21 trees. How many trees did the grove workers plant today?
A: 
Calculate the difference between trees after planting and trees currently in the grove:
21 - 15 = 6

The final answer is 6.


Q: If there are 3 cars in the parking lot and 2 more cars arrive, how many cars are in the parking lot?
A: 
Calculate the sum of cars originally in the parking lot and the arriving cars:
3 + 2 = 5

The final answer is 5.


Q: Leah had 32 chocolates and her sister had 42. If they ate 35, how many pieces do they have left in total?
A: 
First, calculate how many chocolates the sisters originally had in total:
32 + 42 = 74
Next, subtract the total number of chocolates they ate:
74 - 35 = 39

The final answer is 39.


Q: Jason had 20 lollipops. He gave Denny some lollipops. Now Jason has 12 lollipops. How many lollipops did Jason give to Denny?
A: 
Calculate the difference between the lollipops Jason originally had and the lollipops he has now:
20 - 12 = 8

The final answer is 8.


Q: Shawn has five toys. For Christmas, he got two toys each from his mom and dad. How many toys does he have now?
A: 
First, calculate how many toys in total Shawn got for Christmas:
2 * 2 = 4
Next, calculate the sum of the number of toys Shawn originally had and the number of new toys he got:
5 + 4 = 9

The final answer is 9.


Q: There were nine computers in the server room. Five more computers were installed each day, from monday to thursday. How many computers are now in the server room?
A: 
First, calculate how many new computers were installed in total:
4 * 5 = 20
Next, calculate the sum of the number of computers originally in the room and the total number of new computers installed:
9 + 20 = 29

The final answer is 29.


Q: Michael had 58 golf balls. On tuesday, he lost 23 golf balls. On wednesday, he lost 2 more. How many golf balls did he have at the end of wednesday?
A: 
First, calculate the total number of golf balls Michael lost:
23 + 2 = 25
Next, calculate the difference between the original number of golf balls and the number of golf balls lost:
58 - 25 = 33

The final answer is 33.


Q: Olivia has $23. She bought five bagels for $3 each. How much money does she have left?
A: 
First, calculate the total cost of the bagels:
5 * 3 = 15
Next, calculate the difference between Olivia's original money and what she paid for the bagels:
23 - 15 = 8

The final answer is 8.


Q: <Question here>
A: 

Structured NL Prompt↩︎

As an expert problem solver, solve the following mathematical questions, detailing the given data, quantity to find, and necessary calculations.


Q: There are 15 trees in the grove. Grove workers will plant trees in the grove today. After they are done, there will be 21 trees. How many trees did the grove workers plant today?
A: 
Given:
Number of trees currently in the grove: 15
Number of trees in the grove after the workers are done planting today: 21

To find: number of trees the grove workers will plant today.

Solution:
Calculate the difference between trees after planting and trees currently in the grove:
21 - 15 = 6

The final answer is 6.


Q: If there are 3 cars in the parking lot and 2 more cars arrive, how many cars are in the parking lot?
A: 
Given:
Number of cars originally in the parking lot: 3
Number of cars arriving in the parking lot: 2
    
To find: total number of cars in the parking lot.

Solution:
Calculate the sum of cars originally in the parking lot and the arriving cars:
3 + 2 = 5

The final answer is 5.


Q: Leah had 32 chocolates and her sister had 42. If they ate 35, how many pieces do they have left in total?
A: 
Given:
Number of chocolates Leah originally had: 32
Number of chocolates Leah's sister originally had: 42
Number of chocolates Leah and her sister ate: 35

To find: total number of chocolates left.

Solution:
First, calculate how many chocolates the sisters originally had in total:
32 + 42 = 74
Next, subtract the total number of chocolates they ate:
74 - 35 = 39

The final answer is 39.


Q: Jason had 20 lollipops. He gave Denny some lollipops. Now Jason has 12 lollipops. How many lollipops did Jason give to Denny?
A: 
Given:
Number of lollipops Jason had originally: 20
Number of lollipops Jason has left: 12

To find: number of lollipops Jason gave to Denny.

Solution:
Calculate the difference between the lollipops Jason originally had and the lollipops he has now:
20 - 12 = 8

The final answer is 8.


Q: Shawn has five toys. For Christmas, he got two toys each from his mom and dad. How many toys does he have now?
A: 
Given:
Number of toys Shawn originally has: 5
Number of toys his mom and his dad each gave him for Christmas: 2

To find: total number of toys Shawn has now.

Solution:
First, calculate how many toys in total Shawn got for Christmas:
2 * 2 = 4
Next, calculate the sum of the number of toys Shawn originally had and the number of new toys he got:
5 + 4 = 9

The final answer is 9.


Q: There were nine computers in the server room. Five more computers were installed each day, from monday to thursday. How many computers are now in the server room?
A: 
Given:
Number of computers originally in the server room: 9
Number of new computers installed each day: 5
Number of days from monday to thursday: 4

To find: total number of computers in the server room now.

Solution:
First, calculate how many new computers were installed in total:
4 * 5 = 20
Next, calculate the sum of the number of computers originally in the room and the total number of new computers installed:
9 + 20 = 29

The final answer is 29.


Q: Michael had 58 golf balls. On tuesday, he lost 23 golf balls. On wednesday, he lost 2 more. How many golf balls did he have at the end of wednesday?
A: 
Given:
Number of golf balls Michael originally had: 58
Number of golf balls lost on tuesday: 23
Number of golf balls lost on wednesday: 2

To find: number of golf balls left at the end of wednesday.

Solution:
First, calculate the total number of golf balls Michael lost:
23 + 2 = 25
Next, calculate the difference between the original number of golf balls and the number of golf balls lost:
58 - 25 = 33

The final answer is 33.


Q: Olivia has $23. She bought five bagels for $3 each. How much money does she have left?
A: 
Given:
Number of dollars Olivia had originally: 23
Number of bagels Olivia bought: 5
Number of dollars each bagel cost: 3

To find: how many dollars Olivia has left.

Solution:
First, calculate the total cost of the bagels:
5 * 3 = 15
Next, calculate the difference between Olivia's original money and what she paid for the bagels:
23 - 15 = 8

The final answer is 8.


Q: <Question here>
A: 

Simple Code Prompt↩︎

As an expert problem solver and Python developer, write Python functions returning numerical solutions to the following mathematical questions.

Q: There are 15 trees in the grove. Grove workers will plant trees in the grove today. After they are done, there will be 21 trees. How many trees did the grove workers plant today?

A:
def solution():
    trees_now = 15
    trees_after_planting = 21

    # calculate the difference between trees after planting and trees originally
    trees_to_plant_today = trees_after_planting - trees_now
    return trees_to_plant_today


Q: If there are 3 cars in the parking lot and 2 more cars arrive, how many cars are in the parking lot?

A:
def solution():
    cars_before = 3
    cars_arriving = 2

    # calculate the sum of cars originally in the parking lot and the arriving cars
    cars_after = cars_before + cars_arriving
    return cars_after


Q: Leah had 32 chocolates and her sister had 42. If they ate 35, how many pieces do they have left in total?

A:
def solution():
    chocolates_leah_before = 32
    chocolates_sister_before = 42
    chocolates_eaten = 35

    # first, calculate how many chocolates the sisters originally had in total
    chocolates_total_before = chocolates_leah_before + chocolates_sister_before
    # next, subtract the total number of chocolates they ate
    chocolates_left = chocolates_total_before - chocolates_eaten
    return chocolates_left


Q: Jason had 20 lollipops. He gave Denny some lollipops. Now Jason has 12 lollipops. How many lollipops did Jason give to Denny?

A:
def solution():
    lollipops_jason_before = 20
    lollipops_jason_now = 12

    # calculate the difference between lollipops Jason had originally had lollipops he has now
    lollipops_given_to_denny = lollipops_jason_before - lollipops_jason_now
    return lollipops_given_to_denny


Q: Shawn has five toys. For Christmas, he got two toys each from his mom and dad. How many toys does he have now?

A:
def solution():
    shawn_toys_before = 5
    new_toys_from_each_parent = 2

    # first, calculate how many toys in total Shawn got for Christmas
    new_toys_total = 2 * new_toys_from_each_parent
    # next, calculate the sum of the number of toys Shawn originally had and the number of new toys he got
    shawn_toys_now = shawn_toys_before + new_toys_total
    return shawn_toys_now


Q: There were nine computers in the server room. Five more computers were installed each day, from monday to thursday. How many computers are now in the server room?

A:
def solution():
    computers_before = 9
    computers_installed_per_day = 5
    number_of_days = 4

    # first, calculate how many new computers were installed in total
    computers_installed_total = number_of_days * computers_installed_per_day
    # next, calculate sum of the number of computers originally in the room and the total number of new computers installed
    computers_now = computers_before + computers_installed_total
    return computers_now


Q: Michael had 58 golf balls. On tuesday, he lost 23 golf balls. On wednesday, he lost 2 more. How many golf balls did he have at the end of wednesday?

A:
def solution():
    golf_balls_before = 58
    golf_balls_lost_tuesday = 23
    golf_balls_lost_wednesday = 2

    # first, calculate the total number of golf balls Michael lost
    golf_balls_lost_total = golf_balls_lost_tuesday + golf_balls_lost_wednesday
    # next, calculate the difference between the original number of golf balls and the number of golf balls lost
    golf_balls_left = golf_balls_before - golf_balls_lost_total
    return golf_balls_left


Q: Olivia has $23. She bought five bagels for $3 each. How much money does she have left?

A:
def solution():
    dollars_before = 23
    bagels = 5
    price_per_bagel = 3

    # first, calculate the total cost of the bagels
    bagels_cost_total = bagels * price_per_bagel
    # next, calculate the difference between Olivia's original money and what she paid for the bagels
    dollars_left = dollars_before - bagels_cost_total
    return dollars_left


Q: <Question here>

A:
def solution():

Structured Code Prompt↩︎

As an expert problem solver and Python developer, write Python functions returning numerical solutions to the following mathematical questions.

Q: There are 15 trees in the grove. Grove workers will plant trees in the grove today. After they are done, there will be 21 trees. How many trees did the grove workers plant today?

A:
def solution():
    # given:
    trees_now = 15  # number of trees currently in the grove
    trees_after_planting = 21  # number of trees in the grove after the workers are done planting today

    # to find: number of trees the grove workers will plant today

    # solution:
    # calculate the difference between trees after planting and trees originally
    trees_to_plant_today = trees_after_planting - trees_now
    return trees_to_plant_today


Q: If there are 3 cars in the parking lot and 2 more cars arrive, how many cars are in the parking lot?

A:
def solution():
    # given:
    cars_before = 3  # number of cars originally in the parking lot
    cars_arriving = 2  # number of cars arriving in the parking lot

    # to find: total number of cars in the parking lot

    # solution:
    # calculate the sum of cars originally in the parking lot and the arriving cars
    cars_after = cars_before + cars_arriving
    return cars_after


Q: Leah had 32 chocolates and her sister had 42. If they ate 35, how many pieces do they have left in total?

A:
def solution():
    # given:
    chocolates_leah_before = 32  # number of chocolates Leah originally had
    chocolates_sister_before = 42  # number of chocolates Leah's sister originally had
    chocolates_eaten = 35  # number of chocolates Leah and her sister ate

    # to find: total number of chocolates left

    # solution:
    # first, calculate how many chocolates the sisters originally had in total
    chocolates_total_before = chocolates_leah_before + chocolates_sister_before
    # next, subtract the total number of chocolates they ate
    chocolates_left = chocolates_total_before - chocolates_eaten
    return chocolates_left


Q: Jason had 20 lollipops. He gave Denny some lollipops. Now Jason has 12 lollipops. How many lollipops did Jason give to Denny?

A:
def solution():
    # given:
    lollipops_jason_before = 20  # number of lollipops Jason had originally
    lollipops_jason_now = 12  # number of lollipops Jason has left

    # to find: number of lollipops Jason gave to Denny

    # solution:
    # calculate the difference between lollipops Jason had originally had lollipops he has now
    lollipops_given_to_denny = lollipops_jason_before - lollipops_jason_now
    return lollipops_given_to_denny


Q: Shawn has five toys. For Christmas, he got two toys each from his mom and dad. How many toys does he have now?

A:
def solution():
    # given:
    shawn_toys_before = 5  # number of toys Shawn originally has
    new_toys_from_each_parent = 2  # number of toys his mom and his dad each gave him for Christmas

    # to find: total number of toys Shawn has now

    # solution:
    # first, calculate how many toys in total Shawn got for Christmas
    new_toys_total = 2 * new_toys_from_each_parent
    # next, calculate the sum of the number of toys Shawn originally had and the number of new toys he got
    shawn_toys_now = shawn_toys_before + new_toys_total
    return shawn_toys_now


Q: There were nine computers in the server room. Five more computers were installed each day, from monday to thursday. How many computers are now in the server room?

A:
def solution():
    # given:
    computers_before = 9  # number of computers originally in the server room
    computers_installed_per_day = 5  # number of new computers installed each day
    number_of_days = 4  # number of days from monday to thursday

    # to find: total number of computers in the server room now

    # solution:
    # first, calculate how many new computers were installed in total
    computers_installed_total = number_of_days * computers_installed_per_day
    # next, calculate sum of the number of computers originally in the room and the total number of new computers installed
    computers_now = computers_before + computers_installed_total
    return computers_now


Q: Michael had 58 golf balls. On tuesday, he lost 23 golf balls. On wednesday, he lost 2 more. How many golf balls did he have at the end of wednesday?

A:
def solution():
    # given:
    golf_balls_before = 58  # number of golf balls Michael originally had
    golf_balls_lost_tuesday = 23  # number of golf balls lost on tuesday
    golf_balls_lost_wednesday = 2  # number of golf balls lost on wednesday

    # to find: number of golf balls left at the end of wednesday

    # solution:
    # first, calculate the total number of golf balls Michael lost
    golf_balls_lost_total = golf_balls_lost_tuesday + golf_balls_lost_wednesday
    # next, calculate the difference between the original number of golf balls and the number of golf balls lost
    golf_balls_left = golf_balls_before - golf_balls_lost_total
    return golf_balls_left


Q: Olivia has $23. She bought five bagels for $3 each. How much money does she have left?

A:
def solution():
    # given:
    dollars_before = 23  # number of dollars Olivia had originally
    bagels = 5  # number of bagels Olivia bought
    price_per_bagel = 3  # number of dollars each bagel cost

    # to find: how many dollars Olivia has left?

    # solution:
    # first, calculate the total cost of the bagels
    bagels_cost_total = bagels * price_per_bagel
    # next, calculate the difference between Olivia's original money and what she paid for the bagels
    dollars_left = dollars_before - bagels_cost_total
    return dollars_left


Q: <Question here>

A:
def solution():

10 Supplementary Results↩︎

In this section, we present supplementary figures and tables detailing the results of our variant effect and large number effect testing experiments.

10.1 Ex1: Reproducing the GSM-Symbolic Benchmark↩︎

Tables 2 and 3 show mean accuracies, variant performance deltas, and results corresponding statistical significance testing following our reproduction of the GSM-Symbolic benchmark. The variant effect is statistically significant in 50% of the tested models (10% if applying the Holm-Bonferroni correction; see Section 3.5).

Tables 4 and 5 include comparison of our results with those reported by [1]. The results differ up to 22 percentage points (pp) on GSM-Base, 27.16 percentage points on GSM-Variants, and 24.1 percentage points in variant performance deltas. However, no systematic shift can be seen; the differences of variant performance deltas (i.e. delta of deltas, ours vs Mirzadeh) has a mean of -0.54 percentage points, median of -0.8, and standard deviation of 8.34. There are multiple factors possibly contributing to the observed differences. A primary culprit is likely model versioning - the exact model versions on Hugging Face might have changed since [1]’s study was published; the authors did not mention the exact checkpoints or revision hashes and we defaulted to the most recent versions available. Model sensitivity to minor prompt formatting and decoding could also be a source of differences if despite trying to reproduce the study as closely as possible, we missed fine details such as trailing white spacing in the prompts.

10.2 Alternative Prompt Formats↩︎

This section presents supplementary results for variant effect testing when using the four alternative prompt formats (see Section 3.3).

10.2.1 Ex2.A: Simple NL Prompt↩︎

Figure 3, as well as Tables 6 and 7, present results of Ex2.A, i.e. variant effect testing on the simple NL prompt (see Section 3.3 for prompt formats definitions). This adaptation of the original prompt format removes significance of variant performance delta in all but two of the models: phi-2 and gemma-2b.

10.2.2 Ex2.B: Structured NL Prompt↩︎

Figure 4, supported by Tables 6 and Table 7, depicts the results for experiment Ex2.B: structured NL prompt (see Section 3.3 for prompt formats definitions). The variant performance delta remains statistically significant for phi-2 and resurfaces as significant for Meta-Llama-3-8B-Instruct and gemma-2-2b.

10.2.3 Ex2.C: Simple Code Prompt↩︎

Variant performance delta results for the simple code prompt (see Section 3.3) are shown in Figure 5 and Tables 10 and 11. The variant effect is significant for gemma-7b-it, but also Meta-Llama-3-8B which in this case showed a statistically significant improvement on GSM-Variants vs GSM-Base data (disappearing under the Holm-Bonferroni correction).

10.2.4 Ex2.D: Structured Code Prompt↩︎

Figure 6 and Tables 12 and 13 present variant effect testing results for the structured code prompt (see Section 3.3). Variant effect is significant for one model - gemma-2-2b.

10.3 Additional Information on the Degenerate gemma-2-2b Fit↩︎

As mentioned in Sections 4.1 and 4.2, GLMM 1 results on gemma-2-2b are degenerate on the GSM prompt, structured NL prompt, and structured code prompt. Our further analysis shows that the coefficient estimates for gemma-2-2b are consistent and stable across all optimizers available in lme4 and all prompt formats (variant coefficient ranging from -0.24 to -0.45 log-odds across formats), as shown by the following results report:

GSM prompt
                              (Intercept) is_variant
bobyqa                          -2.358318 -0.3157934
Nelder_Mead                     -2.358322 -0.3157909
nlminbwrap                      -2.358319 -0.3157938
nloptwrap.NLOPT_LN_NELDERMEAD   -2.358201 -0.3158539
nloptwrap.NLOPT_LN_BOBYQA       -2.405811 -0.3077173

====================

Simple NL prompt
                              (Intercept) is_variant
bobyqa                          -2.968160 -0.2415366
Nelder_Mead                     -2.968176 -0.2415160
nlminbwrap                      -2.968165 -0.2415333
nloptwrap.NLOPT_LN_NELDERMEAD   -2.968158 -0.2413823
nloptwrap.NLOPT_LN_BOBYQA       -2.968152 -0.2415481

====================

Structured NL prompt
                              (Intercept) is_variant
bobyqa                          -2.006092 -0.4461784
Nelder_Mead                     -2.006096 -0.4461799
nlminbwrap                      -2.006091 -0.4461776
nloptwrap.NLOPT_LN_NELDERMEAD   -2.062258 -0.4470053
nloptwrap.NLOPT_LN_BOBYQA       -2.044976 -0.4464479

====================

Simple code prompt
                              (Intercept) is_variant
bobyqa                          -2.252103 -0.2725128
Nelder_Mead                     -2.252106 -0.2725081
nlminbwrap                      -2.252113 -0.2725396
nloptwrap.NLOPT_LN_NELDERMEAD   -2.251930 -0.2726682
nloptwrap.NLOPT_LN_BOBYQA       -2.252078 -0.2725294

====================

Structured code prompt
                              (Intercept) is_variant
bobyqa                          -2.383680 -0.2927124
Nelder_Mead                     -2.469397 -0.2791749
nlminbwrap                      -2.383678 -0.2927097
nloptwrap.NLOPT_LN_NELDERMEAD   -2.423186 -0.2933606
nloptwrap.NLOPT_LN_BOBYQA       -2.438184 -0.2782190

The estimates are unreliable for the three formats likely due to Hessian issues, not due to optimizer failure or identifiability problems. However, the underlying effect size is real and consistent; it is only the inferential statistics that should be deemed unreliable.

We acknowledge that alternative estimation frameworks—such as Bayesian GLMMs employing weakly informative priors or frequentist approaches utilizing Firth’s penalized likelihood—could theoretically regularize the fit and yield bounded uncertainty estimates for this degenerate case. However, applying a fundamentally different statistical regularization framework to rescue a single model’s estimates would introduce methodological inconsistency into our benchmark reproduction. Therefore, to maintain a rigorously uniform frequentist evaluation standard across all evaluated LLMs, we conservatively elect to report the degenerate fit and exclude gemma-2-2b from our primary inferential analyses.

10.4 The Large Number Effect↩︎

Figure 7 visualises the difference in distribution of integers occurring in questions from the GSM-Base and GSM-Variants datasets. In the latter, the distribution is shifted towards larger numbers with K-S statistic of 12 percentage points and high statistical significance (\(p < 0.001\)), as reported in Section 3.4.

Tables 14 and 15 present the results of the large number effect significance testing.

Figure 8 summarises the results of the large number effect test (see Sections 3.4 and 3.5).

11 Failure Modes↩︎

In both natural-language and code prompt experiments, we predefine classes of errors and issues we anticipated to see when evaluating LLMs answers - in addition to miscalculation errors (where an incorrect numerical answer is returned; in the visualisations which follow, this is marked as Wrong answer). For natural language prompts, the classes are:

• Empty response - if the response is empty (zero-length);

• Empty (after trimming) - if the response only contains whitespaces and, optionally, the ‘Q:’ token, indicating that the model is starting to generate a new question following the few-shot format;

• No number found - if all of our numerical answer extraction methods fails on a given answer;

• Wong answer - last number - if the incorrect numerical answer is also associated with the model not following the few-shot formatting guidelines, resulting in the answer being extracted through a simple regular expression capturing the last number in the text.

For code prompts, we define the following classes:

• No function - if a function definition can not be found in the model’s answer using a simple regular expression approach;

• Forbidden string - if the answer contains any string with potentially dangerous effects if executed, such as open(...) or eval(...);

• Syntax error - if executing the extracted function results in a SyntaxError;

• Name error - if a NameError is raised;

• Type/Value error - if a TypeError or ValueError is raised;

• Zero division error - if a ZeroDivisionError is raised;

• Attribute error - if an AttributeError is raised;

• None returned - if the function does not return any value, either by skipping a return line, using empty return, or returning None explicitly;

• Not a number - if the value returned from the function is not numerical and other than None;

• Unclassified - all remaining error cases encountered when running the extracted function.

11.1 Failure Modes in NL Prompts↩︎

As shown in Figure 9, incorrect numerical response constitutes the overwhelming majority of errors in NL prompts experiments. There is also a considerable portion of the last-number numerical results (in total: 9% in the GSM prompt; 4% in the simple NL prompt; 5% in the structured NL prompt). The failed formatting cases varies in the underlying patterns. In some cases, the models indeed simply “forget” to introduce the answer with the required line (The final answer is...). In other cases, as shown below, the lack of adherence to formatting is a result of a cognitive failure, with the model getting lost in a potentially infinite loop of operations, often resulting in unreasonably high numerical answers.

The first paid hour is $13. The second paid hour is $13 * 2 = $26. The third paid hour is $26 * 2 = $52. The fourth paid hour is $52 * 2 = $104. The fifth paid hour is $104 * 2 = $208. The sixth paid hour is $208 * 2 = $416. The seventh paid hour is $416 * 2 = $832. The eighth paid hour is $832 * 2 = $1664. The ninth paid hour is $1664 * 2 = $3328. The tenth paid hour is $3328 * 2 = $6656. The eleventh paid hour is $6656 * 2 = $13112. The twelfth paid hour is $13112 * 2 = $26224. The thirteenth paid hour is $26224 * 2 = $52448. The fourteenth paid hour is $52448 * 2 = $104896. The fifteenth paid hour is $104896 * 2 = $2087328. The sixteenth paid hour is $2087328 * 2 = $41657656. The seventeenth paid hour is $41657656 * 2 = $81315352. The eighteenth paid hour is $81315352 * 2 = $162630704. The nineteenth paid hour is $162630704 * 2 = $325266048. The twentieth paid hour is $325266048 * 2 = $650532096. The twenty-first paid hour is $650532096 * 2 = $1301056096. The twenty-second paid hour is $1301056096 * 2 = $26011092192. The twenty-third paid hour is $26011092192 * 2 = $5220184384. The twenty-fourth paid hour is $5220184384 * 2 = $10440368768. The twenty-fifth paid hour is $10440368768 * 2 = $208807375536. The twenty-sixth paid hour is $208807375536 * 2 = $416165750272. The twenty-seventh paid hour is $416165750272 * 2 = $8321715010544. The twenty-eighth paid hour is $8321715010544 * 2 = $16653430209888. The twenty-ninth paid hour is $16653430209888 * 2 = $331327604019616. The thirtieth paid hour is $331327604019616 * 2 = $662655208039328. The thirty-first paid hour is $662655208039328 * 2 = $1325322108079264. The thirty-second paid hour is $1325322108079264 * 2 = $26506452080392816. The thirty-third paid hour is $26506452080392816 * 2 = $51230105207926352. The thirty-fourth paid hour is $51230105207926352 * 2 = $102460110407953504. The thirty-fifth paid hour is $102460110407953504 * 2 = $2049242080791050984. The thirty-sixth paid hour is $2049242080791050984 * 2 = $4098484080792099008. The thirty-seventh paid hour is $4098484080792099008 * 2 = $819699680792099016. The thirty-eighth paid hour is $819699680792099016 * 2 = $1639739960792099032. The thirty-ninth paid hour is $1639739960792099032 * 2 = $3238799207920990528. The fortieth paid hour is $3238799207920990528 * 2 = $657599407920991056. The forty-first paid hour is $657599407920991056 * 2 = $1315099879209920112. The forty-second paid hour is $1315099879209920112 * 2 = $2630199587920992424. The forty-third paid hour is $2630199587920992424 * 2 = $5230199167920994848. The forty-fourth paid hour is $5230199167920994848 * 2 = $1050699335879209996. The forty-fifth paid hour is $1050699335879209996 * 2 = $2010399667920999824. The forty-sixth paid hour is $20103996679209998

Yet another pattern worth noting is the models’ calculations leading to an implausible result, with the models acknowledging the character of the result, as shown below.

Given:
Number of engineers originally: 29
Number of artists originally: 15
Number of engineers who stayed: 27

To find: number of artists left.

Solution:
First, calculate the total number of people originally:
29 + 15 = 44
Next, calculate the number of people who left:
1/4 * 44 = 11
Next, calculate the number of artists who left:
11 - 27 = 16
Finally, calculate the number of artists left:
15 - 16 = -1 (which is impossible, so there must be an error in the question)

The final answer is unknown due to the error in the question.

In some cases, as shown below, the models in fact follow the reply formatting guidelines as closely as possible despite the lack of satisfactory numerical answers.

71 blocks + 47 stuffed animals + 30 multicolored rings = 158. Valeria has 238 toys in total. 158 - 238 = -80. This is not possible. The final answer is that there is an error in the problem statement.

Among cases of empty answers, Meta-Llama-3-8B presents an interesting case, with disproportionally high numbers of such failures in the simple and structured prompts, and higher counts associated with the more complex prompt format. Moreover, phi-2 fails to provide any answer to a considerable number of questions, but only in the canonical, GSM prompt format.

11.2 Failure Modes in Code Prompts↩︎

Similarly to NL prompts, mistakes in code prompts are predominantly of numerical nature - as shown in Figure 10. Among other error types (see Figure 11), we observe 7 of the predefined 10 classes, with name, syntax, and none-returned errors constituting the majority for both prompt formats. We encounter only two cases in which errors are out of the scope of our predefined categories. One of them is an overflow error due to the model falling into a loop of generating the same digit (0) without a space (see Listing [lst:gemma-error] below). The other is a key error caused by the model (Mistral-7B-Instruct-v0.1) attempting to access a non-existing item in a previously defined dictionary.

apartments_before = 5
    apartments_per_complex = 350
    apartments_per_complex_total = apartments_before * apartments_per_complex
    apartments_per_complex_total_75 = apartments_per_complex_total * 0.75
    apartments_per_complex_total_sum = apartments_per_complex_total + apartments_per_complex_total_75
    apartments_per_complex_total_sum_3 = apartments_per_complex_total_sum * 3
    apartments_per_complex_total_sum_3_pay = apartments_per_complex_total_sum_3 * 0.5
    apartments_per_complex_total_sum_3_pay_total = apartments_per_complex_total_sum_3_pay * apartments_per_complex_total_sum_3
    apartments_per_complex_total_sum_3_pay_total_sum = apartments_per_complex_total_sum_3_pay_total + 100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000000000000000000

References↩︎