1 Introduction↩︎

Mathematical reasoning has emerged as one of the most fundamental and challenging capabilities for evaluating the cognitive abilities of Large Language Models (LLMs) [1]–[5]. Unlike simple fact recall or basic language understanding, mathematical word problem solving requires a complex integration of multiple cognitive skills: natural language comprehension, logical deduction, numerical computation, multi-step planning, and the ability to maintain coherent reasoning chains over extended sequences [6]–[8]. These requirements make mathematical reasoning an ideal benchmark for assessing the depth and robustness of LLM capabilities, revealing not just what models know, but how they think [9]–[11].

In recent years, the research community has developed numerous benchmarks to systematically evaluate mathematical reasoning [12]–[15]. The GSM8K dataset [1] introduced a collection of grade-school math problems that has become a standard reference point. The MATH dataset [2] expanded this by including more challenging problems across diverse mathematical domains. Subsequently, datasets such as LILA [15], MathVista [16], SVAMP [17], FrontierMath [18], and MMLU-Math [19], MATH-Perturb [20] and Omni-math [13] have further enriched the evaluation landscape, each targeting different aspects of mathematical reasoning such as robustness to numerical variations, program synthesis, and multi-domain coverage [21].

A significant advancement came with the introduction of GSM-Plus [22], a benchmark specifically designed to evaluate model robustness through linguistic perturbations. Unlike traditional benchmarks that measure only accuracy on static problem formulations, GSM-Plus systematically tests model resilience across multiple perturbation types. The authors identify five perspectives to guide the development of GSM-Plus: (1) Numerical Variation (numerical substitution, digit expansion, and integer-decimal-fraction conversion), (2) Arithmetic Variation (adding operation and reversing operation), (3) Problem Understanding (rephrasing text descriptions), (4) Distractor Insertion (inserting topic-related but useless sentences), and (5) Critical Thinking (focusing on question or doubt ability when the question lacks necessary statements).

The importance of robustness evaluation has been increasingly recognized as LLMs are deployed in real-world applications where input variations are inevitable [23], [24]. Users naturally express problems in diverse ways, and models that are brittle to paraphrasing or minor structural changes are unlikely to provide reliable service in production environments [25], [26]. Furthermore, robustness to perturbations serves as a proxy for true understanding: models that merely memorize problem patterns rather than genuinely comprehend mathematical structures will fail when those patterns are disrupted [27], [28].

Despite the substantial progress in English-language mathematical reasoning evaluation, the research landscape for low-resource languages remains severely underexplored [29], [30]. This disparity is not merely a matter of academic interest but has profound implications for the equitable development and deployment of AI technologies across global linguistic communities [31]. As LLMs increasingly power applications used worldwide, the lack of robust evaluation frameworks for languages other than English creates significant risks of biased performance and exclusion [32]. Bangla, the seventh most spoken language in the world with over 300 million native speakers primarily in Bangladesh and the Indian state of West Bengal, exemplifies this disparity[33],[34]. Despite its vast speaker population, Bangla remains severely underrepresented in NLP research, particularly for complex reasoning tasks [35], [36]. This underrepresentation is striking when compared to high-resource languages like English, Chinese, or Spanish, which enjoy abundant datasets, models, and evaluation frameworks [37].

The scarcity of Bangla resources extends across multiple dimensions of NLP. In terms of language models, while English boasts hundreds of pretrained models, Bangla has only a handful such as BanglaBERT [38] and BanglaBERT-Base [39]. For benchmark datasets, initiatives like XTREME [40], and XGLUE [41] have included only a limited set of low-resource languages [42], with Bangla often absent or represented only in token amounts. For named entity recognition, the BanglaNER dataset [43] provides some coverage, but these are fundamentally different from the reasoning tasks we investigate. Recent work has introduced BnMMLU [44], a benchmark to evaluate multitask language understanding in Bengali across 23 domains, revealing significant performance gaps that highlight the need for improved pre-training and fine-tuning strategies tailored to Bengali data.

The gap is particularly acute in mathematical reasoning. While English-language mathematical reasoning has been extensively studied through benchmarks like GSM8K [1], MATH [2], GSM-Plus [22], MathQA [45], and others, the equivalent resources for Bangla are limited. Recent efforts have emerged, including BanglaMATH [35] and SOMADHAN [36]. However, these resources remain limited in scope and lack the comprehensive perturbation-based robustness evaluation that has become standard in English-language benchmarks.

Recognizing this critical gap, we introduce GSM-PLUS-BN, a Bangla-language counterpart to the English GSM-Plus dataset. This publicly available resource provides a parallel, perturbed mathematical reasoning benchmark for Bangla, enabling systematic cross-lingual evaluation of LLM robustness. The dataset comprises 10,544 perturbed question variations derived from 1,318 seed questions from GSM-Plus, incorporating eight distinct perturbation types: numerical substitution, digit expansion, integer-decimal-fraction conversion, adding operation, reversing operation, problem understanding, distraction insertion, and critical thinking. These perturbations are specifically designed to probe different aspects of model robustness, from handling numerical variations to resisting semantically coherent but irrelevant information. As shown in Figure 1, perturbations are applied to a seed example to produce altered versions that can be used to analyze model behavior under different input conditions. This paper conducts a comprehensive comparative evaluation of six diverse LLMs using the GSM-PLUS-BN dataset. The models in our study span a range of architectures and parameter scales, providing a representative cross-section of current LLM capabilities: Qwen3-32B [46], [47], Llama-3.1-8B-Instant, Llama-3.3-70B-Versatile, Llama-4-Scout-17B-16E-Instruct [48], [48]–[50], GPT-OSS-120B, and GPT-OSS-20B [51]. We evaluate all models using two standardized prompting strategies: standard prompting (direct problem presentation without examples) and chain-of-thought prompting (encouraging step-by-step reasoning) [6], [52], [53]. This dual approach enables assessment of both base reasoning capabilities and the effectiveness of prompting techniques in Bangla.

Through this work, we make the following key contributions:

• This paper introduces GSM-PLUS-BN, the first comprehensive perturbed mathematical reasoning benchmark for Bangla, comprising 10,544 question variations across eight perturbation types.

• A detailed analysis of model performance across eight perturbation types is conducted, identifying which types pose the greatest challenges in the Bangla context. The analysis reveals how different perturbation mechanisms from numerical substitutions to distraction insertion affect model performance.

• Insights into how model scale and architectural choices affect performance in low-resource language settings are provided, offering practical guidance for model selection when deploying reasoning systems for Bangla.

• The effectiveness of standard prompting and chain-of-thought prompting strategies in Bangla is evaluated, revealing optimal prompting approaches for different model sizes and perturbation types.

2 Related Work↩︎

This section provides a comprehensive review of prior work, systematically analyzing recent foundational studies that collectively inform our understanding of mathematical reasoning in LLMs. With particular attention to prompting techniques, benchmark development, Bengali language resources, robustness evaluation, multilingual reasoning, multimodal reasoning, and advanced theorem proving. Table 1 presents a comprehensive summary of prior work on mathematical reasoning.

2.1 Prompting Techniques for Mathematical Reasoning↩︎

The foundational work of [54] introduced Zero-shot Chain-of-Thought (Zero-shot-CoT) prompting, establishing that reasoning capabilities can be elicited by simply adding the phrase “Let’s think step by step” to prompts. Their experiments demonstrated significant performance improvements across eight reasoning benchmarks without requiring examples or fine-tuning. The method proved particularly effective because it leverages the emergent reasoning capabilities of LLMs through a simple yet powerful prompting technique. However, the effectiveness depends substantially on model size, with smaller models showing limited improvements, suggesting that certain reasoning capabilities only emerge above specific parameter thresholds. Building upon this foundation, proposed MathPrompter [55], a framework that enhances mathematical reasoning by generating multiple solution paths using Zero-shot Chain-of-Thought prompting. Their approach achieved a notable accuracy improvement from 78.7% to 92.5% on the MultiArith dataset using a 175B parameter GPT-based model.

However, [55] acknowledged that both algebraic and Python-based solutions may still be incorrect if they share the same underlying reasoning error, highlighting the need for more robust and fundamental error-checking mechanisms. The extended prompting [36] research by investigating Tree-of-Thought (ToT) prompting for Bengali mathematical word problems from the SOMADHAN dataset, comparing standard prompting, Chain-of-Thought, and ToT strategies. Their work demonstrated that ToT achieved the highest accuracy of 88% with GPT-OSS-120B and improved performance for medium-to-large models compared to CoT. However, smaller models showed unstable performance under ToT, and the study was limited to only 100 representative problems.

Conducted [14] a comprehensive evaluation of four large language models using five benchmark datasets and seven prompting strategies, finding that GPT-4o achieved the best overall results while larger models were less sensitive to prompting techniques than smaller models. Critically, they discovered that no single prompting strategy consistently outperformed others across all tasks, suggesting that optimal prompting approaches are task-dependent.

2.2 Comprehensive Surveys on Mathematical Reasoning↩︎

Provide [56] a comprehensive survey of large language models for mathematical reasoning, organizing prior work using a dual cognitive framework of comprehension and generation. Their review covers extensive datasets and methodologies including Chain-of-Thought prompting, supervised fine-tuning, and inference scaling, demonstrating that LLMs achieve strong performance across established benchmarks such as GSM8K and MATH. However, as a review paper, no single accuracy metric is reported, and the authors identified the lack of standardized evaluation as a major limitation, highlighting the urgent need for unified benchmarks to enable reliable cross-model assessment.

Present [57] a foundational and comprehensive analysis of LLMs in mathematical reasoning, systematically categorizing methodologies from basic prompting to active fine-tuning across diverse datasets. Their holistic review unifies a highly fragmented research landscape, demonstrating that models utilizing specialized mathematical tokenization and Program-of-Thought architectures achieve significantly superior accuracy and verifiable explainability. However, [57] identified inherent brittleness and limited generalization as persistent challenges, noting that performance remains highly inconsistent when models encounter adversarial phrasing or shifting grade levels, and that these models still lack human-centric pedagogical adaptability.

2.3 Bengali Mathematical Reasoning Resources↩︎

The development of resources for Bengali mathematical reasoning represents a recent but rapidly growing area of research, addressing a critical gap for the world’s seventh most spoken language. Despite Bengali’s significant global footprint, computational resources and benchmarks in the language have historically been scarce, creating substantial barriers for developing and evaluating AI systems tailored to Bengali-speaking populations.

Made [36] a seminal contribution by investigating the effectiveness of LLMs for solving Bengali math word problems using Chain-of-Thought reasoning on the SOMADHAN dataset. Their experiments demonstrated that LLaMA-3.3 70B achieved the best performance with 88% accuracy using few-shot CoT prompting, demonstrating strong potential for multilingual mathematical reasoning. The dataset, containing 8,792 complex Bengali math word problems with manually written, step-by-step solutions, represents a significant advancement as no human-annotated Bengali dataset had previously addressed multi-step mathematical reasoning. However, [36] revealed limitations including a relatively small evaluation set and weaker performance of smaller models in understanding Bengali language structures, and noted that fine-tuning benefits were limited for smaller models, highlighting persistent challenges in low-resource language adaptation.

Introduced the BenNumEval [58], a Bengali numerical reasoning benchmark containing 3,255 samples across six task types including Causal Reasoning, Data Sufficiency, Cause and Effect Questions, Fill in the Blanks, Question-Answering Natural Language Inference, and Arithmetical Word Problems. Their evaluation using Bengali Native Prompting (BNaP), Cross-Lingual Prompting (XLP), and Cross-Lingual Chain-of-Thought (XCoT) strategies showed that Gemini 2.0 Flash achieved the best average accuracy of 79.63% with XLP, while the human baseline reached 98.05%. This substantial performance gap between state-of-the-art LLMs and human performance reveals significant challenges remaining in Bengali numerical reasoning. However, [58] noted that the dataset size is relatively moderate and limited to Bengali, with mostly zero-shot evaluation, and some translated questions may introduce bias.

Contributed BanglaMATH [35], a benchmark dataset of 1,700 Bangla math word problems for grades 6-8, carefully curated from Bangla school textbooks and exams with problems annotated by grade, reasoning steps, and digit count. Their evaluation revealed that DeepSeek V3 achieved the highest accuracy (86.9% for Grade 6, 82.2% for Grade 8), while Grok 3 performed the lowest. Critically, when augmenting problems with distracting information and translating them to English, both top-performing models demonstrated significant robustness failures and performance bias in Bangla, highlighting the vulnerability of LLMs to linguistic and contextual challenges. This paper [35] acknowledged limitations due to budget and API constraints, leaving many open-source and multilingual LLMs untested. Presented an end-to-end Bangla AI system for solving Math Olympiad problems [58] by introducing the BDMO dataset and integrating fine-tuned LLMs with RAG and Tool-Integrated Reasoning (TIR). Their best-performing model achieved a score of 77/100 on the test set, demonstrating strong potential for Bangla numerical reasoning. However, [58] noted that keyword-based RAG did not significantly improve performance and that models still perform weaker in Bangla compared to English, with a key limitation being the small size of curated Bangla benchmarks restricting deep reasoning and generalization.

Introduced the PatiGonit [59] dataset of 10,000 Bengali math word problems, fine-tuning transformer models including mT5, mBART50, BanglaT5, and a basic transformer to generate equations from text. Their approach achieved top accuracies of 97.30% with mT5 and 97.20% with mBART50 alongside high BLEU scores exceeding 94. However, [59] faced challenges in cultural translation nuances, dataset limitations to basic arithmetic without diverse multi-step problems, and resource constraints hindering full dataset expansion or exhaustive hyperparameter tuning. Introduced the BMWP [60] dataset of 8,653 Bengali math word problems aimed at operation prediction and problem-solving, applying RNN-based models with GloVe embeddings and fine-tuned GPT-2, achieving the highest validation accuracy of 90.42% with LSTM. However, [60] noted that the dataset is limited to single-step arithmetic problems and lacks complex, multi-step, or real-world scenarios, and LLMs like GPT-2 or mT5 struggled in zero-shot settings.

Presented MathMist [61], a parallel multilingual benchmark with 2,890 gold-standard Bangla-English math problems from secondary textbooks, expanded synthetically to 13 languages via LLMs, evaluating GPT-OSS under zero-shot, CoT, and code-switched paradigms, yielding strong Bangla performance of 88.97% overall accuracy and slightly higher English results at 90.20%. Despite these encouraging results, [61] identified limitations including restricted coverage to 13 languages excluding many low-resource ones, secondary-level difficulty without advanced topics, narrow error analysis, and metrics overlooking subtle reasoning flaws. Introduced the Bangla-Bayanno [62], a Bengali Visual Question Answering dataset containing 52,650 question-answer pairs translated from English VQA v2 using ChatGPT-4 and a multi-stage validation process. Human evaluation showed that the proposed translation approach outperformed manual and conventional machine translation in preserving semantic quality. However, [62] noted that some translated terms remain language-dependent and visual diversity is limited.

Introduced MathQuest [63], a benchmark for evaluating mathematical reasoning using NCERT Class 11 and 12 mathematics problems, fine-tuning LLaMA-2, WizardMath, and MAmmoTH using QLoRA, with MAmmoTH-13B achieving the best performance. However, [63] acknowledged that the benchmark is limited to school-level mathematics and a few open-source models. Introduced Distract Math-BN [61], a Bangla benchmark augmenting MGSM and MSVAMP with semantically coherent but computationally irrelevant distractors, evaluating seven models from 3B to 12B parameters. Their study revealed substantial performance degradation, with standard models dropping up to 41 points and reasoning-specialized models declining 14-20 points despite consuming 5x more tokens. These findings underscore the vulnerability of even advanced models to adversarial modifications, particularly in low-resource language settings.

2.4 Robustness and Perturbation in Mathematical Reasoning↩︎

Evaluating the robustness of mathematical reasoning under adversarial and perturbed conditions has emerged as a critical research direction.Introduced the GSM-PLUS [22], a robust benchmark designed to evaluate the mathematical reasoning abilities of LLMs under adversarial conditions. Their results demonstrate that although models like GPT-3.5-Turbo achieve around 61% accuracy, performance drops significantly compared to original GSM8K, indicating limited robustness to small perturbations. This suggests that many LLMs rely on surface-level patterns rather than deep reasoning. [22] acknowledged that GSM-PLUS is derived only from modified GSM8K problems, lacking entirely new problem types and higher-level mathematical complexity. Proposed Math-RoB [64], a benchmark specifically designed to evaluate the robustness of mathematical reasoning by modifying problem formulations while preserving the reasoning process. Experimental results showed that all evaluated models experienced noticeable performance degradation, revealing fundamental weaknesses in reasoning robustness. [64] recommended extending the benchmark to broader reasoning domains and developing more robust training strategies.

Introduced MATH-Perturb [65], a benchmark that applies numerical, semantic, and structural perturbations to assess reasoning stability. Experimental results showed that all models experienced performance degradation, with OpenAI o1 demonstrating the strongest robustness, and the largest accuracy drops occurring under semantic rewriting and problem restructuring. [65] noted that the benchmark is limited to text-based mathematics and manually designed perturbations, recommending extension with multilingual datasets. Proposed FormalMATH [66], a benchmark for evaluating formal mathematical reasoning using the Lean 4 theorem prover, containing 5,560 formally verified mathematical problems requiring machine-verifiable proofs. Results showed that OpenAI o1 achieved the highest proof success rate, although many generated proofs failed due to logical or syntactic errors. Investigated [67] whether LLMs rely on reasoning or memorization by introducing the Knights & Knaves, DynamicZebra, and LiMem benchmarks. Results showed that fine-tuned models memorized training data but still demonstrated improved reasoning on unseen tasks, concluding that memorization and reasoning complement each other.

2.5 Multilingual Mathematical Reasoning↩︎

Multilingual mathematical reasoning represents a frontier challenge in AI, as models must navigate linguistic diversity while maintaining mathematical competence across languages. Introduced PolyMath [68], a multilingual mathematical reasoning benchmark covering 18 languages and multiple difficulty levels, finding that Qwen-3-235B-A22B-Thinking achieved the highest performance while significant performance gaps persisted between high-resource and low-resource languages. The proposed MMATH [69], covering 10 languages with competition-level problems, identifying off-target language generation issues, and showing that reasoning in English before generating answers in the target language improved performance.

MathEval [70], which integrates 22 mathematical reasoning datasets in English and Chinese, shows that reasoning-specialized models outperformed general-purpose models. FrontierMath [71] for advanced mathematical reasoning using research-level problems, showing that OpenAI o1-preview achieved the highest accuracy at 25.2% while most other models solved fewer than 2% of problems. Introduced OMNI-MATH [13] containing 4,428 Olympiad-level mathematics problems, showing that OpenAI o1-preview achieved approximately 58% accuracy but all models remained well below expert human performance. Introduced LILA[15], a unified benchmark integrating 23 mathematical reasoning tasks, and proposed the BHASKARA multi-task learning model achieving a 21.83% improvement in F1 score compared to task-specific models.

3.8pt

2.6 Multimodal Mathematical Reasoning↩︎

The integration of visual and textual information for mathematical reasoning has emerged as another important research direction. [16] developed the MathVista benchmark, aggregating 6,141 multimodal math problems from 28 datasets, evaluating 12 foundation models, with GPT-4V leading at 49.9% accuracy. Introduced the MATH-Vision [72] dataset with 3,040 visually grounded problems from real math competitions, showing that current models still perform significantly below human level especially in complex domains like geometry and topology. The proposed Of MultiMath [73], a multimodal framework achieving 58.5% accuracy on MathVista and 48.0% on MathVerse, though reporting limited generalization to out-of-domain datasets.

2.7 Advanced Mathematical Reasoning and Fine-Tuning↩︎

Introduced the DeepSeekMath [74], an open-source mathematical reasoning framework trained on a large-scale corpus, achieving nearly 60% accuracy and supporting both English and Chinese benchmarks. However, [74] noted that DeepSeekMath still lags behind GPT-4 in few-shot generalization. Investigated advancing math [75] reasoning in 8B Llama-based models through continued pretraining with problem-solving data versus general corpora, yielding MathGPT-8B with superior GSM8K (64.44%) and MATH (35.88%) scores. Identified [75] that supervised fine-tuning was inferior to continued pretraining due to poor out-of-distribution handling and hard data adaptation. Introduced the CARP [76] benchmark to systematically evaluate computation-intensive mathematical reasoning in tool-augmented language models, achieving 73.46% accuracy on the CARP dataset using their DELI approach.

However, [76] noted limitations in the scale of the dataset and restricted access to certain models. Proposed MARIO [77], a reproducible pipeline that integrates REACT-style prompting with code interpreter outputs, achieving 69.69% accuracy on GSM8K. However, they identified data quality and format inconsistencies along with the need for human verification as key limitations. Introduced MALT [78], a multi-agent post-training framework that teaches specialized LLMs to collaborate via tree-search and value-iteration, achieving relative improvements of 15.66% on MATH, 9.40% on CSQA, and 7.42% on GSM8K. However, the approach is constrained by severe computational intensity and is currently limited to deterministic domains.

2.8 Research Gap↩︎

Based on our systematic analysis of reviewed papers, we identify two critical research gaps that motivate our study:

First, there has been not too much work done on Bengali language or Bengali Math before. While papers by [36], [58], [35], [59], [60], [61], [62], [58], and [79] have made foundational contributions to Bengali mathematical reasoning, the benchmarks remain limited in scale, scope, and problem diversity. Most existing Bengali datasets are restricted to basic arithmetic [59], [60], school-level mathematics [35], [63], or small evaluation sets [36], [80]. The Bengali mathematical reasoning landscape remains significantly under-resourced compared to English and other high-resource languages.

Second, and critically, there has been no work done on perturbed Bengali Math before. While robustness and perturbation research has progressed significantly in English through benchmarks like GSM-PLUS [22], Math-RoB [64], and MATH-Perturb [65], and while Bengali robustness has been preliminarily explored through distractors [79], no study has systematically translated, evaluated, and benchmarked adversarial or perturbed mathematical reasoning in Bengali. The intersection of Bengali language and perturbed mathematical reasoning-specifically, how LLMs respond to numerical, semantic, and structural perturbations in Bengali-remains entirely unexplored.

3 Dataset↩︎

To systematically evaluate the mathematical reasoning robustness of Large Language Models (LLMs) in Bengali, we introduce GSM-PLUS-BN adaptation of the GSM-PLUS dataset. Built upon the widely-adopted GSM8K benchmark [1], GSM-PLUS-BN comprises eight perturbation categories spanning five distinct dimensions. These perturbations are designed to probe specific vulnerabilities in LLM reasoning capabilities, ranging from numerical manipulation to logical fallacies. Figure 4 provides a comprehensive overview of all perturbation types with representative examples. Figure 2 provides a comprehensive overview of the development of our proposed dataset.

3.1 Perturbation Taxonomy↩︎

The GSM-Plus-BN dataset employs a systematic perturbation framework designed to evaluate the robustness and reasoning consistency of large language models (LLMs) in Bengali mathematical reasoning tasks. Each seed question serves as the base problem, from which eight distinct perturbation types are generated. These perturbations modify the surface form, numerical values, structure, or contextual elements of the problem while preserving the underlying mathematical logic and the correct answer. This controlled variation enables researchers to determine whether models genuinely understand mathematical concepts or rely on superficial pattern recognition.

3.1.1 Numerical Variation↩︎

This perturbation family examines whether LLMs rely on superficial numerical patterns rather than genuine mathematical reasoning. We implement three transformation strategies:

Numerical Substitution replaces numerical values with different numbers while preserving the same digit length. For example, a value of 2 may be replaced with 3 or 4. This tests whether models memorize specific numerical relationships or truly understand the underlying arithmetic operations.

Digit Expansion increases the number of digits while maintaining the same mathematical relationship. For example, 2 becomes 200 or 2,000. This perturbation evaluates whether models remain robust when numerical scales change.

Integer-Decimal-Fraction Conversion changes the representation of numerical values across equivalent or related formats. For example, an integer such as 2 may be represented as 2.0, while other values may appear in decimal or fractional form (e.g., 0.5 or 1/2). This evaluates whether models can correctly interpret different numerical representations.

Example Sequence:

• Seed: A shirt requires 2 bolts of blue fiber and half as many bolts of white fiber. How many bolts of fiber are needed in total?

• Numerical Substitution: A shirt requires 3 bolts of blue fiber and half as many bolts of white fiber. How many bolts are needed in total?

• Digit Expansion: A shirt requires 200 bolts of blue fiber and half as many bolts of white fiber. How many bolts are needed in total?

• Conversion: A shirt requires 2 bolts of blue fiber and 0.5 times as many bolts of white fiber. How many bolts are needed in total?

3.1.2 Arithmetic Variation↩︎

These perturbations evaluate whether models can flexibly apply arithmetic reasoning under modified problem structures.

Adding Operation introduces an additional arithmetic step that must be incorporated into the solution. This tests whether models can correctly perform multi-step reasoning without losing track of the original objective.

Reversing Operation converts the problem into its inverse form. Instead of asking for the final result given the inputs, the problem provides the result and asks the model to infer one of the original inputs. This evaluates bidirectional mathematical reasoning.

Example Sequence:

• Seed: A shirt requires 2 bolts of blue fiber and half as many bolts of white fiber. How many bolts of fiber are needed in total?

• Adding Operation: A shirt requires 2 bolts of blue fiber and half as many bolts of white fiber. Each bolt costs $12. What is the total cost?

• Reversing Operation: A shirt requires some blue fiber and half as much white fiber. The total number of bolts used is 3. How many bolts of blue fiber are required?

3.1.3 Problem Understanding↩︎

This category investigates whether models are sensitive to changes in wording while preserving the underlying mathematical meaning. We reformulate problems using different vocabulary and sentence structures without altering the required reasoning. This evaluates whether models genuinely understand problem semantics rather than relying on memorized textual patterns.

Example:

• Seed: A shirt requires 2 bolts of blue fiber and half as many bolts of white fiber. How many bolts of fiber are needed in total?

• Problem Understanding: If a shirt requires 2 units of blue material and half as many units of white material, how many units of material are required altogether?

3.1.4 Distractor Insertion↩︎

We insert topic-relevant but mathematically irrelevant statements that often include additional numerical values. This perturbation evaluates whether models can distinguish essential information from distracting details, a key requirement for solving real-world mathematical problems containing extraneous information.

Example:

• Seed: A shirt requires 2 bolts of blue fiber and half as many bolts of white fiber. How many bolts of fiber are needed in total?

• Distractor Insertion: A shirt requires 2 bolts of blue fiber and half as many bolts of white fiber. Each bolt normally costs $5, but today there is a special discount that reduces the price of each bolt by $5. How many bolts of fiber are needed to make the shirt?

3.1.5 Critical Thinking↩︎

Inspired by Polya’s problem-solving principles, this perturbation removes essential information from the problem statement. Instead of generating an unsupported answer, models are expected to recognize that the available information is insufficient and explicitly identify the missing information. This evaluates higher-order reasoning and resistance to producing unjustified responses [6].

Example:

• Seed: A shirt requires 2 bolts of blue fiber and half as many bolts of white fiber. How many bolts of fiber are needed in total?

• Critical Thinking: A shirt requires 2 bolts of blue fiber and some amount of white fiber. How many bolts are needed in total?

3.2 Data Translation↩︎

In this section, we describe the dataset translation process for GSM-PLUS-BN. We employed a hybrid approach combining AI-based tools with expert human refinement. An automated translation system first generated initial Bengali versions of all mathematical problems from the original GSM-PLUS dataset, which is in English. Subsequently, expert translators with strong backgrounds in both linguistics and mathematics manually reviewed and corrected each output. They followed strict guidelines to preserve mathematical expressions, contextual nuances, and step-by-step reasoning chains. This two-stage methodology effectively mitigated automated translation errors while ensuring logical coherence. The final dataset achieves high linguistic fluency and retains complete reasoning fidelity for reliable model evaluation.

3.3 Translator Demographics and Expertise↩︎

We engaged six skilled English-to-Bengali translators with expertise in both linguistics and mathematical reasoning. The entire corpus was evenly distributed among the translators to ensure efficiency and maintain uniform quality across the translated content. Table 2 provides detailed information about their backgrounds and experience. The team comprised one graduate, one postgraduate, and four undergraduate students, all specializing in Natural Language Processing (NLP), with ages spanning 22 to 30 years. Their research experience ranged from 1 to 4 years, with senior members overseeing the work of junior translators to ensure translation quality and consistency. This structured expertise distribution enabled effective quality control and maintained linguistic fidelity throughout the dataset refinement process.

Together, this combination of expert translation and multi-stage human refinement ensures that GSM-PLUS-BN maintains both linguistic fluency and logical coherence, making it a reliable benchmark for evaluating Bengali mathematical reasoning in large language models.

3.3.1 Translation Guidelines↩︎

To guarantee translation quality, cultural relevance, and linguistic consistency, the following principles were established and communicated to the translators prior to the translation process. Figure 3 provides an overview of the challenges encountered during translation.

1. Translations preserve the original reasoning process and logical structure of each problem. Mathematical relationships, problem constraints, and contextual information remain unchanged to ensure semantic equivalence with the source text.

2. Personal names are localized using culturally appropriate Bengali names to improve familiarity and naturalness for native readers. For example, the English name “Janet” is translated as “Mina”.

3. Currency references are adapted to the local context by replacing the Dollar symbol ($) with the Bangladeshi Taka (BDT). For example, “$100” is translated as “100 BDT”.

4. Arabic numerals are converted to Bengali numerals throughout the translated text to maintain orthographic consistency with the Bengali writing system.

5. Location names and culturally specific entities are replaced with locally familiar equivalents whenever appropriate. These substitutions preserve the original problem structure and intended meaning while improving cultural relevance. For example, a foreign city name may be replaced with “Dhaka,” the capital of Bangladesh.

3.4 Dataset Characteristics↩︎

The GSM-Plus-BN dataset comprises a total of 10,544 problem instances in Bengali. These instances were generated from 1,318 seed questions from the GSM8K test set. Following the original GSM-Plus methodology, each seed question was expanded into eight distinct adversarial variants. These variants preserve the correct numerical answer of the original seed question while introducing linguistic or structural perturbations designed to challenge model robustness without altering mathematical correctness.

3.5 Data Structure↩︎

GSM-Plus-BN is organized into seven following structured columns:

• question: The Bengali text of the perturbed math word problem. This is the input that models are expected to reason over, incorporating one of the eight adversarial perturbations while maintaining mathematical correctness.

• solution: The step-by-step reasoning chain and final answer derivation for the perturbed question, provided as a complete solution trace for training or evaluation purposes.

• answer: The final numerical answer to the perturbed problem, serving as the primary metric for evaluating model correctness.

• perturbation_type: The specific category of adversarial modification applied to the seed question, enabling fine-grained analysis of model performance across different perturbation types.

• seed_question: Each seed question serves as the base question from which all perturbations are generated, ensuring traceability to the original source.

• seed_solution: The step-by-step reasoning solution for the original seed question, enabling comparative analysis between perturbed and unperturbed versions.

• seed_answer: The final answer of the original seed question, which remains identical across all eight adversarial variants to ensure perturbations do not alter the underlying mathematics.

3.6 Summary of Perturbation Examples↩︎

Both GSM-Plus and GSM-Plus-BN employ the same adversarial perturbation framework to evaluate the robustness of large language models (LLMs) in mathematical reasoning. Since GSM-Plus-BN is a direct Bengali translation of GSM-Plus, the perturbation structure and reasoning logic remain identical across both datasets, enabling consistent cross-lingual evaluation. Figure 4 provides a overview of the Summary of all perturbation types with Bengali examples.

The perturbations modify the surface form of a problem while preserving its underlying mathematical semantics and correct answer. These variations include numerical changes, arithmetic operation modifications, paraphrasing for problem understanding, distractor insertion, and critical reasoning prompts. Such controlled variations help assess whether models rely on genuine reasoning rather than memorized patterns.

By generating multiple semantically equivalent variants of each seed problem, the perturbation framework enables systematic analysis of reasoning consistency across different linguistic and structural forms. Maintaining identical perturbation strategies across the English and Bengali datasets allows our study to directly compare model robustness and reasoning performance in both languages.

3.7 Dataset Availability↩︎

The GSM-Plus-BN dataset is publicly available for the research community. It is hosted on the Mendeley Data repository, and this release represents the third Version (V3) of the dataset. The GSM-Plus-BN dataset is openly available in Mendeley Data at https://data.mendeley.com/datasets/74dscnmrhv/3

4 Methodology↩︎

In this study, we present a comprehensive evaluation framework for benchmarking the mathematical reasoning capabilities of six open-source Large Language Models (LLMs) on perturbated Bengali mathematical problems. The framework evaluates model performance under two prompting strategies: Standard prompting and Chain-of-Thought (CoT) prompting. Each model is tested on the GSM-Plus-BN dataset using identical experimental settings to ensure a fair and reproducible ablation study. Figure 5 displays the schematic diagram of our proposed methodology.

4.1 Selected Large Language Models↩︎

In this study, we evaluate six open-source Large Language Models (LLMs) available through the Groq API platform [81]. The selected models include both dense transformer and Mixture-of-Experts (MoE) architectures developed by Alibaba, Meta, and OpenAI. All models are instruction tuned and support multilingual reasoning, making them suitable for evaluating mathematical reasoning in both English and Bangla. The models were selected based on three primary criteria:

1. Open Source and Free Accessibility: All selected models are open-source and freely available to the research community, ensuring reproducibility and accessibility for future research. The models can be accessed without cost through the Groq API, making them ideal for academic research with limited budgets.

2. Parameter Range (8B to 120B): We selected models spanning a wide parameter range from 8 billion to 120 billion parameters. This diversity allows us to investigate how model size influences performance on perturbated Bengali mathematical reasoning tasks. Smaller models (8B-20B) help us understand the capabilities and limitations of computationally efficient models, while larger models (70B-120B) serve as performance benchmarks.

3. Suitability for Bengali Mathematical Reasoning: All selected models support multilingual reasoning capabilities, making them suitable for evaluating mathematical reasoning in Bengali. The models represent different architectural families, including dense transformer and Mixture-of-Experts (MoE) architectures, enabling a comprehensive analysis of how architectural design influences reasoning performance.

All models were evaluated using API-based inference through the Groq platform without additional fine-tuning or parameter modification. The following subsections provide detailed descriptions of each selected model. To ensure a fair comparison, all models were evaluated using API-based inference without additional fine-tuning or parameter modification. Table 3 summarizes the selected models and the reasons for their inclusion in this study.

4.1.1 Qwen3-32B↩︎

Qwen3-32B is an open-source large language model developed by Alibaba’s Qwen team and released under the Apache 2.0 license [82]. The model is designed for multilingual instruction following, logical reasoning, and complex problem solving, demonstrating strong performance across a wide range of language understanding and mathematical reasoning benchmarks, including GSM8K, MATH, AIME, and MGSM. The model supports more than one hundred languages, making it well suited for multilingual evaluation tasks. In this study, Qwen3-32B was accessed through the Groq API, which provides efficient inference for large-scale experiments. The model was selected because it offers a strong balance between reasoning capability, multilingual understanding, and computational efficiency. With 32 billion parameters, it represents a mid-range model in our evaluation, allowing us to examine how moderate parameter scales perform on perturbated Bengali mathematical reasoning tasks. Its proven ability to handle cross-lingual reasoning and instruction-following tasks makes it particularly suitable for evaluating mathematical reasoning in Bengali. Furthermore, as an open-source model freely available through the Groq API, it enables reproducible research without financial barriers, which is essential for academic studies with limited resources.

4.1.2 Llama-3.1-8B-Instant↩︎

Llama-3.1-8B-Instant is an open-source instruction-tuned language model developed by Meta AI and released under an open-source license [83]. The model has been optimized for efficient inference while maintaining competitive reasoning and instruction-following capabilities. The model supports multilingual text generation and performs well on a variety of natural language processing tasks, including question answering, summarization, and mathematical reasoning. In this work, Llama-3.1-8B-Instant was accessed through the Groq API to ensure consistent and low-latency inference. The model was selected as the smallest model in our evaluation with 8 billion parameters to serve as a lightweight baseline. Including this model allows us to investigate the minimum parameter threshold required for reasonable performance on perturbated Bengali mathematical reasoning tasks. Its efficient computational requirements make it a practical choice for real-world applications where hardware resources and inference speed are important considerations. As an open-source model freely available through the Groq API, it provides an accessible entry point for researchers and developers working with limited computational resources. Comparing its performance with larger models provides valuable insight into the relationship between model size and reasoning ability in the context of Bengali perturbated mathematical problems.

4.1.3 Llama-3.3-70B-Versatile↩︎

Llama-3.3-70B-Versatile is a high-capacity open-source instruction-tuned language model developed by Meta AI and released under an open-source license [84]. The model is designed to deliver strong performance across complex reasoning, multilingual understanding, coding, and instruction-following tasks. Owing to its larger parameter scale, it consistently achieves competitive results on several reasoning and mathematical benchmarks while maintaining robust performance across multiple languages. In this study, the model was accessed through the Groq API to support efficient inference during large-scale evaluation. Llama-3.3-70B-Versatile was selected as a high-capacity dense transformer model with 70 billion parameters to serve as a performance benchmark. Including this model allows us to investigate how increased model capacity influences perturbated Bengali mathematical reasoning performance under different prompting strategies. It also serves as an important reference for comparing the performance of smaller models against a larger instruction-tuned LLM. As an open-source model freely available through the Groq API, it enables high-quality research without the financial constraints associated with proprietary models. Its proven performance on multilingual reasoning tasks makes it particularly suitable for evaluating Bengali mathematical reasoning, where linguistic complexity presents additional challenges for language models.

4.1.4 Llama-4-Scout-17B-16E-Instruct↩︎

Llama-4-Scout-17B-16E-Instruct is an open-source instruction-tuned language model developed by Meta AI based on a Mixture-of-Experts (MoE) architecture [85]. Unlike conventional dense transformer models, the MoE architecture activates only a subset of experts during inference, improving computational efficiency while maintaining strong reasoning performance. The model is released under an open-source license and is designed to support multilingual understanding, long-context processing, and complex instruction-following tasks. In this study, Llama-4-Scout-17B-16E-Instruct was accessed through the Groq API and included as the representative MoE model among the evaluated language models. With 17 billion parameters and 16 experts, it represents an innovative architectural approach that balances computational efficiency with reasoning capability. The model was selected to examine whether sparse expert activation provides advantages over dense transformer architectures when solving perturbated Bengali mathematical reasoning problems. Comparing this model with dense LLMs enables a broader analysis of how different architectural designs influence reasoning accuracy, computational efficiency, and robustness. As an open-source model freely available through the Groq API, it allows researchers to explore the benefits of MoE architecture without the barriers of proprietary systems. Its inclusion in our study provides valuable insights into whether architectural innovations can compensate for smaller parameter counts in the context of Bengali perturbated mathematical reasoning.

4.1.5 GPT-OSS-20B↩︎

GPT-OSS-20B is an open-weight instruction-tuned large language model developed by OpenAI and released under an open-source license [86], [87]. The model is designed for advanced reasoning, instruction following, and general-purpose natural language understanding. Despite its moderate parameter size, the model demonstrates competitive performance across various reasoning tasks, including mathematical problem solving, logical inference, and multilingual language understanding. In this study, GPT-OSS-20B was accessed through the Groq API, which provides fast and reliable inference for large-scale evaluation. The model was selected as a medium-scale model with 20 billion parameters to investigate how model size influences perturbated Bengali mathematical reasoning performance under different prompting strategies. Its inclusion enables a direct comparison with the larger GPT-OSS-120B model while maintaining the same experimental environment, providing insights into the trade-off between computational efficiency and reasoning capability. As an open-source model freely available through the Groq API, it offers researchers a balance between performance and resource requirements. Evaluating GPT-OSS-20B alongside the selected Qwen and Llama models provides valuable insights into how different model families and training approaches influence mathematical reasoning in Bengali. The model’s proven reasoning capabilities make it a strong candidate for evaluating the challenges of perturbated Bengali mathematical problems.

4.1.6 GPT-OSS-120B↩︎

GPT-OSS-120B is a high-capacity open-weight large language model developed by OpenAI and released under an open-source license [87], [88]. The model is designed for complex reasoning, multilingual understanding, and advanced instruction-following tasks. Owing to its substantially larger model capacity, GPT-OSS-120B is expected to provide stronger reasoning ability and better generalization on challenging mathematical problems than smaller language models. The model supports an extensive context window, enabling it to process long reasoning chains and complex problem formulations. In this study, the model was accessed through the Groq API, ensuring a consistent inference environment for all evaluated models. GPT-OSS-120B was selected as the largest model in our experiments with 120 billion parameters to examine the impact of model scale on perturbated Bengali mathematical reasoning performance. Its inclusion provides an upper-bound reference for comparing reasoning performance across different model families and prompting strategies. As an open-source model freely available through the Groq API, it represents the state-of-the-art in accessible large language models for academic research. By evaluating the largest available open-source model under the same experimental settings as the other selected LLMs, this study systematically analyzes how increased model capacity influences mathematical reasoning in perturbated Bengali. The model’s high-capacity architecture makes it particularly suitable for handling the complex reasoning required by perturbated mathematical problems, where multiple reasoning steps and careful attention to linguistic nuances are essential for accurate problem solving.

4.5pt

4.2 Prompting Strategy↩︎

The design of effective prompting strategies is fundamental to evaluating the mathematical reasoning capabilities of Large Language Models (LLMs). Prior research has demonstrated that LLM performance is highly sensitive to prompt design, with variations in instruction formulation, formatting, and contextual cues significantly influencing model outputs [6], [54]. To ensure a fair and consistent evaluation across all models in our study, we developed a standardized prompting framework that provides clear, structured, and unambiguous instructions for solving mathematical reasoning tasks. This methodological rigor is essential for attributing performance differences to the models’ inherent reasoning abilities rather than to variations in prompt engineering. We employed identical prompts across all evaluated models, including Qwen3-32B, Llama-3.1-8B-Instant, Llama-3.3-70B-Versatile, Llama-4-Scout-17B-16E-Instruct, GPT-OSS-20B, and GPT-OSS-120B.

4.2.1 Standard Prompting↩︎

Standard prompting, also referred to as direct answering (DA), represents the most fundamental interaction paradigm with a Large Language Model (LLM). In this configuration, the prompt is formulated as a clear, unambiguous instruction or direct question, without explicitly guiding the model’s internal reasoning process or providing any worked examples. The model is expected to generate a final response solely based on its pre-trained parametric knowledge and the immediate context provided within the prompt.

This strategy serves a critical methodological function as the baseline condition in our comparative analysis. By establishing a performance benchmark under standard prompting, we create a control against which the efficacy of more sophisticated prompting techniques such as Chain-of-Thought can be quantified. Figure 6 provides an illustration of the Standard prompting strategy. Previous studies, including the work of Paul et al., have employed this structure to demonstrate the performance ceiling of LLMs when no explicit reasoning scaffolding is provided [36]. This approach is efficient in terms of token usage and computational cost, making it suitable for straightforward tasks where the model’s inherent knowledge is sufficient. Its primary limitation, however, is that it does not scaffold the model’s internal computation for complex, multi-step problems, making it vulnerable to errors on tasks requiring intricate reasoning.

4.2.2 Chain-of-Thought (CoT) Prompting↩︎

Chain-of-Thought (CoT) prompting is an advanced technique designed to enhance the model’s performance on complex reasoning tasks by eliciting a structured sequence of intermediate reasoning steps prior to the final answer. This method, initially formalized by Wei et al., compels the model to decompose a problem into smaller, manageable sub-problems and address them sequentially, thereby mimicking human cognitive processes [6]. The prompt explicitly instructs the model to "think step by step," requiring the generation of a transparent reasoning pathway that logically connects the initial query to the final conclusion. In our implementation, we adopt the zero-shot variant, wherein the trigger phrase "Let’s think step by step" is appended to the task instruction without the provision of any example rationales. Figure 7 provides an illustration of the Chain-of-Thought prompting strategy, demonstrating the step-by-step reasoning process generated by the LLM.

The primary justification for incorporating CoT prompting is its proven efficacy in improving performance on tasks that necessitate mathematical, logical, and symbolic reasoning [54]. By externalizing the reasoning chain, the model reduces the likelihood of logical errors and enhances the interpretability and transparency of its output. Our methodology leverages this strategy to investigate the model’s ability to perform execution steps the process of performing computation and symbolic manipulation which represents a critical bottleneck in standard prompting scenarios. This approach is particularly valuable in our experimental design, as it allows us to assess whether the LLM’s performance on complex reasoning tasks can be improved through structured prompting, a finding consistently reported in prior work [6], [36]. It is important to note, however, that the utility of CoT prompting may be domain-specific; studies indicate that its benefits are most pronounced in symbolic reasoning tasks and may not consistently generalize to non-symbolic or commonsense reasoning problems.

5 Results Analysis↩︎

This section presents the performance evaluation of six Large Language Models (LLMs) across eight distinct perturbation types designed to test reasoning robustness. We analyze the models’ accuracy under two prompting strategies: Chain-of-Thought (CoT) and Standard Prompting. A total of 9000 samples were taken for the evaluation, where 1000 samples were from seed questions and the other 8000 are the variations of them. The analysis aims to identify which models exhibit superior reasoning capabilities and which perturbation types pose the greatest challenge.

5.1 Performance Overview: Seed Questions↩︎

To establish a baseline for comparison, we first evaluate model performance on unperturbed seed questions. Table 4 presents the accuracy percentages for all models under both prompting strategies. Parenthetical values indicate the performance change with CoT relative to Standard prompting, with \(\uparrow\) denoting improvement and \(\downarrow\) denoting decline. The highest accuracy in each row is bolded.

The seed question results reveal several important patterns. Under Standard prompting, most models demonstrate strong performance, with GPT-OSS-20B achieving the highest accuracy (96.08%), closely followed by Llama-3.3-70B-Versatile (95.90%) and Llama-4-Scout (95.68%). However, Qwen3-32B exhibits exceptionally poor performance (13.98%), suggesting a significant deficiency in its inherent reasoning capabilities for the given tasks.

Under CoT prompting, performance improves substantially for most models. GPT-OSS-120B achieves the highest accuracy (92.70%), followed by Llama-3.3-70B-Versatile (90.25%) and Llama-4-Scout (90.55%). Notably, Qwen3-32B shows the largest improvement (from 13.98% to 76.25%), indicating that CoT prompting is particularly beneficial for models with weaker inherent reasoning capabilities. Conversely, Llama-3.1-8B-Instant shows a decline in performance under CoT (from 90.15% to 66.88%), suggesting that this model may not be well-aligned with the CoT prompting format for seed questions.

5.2 Performance Overview: Standard Prompting↩︎

Table 5 presents the accuracy percentages for each model on perturbated questions under Standard prompting. The highest accuracy in each row is bolded.

Under Standard prompting, the Llama-4-Scout and GPT-OSS-120B models demonstrate the strongest performance across most perturbation types, consistently achieving accuracy rates exceeding 80% on simpler perturbations. The Llama-3.3-70B-Versatile model also performs competitively, while Qwen3-32B exhibits significantly lower performance, with accuracy rates dropping below 25% on more challenging perturbations.

The Critical Thinking perturbation proves to be the most challenging across all models, with accuracy rates ranging from a low of 6.60% (Qwen3-32B) to a high of only 31.80% (Llama-3.3-70B). This suggests that current LLMs lack the fundamental reasoning capabilities required for complex, multi-step logical problems. The Adding Operation and Integer-Decimal-Fraction Conversion perturbations also present significant challenges, with accuracy rates consistently below 80% for most models.

The Problem Understanding perturbation yields the highest accuracy across models, with GPT-OSS-120B achieving 89.82% and Llama-4-Scout achieving 87.43%. The Reversing Operation and Numerical Substitution perturbations also show strong performance, with top models exceeding 85% accuracy. This indicates that models possess relatively robust syntactic and semantic processing capabilities.

5.3 Performance Overview: CoT Analysis↩︎

Table 6 presents the accuracy percentages for each model on perturbated questions under Chain-of-Thought prompting. Parenthetical values indicate the performance change relative to Standard prompting, with \(\uparrow\) denoting improvement and \(\downarrow\) denoting decline.

CoT prompting substantially improves model performance across all perturbation types. The Llama-3.3-70B-Versatile and Llama-4-Scout models achieve the highest overall accuracy, consistently exceeding 85% on most perturbations. The GPT-OSS-120B model also performs competitively, while Qwen3-32B and Llama-3.1-8B-Instant show moderate improvements but remain the weakest performers. Under CoT, Qwen3-32B achieves the lowest average accuracy (54.28%), closely followed by Llama-3.1-8B-Instant (53.38%).

The most substantial improvements from CoT prompting are observed in arithmetic and numeric perturbations, particularly for Qwen3-32B. Qwen3-32B shows dramatic gains across all perturbation types, with improvements ranging from +20.56 (Distraction Insertion) to +30.52 (Integer-Decimal-Fraction Conversion) percentage points. This confirms that explicit step-by-step reasoning instructions effectively guide models through arithmetic and logical steps disrupted by numeric perturbations. The average improvement of +23.85 percentage points for Qwen3-32B suggests that CoT prompting is particularly beneficial for models with weaker inherent reasoning capabilities.

In contrast, the larger models show more modest improvements. Llama-3.3-70B-Versatile and Llama-4-Scout achieve average gains of +3.92 and +2.50 percentage points, respectively, indicating that these models already possess strong inherent reasoning capabilities. Interestingly, GPT-OSS-120B shows a marginal decline (-0.03 percentage points average), suggesting the model may be optimized for direct answer generation rather than step-by-step reasoning. Llama-3.1-8B-Instant demonstrates the most concerning pattern, with an average decline of -1.60 percentage points, suggesting the model is not well-aligned with the CoT prompting format.

Despite CoT prompting, the Critical Thinking perturbation remains exceptionally challenging, with accuracy rates ranging from 11.00% (Qwen3-32B) to 33.00% (GPT-OSS-20B). The minimal improvement from CoT (0.00 to 7.80 percentage points) suggests that current LLMs fundamentally lack the capabilities required for complex logical reasoning tasks. The Adding Operation perturbation also continues to pose difficulties, with accuracy rates below 77% for all models. This indicates that while CoT prompting is beneficial for many tasks, it cannot fully compensate for fundamental reasoning deficiencies in current LLM architectures.

(Numerical Substitution, Digit Expansion, Integer-Decimal-Fraction Conversion, Adding Operation) consistently yield the lowest accuracy rates across both prompting strategies. The Llama-4-Scout model demonstrates the strongest performance on these perturbations, achieving the highest scores on Numerical Substitution (88.42%), Digit Expansion (84.20%), and Integer-Decimal-Fraction Conversion (79.52%) under CoT. However, all models struggle with Adding Operation, suggesting fundamental limitations in arithmetic reasoning.

(Reversing Operation, Problem Understanding) demonstrate the highest resilience across all models, with CoT accuracy rates exceeding 86% for top performers. Problem Understanding achieves the highest accuracy, with GPT-OSS-120B reaching 91.82% under CoT. The high baseline Standard performance (71.86% to 89.82%) suggests models possess relatively robust syntactic and semantic processing capabilities.

(Distraction Insertion) shows moderate improvement with CoT, with gains ranging from 0.40 to 20.56 percentage points. GPT-OSS-120B and Llama-3.3-70B achieve the highest performance under CoT (84.43% each). However, Standard performance (33.53% to 83.83%) indicates models often get misled by inserted distractions, and CoT helps systematically evaluate and disregard irrelevant content.

(Critical Thinking) shows uniformly poor performance across all models and prompting strategies. Even the best-performing models achieve only 31.80% to 33.00% under CoT, with minimal improvement from Standard (6.60% to 31.80%). This indicates that while CoT prompting is effective for many tasks, it cannot compensate for missing reasoning abilities required for complex logical problems.

5.4 Detailed Model-wise Analysis↩︎

5.4.1 Llama-3.3-70B-Versatile: The Overall Best Performer↩︎

The Llama-3.3-70B-Versatile model demonstrates the highest overall performance under CoT prompting, achieving accuracy rates exceeding 85% on most perturbation types. The model achieves its highest performance on Problem Understanding (89.22%), Reversing Operation (87.00%), and Numerical Substitution (87.03%). Its performance on seed questions (90.25% under CoT) establishes a strong baseline.

However, the model shows notable weaknesses in Adding Operation (76.75%) and Critical Thinking (31.80%). The significant drop in performance on arithmetic tasks indicates that even state-of-the-art models struggle with mathematical reasoning when numbers are presented in perturbated formats. The CoT prompting provides limited improvement on Critical Thinking (0.00 percentage points), confirming that complex logical reasoning remains a significant challenge.

5.4.2 Llama-4-Scout-17B-16E-Instruct: Strong and Consistent Performer↩︎

The Llama-4-Scout model achieves performance comparable to Llama-3.3-70B, with accuracy rates exceeding 84% on most perturbations. Its highest accuracy is achieved on Numerical Substitution (88.42%), Problem Understanding (88.82%), and Reversing Operation (88.00%). The model demonstrates particular strength in handling numeric format conversions, achieving 79.52% on Integer-Decimal-Fraction Conversion.

Similar to other models, Llama-4-Scout struggles with Adding Operation (75.95%) and Critical Thinking (27.40%). The model shows moderate improvement with CoT (average +2.50 percentage points), suggesting that it is well-aligned with explicit reasoning instructions but already possesses strong inherent capabilities.

5.4.3 GPT-OSS-120B: Strong Inherent Capabilities↩︎

The GPT-OSS-120B model achieves 92.70% accuracy on seed questions under CoT, the highest among all models. On perturbated questions, the model excels at Problem Understanding (91.82%) and Reversing Operation (86.60%). However, its performance on arithmetic perturbations is notably lower, with Adding Operation at 70.54% and Numerical Substitution at 83.23%.

Interestingly, GPT-OSS-120B shows a slight decline in performance with CoT prompting on several perturbation types, including Numerical Substitution (-0.80%), Digit Expansion (-2.00%), and Integer-Decimal-Fraction Conversion (-2.21%). This suggests that the model may have been optimized for direct answer generation and does not benefit from explicit reasoning instructions.

5.4.4 GPT-OSS-20B: Competitive Mid-tier Performer↩︎

The GPT-OSS-20B model achieves competitive performance on seed questions (87.80% under CoT). On perturbated questions, the model achieves its highest accuracy on Problem Understanding (86.63%) and Numerical Substitution (81.64%). The model shows modest improvement with CoT (average +0.60 percentage points) and achieves the highest Critical Thinking performance under CoT (33.00%), surpassing all other models.

5.4.5 Qwen3-32B: Most Improved with CoT↩︎

Qwen3-32B exhibits the most dramatic improvement with CoT prompting, with an average gain of 23.85 percentage points. On seed questions, the model improves from 13.98% (Standard) to 76.25% (CoT), representing a 62.27 percentage point increase. This suggests that Qwen3-32B possesses latent reasoning capabilities that can be effectively unlocked through explicit prompting.

However, even with CoT, Qwen3-32B remains the weakest performer on most perturbations, with accuracy rates below 70% on all tasks except Problem Understanding (77.05%) and Numerical Substitution (68.06%). The model struggles particularly with Critical Thinking (11.00%), Adding Operation (45.49%), and Distraction Insertion (54.09%).

5.4.6 Llama-3.1-8B-Instant: Mixed Performance↩︎

Llama-3.1-8B-Instant shows a decline in performance with CoT prompting on most perturbation types, with an average decrease of 1.60 percentage points. On seed questions, the model drops from 90.15% (Standard) to 66.88% (CoT), representing a 23.27 percentage point decline. This suggests that the model is not well-aligned with the CoT prompting format.

However, the model achieves competitive performance on some perturbations under CoT, including Problem Understanding (69.06%) and Reversing Operation (68.00%). The model’s performance on Critical Thinking is the lowest among all models under CoT (16.40%), indicating significant limitations in complex reasoning.

5.5 Seed vs. Perturbated Performance Analysis↩︎

To understand the impact of perturbations on model performance, we compare accuracy on seed questions against perturbated questions. Table 7 presents the performance gap between seed and perturbated questions.

The seed-perturbation gap reveals several important patterns. Under Standard prompting, all models show substantial performance degradation on perturbated questions, with average gaps ranging from 20.22 to 39.92 percentage points. The Llama-3.1-8B-Instant shows the largest gap (39.92%), followed by Qwen3-32B (28.61%).

Under CoT prompting, the performance gap narrows considerably for most models. The largest improvement is observed for Qwen3-32B, which reduces its gap from 28.61 to 20.26 percentage points. The Llama-3.1-8B-Instant shows the most dramatic reduction, from 39.92 to 12.70 percentage points. This suggests that CoT prompting helps models better handle perturbations by providing structured reasoning frameworks.

The GPT-OSS models show the smallest gaps under both prompting strategies, indicating greater robustness to perturbations. The GPT-OSS-120B achieves the smallest gap under Standard (20.22%), while GPT-OSS-20B achieves the smallest gap under CoT (11.36%).

5.6 Discussion↩︎

The experimental results reveal several critical insights into the reasoning robustness of contemporary Large Language Models. The differential impact of Chain-of-Thought prompting across models is particularly noteworthy weaker models like Qwen3-32B show dramatic improvements (averaging +23.85 percentage points), suggesting they possess latent reasoning abilities that remain dormant without explicit guidance. In contrast, larger models like Llama-3.3-70B and Llama-4-Scout demonstrate strong inherent robustness with only modest CoT gains, while GPT-OSS-120B and Llama-3.1-8B-Instant surprisingly show performance declines, indicating that not all models benefit uniformly from reasoning instructions and highlighting the importance of model-specific prompting alignment. Persistent challenges in arithmetic reasoning across all models represent a fundamental bottleneck, with perturbations like Adding Operation and Integer-Decimal-Fraction Conversion consistently yielding the lowest accuracy rates below 80% even under CoT. This systematic failure suggests LLMs rely on surface-level pattern recognition rather than genuine mathematical understanding, with tokenization and attention mechanisms prioritizing linguistic patterns over numerical precision a concerning limitation for applications requiring reliable computation.

Most alarmingly, the uniformly poor performance on Critical Thinking perturbations (accuracy below 34% across all models) with minimal CoT improvement (0.00-7.80 percentage points) reveals that current LLMs fundamentally lack complex, multi-step logical reasoning capabilities. This limitation persists regardless of prompting strategy, indicating architectural constraints rather than prompt engineering issues, with profound implications for deployment in legal, medical, or strategic domains requiring genuine reasoning beyond pattern matching.

The seed versus perturbated performance gap analysis shows substantial degradation under standard prompting (20.22-39.92 percentage points), confirming high sensitivity to surface-level input variations. While CoT narrows this gap, persistent degradation suggests models rely on format-specific patterns rather than true reasoning principles. GPT-OSS models demonstrate the greatest robustness, maintaining the smallest performance gaps and suggesting that training diversity improves perturbation resistance. Syntactic and structural transformations show the highest resilience, with Problem Understanding reaching 91.82% under CoT, indicating robust semantic processing. However, arithmetic and higher-order reasoning tasks expose fundamental fragility, with implications for model selection Llama-3.3-70B and Llama-4-Scout are recommended for robust reasoning applications, while Qwen3-32B may serve as a cost-effective option when CoT can be applied. The persistent challenges in arithmetic and critical thinking underscore urgent research directions, including specialized arithmetic training, novel reasoning frameworks, and architectural innovations that support genuine logical reasoning rather than pattern recognition.

6 Limitations↩︎

While this study provides the first comprehensive perturbed mathematical reasoning benchmark for Bangla, several limitations should be acknowledged.

First, our evaluation is restricted to six open-source models due to budgetary constraints. Proprietary models such as GPT-4, Claude, and Gemini were not included, which may limit the generalizability of our findings to state-of-the-art commercial systems. Future work should expand the model roster to include both proprietary and domain-specialized models. Second, we employ only two prompting standard prompting and chain-of-thought prompting. More advanced techniques such as few-shot prompting, self-consistency, Tree-of-Thought, and Program-of-Thought were not explored. A broader investigation of prompting variants could reveal optimal strategies for Bangla mathematical reasoning. Third, the perturbation types evaluated while systematically designed represent only a subset of possible input variations. Real-world user queries exhibit a wider range of linguistic diversity including misspellings, code-switching, and colloquial expressions not captured here.

Finally, our primary evaluation metric is exact match accuracy, which does not capture partial correctness or reasoning quality. Finer-grained error analysis would provide deeper insights into model failure modes.

7 Future Work↩︎

Despite these limitations, GSM-PLUS-BN serves as a valuable foundation for future research. First, we plan to expand our evaluation to include proprietary models (GPT-4, Claude, Gemini) and domain-specialized reasoning models, as our current study was limited to open-source models due to budgetary constraints. Second, we intend to explore a broader range of prompting strategies, including few-shot prompting, self-consistency, Tree-of-Thought, Program-of-Thought, and cross-lingual prompting, to identify optimal approaches for Bangla reasoning tasks. Finally, longitudinal studies tracking model performance over time will reveal whether robustness gaps for low-resource languages are narrowing or widening.

8 Conclusion↩︎

In this study, we evaluated six Large Language Models across eight perturbation types Numerical Substitution, Digit Expansion, Integer-Decimal-Fraction Conversion, Adding Operation, Reversing Operation, Problem Understanding, Distraction Insertion, and Critical Thinking designed to systematically test reasoning robustness from our proposed dataset (GSM-PLUS-BN). We employed Standard prompting and Chain-of-Thought prompting strategies to evaluate model performance, with GPT-OSS-120B achieving the highest seed question accuracy at 92.70%, while Llama-3.3-70B-Versatile emerged as the most consistent performer on perturbated questions, exceeding 85% accuracy on most perturbations under CoT. Qwen3-32B showed the most dramatic improvement with CoT (average +23.85 percentage points), whereas Llama-3.1-8B-Instant demonstrated performance degradation under CoT, highlighting model-specific prompting sensitivities. Critical Thinking perturbations proved universally challenging, with accuracy remaining below 34% across all models, revealing fundamental limitations in complex logical reasoning. Arithmetic perturbations consistently yielded the lowest accuracy, indicating that current LLMs rely on pattern recognition rather than genuine mathematical understanding. CoT prompting effectively narrowed the seed-perturbation performance gap but could not fully compensate for architectural limitations in mathematical and logical reasoning. These findings provide practical guidance for model selection while highlighting critical directions for advancing LLM reasoning capabilities.

References↩︎