1 Introduction↩︎

While larger language models consistently outperform smaller ones on complex reasoning benchmarks, the reasoning processes underlying this advantage remain insufficiently understood. Existing evaluations focus primarily on aggregate outcomes such as accuracy, which reveal little about where larger models excel or how their reasoning differs from that of smaller models[1].

Prior works primarily focus on enhancing smaller models via distillation to address reasoning bottlenecks [2], [3]. While some evaluate reasoning through process supervision [1] or holistic benchmarks [4], the field still lacks a systematic framework for empirically analyzing the qualitative reasoning advantages of larger models.

In this work, we present a cross-scale analysis of reasoning, specifically comparing larger models against their smaller counterparts within the same model family. Inspired by [5], we develop AdvCluster, an automated advantage discovery framework that first identifies benchmark questions where the larger model consistently outperforms the smaller model, and then compares their reasoning traces to extract advantage descriptions. We then organize these descriptions into clusters through a semantic processing pipeline, and use quantitative metrics together with a reviewer model to select among candidate clustering solutions based on its consistency, distinctness, and granularity. Figure 1 illustrates the AdvCluster framework on a representative math example, showing how the reasoning advantages of larger models emerge naturally from the data.

Our analysis categorizes these reasoning advantages into two types: common advantages, which recur across multiple domains, and specialized advantages, which are tied to domain-specific knowledge. A primary common pattern identified by AdvCluster across various domains is Constraint-Guided Reasoning. As illustrated in Figure 1, while smaller models often rely on unguided trial and error, larger models tend to reformulate problems using explicit constraints to systematically guide the solution process.

2 Motivation and Preliminary Observations↩︎

Chain-of-thought prompting improves performance on complex reasoning tasks while exposing intermediate reasoning steps, making reasoning traces analyzable [6], [7]. This increased transparency has motivated a growing body of work on reasoning trajectory analysis [8], [9].

Larger models generally perform better. A separate line of work shows that language model capability generally improves with scale. [10] characterize these gains through neural scaling laws over parameters, data, and compute. Subsequent studies further suggest that some complex capabilities, including multi-step reasoning, emerge only at sufficiently large scales. [11] describe such behaviors as emergent abilities, namely capabilities absent in smaller models but present in larger ones.

Together, these findings indicate that larger models tend to achieve stronger reasoning performance than their smaller counterparts. However, a central question remains: where do larger models exhibit consistent reasoning advantages over smaller ones?

Most existing work instead focuses on improving smaller models, for example by addressing specific bottlenecks such as reasoning data quality through distillation [12]. While such efforts may indirectly suggest where larger models hold advantages, they do not provide a systematic empirical account of how reasoning behavior differs between larger and smaller models.

Preliminary results. We first conduct a preliminary experiment across four domains: mathematics, physics, chemistry, and programming. Our evaluation covers a diverse set of reasoning benchmarks, including HHMT [13], Omni-MATH [14], JEEBench [15], OlympiadBench [16], GPQA [17], and CRUXEval [18]. We evaluate two reasoning model families at different scales: Qwen3-8B vs.Qwen3-32B, and GPT-OSS-20B vs.GPT-OSS-120B [19], [20].

Our preliminary results show a consistent improvement from smaller models to their larger counterparts on these reasoning benchmarks. On average across all evaluated datasets, Qwen3-32B outperforms Qwen3-8B by 6.43%, while GPT-OSS-120B outperforms GPT-OSS-20B by 7.38%. These preliminary results confirm the performance gap between larger and smaller models in our setting and motivate a closer investigation into where these gains arise in the reasoning process. Detailed results per data set are provided in Appendix 7.4.

3 Methodology↩︎

3.1 Dynamic Advantage Classification↩︎

Reasoning advantages are difficult to categorize using a predefined taxonomy, since the forms of model superiority vary across tasks and domains. This makes static category assignment by an LLM judge restrictive and potentially unreliable. [5] address a related challenge in mathematical error analysis with a dynamically adaptive framework, in which categories are induced from data rather than specified in advance. Inspired by this idea, we develop a data-driven pipeline for reasoning advantage analysis. Instead of imposing a fixed taxonomy, we first extract fine-grained advantage descriptions from empirical comparisons between larger and smaller models’ reasoning traces, and then organize these descriptions through semantic clustering. This allows recurring reasoning advantage categories to emerge naturally from the data.

3.2 Our Framework: AdvCluster↩︎

AdvCluster consists of three stages: Analysis Question Set, Advantage Extraction, and Semantic Clustering. In the first stage, we construct an analysis question set by identifying benchmark questions on which the larger model consistently outperforms the smaller model across repeated runs. In the second stage, we construct paired comparisons between the larger and smaller models’ reasoning traces for each question in the analysis set, and use a LLM as an Advantages Extractor to produce fine-grained advantage descriptions from these comparisons. In the third stage, we embed and cluster the extracted descriptions to induce an interpretable taxonomy of larger model reasoning strengths.

3.2.1 Analysis Question Set↩︎

Gap-Based Filtering. Because our goal is to compare reasoning differences between larger and smaller models, we first identify questions for which the larger model consistently outperforms the smaller one across repeated trials. This step isolates analysis questions that reflect stable performance differences.

For each question \(q\), we evaluate both models over \(T\) independent trials. Let \(c_M(q,t)=1\) if model \(M\) answers \(q\) correctly on trial \(t\), and \(0\) otherwise. We define the pass rate of model \(M\) on \(q\) as \[\mathrm{PassRate}_M(q)=\frac{1}{T}\sum_{t=1}^{T} c_M(q,t),\] i.e., the proportion of trials on which \(M\) answers \(q\) correctly.

We define the performance gap for question \(q\) as \[\Delta(q)=\mathrm{PassRate}_{M_L}(q)-\mathrm{PassRate}_{M_S}(q),\] where \(M_L\) and \(M_S\) denote the larger and smaller models, respectively. This quantity measures how much more often the larger model answers question \(q\) correctly than the smaller model across repeated trials.

We retain questions with sufficiently large \(\Delta(q)\) to form the analysis question set. The next stage constructs reasoning comparisons only on this filtered set.

3.2.2 Advantage Extraction↩︎

For each question \(q\) in the analysis question set, we construct reasoning comparisons between the larger and smaller models. Each comparison instance consists of a pair of reasoning traces generated on the same question, one from the larger model and one from the smaller model.

Let \(r^{L}_{q,i}\) and \(r^{S}_{q,i}\) denote the larger model and smaller model reasoning traces in the \(i\)-th comparison instance for question \(q\), respectively. We then use an Advantages Extractor \(\mathcal{J}\) to compare the paired traces and produce a set of advantage descriptions: \[Z_{q,i}=\mathcal{J}(q, r^{L}_{q,i}, r^{S}_{q,i}),\] where each element \(z \in Z_{q,i}\) is a natural-language description of a reasoning advantage exhibited by the larger model over the smaller model in that comparison. In this way, each reasoning pair generates a small set of advantage descriptions.

Concretely, the extractor is implemented with the following prompt:

The complete prompt is provided in Appendix [llmanalyzeprompt]

3.2.3 Semantic Clustering↩︎

After advantage extraction, we obtain a collection of natural-language advantage descriptions from all comparison instances constructed on the analysis question set. These descriptions summarize how the larger model outperforms the smaller model across questions from different domains. Let \[\mathcal{Z}=\{z\}\] denote the set of all extracted advantage descriptions.

Advantage Encoding and Preprocessing. We encode each description \(z \in \mathcal{Z}\) into an embedding vector \[e=\phi(z)\in\mathbb{R}^{d},\]where \(\phi\) denotes the embedding model.

As multiple comparison instances for the same question may yield highly similar advantage descriptions, directly clustering all descriptions can bias cluster centroids toward duplicated patterns. We therefore perform deduplication in the embedding space using a greedy procedure with a cosine-similarity threshold of \(0.95\) [21], [22].

We then reduce the embedding dimensionality via PCA before clustering to mitigate the difficulty of clustering in high-dimensional vector spaces, following common embedding-based text clustering pipelines [23], [24]. \[\tilde{e}=\psi(e)\in\mathbb{R}^{d'}, \qquad d' < d.\] Let \[\mathcal{E}=\{\tilde{e}\}\] denote the deduplicated and low-dimensional advantage embedding vectors.

Clustering and Candidate Generation. We apply K-means clustering to \(\mathcal{E}\) under multiple candidate settings \((d', K)\), where \(d'\) is the PCA dimension and \(K\) is the number of clusters. For each fixed \(d'\), we sweep over a range of \(K\) values and quantitatively evaluate the resulting candidate clustering solutions using Davies–Bouldin Index (DBI) as the primary criterion and the Silhouette score as a secondary reference. Based on this quantitative evaluation, we retain a smaller set of quantitatively favorable candidate clustering solutions.

For each retained candidate clustering solution, we map the low-dimensional embeddings assigned to each cluster back to their underlying natural-language advantage descriptions. We then use a summarization model to produce, for each cluster:

• a short tag;

• a concise definition of the shared reasoning advantage pattern.

The detailed prompt for summarization model is provided in Appendix 7.2.2.

Final Clustering Selection. The quantitative evaluation above narrows the options to a smaller set of retained candidate clustering solutions. However, for downstream results analysis, we seek a clustering solution that is not only quantitatively supported but also semantically well separated, interpretable, and appropriately granular. We therefore use a reviewer model to compare the retained candidate clustering solutions and assess which one is most suitable overall, based on criteria such as internal coherence, separation between clusters, semantic interpretability, and granularity. The prompt for reviewer model is provided in Appendix 7.2.3.

Finally, based on both the quantitative metrics and the reviewer model’s assessment, we select the clustering solution used for downstream results analysis. The selected clustering solution determines the final number of clusters \(K\) and yields \[\{(C_k, t_k, d_k)\}_{k=1}^{K},\] where \(C_k\) is the set of advantage descriptions assigned to the \(k\)-th cluster, \(t_k\) is its tag, and \(d_k\) is its definition.

4 Experiment↩︎

4.1 Setup↩︎

4.1.0.1 Models and Benchmarks.

We study two larger–smaller model pairs from the same family: Qwen3-32B vs.Qwen3-8B, and GPT-OSS-120B vs.GPT-OSS-20B. We evaluate them on benchmarks from four domains: mathematics (HHMT, OMNI, JEEBench), physics (GPQA, JEEBench, Olympiad-level benchmarks), chemistry (GPQA, JEEBench), and programming (CRUXEval).

4.1.0.2 Repeated Inference and Analysis Question Set Construction.

For each model on each dataset, we perform \(T=10\) independent runs. For each question \(q\), we compute the larger–smaller performance gap \[\Delta(q)=\mathrm{PassRate}_{M_L}(q)-\mathrm{PassRate}_{M_S}(q),\] and retain questions with \(\Delta(q)\geq 0.6\). These retained questions form the analysis question set.

4.2 Advantage Analysis Pipeline↩︎

4.2.0.1 Advantage Extraction and Clustering.

Applying the above filtering procedure yields analysis question sets of 115 questions for Qwen3 and 106 questions for GPT-OSS. For each question in the analysis question set, we construct paired reasoning comparisons between the larger and smaller models and use Gemini 3 Pro as the advantage extractor to generate advantage descriptions.

We encode the extracted advantage descriptions using OpenAI’s text-embedding-3-large, then perform semantic deduplication and PCA before clustering the resulting embeddings under multiple candidate settings \((d', K)\). After deduplication, the final analysis corpus contains 1824 advantage descriptions for Qwen3 and 1963 for GPT-OSS.

For each retained candidate clustering solution, we use gpt-5.2 as the summarization model to generate cluster tags and definitions. We then use gpt-5.2 as a reviewer model to assess candidate clustering solutions in terms of internal coherence, inter-cluster distinctness, interpretability, and granularity. Based on this assessment together with quantitative clustering metrics, we select the final clustering solution used in our results analysis.

Detailed domain level statistics and deduplication results are provided in Appendix 7.4.2.

5 Results and Analysis↩︎

On the basis of the quantitative clustering evaluation and reviewer model assessment described in Section 3.2.3, we adopt the final clustering solutions of \(d'=8\), \(K=6\) for Qwen3 and \(d'=4\), \(K=6\) for GPT-OSS. These settings provide the best balance between quantitative cluster quality and semantic interpretability. The tags, definitions, and sizes of the final clusters are provided in Appendix 7.3.

We then use heatmaps to examine how the resulting advantage clusters are distributed across domains. This analysis is motivated by the possibility that different domains may nonetheless share similar underlying reasoning patterns. As shown in Figure 2, some clusters have substantial presence across multiple subjects, whereas others are concentrated in more specific domains. Based on these distributional patterns, we organize the reasoning advantages of larger models into two broad types:

• common advantages recur across multiple subjects.

• specialized advantages are concentrated in a particular subject and are often tied to domain-specific representations or verification demands.

The former indicates that some reasoning strengths are shared across tasks in various domain, while the latter is more evident in specific domains.

For clarity, the names shown in Figure 2 are the cluster tags derived from semantic clustering  3.2.3. We use these tags throughout the following discussion as the corresponding reasoning advantage clusters.

We further find that several clusters can be organized under a higher-level reasoning pattern: Constraint-Guided Reasoning. This refers to cases where the model structures the solution process around explicit or derived constraints, reformulates them into usable conditions, and uses them to restrict the search space, rule out invalid candidates, and support intermediate verification. We begin with this pattern, and then turn to additional reasoning advantages that are more specialized by subject.

5.1 Constraint-Guided Reasoning↩︎

The primary common advantages that emerged in our semantic clustering reveal a similar pattern: Constraint-Guided Reasoning. Rather than relying on trial and error over surface forms, larger models more often begin with a problem’s explicit constraints and implicit constraints and reformulate them into more usable conditions, such as feasibility conditions, boundary conditions, invariant relations, and consistency checks. This allows the larger model to systematically narrow the solution space, remove invalid candidates, and verify intermediate steps in reasoning.

Qwen3 exhibits this pattern most clearly in its common advantages such as #5 Reducing to Existence Constraints and #2 Recasting via Transformations. Similar pattern appears in GPT-OSS through its common advantages like #3 Enforcing Constraint Consistency and #2 Deriving Invariant Parameterizations. Here, the larger model first organizes the problem into a unified set of constraints, then uses invariant relations or parameterized forms to derive exact formulas, inequalities, or bounds. The larger model uses them to maintain consistency with the original problem structure and to rule out conclusions that violate the constraints.

A representation case of Constraint-Guided reasoning is discussed earlier in Figure1. The larger model first performs structural reformulation, then advances the solution through boundary proof and reparameterization. By contrast, the smaller model stays closer to local trial and error over surface cues, making ineffective search and incorrect paths more likely.

This is also revealed in specialized advantages, where it is expressed through the solving tools of a particular domain:

• In chemistry, Qwen3 exhibits #0 Constraining via Invariants. Since chemistry reasoning process relies heavily on invariants–such as balances, conserved totals, configuration states, the model translates descriptions into a formal representation, derives quantities that must remain valid, and uses them to eliminate inconsistent candidates or revise conclusions when general heuristics conflict with harder constraints.

• In mathematics, GPT-OSS naturally exhibits #0 Deriving Tight Bounds and #5 Deriving Modular Periods. The former advantage allows larger model to turn the problem into a counting system based on constraint and propagates local requirements into global feasibility or contradiction. In #5 Deriving Modular Periods, it rewrites the problem as a modular system, tracks invariant residues, and derives periodic behavior from that constraint-guided reformulation.

5.2 Other Reasoning Advantages↩︎

Beyond Constraint-Guided Reasoning, we observed other advantages emerge in specific domains:

Execution Tracking. This is primarily seen in programming tasks. Qwen3 demonstrates this in #4 Simulating Stateful Execution. Larger models track operations step-by-step and update state variables after every single action. Throughout this process, strict control flow is maintained. Finally, intermediate cross-checks effectively eliminate state drift and missed updates.

Format Control. This reasoning advantage appears in programming tasks. For instance, in Qwen3’s #1 Verifying Exact Sequences, the model precisely tracks elements and boundaries; consequently, it verifies the constructed output against properties like total length and delimiter placement. Similarly, in #3 Enforcing Output Conformance, it identifies structural and syntax constraints early on, then seamlessly converts intermediate results into the required format. Furthermore, GPT-OSS exhibits a related pattern in #4 Preserving Output Fidelity. During complex operations, this model maintains integrity at character level and corrects minor local anomalies. Ultimately, these clusters demonstrate that larger models excel at presenting reasoning in the exact required form.

Representation Alignment. In the chemistry domain, a specialized advantage demonstrated by GPT-OSS is #1Reconciling Representations to Outcomes. This comprehensive ability requires the model to seamlessly fuse multiple diverse descriptions of the same system. By jointly processing 1D symbolic stoichiometry, 2D topological connectivity, and dynamic state variables, the model effectively synthesizes these different chemical perspectives. By successfully organizing these features, the larger model can dynamically track complex chemical transformations and accurately project intermediate states into the correct final outcome.

5.3 Validation via SLM Systemic Failures↩︎

Beyond identifying the advantages of larger models, we validate our findings by analyzing where smaller models systematically fail. Using the Error Dispersion Index (EDI, Appendix  7.5) to quantify "First-Error" consistency, we found that both Qwen3-8B and GPT-OSS-20B yield an average EDI \(< 0.4\). This confirms that their failures are not stochastic, but rather stem from fundamental capability deficits.

As shown in Figure 3, these deficits heavily concentrate at the Transformation (Depth 3) and Process Skills (Depth 4) stages. This distribution directly mirrors the core advantage of larger models: Constraint-Guided Reasoning. Lacking the consistent support of this capability, smaller models either fail to abstract problem into bounded mathematical representations (Depth 3), or they resort to unconstrained, localized trial-and-error that inevitably diverges during execution (Depth 4). Ultimately, these bottlenecks provide empirical evidence that Constraint-Guided Reasoning constitutes the primary capability gap between smaller and larger models.

6 Conclusion↩︎

This work aims to better understand how reasoning changes with scale within the same model family. We develop AdvCluster, an automated analysis framework that systematically compares larger and smaller model reasoning traces, extracts advantage descriptions, and organizes them through semantic clustering. This allows recurring patterns to emerge naturally from the data and yields a structured taxonomy of common advantages and specialized advantages.

Our main finding is that larger models across different model families consistently exhibit Constraint-Guided Reasoning: they are better at identifying conditions, organizing them into structured constraints, and using them to guide reasoning and eliminate infeasible paths. We hope this work provides a clearer starting point for understanding the reasoning advantages that emerge with scale.

7 Appendix↩︎

7.1 LLM Usage Summary↩︎

7.2 Prompt↩︎

[1][] colback=gray!20!white, colframe=gray!50!black, title=Prompt, fonttitle = , boxrule=0.2pt

7.2.1 Advantage Extractor prompt↩︎

[
  {
    "type": "failure", 
    "failure_stage": "<ONE OF: Reading | Comprehension | Transformation | 
       Process Skills | Encoding>",
    "failure_description": "<Brief description of Model_B's FIRST failure>"
  },
  {
    "type": "advantage",
    "advantage": "<Action verb + abstract skill (max 10 words)>",
    "evidence": "<Specific text contrast proving the advantage>"
  },
  {
    "type": "advantage",
    "advantage": "<another Action verb + abstract skill (max 10 words)>",
    "evidence": "<another specific text contrast proving the advantage>"
  },...
]

7.2.2 Summarization model prompt↩︎

{ 
  "tag": "...",
  "definition": "..."
}

7.2.3 Reviewer Model prompt↩︎

K = {k}
CLUSTERS = {clusters_section}    

{{
  "Distinctness_score": [1-10],
  "Granularity_score": [1-10],
  "Interpretability_score": [1-10],
  "Balance_score": [1-10],
  "Taxonomy_resolution_score": [1-10],
}}

7.3 Final Clustering Solutions↩︎

Tables 2 and 3 present the tags and definitions for all clusters in the final clustering solutions (Qwen3: \(d'=8\), \(K=6\); GPT-OSS: \(d'=4\), \(K=6\)). These were generated during the semantic clustering stage and are included for reference. Table 4 reports the size of each final cluster.

7.3.1 Tags and Definitions↩︎

7.3.2 Cluster Sizes↩︎

7.4 Supplementary Data↩︎

7.4.1 Accuracy on Benchmarks↩︎

The following four tables contain detailed data from the preliminary results across all datasets, with each table corresponding to one of the four domains: mathematical, physics, chemistry, and programming.

7.4.2 Analysis Question Set↩︎

Table 9 provides the detailed counts of the 115 (Qwen3) and 106 (GPT-OSS) questions retained after gap-based filtering, categorized by domain.

7.4.3 Advantage32Descriptions3232Statistics↩︎

Table 10 summarizes the effect of semantic deduplication on the extracted advantage descriptions for each model pair.

7.5 Experimental Details for First-Error Analysis↩︎

To investigate the systemic bottlenecks of smaller models, we conducted the "First-Error" analysis. Our goal is to determine whether these models make stochastic (random) mistakes or exhibit recurring error.

7.5.0.1 First-Error Extraction Pipeline

We filter for challenging questions for smaller models, where the performance gap between the larger and smaller models is \(\ge 0.6\). For each question, we prompt the Advantage Extractor to identify the exact reasoning step where the smaller model commits its First Error during the Advantage Extraction stage. We adapt Newman’s Error Analysis framework, grouping reasoning failures into five sequential depths (1 to 5):

1. Reading: Failing to recognize basic text, symbols, or vocabulary.

2. Comprehension: Misunderstanding the overall objective or meaning of the problem.

3. Transformation: Failing to translate the problem description into a valid mathematical or logical operation.

4. Process Skills: Committing errors during step-by-step calculations or rule execution.

5. Encoding: Failing to properly format or express the final derived solution.

7.5.0.2 Error Dispersion Index (EDI) Formulation

To quantify error consistency, we propose the Error Dispersion Index (EDI), a question-level metric based on normalized empirical Shannon entropy. For a given question, assuming \(N\) independent reasoning trajectories sampled from a smaller model, let \(c_i\) be the count of "first errors" occurring at depth \(i\) out of \(|D|=5\) possible depths. The empirical probability is \(P(d_i) = \frac{c_i}{N}\).

The empirical Shannon entropy (in bits) for this specific question is calculated as: \[H = - \sum_{P(d_i)>0} P(d_i) \log_2(P(d_i))\]

To ensure the metric is bounded within \([0, 1]\), we normalize \(H\) by the exact empirical maximum entropy, \(\mathcal{H}_{max}(N, |D|)\). Conceptually representing a state of completely random guessing, \(\mathcal{H}_{max}\) is defined as the entropy achieved when the \(N\) discrete errors are distributed as evenly as possible across the \(|D|\) reasoning depths. The per-question EDI is then defined as: \[EDI = \frac{H}{\mathcal{H}_{max}(N, |D|)}\]

• \(EDI \to 1\): Errors are uniformly scattered (stochastic).

• \(EDI \to 0\): Errors isolate to a single depth (deterministic).

7.6 Summary of Error Dispersion Data↩︎

While the EDI is computed independently for each question, we average these scores across all questions within a specific domain to evaluate broader systemic trends. As detailed in Table 11, across all domains, smaller models consistently yield an average EDI below \(0.4\). This confirms that their reasoning failures are predominantly structural deficits rather than stochastic hallucinations.

References↩︎