1 Introduction↩︎

Recent advances in Large Language Models (LLMs) show a promising trend to support a wider range of languages. For example, the pretraining of Qwen3 models [1] included over 100 languages compared to 29 for Qwen2.5 [2]. Nevertheless, their performance is limited to basic capabilities, while their effectiveness in tasks that require complex reasoning remains concentrated in a limited set of languages, such as English and Chinese [3].

Consequently, prior work predominantly leverages the reasoning on a strong language, typically English. For instance, given a question in the source language, the model can be prompted to reason in an English chain-of-thought instead of the native language [4], [5]. As an alternative, the source-language query is translated into English in a first step, to then perform the reasoning in English in a second step. Translations can be provided by the model itself (self-translation), or, alternatively, by an external machine translation engine [4]. Though simple, these strategies limit applicability in realistic scenarios where reasoning steps must be expressed in the source language.

In order to reduce the performance gap when reasoning in the source language, prior work focused on improving the cross-lingual alignment. Some works use supervised fine-tuning (SFT) on translations of questions to and from the reference languages [5] or control the response generation by manipulating the logits to mimic those of English [6], [7]. Other works use post-training methods, namely DPO [8], to push models to generate correct reasoning traces translated from English and discourage incorrect, natively generated ones [9], [10]. These approaches rely on translated questions and reasoning traces for supervision and, as a result, generalization to unseen languages and domains is constrained by both machine-translation quality and the scarcity of high-quality manually translated data [3].

In this paper, we propose to use cross-lingual self-consistency as an unsupervised signal for RL. Our method extends existing monolingual self-consistency approaches, which capture the intuition that different reasoning paths should lead to consistent final answers, to further push the model to produce the same final answer for equivalent problems in different languages (Figure 2). This design guarantees broad practical applicability, since it requires no access to ground truth labels nor pre-translated reasoning traces. This is specially useful given that multilingual reasoning data is only available for evaluation, and training resources are largely monolingual [3]. Our experimental setup only uses English questions.

We demonstrate the effectiveness of our method across three model scales, achieving an average performance improvement of 7.8% to 21.7% over 10 languages on the MGSM benchmark (Figure 1). The improvements can be amplified when high-quality translations for the questions are available (§6.4). Furthermore, our approach exhibits strong generalization, yielding up to 18.2% average improvement on unseen languages within MGSM and extending effectively to other domains with a 4.7% average gain on the mGPQA [11] and 6.8% on MMATH [12] benchmarks (§[sec:gen]). Collectively, these results show that our method is effective at narrowing the multilingual reasoning gap in LLMs, advancing their robustness and equity across languages.

2 Background↩︎

Recent work has shown improvements in the monolingual reasoning capabilities of LLMs by maximizing the consistency of the model [13]–[17]. We next formalize this framework, and present our proposed extension to the multilingual scenario in §3.

Given an input \(x\), let \(\{(c_i, y_i) \sim \pi(\cdot \mid x) \}_{i=1}^N\) denote a set of \(N\) trajectories sampled from the policy model \(\pi\), where \(c_i\) is the chain-of-thought reasoning and \(y_i\) is the final answer following it. The core idea behind these methods is that, even if the reasoning paths \(c_i\) can differ, a strong model should always arrive at the same (ideally correct) final answer \(y_i\). This principle of final answer consistency can be optimized through RL, even if the ground truth answer is not known. To that end, existing methods minimize the entropy of the marginalized answer distribution.

More precisely, let \(\hat{p}_{\pi}(y|x)\) be the empirical probability of a specific final answer \(y\): \[\hat{p}_{\pi}(y|x) = \frac{1}{N}\sum_{i=1}^N \mathbf{1} (y_i=y)\] The aggregated reward over all sampled generations of \(x\) is the negative of the entropy of this distribution: \[\label{eq:r95total95mono} \begin{align} R_{\mathrm{total}}(x) &= -H(\hat{p}_\pi (\cdot|x)) \\ &= \sum_{y} \hat{p}_\pi(y|x) \log \hat{p}_\pi(y|x) \end{align}\tag{1}\] and the reward of each individual generation is decomposed as follows: \[R(x, c_i, y_i) = \log \hat{p}_\pi(y_i|x)\] so it holds that \(R_{\mathrm{total}}(x)=\frac{1}{N}\sum_{i=1}^N R(x, c_i, y_i)\). This reward function is optimized through RL, with GRPO [18] being a common choice.

3 Proposed Method↩︎

In the multilingual scenario, not only do we want models to produce consistent final answers within each language, but we also want them to produce consistent final answers for equivalent inputs in different languages. We next propose a multilingual extension of the framework in §2 that formalizes this principle of cross-lingual self-consistency.

Let \(\mathbf{x}=\{x^{(l)}\}_{l=1}^L\) be a set of parallel inputs in \(L\) languages, and let \(\{(c_i^{(l)}, y_i^{(l)}) \sim \pi(\cdot \mid x^{(l)}) \}_{i=1}^N\) be a set of \(N\) generations for the input in language \(l\). We assume the final answers \(y_i^{(l)}\) are normalized into a shared canonical form (e.g., numeric values), so that equality is well-defined across languages. We define the following aggregated reward for the input in each language \(x^{(l)}\), which replaces the entropy term in Equation 1 with the cross-entropy with respect to a second distribution \(\hat{q}_\pi\): \[\begin{align} R_{\mathrm{total}}(x^{(l)}) &= -H \left(\hat{p}_\pi (\cdot|\mathbf{x}, l), \hat{q}_\pi (\cdot|\mathbf{x}, l) \right) \\ &= \sum_{y} \hat{p}_\pi(y|\mathbf{x}, l) \log \hat{q}_\pi(y|\mathbf{x}, l) \end{align}\] where \(\hat{p}_{\pi}(y|\mathbf{x}, l)\) is the empirical marginalized answer distribution in language \(l\): \[\hat{p}_{\pi}(y|\mathbf{x}, l) = \frac{1}{N}\sum_{i=1}^N \mathbf{1} \left( y_i^{(l)}=y \right)\] and \(\hat{q}_{\pi}(y|\mathbf{x}, l)\) is the empirical target distribution, for which we define several alternatives below. Replacing entropy with cross-entropy allows each language-specific answer distribution to be aligned to a target distribution derived from other languages, thereby operationalizing cross-lingual consistency. The reward for each sampled generation is given by \[R \left(x^{(l)}, c_i^{(l)}, y_i^{(l)} \right) = \log \hat{q}_\pi \left(y_i^{(l)}|\mathbf{x}, l\right)\] so, analogous to the monolingual case, it holds that \(R_{\mathrm{total}}(x^{(l)})=\frac{1}{N}\sum_{i=1}^N R(x^{(l)}, c_i^{(l)}, y_i^{(l)})\).

In our experiments, we explore the following alternatives for \(\hat{q}_\pi\), which differ on the language(s) they use to estimate the target distribution:

1. The source language \(l\): In this case, the source and target distributions are the same: \[\hat{q}_{\pi}(y|\mathbf{x}, l) = \hat{p}_{\pi}(y|\mathbf{x}, l)\] Given that \(H(p, p)=H(p)\), this is equivalent to minimizing the entropy of the marginalized answer distribution for each language without any constraint on cross-lingual consistency. When \(L=1\), this further becomes equivalent to the formulation in §2, making our framework a strict generalization of monolingual self-consistency.

2. The reference language \(\mathrm{en}\): We take the marginalized answer distribution in another (reference) language, which, without loss of generality, we set to English:¹ \[\hat{q}_{\pi}(y|\mathbf{x}, l) = \max \left(\hat{p}_{\pi}(y|\mathbf{x}, \mathrm{en}), \epsilon\right)\] \(\epsilon\) prevents the reward becoming \(-\infty\) when \(\hat{p}_{\pi}(y|\mathbf{x}, \mathrm{en})=0\) and \(\hat{p}_{\pi}(y|\mathbf{x}, l)>0\), while still strongly favoring answers that are probable in English.² This variant can be seen as a form of cross-lingual distillation that, by aligning all languages to the same reference, indirectly makes them more consistent with one another.

3. All languages combined: We take the empirical answer distribution marginalizing over all languages and trajectories: \[\hat{q}_{\pi}(y|\mathbf{x}, l) = \frac{1}{L} \sum_{k=1}^L\hat{p}_{\pi}(y|\mathbf{x}, k)\] This variant directly optimizes cross-lingual self-consistency by pushing the final answer distribution in each language to match the aggregated one across all languages. Unless otherwise indicated, we use this variant in our experiments.

Our approach is unsupervised in that it does not require any parallel data labeled by humans. To obtain these, we adopt self-translation: starting from a set of English questions, we prompt the model itself to translate them into other languages. Although self-translation is noisy, we adopt it for its practicality and scalability, while avoiding external translation systems that could inject additional knowledge into the model. In practice, we find that aggregating multiple generations—e.g., via majority voting at high sample counts—substantially mitigates translation errors and yields reliable baseline performance (see App. [app:maj64k] for details).³

4 Experimental Setup↩︎

4.0.0.1 Base Model

We conduct our experiments and ablations on the Qwen-2.5 family of models [2]. We chose those models since they (i) are pre-trained and supervised fine-tuned on over 29 languages, (ii) come in a handful of practical sizes, facilitating ablations at scales, and (iii) widely used in multilingual research. In addition, we experiment with the Llama3 8B model [19] to compare with prior work that use Llama-based models.

4.0.0.2 Baselines

We evaluate our approach against two categories of baselines.

(A) Prompt-based methods. We consider two prompting strategies: Original model performance using native-cot, where the model performs reasoning and produces answers in the original language of the query, and translate-test, where the query is first translated into English and the model subsequently solves the problem in English [20].

(B) Finetuning methods. We include two of the best-performing approaches that improve reasoning via finetuning. The first is test-time scaling (S1) [21], which applies SFT on English data to enhance reasoning capabilities. The second is LIDR [9], which leverages DPO to improve multilingual reasoning performance. We train both methods on the Qwen2.5 models ourselves using the official code bases. In addition, we report relative performance gains compared to the prior published methods that use Llama base models.

4.0.0.3 Monolingual Training Dataset

Our method only requires English questions for training. We randomly sample 1k questions from the training split of the English-only GSM8k dataset [22]. The final output is parsed using the standard regular expressions from the original benchmark. As discussed in §3, we use self-translation to obtain parallel questions, and analyze the impact of manual translation in §6.4

4.0.0.4 Evaluation Data

Main evaluation and ablations are done on MGSM [23], which consists of 250 math word problems in 10 languages, which were sampled and manually translated from the GSM8K test set. In addition, we evaluate the generalization capabilities on 3 OOD datasets: mGPQA [11], MMath [12], and PolyMath [24].

4.0.0.5 Training Environment and Hyperparameters

We use GRPO as the RL algorithm and use VeRL [25] for the training backend. We use a batch size of 32 parallel prompt sets (making a total of 320 prompts per step across all the 10 languages), with 32 rollouts per language and prompt, resulting in a total of 160 steps. The mini-batch size is set to 64 prompts (5 mini-batches per step), the KL coefficient to 0.001 and the learning rate to 1E-06. We set the temperature to 0.4.⁴ All models are trained using 4\(\times\)A100 GPUs. Other hyperparameters are left at the default values.

5 Results↩︎

5.0.0.1 Main Results

Table [tab:main-res] contains the main results of our method. As show in Table [tab:main-res-qwen], we achieve average gains of 7.8, 15.1, and 21.7 points in Qwen2.5-IT 1.5B, 7B, and 14B, respectively, with improvements across nearly all languages. Notably, gains increase consistently with model size. We note that in Qwen-2.5 14B, Russain (ru) and Telugu (te) achieve the highest relevant performance gains: 102% and 104% respectively. We also observe degradation in Swahili for Qwen-2.5 7Band in English and Telugu in the Qwen-2.5 1.5Bmodel.

Models trained with S1 test-time scaling remain competitive with our approach. In particular, they outperform our method on 3 languages for both Qwen-2.5 14Band Qwen-2.5 7B. However, these gains come at the cost of substantially longer reasoning traces, requiring approximately 3× more tokens. In contrast, our method achieves stronger overall performance with considerably more efficient token usage (see App. [app:token-budget]). Furthermore, the improvements from S1 are driven primarily by the quote-and-think paradigm [21], where the model performs intermediate reasoning in English before generating the final answer in the target language. Consequently, these gains do not necessarily indicate improved intrinsic multilingual reasoning ability (see App. 12.1).

5.0.0.2 Performance Difference w.r.t English

English is the best-performing language in the all baseline models. We examine the performance difference between English and other languages before and after training with our method. We partition languages into high-, medium-, and low-resource groups according to their representation in Common Crawl archives.⁵ and show the maximum performance gap within each tier in Figure 3. Notably, the gap in performance is reduced up to 4 folds in High-resource languages for the Qwen-2.5 14Bmodel. This suggests that cross-lingual performance gaps are not solely attributable to differences in factual or linguistic coverage, but may instead arise from inconsistencies in multilingual reasoning behavior. We observe similar behavior in the Medium-resource tier. Across model sizes, our method yields a comparable performance gap of 23-28% in the Medium-resource group. We observe that the difference increases in Low-resource languages for the Qwen-2.5 7Bmodel, which can be explained by the degradation observed in Swahili for that particular model.

Although our method yields the largest gains for low-performing languages, they still lag behind high-resource languages. This reflects limitations of the multilingual backbone model (Qwen), whose weaker language proficiency in these languages constrains our method’s efficiency.

5.0.0.3 Comparison to Prior Methods

Head-to-head comparison with alternative methods to improve multilingual reasoning is obscured by different test sets, languages or base models. In Table [tab:main-res-prior] we report results on MGSM or other methods including INCLINE [6], Mathoctopus [26], MetaMathOctopus [27], LIDR [9] and MAPO [27].⁶ The methods are applied to Llama2 or Llama3 base models. We run our self-consistency method on Llama3-8B model, and obtain the best results on average.

6 Analysis and Discussion↩︎

Below we analyze whether our method improves the cross-lingual self-consistency of the model, whether it generalizes to unseen domains and languages, the impact of the target distribution and that of multilingual datasets.

6.1 Cross-Lingual Self-Consistency↩︎

Monolingual self-consistency methods work by minimizing the entropy of the marginalized answer distribution (see §2). For easier interpretability, we next analyze the monolingual self-consistency of different models by taking the frequency of the most repeated answer instead: \[\max_i \frac{1}{N} \sum_{j=1}^N \mathbf{1} \left( y_i^{(l)}=y_j^{(l)} \right)\] Similarly, we measure cross-lingual self-consistency for each language \(l\) by taking the maximum frequency with which any of its answers appears across the rest of the languages: \[\max_i \frac{1}{N (L-1)} \sum_{\substack{k=1 \\ k \ne l}}^L \sum_{j=1}^N \mathbf{1} \left( y_i^{(l)}=y_j^{(k)} \right)\] We analyze self-consistency on MGSM before and after applying our method to Qwen-2.5 7B. To that end, we sample 32 parallel responses per example and language, and compute the average monolingual and cross-lingual self-consistency for each language. As shown in Figure 4, our approach greatly improves both consistency measures, with monolingual consistency remaining higher than cross-lingual consistency. In Appendix [app:training-curves] we show that self-consistency improves as training progresses.

We also analyze the consistency of our method on the final accuracy performance using using majority voting (Maj@k) with varying sample sizes. As shown in Figure 5, models trained with our approach reach near-peak performance at relatively low k: Maj@8 achieves over 95% of the accuracy achieved at Maj@32, suggesting our method is optimizing for inference compute. All model sizes exhibit the same behavior, as we show in Appendix 10.1. We observe high inconsistency in the results of S1 trained models at higher values of K, mainly in the medium- and low-resource languages. We also note that prompt-based baselines — particularly self-translation— continue to benefit from increasing k, making it a reliable basis for our method.

6.2 Generalization↩︎

max width=1.0

6.2.0.1 Out-of-distribution Generalization

We evaluate our models trained with the GSM8k sample on three Out-of-distribution benchmarks: mGPQA, MMATH, and PolyMath. As shown in Table [tab:gen-domain], our method achieves the best generalization performance. On mGPQA, we acheive an average gain of 2.1%, 4.3%, and 4.7% for the 1.5B, 7B, and 14B models, respectively, surpassing that of S1-trained models. On MMath, our method acheives up to 6.2% gain in the 14B model. Conversely, LIDR has an adversarial impact on the performance, causing performance degradation on all OOD benchmarks. The PolyMath benchmark is the furthest from the GSM8k distribution, yet our method achieves the best relative performance gains. This corroborates the strength of our method in cross-domain generalization. We also observe that generalization benefits from model scale, indicating that larger models are more capable of leveraging our training signal.

6.2.0.2 Cross-lingual Generalization

We train the Qwen-2.5 1.5BIT and Qwen-2.5 7Bmodels on a random subset of 1, 2, and 3 languages and evaluate their performances on the remaining, unseen languages. As shown in Table 1, our method achieves substantial improvements in the unseen languages, reaching up to 18.2% for the Qwen-2.5 7B-IT model. Notably, when the model is trained on a single source language—e.g., Bengali—the performance gain in that language is maximized. Nonetheless, including additional languages during training enhances the model’s overall generalization to other unseen languages.

6.3 Different Target Distributions↩︎

We now analyze the strategies to estimate the target output distribution \(q(y)\) used in reward modeling (see § 3). As shown in Table [tab:ablate-tgt-dist], using the source language outputs to construct the target distribution yields the least improvement. However, it can be useful in cases where the model is initially indecisive, such as Qwen-2.5 1.5B.

In contrast, constructing the target distribution from the outputs of English or combining the outputs of all languages achieves competitive performance. Specifically, using English outputs yields 1.4% more improvements than the combined outputs for Qwen-2.5 7Bin MGSM. We hypothesize this is due to the less noisy English outputs and that English yields significantly higher base performance (at least 18% above other languages). Nonetheless, we believe that the choice of combined output is favorable, especially in domains and models where English performance is not dominant or other languages perform competitively. In addition, the fact that our method performs well even when considering weaker languages makes it robust for wide applicability.

6.4 Access to multilingual data for training↩︎

Our main approach accounts for domains where reasoning data is only available in one or a few languages by leveraging the translation capability of the model across supported languages. We now analyze the efficiency of our method in cases when multilingual data is accessible.

We post-train our models on the Multilingual Grad School Mathematics (MGSM) dataset [23]. The dataset includes 250 examples from the test set of the GSM8k benchmark, manually translated into 10 languages besides English. We train on 25% of the MGSM sampled uniformly over languages and evaluate on the remaining 75%. As shown in Table [tab:train-mgsm-summarized], this setup improves the results for the smaller models, but gets comparable average results for the largest model. We hypothesize that the largest model has enough quality to self-translate, and that, after all, it is the self-consistency of the trajectories that matters most, and that adequate translation of the questions suffices.

max width=1.0

7 Related Work↩︎

7.0.0.1 Multilingual Reasoning with Supervised Fine-Tuning.

Prior work addresses cross-lingual reasoning gaps by fine-tuning models on translated question pairs to improve alignment across languages [5], [26]. However, the scarcity of high-quality parallel data and the noise introduced by machine translation limit scalability and performance on complex reasoning tasks, making English-only reasoning (English-cot) remain the most effective baseline [4], [28]. Some works add a second step with preference optimization, which requires external translation, and use English trajectories as preferred with respect to native trajectories [9], [27]

7.0.0.2 Multilingual Reasoning with Inference-Time Intervention.

To bypass data limitations, recent studies explore inference-time strategies adapted from monolingual reasoning. Longer chains of thought [21] and multilingual ensembling via majority voting [20] both improve cross-lingual performance, with English often serving as the most reliable reasoning language. As an alternative [6], [7], response generation can be manipulated using the logits to mimic those of English. Nevertheless, they perform poorly compared to supervised fine-tuning methods.

max width=1.0

7.0.0.3 Monolingual Reasoning.

Outcome-based reward modeling has recently shown that enforcing output consistency can improve reasoning without ground-truth supervision, but existing work is confined to English [13], [14], [16]. We extend this paradigm to multilingual reasoning.

8 Conclusion and Future Work↩︎

In this paper, we show that incentivizing cross-lingual consistency improves reasoning in the source language. Our method uses only monolingual questions, applying reinforcement learning with an unsupervised cross-lingual consistency loss to further train an existing model. It assumes no preferred language and enforces consistency across all languages. Experiments on MGSM show substantial gains, especially for weaker languages, while also improving stronger ones and reducing performance gaps. Results on OOD benchmarks indicate good generalization, though with smaller gains. Since our method relies only on monolingual questions, we hypothesize that training on more diverse domains could further enhance generalization.

In the future we would like to explore further domains, and extend our method to non-verifiable tasks. All things considered, our findings pave the way to improve the multilingual reasoning performance at large without any extra data annotation.

Limitations↩︎

Our experiments are limited to sub 20B LLMs, primarily due to compute constraints. While we observe that our method is increasingly effective as the size grows it would be interesting to explore its behaviour for larger models. In addition, we used the Qwen2.5 Models; It would be efficient to see the impact on more recent models such as Qwen3 [1].

Our training and evaluation were highly constrained by the data. The limited availability of reliable multilingual reasoning benchmarks—particularly for low-resource languages—restricts broader validation and remains a hindering challenge. We hope to address some of these limitations in future work.

Acknowledgments↩︎

9 Self-consistency Training Curves↩︎

In Figure 6 (top), we show that our method yields a steady progression for all models. The progression improves steadily for all models, and the upward trend suggests further improvements could be achieved by scaling up the training compute. Concurrent to the improvements in cross-lingual consistency, we show in Figure 6 (bottom) that the entropy of the model’s policy is steadily regularized as training progresses. This supports our view of the method as a cross-entropy regularization technique and links the improvement in cross-lingual consistency with the improvement of the model entropy.

10 Self-consistency Accuracy Performances↩︎

10.1 Majority@k Performances↩︎

Figure 7 shows the performance of Qwen-2.5 IT models using Majority@k, for \(k \in \{1,8,32\}\). Our method achieves near-peak performance at lower samples, making it more efficient with respect to inference budget.

10.2 Detailed Performances↩︎

max width=1.0

In Table [tab:acc-at-k], we show the performance of different methods by computing the accuracy with majority vote at different sample sizes (Maj@k). Ideally, a self-consistent model achieves the best performance at a lower sample size, hence a more efficient inference token consumption. Models trained with our approach achieve their highest performance at lower K values, with Maj@8 achieving more than 95% of that of Maj@32.

In addition, results show that the performance of prompt-based methods, specifically self-translation, improves consistently with higher K, corroborating our assumption that they provide a reliable basis for our method.

11 Detailed Performances↩︎

Tables [tab:gen-domain] – [tab:train-mgsm] show the detailed performances of our method training.

max width=1.0

max width=1.0

max width=1.0

12 Comparison to S1 trained models↩︎

Models trained with S1 test-time scaling in English exhibit impressive improvements in other languages [21]. They achieve competetive performances to our method. We analyze the reasoning traces of the S1 models and ours to better compare the methods.

12.1 Reasoning Traces↩︎

In Box 1, we show traces of Qwen2.5 14B trained with our method and the s1-method on a Spanish question. Models trained with S1 test-time scaling technique exhibit a quote-and-think paradigm where they do reasoning first in English and deliver the final response –and occasionally short chain of thought– in the prompt language. Albeit efficient, this does not imply that the S1 training recipe induces better multilingual reasoning, but rather surfaces the model’s translation capabilities as a proxy for enhanced reasoning.

12.2 Test-time Token Consumption↩︎

max width=1.0

S1 models tend to do longer reasoning in English, while our method arrives at the same answer with shorter and language-consistent chain-of-thought. In Table  [tab:resp-token-len], we show the average and maximum token length of the responses per language using Qwen-2.5 14Btrained with our method and with S1 test-time scaling.

References↩︎

---

1. In practice, one would choose the language that is expected to have the strongest monolingual performance which, in most scenarios (including ours), is English.↩︎

2. We set \(\epsilon=0.01\) in our experiments, but find that it is rarely triggered in practice.↩︎

3. Training on noisy signals is common in machine learning and can improve generalization. For example, although applying dropout at test time can hurt performance, it remains a standard and effective training technique.↩︎

4. Setting the temperature to a higher value sometimes led to unstable training. We leave investigating this for future work.↩︎

5. commoncrawl.github.io/cc-crawl-statistics/plots/languages↩︎

6. Note that Telugu was not reported for any prior model.↩︎