1 Introduction↩︎

Supervised finetuning [1] has emerged as a standard approach for adapting language models (LMs) to specific target languages [2], [3]. Central to the success of SFT is the availability of high-quality training data, consisting of pairs of user prompts and a corresponding response, which is often scarce for less-resourced languages [4]. Generating prompt-response pairs for these languages demands substantial human effort [5], [6], creating a bottleneck for language-specific model development.

To alleviate the challenge of human effort and data scarcity, synthetic data generation using LMs has gained traction as a promising solution for multilingual LM development [7]–[10]. This approach involves leveraging a typically larger teacher model to generate training examples, which are then used to finetune a smaller student model to replicate the knowledge of the teacher [11]. However, existing works often select teacher models arbitrarily, defaulting to the largest state-of-the-art models that excel on benchmarks [12]–[14]. This practice is problematic because these models, despite strong performance, may have significant capability gaps in non-English languages, leading to poor-quality synthetic data that propagates the teacher’s weaknesses rather than its strengths. And so we ask: “what makes an effective multilingual teacher for synthetic data generation, and how can we systematically measure it?”

In this work, we conduct a comprehensive analysis of 10 LMs across 6 typologically diverse languages on three common synthetic data generation methods: responding to a user query or instruction, translating prompts from English to a target language, and generating prompt-response pairs given in-context examples (§2.2). To systematically assess teacher model effectiveness, we evaluate LMs using both intrinsic measures of data quality (§2.2, i.e., the diversity of prompts and responses, the perplexity of the base model on the response, and response quality based on a multilingual reward model) and an extrinsic measure of student model performance on multilingual tasks (§2.3, cultural understanding, mathematical reasoning, general chat). We aggregate these measurements into a single metric called Polyglot Score (PG-Score), in order to provide a holistic assessment of a teacher model’s data generation capabilities. Our contributions are as follows:

• We close the evaluation gap by evaluating 10 teacher models, generating over 1.4M SFT examples and finetuning 240 student models from OLMo 3 7B. We find that Gemma 3 27B consistently ranks within the top three highest PG-Score and that the Gemma 3 model family outperforms other families such as Llama 3.1 and IBM Granite (§3.1). Our PG-Score rankings are consistent across other base model families (Llama 3.1 8B, Qwen 3 8B, Gemma 3 4B, §3.2).

• We provide analyses and insights on the characteristics of a good multilingual teacher model. Our analyses reveal that model scale and benchmark performance, which are common assumptions of a “strong” model, do not significantly predict teacher effectiveness (§4.1). Instead, we find that qualities of the generated data, namely prompt diversity and length coupled with fluent and diverse responses, capture over 93.3% of the variance in intrinsic data quality metrics, and their principal components predict student performance with \(R^2\)=0.664 (§4.2).

• Based on these findings, we recommend a recipe (§5) for generating multilingual synthetic data. For example, we find that matching the model families of the teacher and student is a reliable heuristic for choosing a teacher model (§3.2), and generating responses to existing prompts or translating from English can yield substantial improvements on less-resourced languages compared to a random mix of data generation methods, though gains vary by teacher model (§3.3). ¹

We hope that this work paves the way for developing inclusive and equitable language technologies through quality and cost-effective data. We release our code, data, and models to drive research in multilingual synthetic data generation. We will release our code, data, and models to drive research in multilingual synthetic data generation after the review period.

2 Evaluating Language Models as Multilingual Teachers↩︎

The Polyglot Score (1) of a teacher model \(T\) for a target language \(\ell\) is based on the (1) intrinsic quality of the synthetic data generated by the teacher (§2.2) and the (2) extrinsic performance of a student model \(S\) finetuned on this data (§2.3).

2.1 Creating the seed dataset↩︎

In order to bootstrap the synthetic data generation process, we create a seed dataset \(\mathcal{D}_{\text{seed}, \ell}\) for each target language \(\ell\). We create \(\mathcal{D}_{\text{seed}, \ell}\) by aggregating publicly available multilingual instruction-tuning datasets, including the Aya Collection [3], WildChat 4.8-M [15], EuroBlocks-SFT [9], and Magpie-Align [16]. In order to simulate scenarios where English prompts are translated into a target language, we also include examples from Tülu 3 SFT [17], Helpsteer3 [18], and GSM8K [19]. Detailed seed dataset statistics in Appendix 9.

2.2 Multilingual Data Quality & Diversity↩︎

2.2.0.1 Synthetic data generation

Given a teacher model \(T\), target language \(\ell\), and a seed dataset for language \(\ell\), \(\mathcal{D}_{\text{seed}, \ell}\), we distill a synthetic dataset \(\mathcal{D}_{T,\ell} = \{(x_i, y_i)\}_{i=1}^{N}\) consisting of \(N\) prompt-response pairs \((x_i, y_i)\). We consider three synthetic data generation methods found in literature:

• Generate: we sample \(k\) prompt-response pairs from \(\mathcal{D}_{\text{seed}, \ell}\) as few-shot examples and use \(T\) to generate a new pair \((x_i, y_i)\) conditioned on these examples.

• Translate: we forward-translate English prompts from \(\mathcal{D}_{\text{seed}, \ell}\) to the target language \(\ell\) to obtain \(x_i\), and use \(T\) to generate the corresponding response \(y_i\).

• Respond: we take a prompt \(x_i\) from \(\mathcal{D}_{\text{seed}, \ell}\) and use \(T\) to generate the response \(y_i\).

We provide a brief review of multilingual synthetic data generation methods in §6 and a supplementary survey in Appendix 8.

2.2.0.2 Data quality and diversity metrics

Synthetic data is valuable when it is both high-quality and diverse [20]–[22].² To estimate the value of \(\mathcal{D}_{T,\ell}\), we compute a set of lexical and model-based metrics:

• Diversity of prompts and responses \((d_x, d_y)\): a corpus-level statistic that computes the cosine distance of the prompt and response embeddings. In practice, we use Llama-Embed-Nemotron-8B [23], the top-performing model on the MMTEB leaderboard [24], to embed the texts.

• Perplexity (PPL): the perplexity of a base model on the response \(y_i\) conditioned on the prompt \(x_i\), measuring the fluency and naturalness of the generated text. Lower perplexity indicates more coherent and linguistically natural responses.

• Reward score of a multilingual reward model (R): the verbalized score (1-5) of a multilingual reward model based on rubrics relating to fluency, naturalness, and instruction-following. In practice, we prompt M-Prometheus 14B [25] as an LM judge to score the quality of the prompt-response pair (13). We choose M-Prometheus because of its high performance on human-aligned evaluation benchmarks, suggesting that the reward model aligns well with native speakers.

We combine these intrinsic metrics by scaling each metric using z-score normalization and averaging them as shown in 1 .

\[\begin{align} \text{Intrinsic}_{T,\ell} & = \frac{1}{\vert M \vert} \sum_{m \in M} \text{z-score}(m(\mathcal{D}_{T,\ell})) \\ \text{where } M & = \{d_x, d_y, -\log(1+\text{PPL}), R\} \end{align} \label{eq:intrinsic95score}\tag{1}\]

2.3 Student Model Performance↩︎

We perform supervised finetuning of a base model \(S_{\phi}\) on the synthetic dataset \(\mathcal{D}_{T,\ell}\) to obtain a student model \(S_{T,\ell}\). Then, we evaluate \(\mathcal{S}_{T,\ell}\) on a suite of multilingual tasks to assess how well the student has learned from the teacher. These tasks include:

• Cultural and factual understanding (Culture): we evaluate on Global-MMLU Lite [26], containing culturally diverse and relevant questions that were localized by native speakers from English [27].

• General chat (Chat): we evaluate on M-RewardBench [28] which measures the alignment of models with human preferences in conversational settings.

• Mathematical reasoning (Math): we evaluate on M-GSM [29], a multilingual version of the GSM8K dataset [19] that tests the model’s ability to solve mathematical word problems.

Inspired by [30], we compute the Performance Gap Recovered (PGR) that measures the improvement of \(S_{T,\ell}\) over a base model \(S_{\phi}\) on a benchmark \(b\) relative to a reference model \(S_{\text{REF}}\) (2 ).

\[\begin{align} \text{Extrinsic}_{T,\ell} & = \frac{1}{\vert B \vert} \sum_{b\in B} \dfrac{\text{score}_b(S_{T,\ell}) - \text{score}_b(S_{\phi})}{\text{score}_b(S_{\text{REF}}) - \text{score}_b(S_{\phi})} \\ \text{where } B & = \{\mathrm{\small Culture}, \mathrm{\small Chat}, \mathrm{\small Math}\} \end{align} \label{eq:extrinsic95score}\tag{2}\]

2.4 Computing the Polyglot Score↩︎

To provide straightforward comparisons between teacher models, PG-Score reports a single score that combines both extrinsic and intrinsic metrics as shown in 3 .

\[\text{\mathrm{\small PG-Score}{}}_{T,\ell} = \text{z-score}(\text{Intr.}_{T,\ell} + \text{Extr.}_{T,\ell}) \\ \label{eq:pg95score}\tag{3}\]

We combine both intrinsic and extrinsic metrics because they capture complementary aspects of teacher quality. Extrinsic metrics alone may overlook the quality of synthetic data that propagates through the ecosystem, while intrinsic metrics alone do not guarantee that the student model achieves strong downstream performance. The resulting PG-Score is z-score normalized, where \(0\) indicates average teacher effectiveness, and higher scores indicate better synthetic data quality and student performance for that language. We adopt equal weighting as a baseline; we show that teacher rankings are robust to alternative weighting schemes in Appendix 14.4.

3 Experiments: Evaluating LMs and PG-Score Generalization↩︎

In this section, we measure the Polyglot Score of state-of-the-art LMs (§3.1). Then, we test whether our findings are consistent across other base models (§3.2). Finally, we determine if a certain data generation method is more effective in multilingual settings (§3.3). We conduct additional experiments and ablations in Appendix 14.

3.1 Which State-of-the-Art LMs Are Good Multilingual Teachers?↩︎

3.1.0.1 Setup

In order to evaluate the effectiveness of different LMs as multilingual teachers, we select 10 state-of-the-art models that vary in scale, architecture, and training data, then evaluate them on 6 typologically diverse languages by generating 10.5k prompt-response pairs for each teacher-language pair where each data generation (§2.2) method is equally represented. We repeat the data generation process three times with different random seeds to account for variability in LM outputs. Then, we finetune a pretrained OLMo 3 7B model [31] on each \(\mathcal{D}_{T,\ell}\) to obtain \(S_{T,\ell}\). Appendix 12.1 describes SFT information.

3.1.0.2 Teacher Models

We include Llama 3.1 [32], Gemma 3 [33], Command A [34], Aya Expanse 32B [35], and IBM Granite [36]. In addition, we also include GPT 4o mini [37] as a representative closed-source model. See 7 in Appendix 11 for detailed model information.

3.1.0.3 Target Languages

We select 6 typologically diverse languages: Arabic (ar), Czech (cs), German (de), Spanish (es), Indonesian (id), and Japanese (ja). These languages are chosen due to their variation in resource availability, script, and family. This language choice is also supported by prior work on informed sampling [38] that considers typological variety of the chosen languages. See 8 in Appendix 11 for language statistics.

2pt

3.1.0.4 Results

1 shows the PG-Score of each teacher model across all target languages. The results suggest the following:

• Gemma 3 27B and Aya Expanse 32B are the most effective teachers. Gemma 3 27B achieves the highest average PG-Score (0.726), followed closely by Aya Expanse 32B (0.706), both outperforming larger models like Llama 3.1 70B Inst. (0.140), suggesting that model scale alone does not determine teacher effectiveness. We also observe that the Gemma 3 family dominates the top ranks, while the Llama 3.1 family underperforms on most languages.

• Smaller LMs can be effective multilingual teachers. Gemma 3 12B (0.595) and 4B (0.469) rank among the top-5 teachers, while the Llama 3.1 70B Inst. (0.140) ranks ninth, suggesting that smaller LMs can match or exceed larger LMs in data generation capabilities.

• Teacher performance varies significantly by language. German and Spanish consistently show the highest scores across all models, while Arabic proves challenging with most teachers yielding negative scores, suggesting that language-specific factors influence teacher effectiveness. We hypothesize that a language’s resource status or presence in pretraining data may contribute to this variability (§14.5).

3.2 Generalization of PG-Score Across Different Base Models↩︎

3.2.0.1 Setup

Instead of using OLMo 3 7B as the base model (\(S_{\phi}\)) for student finetuning, we use (1) Llama 3.1 8B, (2) Gemma 3 4B PT, and (3) Qwen 3 8B Base [39]. We recompute \(S_{\phi}\)-dependent metrics such as perplexity and PGR. To reduce computational costs, we focus on three languages: German (high PG-Score), Indonesian (mid-range), and Arabic (low PG-Score).

3.2.0.2 Results

2 shows the average PG-Score of each teacher model across different base models while 13 shows the percentage increase of family-matched teacher-student pairs compared to the OLMo 3 7B (mismatch) baseline. We observe that the best teacher models remain consistent across different student base models, with Gemma 3 27B and Aya Expanse 32B consistently ranking among the top three teachers. Furthermore, the Gemma 3 family continues to outperform other model families. In addition, we find that the model rankings vary slightly depending on the base model used, as Spearman rank correlation ranges from \(\rho\)=0.57 (moderate) to \(\rho\)=0.87 (strong). We hypothesize that this variation may be due to differences in architecture and pretraining data between base models. Despite this variation, we observe that teacher-student model family alignment is a reliable heuristic for achieving good PG-Score. For example, Gemma 3 teachers consistently perform well with Gemma 3 student bases, with family-matched pairs achieving at least +20.5% higher PG-Score compared to the worst pair (see 13). This finding is interesting but reasonable given that models from the same family likely share similar tokenization schemes, leading to easier transfer from teacher to student. In addition, family-matching is not a hard constraint unlike in other distillation settings [40], [41], but it remains a reliable heuristic for teacher selection when the optimal teacher is unknown. For our core experiment, we use OLMo 3 7B as the base model for finetuning to control the effect of model family alignment when evaluating teacher quality.

3.3 Effect of Synthetic Data Generation Method on PG-Score↩︎

3.3.0.1 Setup

In order to determine if a data generation method is more effective than others, we generate 10k prompt-response pairs for each method in §2.2 and compare the PG-Score of each mix. We recompute intrinsic data quality metrics and finetune OLMo 3 7B to obtain a student model and evaluate the teacher’s PG-Score. We also compare each mix against a baseline consisting of 10k instances with roughly equal number of samples (\(\approx\)​3.3k) from each method. To reduce computational costs, we conduct this experiment on three representative teachers (Gemma 3 27B, Aya Expanse 32B, and Llama 3.1 70B) spanning high to low PG-Score, and three languages (German, Indonesian, Arabic) covering diverse resource levels.

3.3.0.2 Results

2 shows the PG-Score of each data generation (see 14 for baseline comparisons). We observe that for a high-resource language like German, the Generate method yields the highest PG-Score, while for less-resourced languages like Arabic and Indonesian, the Respond or Translate methods are more effective. We hypothesize that this occurs because the Generate method depends on few-shot examples from the seed dataset, which are typically of higher quality in high-resource languages. Overall, our findings suggest that selecting a data generation method can have an impact on teacher effectiveness. In our core experiment, we sample an equal mix of all three methods (3.5k each) to control their effect when evaluating teacher model quality.

4 Analysis: What Makes a Good Polyglot Teacher?↩︎

We investigate the factors that contribute to effective multilingual teachers. We start by analyzing common assumptions about teacher model performance, such as size and benchmark scores (§4.1), then determine which intrinsic factors drive student performance (§4.2). Lastly, we examine language properties that might influence a teacher’s PG-Score (§14.5).

4.1 Do stronger models make better teachers?↩︎

4.1.0.1 Setup

In order to determine if there is a relationship between a model’s size or benchmark performance (i.e., common assumptions to assess a model’s “strength”) to its effectiveness as a multilingual teacher, we fit a mixed-effects model regressing PG-Score on (a) parameter size (N=27, 9 models, excluding GPT-4o-mini with unknown size \(\times\) 3 trials), and (b) average multilingual benchmark performance on Global-MMLU Lite, M-GSM, and M-RewardBench (N=180, 10 models \(\times\) 6 languages \(\times\) 3 trials).

4.1.0.2 Results

3 shows the regression results. We observe that neither parameter size nor average multilingual benchmark performance significantly predict PG-Score (p\(>\)​0.05). Specifically, a 1-unit increase in \(\log(\text{Param. Size})\) corresponds to a non-significant 0.053 increase in PG-Score. Although this finding confirms the results of [12] and [30] for English-based tasks, we show that “stronger” models do not necessarily make better multilingual teachers.

4.2 Which intrinsic metrics determine extrinsic student model performance?↩︎

4.2.0.1 Setup

In order to identify latent factors from the intrinsic metrics that explain student performance, we perform principal component analysis (PCA) on the intrinsic metrics described in §2.2. Then, we fit a regression model to predict extrinsic student performance based on the principal components (PCs) obtained from PCA: we split 180 data points (10 models \(\times\) 6 languages \(\times\) 3 trials) into 80% train and 20% test, then train a linear regression model with the PCs as the features and the student performance as the target.

4.2.0.2 Results

4 shows how much of the variance is explained by each principal component while 3 shows the loading strength of each intrinsic metric on the principal components. We observe that the first four PCs explain over 93.3% of the variance in the intrinsic data quality metrics. Specifically, PC 1 (42.2%) captures characteristics such as lower response perplexity and high distinctiveness, PC2 (22.1%) captures variance in characteristics such as higher prompt diversity and length, whereas PC3 (16.5%) and PC4 (12.6%) capture variance that reinforce trends on prompt length and diversity. In addition, 4 shows the fit of a linear model on the test set when the PCs learn to predict student performance. We observe that interactions within the intrinsic metrics can predict extrinsic student performance decently, with \(R^2=0.664\) and \(\text{RMSE}=0.440\). This finding suggests that even with a simple linear model, our chosen intrinsic metrics are predictive of student performance. In practice, these insights can help practitioners select teacher models based on intrinsic metrics alone, which are cheaper to compute than extrinsic student evaluations.

5 Discussion: Towards a Recipe for Multilingual Synthetic Data Generation↩︎

Our results provide actionable insights for selecting and effectively using teacher models in multilingual synthetic data generation. First, we find that model scale does not significantly predict teacher effectiveness: Llama 3.1 70B Instruct, despite being the largest model evaluated, ranks at the bottom half in PG-Score across all student base models we tested (§3.1, §3.2). Our analyses suggest that what matters instead is the quality of generated data: prompt diversity, response fluency, and length collectively capture over 93% of the variance in intrinsic data quality and predict student performance with \(R^2\)=0.664 (§4.2), offering practitioners a cheaper alternative to full student training runs for screening teacher candidates.

Second, when the optimal teacher is unknown, matching model families offers a reliable heuristic for teacher selection. Gemma teachers paired with Gemma students, and Llama teachers with Llama students, outperform a mismatched baseline by at least 20% (2). We hypothesize this finding reflects shared tokenization and similar pretraining distributions, though disentangling these factors remains future work.

Finally, we find that there are language-dependent considerations for data generation. For high-resource languages like German, where seed data quality is high, the Generate method performs best. For less-resourced languages like Arabic and Indonesian, methods that leverage existing prompts (Respond) or transfer from English (Translate) can yield substantial gains over a uniform mix of methods, though the magnitude varies by teacher (2). For truly low-resource languages, we recommend combining synthetic data generation with targeted data collection.

As a supplementary, we demonstrate the applicability of our findings by building a multilingual synthetic data recipe for a held-out language, Tagalog, in Appendix 16. We show that models trained using our recipe (based on analyses from PG-Score) have better performance on an unseen Filipino-centric benchmark, and that each component of our recommendation (e.g., choose top teacher from 1, match model families, etc.) resulted in observable performance gains. This suggests that our evaluation protocol is robust that the insights transfer to an unseen language, even when measured with a different set of downstream metrics.

6 Related Works↩︎

6.0.0.1 Synthetic Data Generation for Multilingual SFT

In order to offset the high costs of recruiting language experts for data collection, prior works relied on generating synthetic datasets. This effort resulted in large multilingual datasets such as Bactrian-X [42], MultiAlpaca [43], and xP3 [44] that were created through various data generation methods. These works have different data generation recipes, and so we provide a brief survey of these works and their recipes in Appendix 8, then classify them across the three strategies / archetypes (Generate, Translate, Response; 2.2). Building on these prior efforts, we examine the three core strategies for multilingual synthetic data generation and test each in isolation. This setup enabled us to provide practitioners with empirically-grounded recipe on selecting teacher LMs that we hope to be applicable across any generation method.

6.0.0.2 Evaluating and Improving the Synthetic Data Pipeline

While prior works have evaluated aspects of the synthetic data pipeline, they typically do so in isolation (i.e., intrinsic \(\oplus\) extrinsic) or focus exclusively on English [14]. For instance, [30] evaluated teacher models solely as a function of extrinsic student performance on English tasks (e.g., reasoning and coding), while [45]’s OpenDataArena focuses on intrinsic data quality (model-based and heuristic) to score models. Signals of multilingual data quality are often a function of corpus-level diversity [24], [46], [47] and generation quality [25], [48]. On the other hand, multilingual LMs are typically evaluated on general-knowledge and culture-specific benchmarks [33], [49], [50]. These practices informed our choice of intrinsic and extrinsic metrics throughout this work. More importantly, PG-Score provides a holistic analysis that combines both intrinsic data quality and extrinsic student downstream performance to evaluate teacher models across various generation methods.

7 Conclusion↩︎

We conduct a comprehensive evaluation of state-of-the-art LMs as multilingual teachers for synthetic data generation by assessing both intrinsic data quality and extrinsic student model performance. We find several properties that contribute to teacher effectiveness outside of model size or benchmark performance, such as prompt-response diversity, fluency, and language representation. Finally, we outline practical recommendations for creating a multilingual synthetic data generation recipe. We hope our findings guide future work on developing inclusive language technologies through high-quality synthetic data.

Limitations↩︎

Our work comes with some limitations and open questions left for future work. For example, our language set encompasses six languages. Although we chose these languages carefully based on (1) whether they can be evaluated on publicly-available LM benchmarks and (2) prior theoretical work on principled test language selection [38], validating our findings across a broader language sample remains important future work. In addition, our Translate data generation method assumes access to English prompts that can be meaningfully translated to target languages. This approach inherits limitations from LM-based techniques such as localizing culture-specific references, introducing translationese artifacts.

Ethics Statement↩︎

Synthetic data generation risks amplifying biases present in teacher models. If a teacher model underperforms on certain languages or exhibits cultural biases, these weaknesses propagate to student models trained on its outputs. Our finding that teacher effectiveness correlates with CommonCrawl representation ( Appendix 14.5) suggests that already underrepresented languages may be further disadvantaged in synthetic data pipelines, potentially widening the performance gap between high- and low-resource languages.

Acknowledgments↩︎

LJVM and AK acknowledge the support of the UKRI Frontier Grant EP/Y031350/1 (EQUATE). This work was performed using joint resources provided by the Cambridge Service for Data Driven Discovery (CSD3) EP/T022159/1, Isambard AI National AI Research Resource (AIRR) ST/AIRR/I-A-I/1023, and the Microsoft Research Grant. LJVM would also like to thank Songbo Hu, Chen Cecilia Liu, Millicent Ochieng, and Felermino Ali for helpful and productive discussions on the project.

Appendix↩︎

8 Multilingual Synthetic Data Generation↩︎

We present an overview of prior works in 5 that used synthetic data to train multilingual LMs. In general, we find that most data generation methods fall into one of the three categories described in §2.2.0.1, i.e., Generate, Translate, or Respond, which we tested in our experiments. Our survey suggests that our choice of data generation methods are grounded in prior work and covers the majority of approaches used in synthetic data generation.

9 Seed Dataset Statistics↩︎

6 shows the statistics of the seed dataset used for synthetic data generation.

1.2pt

10 The Polyglot Collection↩︎

In order to facilitate future research on multilingual synthetic data generation, we introduce the Polyglot collection, a collection of synthetic datasets and student models generated by the best teacher model across all target languages. The Polyglot collection includes:

• PolyglotTeachers-SFT-Synth-Data: Synthetic datasets for each target language generated by each teacher model using all three data generation methods (§2.2).

• PolyglotTeachers-Multilingual-Instruct: A set of 8B student models finetuned on each synthetic dataset from the OLMo 3 7B base model using the Gemma 3 27B (highest-scoring model) teacher.

Because synthetic data generation can be expensive [57], especially when accounting teacher inference costs, we hope that the Polyglot Collection can help amortize the need for quality multilingual data without re-running these systematic experiments. We publicly release these artifacts in HuggingFace.³ We will publicly release the Polyglot Collection after the review period.

11 Teacher Model and Target Language Details↩︎

In this section, we provide additional details about the teacher models and target languages used in our experiments. 7 summarizes the key characteristics of each teacher model. On the other hand, 8 provides information about the target languages, including language family, number of speakers, and resource availability.

12 Experimental Details↩︎

12.1 Supervised Finetuning↩︎

9 summarizes the hyperparameters used for finetuning student models. We train models using the Unsloth framework [60] using a cluster of Grace Hopper GH200 Superchips. Full finetuning (7B) takes around 1.5 hours (wall clock) for 2 epochs and 2 nodes.

12.2 Model Evaluation↩︎

We used the Lighteval framework [61] for evaluation. 10 summarizes the benchmarks used for evaluating student models. We decided to use Global-MMLU Lite instead of Global-MMLU becaue the former contains actual native speaker annotations that localized the benchmark into different cultural contexts.

For Global-MMLU Lite and M-RewardBench, we use the Multiple-Choice Formulation (MCF) with character normalization. In addition, we also follow the corpus-level metric in M-RewardBench which uses a weighted accuracy for each data subset and category [28]. For M-GSM, we show 5 few-shot examples from the training set in order for the model to properly generate the answer. We run all evaluation experiments for three trials with different random seeds and report the average and standard deviation.

13 Full Results for Intr. and Extr. Metrics↩︎

¿tbl:table:full95intrinsic95results? shows all the data quality metrics for each teacher model across all languages. 11 shows the full results of student models finetuned on synthetic datasets generated by each teacher model across all target languages.

13.0.0.1 Percentage Increase Tables

We provide additional tables from the main experiments in §3 and §4. 13 shows the percentage increase in PG-Score when using family-matched teacher-student pairs compared to the OLMo 3 7B baseline (see §3.2). 14 shows the percentage increase in PG-Score when using the best data generation method for each teacher-language pair compared to an equal mix baseline (see §3.3).

1pt

4pt

14 Additional Experiments and Ablations↩︎

In this section, we ablate several aspects of our evaluation protocol that may affect a teacher model’s PG-Score.

14.1 Effect of Data Scale on Student Model Performance↩︎

One component of PG-Score is the extrinsic student performance metric (§2.3) as measured by PGR. Scaling laws suggest that this performance improves with more data [62]. Then, it is possible to inflate PG-Score by simply using more synthetic data. In order to control for this variable, we conduct an experiment to determine how much synthetic data is needed to reliably compute PG-Score.

14.1.0.1 Setup

We finetune an OLMo 3 7B base model on \(n\) SFT instances where \(n \in \{\text{1k}, \text{5k}, \text{10k}, \text{25k}, \text{50k}\}\). To reduce computational costs, we perform this experiment only on a single teacher model (Gemma 3 27B Instruct) on three target languages that represent diverse scripts and resource availability: Arabic, German, and Indonesian. Similar to the main experiments, we represent each data generation method equally when creating the SFT datasets. Then, we recompute the intrinsic metrics and finetune student models and measure their performance across three benchmarks (§2.3).

14.1.0.2 Results

5 shows the average student model performance as a function of the number of SFT instances. We observe that student performance improves with more synthetic data, but gains diminish beyond 10k examples. This finding suggests that using 10k synthetic examples per language is sufficient to reliably compute PG-Score without inflating the metric by increasing the number of samples. In our experiments, we use 10k synthetic examples per language when computing PG-Score. Specifically, we show that 10k synthetic examples from a strong teacher are sufficient to finetune a student model to achieve reasonable performance across multiple benchmarks.

14.2 Generalization Across Model Size↩︎

14.2.0.1 Setup

In order to test whether PG-Score generalizes beyond 8B parameter size models, we use an OLMo 32B base model (\(S_{\phi}\)) and recompute the intrinsic and extrinsic metrics to obtain the PG-Score. To save computational costs, we train student models across three teachers (Gemma 3 27B Instruct, Aya Expanse 32B, Llama 30B Instruct) and all 6 target languages.

14.2.0.2 Results

15 shows the PG-Score scores for three teacher models when using OLMo 3 32B as the student model. We find that Gemma 3 27B Instruct remains the highest-scoring teacher in this comparison, achieving the highest average PG-Score of 0.805 across all languages. This result is consistent with our findings using the 8B student model (§3), demonstrating that the superior data quality generated by Gemma 3 27B generalizes across model scales. Aya Expanse 32B achieves a positive average PG-Score of 0.227, while Llama 3.1 70B Instruct shows a negative average of \(-\)​0.267.

Furthermore, the language-dependent effects observed in the 8B experiments remain consistent at 32B scale. German continues to show the highest PG-Score values across all three teachers (2.389 for Gemma, 1.979 for Aya, 0.838 for Llama), suggesting that certain languages benefit more from synthetic data regardless of student model size. Similarly, Spanish exhibits strong performance across all teachers, with PG-Score values ranging from 1.353 to 1.855. In contrast, Arabic shows the most variable results, with Gemma achieving slightly negative scores (\(-\)​0.239) while Aya and Llama show substantially lower performance (\(-\)​0.872 and \(-\)​1.688, respectively). Overall, these findings demonstrate that PG-Score and teacher model rankings generalize to the 32B parameter range.

14.3 Effect of Translation Method (Prompting an LM vs. Translation Model)↩︎

An alternative to using an LM for translating texts from English to a target language is via a translation model such as NLLB [63]. In this section, we examine the effect of the translation method on the PG-Score of teacher models.

14.3.0.1 Setup

First, we filter and sample 10k English prompt-response pairs from the Tülu 3 SFT dataset.⁴ Then, using the NLLB model (nllb-200-distilled-600M), we perform two translation methods: (1) NLLB-Translate-then-Respond: translate the prompts to each target language and prompt Gemma 3 27B Instruct to generate a response, and (2) NLLB-Translate-Both: translate both the prompts and responses from English to the target language. We choose the 600M version due to its computational efficiency and popularity among practitioners, as measured by HuggingFace downloads and community likes.

We compare these methods against our original Translate method, i.e., prompting Gemma 3 27B Instruct to directly translate the prompt and generate the response in the target language (LM-Translate). Then, we compute the intrinsic data quality metrics and finetune OLMo 3 7B student models on each synthetic dataset to compute PG-Score.

14.3.0.2 Results

6 shows the PG-Score and average benchmark performance of the student model for each translation method across Arabic, German, and Indonesian. We find that LM-Translate outperforms both NLLB-based approaches, achieving an average PG-SCORE of 1.36 compared to 0.85 for NLLB-Translate-Both and 0.80 for NLLB-Translate-then-Respond. This pattern holds across all three languages, with the largest gap observed for German (2.09 vs 1.26/1.68).

Our findings suggest that prompt naturalness, rather than response quality, is a bottleneck in translation-based pipelines: having an LM generate responses to NLLB-translated prompts provides no improvement over pure NLLB translation (0.80 vs 0.85), indicating that translated prompts fail to elicit the same quality of responses as LM-translated prompts.

14.4 Weighing of Intrinsic and Extrinsic Metrics in PG-Score↩︎

Our PG-Score formulation uses an assumption-free and equal weighing scheme between the intrinsic (\(\mathcal{I}\)) and extrinsic (\(\mathcal{E}\)) metrics. In this section, we test whether these two metrics capture (1) complementary aspects of teacher effectiveness and (2) how model rankings differ if one metric is weighted more than the other.

14.4.0.1 Setup

In order to test whether each metric captures complementary aspects of teacher effectiveness, we compute the Spearman rank correlation (\(\rho\)) between the intrinsic and extrinsic metrics across all teacher-language pairs (N=60, 10 models \(\times\) 6 languages). In addition, in order to test the effect of weighing one metric against the other, we formulate a generalized version of PG-Score:

\[\begin{align} \text{\mathrm{\small PG-Score}{}}_{T,\ell} & = \alpha \mathcal{I} + (1-\alpha) \mathcal{E} \\ & \text{where } 0 \leq \alpha \leq 1 \end{align}\]

Note that the experiments in §3 and §4 assume \(\alpha=0.5\). We compute the PG-Score across \(\alpha = \{0.00, 0.25, 0.50, 0.75, 1.00\}\) and then test the resulting model ranks’ \(\rho\) across all pairs of \(\alpha\). We perform this experiment on all teacher-language pairs where students are finetuned from the OLMo 3 7B base model (N=30, 10 models \(\times\) 6 languages).

14.4.0.2 Results

Intrinsic and extrinsic metrics show a moderate positive correlation (Spearman \(\rho\) = 0.41, p < 0.01), suggesting that data quality metrics are predictive of student performance while capturing complementary information. This finding motivates our combined PG-Score computation. In addition, teacher rankings are stable for nearby weighting schemes (\(\rho \geq 0.90\) for adjacent \(\alpha\) values) as shown in 7. Our finding suggests that model rankings are robust to small changes in the weighing of intrinsic and extrinsic metrics. Our equal weighting (\(\alpha=0.5\)) balances both perspectives, correlating strongly with extrinsic-focused (\(\rho = 0.89\)) and reasonably with intrinsic-focused (\(\rho = 0.74\)) rankings.

14.5 Effect of language resource levels on PG-Score↩︎

14.5.0.1 Setup

For each language, we consider the following properties drawn from prior work: CommonCrawl (CC) percentage as a proxy for presence in pretraining data [59], and linguistic resource availability [58]. We compute the Spearman rank correlation (\(\rho\)) between each property and PG-Score across all teacher-language pairs (N=60, 10 models \(\times\) 6 languages).

14.5.0.2 Results

8 shows the relationship between a language’s percentage in CommonCrawl and PG-Score. We observe a suggestive positive trend between CommonCrawl representation and PG-Score (\(\rho=\)​0.886, \(p<\)​0.05). This finding suggests that languages with greater presence in pretraining data enable teacher models to generate higher-quality synthetic data that leads to better student performance. This finding is unsurprising, but it provides empirical evidence of a structural gap that inhibits quality synthetic data generation for long-tail languages. In contrast, we do not find a significant correlation between resource availability and PG-Score (\(\rho=\)​0.372, \(p=\)​0.468). Our findings suggest that teacher model generation quality depends more heavily on pretraining exposure than linguistic resources. Additionally, the data sources from [58] do not reflect the current landscape: recent LMs are trained on either publicly-available datasets from HuggingFace or in-house datasets. While our work includes 6 diverse languages, the sample size remains limited; we encourage future work to expand the number of languages to validate these findings.

15 Disclosure on the Use of LLMs↩︎

We used Claude [66] to assist with editing, title ideation, and proofreading portions of this work. All scientific claims and interpretations are solely our own. We reviewed and revised all LLM-assisted text.

16 Multilingual Synthetic Data Recipe: Case Study on Tagalog↩︎

As an application of our findings and discussion in §5, we present a case study on developing a multilingual synthetic data recipe on a held-out language: Tagalog. It is a mid-resource language (Category 3 in [58]’s taxonomy) and the standardized form of Filipino, the national language of the Philippines.

16.1 Setup: Recipe Design and Evaluation↩︎

16.1.0.1 Data

We collect Filipino seed data from various publicly-available SFT datasets such as WildChat 4.8M and the Aya Collection. In addition, we also include English data from the Tülu 3 SFT dataset for the Translate method. 17 shows the statistics of the seed dataset used for Tagalog synthetic data generation. Then, we implement the following data interventions based on our findings:

• Teacher Model: we use Gemma 3 27B Instruct as the teacher model, as it was the best-performing model across most target languages we evaluated (§3).

• Data Generation Method: we use the Translate and Respond methods, as they were the best-performing methods for mid-resource languages like Indonesian (§3.3). In addition, we add a small sample of prompt-response pairs synthesized via the Generate method.

• Synthetic Data Scale: we generate 10k synthetic examples using the selected teacher and data generation method, as we found that this scale is sufficient to achieve strong student performance ( Appendix 14.1). However, we also test on finetuning a model with 25k synthetic examples to see if more data improves performance.

• Student Base Model: we finetune using the Gemma 3 4B model, as we find that family-matched teacher-student pairs yield higher PG-Score (§3.2).

1.2pt

For the purposes of this report, we will designate the model finetuned on Gemma 3 4B using our synthetic recipe as 10K-Polyglot-TL, where “10K” indicates the number of SFT instances used during finetuning.

16.1.0.2 Evaluation

We evaluate on FilBench [68], a benchmark for LMs that includes Filipino-centric multiple-choice and generative tasks. It measures an LM’s performance across four categories such as classical NLP, cultural knowledge, reading comprehension, and generation, alongside an aggregated FilBench score.

We also compare against two data mix baselines:

1. 10K-Public: we sample 10k Tagalog prompt-response pairs from the seed dataset. This baseline aims to simulate a non-synthetic data approach to training multilingual LMs.

2. 10K-GPT-4oM: we synthesize 10k instances using an off-the-shelf teacher model (GPT-4o-mini). This baseline simulates a typical data generation approach of choosing a teacher in an ad hoc manner due to its perceived strength (size or benchmark performance) or ease of use.

For all methods, we finetune a Gemma 3 4B base model using the same training settings indicated in Appendix 12.1.

16.2 Results: Leaderboard Scores and Ablations↩︎

18 shows the FilBench score of our optimal synthetic recipe compared to other models in the same parameter range. We find that 10K-Polyglot-TL is competitive against 10K-GPT-4oM (+1.85pp), and has better performance compared to 10K-Public (+2.28pp). These results suggest that (1) synthetic data generation is a viable approach for building less-resource language models, and (2) our finding that selecting strong teacher models based on PG-score is effective, as larger models do not always produce better training data (§3).

In addition, comparing 10K-Polyglot-TL to other models in the FilBench leaderboard⁵ shows that the former is competitive against Qwen 3 4B and Llama 3.1 8B Instruct. We highlight that our 4B models are competitive against other models with larger parameter sizes, suggesting that a multilingual synthetic data recipe based on our PG-Score findings is data-efficient. We also find that increasing the number of SFT instances (10k to 25k) led to a performance increase of 0.21pp. While we previously found that 10K instances showed diminishing returns (see Appendix 14.1), the continued gains from scaling to 25K instances on FilBench suggest that saturation points may depend on task diversity. FilBench covers a broader range of NLP tasks (e.g., named-entity recognition) compared to our experimental benchmarks in §3 and Appendix 14, indicating that practitioners working with diverse task distributions may benefit from exploring larger synthetic datasets beyond the 10K threshold.

16.3 Analysis: Ablation Experiments↩︎

In order to measure the contribution of our findings and recommendations in §5, we perform the following ablation experiments as shown in 9. Note that the interventions described below are additive.

16.3.0.1 Curation of publicly-available data vs. Synthetic data generation

We compare student models trained on (1) publicly-available Tagalog SFT data and (2) synthetic SFT instances generated by a GPT-4o teacher (note that these are also the same baselines in Appendix 16.2). We find that the performance of these two baselines are similar (\(\Delta=0.5\text{pp}\)), suggesting that there is no significant advantage to using a synthetic data pipeline if the teacher model is not optimal. We also hypothesize that some publicly-accessible datasets in Tagalog were semi-synthetic (e.g., TaCO uses a synthetic pipeline akin to the Translate method, but using chain-of-thought to improve the quality of translations), making it difficult to perform a fair comparison.

16.3.0.2 Using a teacher with a higher PG-Score

We then swap the GPT-4o-mini teacher with Aya Expanse 32B, a teacher with a higher PG-Score based on our main findings (0.461 vs. 0.706, c.f. §3, 1). We observe a slight performance improvement in this intervention, suggesting that the PG-Score metric is generalizable across an unseen language.

16.3.0.3 Matching teacher and student model families

One of our key findings and recommendation is to match the model families of the teacher and the student (§3.2). We use a Gemma 3 Instruct 27B teacher model to match the family of the Gemma 3 4B base model. This intervention yields a substantial performance improvement, demonstrating that family alignment is a reliable heuristic for teacher selection. The improvement from family matching is consistent with our findings that family-matched pairs achieve at least +20.5% higher PG-Score compared to mismatched pairs, likely due to shared tokenization schemes and architectural similarities that facilitate better knowledge transfer from teacher to student.

16.3.0.4 Increase data scale

We increase the number of synthetic instances from 10k to 25k to assess whether additional data continues to improve performance. We observe a modest gain of 0.21pp, which is smaller than the improvements from teacher model selection and model family matching. This finding aligns with our earlier observation that gains diminish beyond 10k examples ( Appendix 14.1), though the continued improvement on FilBench’s diverse task distribution suggests that saturation points may be task-dependent.

16.3.0.5 Increase model scale

Finally, we explore whether scaling the student model from 4B to 12B (and 27B) parameters provides additional performance gains. We find that the larger student model achieves higher performance, demonstrating that our synthetic data recipe benefits from increased model capacity. This result is consistent with our generalization experiments ( Appendix 14.2), where we showed that PG-Score generalizes across different model sizes while maintaining the relative ranking of teacher models. However, we note that the performance of our best models are still behind Gemma 3 27B Instruct and Gemma 3 12B Instruct (18). Given that observation, we still argue that our synthetic pipeline, which uses 25K instances trained only via SFT, can be considered data and resource-efficient compared to the post-training interventions done in Gemma 3, which involved instruction-tuning and reinforcement learning objectives [33].

16.4 Analysis: Instance-Level Qualitative Inspection of Generated Data↩︎

16.4.0.1 Setup

We also inspect the quality of generated Tagalog data across teacher models with different PG-Scores. To do so, we sample 50 instances from a dataset generated by the following models (in increasing PG-Score): GPT-4o, Aya Expanse, and Gemma 3 27B.We evaluated these instances in terms of the following criteria:

• Fluency: whether the generated text reads naturally in Tagalog, with grammatical correctness and idiomatic word choice rather than literal translations or code-switched constructions that a native speaker would find awkward.

• Cultural accuracy: whether the content reflects appropriate Filipino cultural context, including correct references to local entities, customs, and norms, as opposed to generic or Western-centric framings that have been superficially translated into Tagalog.

The scoring is from 1–5, using the rubric:

1. Unacceptable: ungrammatical, code-switched, or reads as a literal translation (fluency); content is absent, generic, or Western-centric (cultural accuracy).

2. Poor: frequent grammatical or idiomatic errors (fluency); cultural references are mostly missing or incorrect (cultural accuracy).

3. Adequate: comprehensible but noticeably awkward (fluency); some appropriate grounding mixed with generic or Western framings (cultural accuracy).

4. Good: mostly natural with only minor lapses (fluency); cultural context is largely accurate with small inaccuracies (cultural accuracy).

5. Excellent: fully fluent and idiomatic, indistinguishable from native writing (fluency); consistently and accurately grounded in Filipino cultural context (cultural accuracy).

These annotations were performed by a native-speaker who is one of the authors in this work. To reduce bias, the annotator only sees a prompt-response pair and its generation method without knowledge of which teacher model it was distilled from. Finally, we also include some observations by the annotator during the annotation process.

16.4.0.2 Results

19 shows the annotation results for the 150 instances generated by each teacher model. We find that for Tagalog, teacher models with a higher PG-Score also produce higher-quality Tagalog data as judged by a native speaker. Gemma 3 27B, which has the highest PG-Score_tl (1.049), achieves the strongest ratings on both criteria. This pattern is most consistent on cultural accuracy, which increases monotonically with PG-Score, suggesting that PG-Score captures aspects of cultural grounding that are perceptible to native speakers.

However, we observe that fluency does not follow the same trend: Aya Expanse receives a lower fluency score than GPT-4o (2.480 vs.) despite a higher PG-Score_tl. We also note the substantial gap between Gemma 3 27B and the other two teachers on both dimensions (\(\Delta\geq1.3\) on fluency, \(\Delta\geq0.9\) on cultural accuracy), reinforcing our earlier finding that matching the teacher model family with the student is a reliable heuristic for selecting strong teachers (§3.2).

17 Inference Details↩︎

17.0.0.1 Prompt templates

10 to 12 show the prompt templates used for each data generation method. In addition, 13 shows the prompt template used for the LLM-as-a-judge method to evaluate text quality.

17.0.0.2 Inference settings

We use vLLM [69] and Curator [70] for inference. For each teacher model, we check whether the model provider recommended best settings for usage. If not, then we set a default configuration (temperature=0.8, top_p=0.9). 16 summarizes the inference settings we used for each teacher model.

References↩︎

---

1. As a supplementary, we show that our recipe improves performance on a held-out language, Tagalog,et on a language-specific benchmark ( Appendix 16).↩︎

2. We use “data quality” to refer to both aspects hereafter.↩︎

3. : ljvmiranda921/polyglot-teachers↩︎

4. Tülu 3 also contains non-English data. We perform English-language filtering using fastText [64], [65] and the staticvectors library.↩︎

5. Official FilBench leaderboard: https://hf.co/spaces/filbench/filbench-leaderboard↩︎