1 Introduction↩︎

Openly available pre-training data at scale exist primarily for English [1], [2], with most other languages lacking volume or quality (see Section 2 for details). This shortfall has motivated growing interest in machine translation as a means of producing multilingual pre-training data at scale, but the practice raises legitimate concerns: translated text can carry stylistic artifacts known as “translationese” and inherits the cultural reference frame of the source language, with names, places, idioms and culturally grounded knowledge in the English source remaining English-anchored after translation [3], [4].

We address this resource limitation by introducing MultiSynt/MT, an open multilingual synthetic parallel corpus of approximately 4.8 trillion target-language tokens covering 36 languages, produced by translating a 100B-token sample of high-quality web data from Nemotron-CC with open translation models, including Tower+ 9B and 72B [5] and OPUS-MT [6]. MultiSynt/MT is the largest openly available pre-training corpus to date for many European low- and medium-resource languages, exceeding the largest comparable native resource (HPLT 3.0; [7]) by more than an order of magnitude in the lowest-resource ones (Figure 1). For a subset of languages, we release row-aligned parallel translations from multiple systems, enabling controlled comparison under matched source data, and the corpus is released under a permissive open license.

Alongside the corpus, we present a balanced empirical characterization reporting both where reference LLMs trained on MultiSynt/MT outperform native multilingual baselines and where they fall short, and drawing attention to evaluation practices that obscure these differences. On standard multilingual base-model benchmarks, reference LLMs pre-trained on MultiSynt/MT reach the final score of a native-data baseline (HPLT 2.0¹), using approximately 72% fewer training tokens, and improve over it by approximately 15% relative at a matched 100B-token training budget (Section 4). A fluency-sensitive LLM-as-judge evaluation on the trained LLMs cleanly recovers an MT-system quality ranking that standard multiple-choice benchmarks fail to discriminate, exposing a blind spot in standard evaluation rather than a fluency deficit in MultiSynt (Section 5.1); a complementary embedding-space diagnostic finds that standard benchmark items tend to have more MultiSynt/MT than HPLT 2.0 documents among their nearest neighbors (Section 5.2). On Norwegian tasks targeting idiomatic and culturally grounded knowledge, models trained on native data outperform translated-data models throughout training, while on Norwegian commonsense reasoning the translated data closes the gap and eventually overtakes the native baseline; direct evidence that translated text is not a substitute for native corpora on all phenomena (Section 5.3).

2 Related Work↩︎

2.0.0.1 Multilingual pre-training corpora.

Open multilingual pre-training corpora have evolved from filtered Common Crawl pipelines (CCNet, mC4, OSCAR, ROOTS) to larger and better-filtered resources including MADLAD-400, CulturaX, Glot500, HPLT, FineWeb2 and EMMA-style adaptation [8]–[20], but a persistent volume and quality imbalance remains for lower-resource languages [21]. MultiSynt/MT targets that part of the language distribution as a complement to these corpora; we use HPLT [7], the strongest openly available native multilingual baseline at our scale, as our principal point of comparison.

2.0.0.2 High-quality English source data.

Translation-based corpora inherit the quality distribution of their source: recent reference-baseline work shows that Nemotron-CC HQ [2] produces the strongest downstream performance across model and token scales, ahead of established English corpora including C4, The Pile, RefinedWeb, FineWeb-Edu and DataComp-LM [1], [22]–[26], motivating its use as the MultiSynt/MT source distribution.

2.0.0.3 Translated corpora for language-model pre-training.

Studies on Basque, Indian languages, Arabic, Indonesian and Tamil consistently find that machine-translated pre-training data is competitive with native data on standard NLU but underperforms on culturally nuanced tasks [27]–[30]. Closest in scale, TransWeb-Edu / CuatroLLM and its extension TransWebLLM translate up to 1.7T tokens from FineWeb-Edu into nine languages [31], [32], and the contemporaneous FineTranslations reverses the direction by translating 500+ non-English languages into English [33]. A parallel line treats translation as an explicit pre-training objective rather than a corpus-construction mechanism [34]–[37]; we instead train standard next-token language models on translated target-language documents. MultiSynt/MT extends this line in three ways: 36-language coverage at approximately 4.8T target-language tokens, a Nemotron-CC HQ rather than FineWeb-Edu source, and row-aligned outputs from multiple open MT systems (translation-specialized LLMs in the Tower family [5], [38] and classical NMT systems [6], [39], [40] supporting controlled comparison under matched source data.

2.0.0.4 Translationese, cultural grounding, and native-authored evaluation.

Translated text differs systematically from natively authored text [41]–[44], with downstream effects on cross-lingual benchmark interpretation [4], [45]–[48], and multilingual models in turn often lack the cultural and idiomatic knowledge of the target-language community [49]–[54]. Multilingual evaluation is itself often translation-derived (Belebele, MMLU-ProX, XNLI, XTREME) [55]–[58]; native-authored alternatives such as TyDi QA, Global MMLU and NorEval are an essential complement [59]–[63], and LLM-as-a-judge protocols [64]–[67] provide a fluency-sensitive complement that we exploit in Section 5.

3 Building MultiSynt/MT↩︎

3.1 Source data: high-quality English web text↩︎

We translate from the actual high-quality split of Nemotron-CC [2], which contains non-synthetic Common Crawl documents receiving the highest scores from its ensemble quality classifier. Recent reference-baseline work [26] finds that Nemotron-CC HQ produces the strongest average downstream performance among established open English pre-training corpora across multiple model and token scales. We draw a uniform random sample of approximately 155 million documents (on the order of 100 billion English tokens), sized so that each per-language translation yields outputs at the same order of magnitude, and apply no topical, length, or quality stratification beyond the underlying quality classifier. Translations preserve a stable mapping to their source documents, so the resulting corpus is usable both as per-language pre-training data and as a multi-way parallel resource for cross-language and cross-system experiments (Section 3.4).

3.2 Translation system pool and selection↩︎

We require open translation systems that combine strong target-language quality with throughput feasible at the 100B-token scale, and that together cover the languages targeted by MultiSynt/MT, including medium- and lower-resource European languages where openly available native pre-training data is scarce. No single family meets all three requirements: translation-specialized LLMs in the Tower family [5], [38] and general-purpose open instruction-tuned LLMs [68]–[70] concentrate on higher-resource European languages, while OPUS-MT and HPLT-MT [6], [17] extend coverage to lower-resource languages. We therefore use LLM-based translation and OPUS-MT/HPLT-MT jointly as the production pool.

To rank candidate systems, we ran a small human-judged quality study covering seven languages (Czech, French, Finnish, German, Italian, Polish, Spanish). For each language, a single native speaker rated translations of the same 10 English source documents from each of six candidate systems on a fluency-focused 1–5 scale, blind to system identity, additionally penalizing hallucinations, repetitions, and extra or missing text. The full protocol, KEOPS annotation interface, and per-language results are in Appendix 12. Tower+ (we use the 72B model) is rated highest in every language, the general-purpose LLMs (EuroLLM-9B-Instruct, Qwen3-32B, Gemma-3-4b-it, Mistral-Small-3.2-24B-Instruct) bunch well below it, and OPUS-MT is close to the general-purpose LLMs on average (Figure 3); the ranking is consistent with the public Tower MT leaderboard and with per-language Flores-200 BLEU on the OPUS-MT dashboard.² Based on this ranking the general-purpose LLMs were dropped from the production pool.

We ship translations from Tower+ 9B (Gemma-based, 16 languages)³ and Tower+ 72B (Qwen-based, 5 languages: German, Finnish, Italian, Spanish, Swedish).⁴ A reference-LLM comparison between the two on the 5-language overlap (Section 4) showed a small enough downstream gap that we treat 9B as the default LLM-based system and retain 72B on the high-resource subset where any residual gains can be isolated. OPUS-MT supplies the production models for 31 of our 36 target languages from the Tatoeba Translation Challenge collection [71], and HPLT-MT supplies the remaining 5 (Basque, Irish, Icelandic, Maltese, Albanian) where no comparable OPUS-MT model is available. These NMT systems are orders of magnitude smaller than the LLM-based candidates (\(\sim\)​75M–230M parameters) with corresponding throughput advantages, at the cost of a sentence-level constraint addressed by the pipeline of Section 3.3. The released corpus therefore contains translations from Tower+ 9B (16 languages), Tower+ 72B (5 languages), and OPUS-MT/HPLT-MT (36 languages); for the languages where multiple systems are shipped, the corpus supports controlled comparison of translation choices under matched source data, which we exploit in Sections 4 and 5.

3.3 Translation pipeline at scale↩︎

We run two parallel translation pipelines on the same 155 million source documents, both supporting per-shard resumption and emitting Parquet files that preserve source document identifiers and order so that the released corpus remains a synchronized multi-parallel resource. The LLM pipeline deploys Tower+ 9B and 72B on Leonardo HPC cluster (CINECA, NVIDIA A100 nodes) with vLLM under a Slurm-native orchestration toolkit released as open-source code,⁵ consuming approximately 3.1 million A100 GPU-hours in aggregate. The NMT pipeline decodes OPUS-MT and HPLT-MT models with Marian-NMT [40] on LUMI cluster (AMD MI250x), with sentence-splitting pre-processing and document-reassembly post-processing that preserve both per-sentence and per-document alignments; the pipeline is released as open-source code at github.com/Helsinki-NLP/Opus-MT. Per-shard throughput, decoding settings, energy, and carbon-footprint estimates for both pipelines are reported in Appendix 8.

3.4 Resource overview↩︎

MultiSynt/MT contains approximately 4.8 trillion target-language tokens across 36 languages, derived from the single shared sample of 155M English documents introduced in Section 3.1, and is the largest openly available pre-training corpus to date for many European low- and medium-resource languages (Figure 1). Counting each translation system’s output separately, the released artifact totals approximately 7.1T tokens (1.7T from Tower+ 9B, 0.5T from Tower+ 72B, 4.8T from OPUS-MT/HPLT-MT, all under Gemma-3 tokenization); per-language and per-system breakdowns are in Appendix 11.

3.4.0.1 Splits and schema.

For each language we release three splits: The parallel split contains the documents translated by all systems released for that language, row-aligned by index so that row \(i\) of any language file refers to the same source document, supporting cross-language and cross-system controlled experiments. The additional split contains further translated documents that are not present for every language, enabling larger language-specific volumes at the cost of cross-language alignment, and the all split is the union of the two. Each row contains the source warc_record_id, the translated text, its token count, and the source partition identifier.

3.4.0.2 Availability and license.

MultiSynt/MT is released under a Creative Commons Zero (CC0) license applied to the corpus as a database (underlying text content remains subject to Common Crawl source-content terms), at hf.co/datasets/MultiSynt/MT-Nemotron-CC. The OPUS-MT/HPLT-MT subset is additionally available on Hugging Face at hf.co/datasets/Helsinki-NLP/nemotron-cc-translated and as a sentence-aligned OPUS release.⁶

4 Effectiveness for Multilingual Pre-training↩︎

4.1 Reference model setup↩︎

We evaluate MultiSynt/MT as pre-training data by training reference language models on each MultiSynt/MT translation variant and on a native multilingual baseline dataset. Across variants, we hold the model architecture, tokenizer, optimizer settings, and total token budget fixed; the variants differ in the pre-training corpus itself—specifically, the source data and the translation system (if any)—and in the set of languages each corpus covers.

Our reference architecture follows that of [1]: we use a dense Llama-like transformer with 24 layers, hidden size 2048, 32 attention heads, RoPE positional embeddings, and RMSNorm, with 1.61B parameters in transformer layers. We use the google/gemma-3-27b-pt tokenizer [70] with a vocabulary size of 262144 for its broad multilingual coverage across the languages in MultiSynt/MT. We tie word embeddings, giving a total of 2.15B parameters. The optimization recipe uses Adam with \(\beta_1=0.9\), \(\beta_2=0.95\), weight decay 0.05, gradient clipping at 1, global batch size 1024, and a linear WSD learning-rate schedule with 10% warmup and 20% cooldown. Full hyperparameters are reported in Appendix 9, Table 1.

Training is conducted with Megatron-LM [72] on the LUMI supercomputer, using 16 nodes with 4 AMD MI250x GPUs per node. For every pre-training data variant, we train a separate model on a budget of 100 billion tokens, saving checkpoints every 1,000 steps. Each variant is trained once; we do not report variance over training seeds, and the curves shown in Section 4.2 should be read as single-seed point estimates.

We compare four pre-training data variants under this setup: HPLT 2.0 [18], a strong openly available native multilingual baseline at the relevant scale, and MultiSynt/MT translated by Tower+ 9B (16 languages), Tower+ 72B (5 languages), and OPUS-MT/HPLT-MT (36 languages). Holding architecture, optimizer, and budget fixed lets us attribute downstream differences to the pre-training corpus. Within the MultiSynt/MT family, this isolates the choice of translation system; between MultiSynt/MT and HPLT 2.0 the comparison additionally varies the source corpus and its quality-filtering pipeline, a confounding factor we discuss in Section 6.

4.1.0.1 Evaluation.

Downstream evaluation uses two open frameworks, LightEval [73] and LM-evaluation-harness [74]. We evaluate each checkpoint on a broad multilingual benchmark suite: belebele [57], arc:challenge [75], [76], hellaswag [76], [77], goldenswag [78], xstory-cloze [79], global-mmlu [60], mmlu [76], [80], exams [81], xcodah [82], xcsqa [83], MultiBlimp [84], enem [85], faquad [86], oab-exams [87], glianorex [88], xnli [55], pawsx [89], Finbench [90]. We apply the same evaluation protocol across all variants.

Per-language scores are aggregated into a single mean across the languages covered by every variant under comparison; per-language and per-benchmark breakdowns are provided as supplementary data.

4.1.0.2 Decontamination check.

A controlled training experiment with vs.without n-gram decontamination of the Nemotron-CC source rules out source-document contamination as an explanation for the Section 4.2 gains (the released MultiSynt/MT corpus is itself not decontaminated due to the low risk of actual contamination; see Appendix 10).

4.2 Gains over native baselines↩︎

Figure 2 reports aggregate benchmark performance over training tokens, comparing reference LLMs pre-trained on the HPLT 2.0 native baseline against reference LLMs pre-trained on MultiSynt/MT OPUS-MT, averaged over the five languages where both baselines have evaluation data (Danish, Dutch, Italian, Portuguese, Swedish). Models pre-trained on MultiSynt/MT reach the HPLT 2.0 endpoint score at approximately 28B training tokens, roughly \(72\%\) fewer than the native baseline; at the matched 100B-token training budget, MultiSynt/MT exceeds the HPLT 2.0 endpoint by approximately 15% relative.

Figure 4 compares the three MultiSynt/MT variants (OPUS-MT, Tower+ 9B, Tower+ 72B) on Swedish, the single language for which a reference LLM was trained on every variant. The three MultiSynt/MT curves cluster tightly together throughout training and all sit well above the HPLT 2.0 baseline, indicating that on these standard multilingual benchmarks the gain over native data is largely independent of the choice of MT system within MultiSynt/MT. The downstream gap among Tower+ 9B, Tower+ 72B, and OPUS-MT is therefore small enough that we treat the 9B variant as our default LLM-based system and retain 72B only on the high-resource subset where any residual gains can be isolated (Section 4.1).

Per-language and per-benchmark breakdowns are reported in Appendix 11.

5 Interpreting the Gains: Fluency, Benchmark Alignment, and Native-Authored Tasks↩︎

Section 4 showed that training on MultiSynt/MT yields large gains over native data on standard multilingual benchmarks. We now interpret those gains along three axes the standard benchmarks do not surface: an LLM-judge fluency probe that exposes MT-system differences flattened by multiple-choice scoring (Section 5.1), an embedding-space diagnostic of benchmark-corpus alignment (Section 5.2), and a Norwegian case study on native-authored idiomatic and culturally grounded tasks (Section 5.3). We draw the methodological consequences in Section 6.

5.1 What standard benchmarks miss about MT-system choice↩︎

The MT-variant indistinguishability of Figure 4 is surprising: the same three translation systems are clearly separated by human raters (Figure 3) and by the public Tower MT leaderboard. A plausible explanation is that translationese artifacts in the MT output propagate to the trained LLM in a form the standard multilingual benchmarks of Section 4, predominantly multiple-choice QA and selection, cannot detect: those benchmarks reward picking the right token under strong exploitable cues, while the surface fluency of the model’s own free-form generations does not enter the score.

To probe this hypothesis, we evaluate the pre-trained models with an LLM-as-a-judge protocol that scores fluency of free-form generations. For each of German, Spanish, Finnish, Italian and Swedish we collect 100 natural-language cut-out sentences (e.g., “Das Grundgesetz der Bundesrepublik Deutschland garantiert…”). Each candidate pre-trained model produces a continuation that DeepSeek V3.1 then compares against a continuation from a monolingual HPLT 2.0 1.7B native-data baseline; we report the candidate’s winrate averaged over prompts and languages.

As a sanity check, the off-the-shelf Qwen 2.5 series at five sizes from 0.5B to 14B parameters shows winrate rising monotonically with model size against the HPLT 2.0 baseline (Appendix 13, Figure 9), confirming the judge responds to generation quality. The reference LLMs pre-trained on MultiSynt/MT with matched data and budget produce a clear ranking, Tower+ 72B \(\geq\) Tower+ 9B \(>\) OPUS-MT (Figure 5), consistent with the human evaluation of the underlying MT outputs (Figure 3) and the public Tower MT leaderboard. All three MultiSynt/MT variants match or exceed the HPLT 2.0 baseline on the judge, so MultiSynt/MT is not fluency-deficient; the finding is that standard benchmarks are blind to translation-quality differences the trained LLMs nonetheless inherit.

5.2 Benchmark-corpus alignment in embedding space↩︎

As a complementary diagnostic, we compare the embedding neighborhoods of standard benchmark items against MultiSynt/MT and HPLT 2.0 documents. Across most benchmarks in the suite, benchmark items have more MultiSynt/MT documents among their nearest embedding neighbors, suggesting that translated high-quality English-source data provides strong coverage of many benchmark-adjacent regions; this is not a causal test of the gains, but it indicates that benchmark-corpus alignment is one reason those gains should be interpreted with care (Appendix 14).

5.3 Where translated data falls short: a Norwegian case study↩︎

Named entities, idiomatic constructions, and culturally specific knowledge in English text remain English-anchored after translation, so we expect translated data to underperform native data on target-language tasks probing local idiomaticity or culture even when the translations are fluent. We test this by training four reference LLMs sharing the architecture and recipe of Section 4 and differing only in Norwegian pre-training data: HPLT 2.0 (native), HPLT 3.0 (native), MultiSynt/MT translated by Tower+ 9B, and MultiSynt/MT translated by OPUS-MT, evaluated across 11 training checkpoints on two Bokmål tasks from NorEval⁷ [61]: NorIdiom (idiomatic expressions) and NorCommonsenseQA (commonsense reasoning), selected as illustrative examples following the criteria in [7].

On NorIdiom (Figure 6, left), both native-data models outperform both translated-data models with a stable gap throughout training, consistent with idiomatic constructions being a long-tail phenomenon of the target language that the translated corpus does not generate at the rate native text does. On NorCommonsenseQA (Figure 6, right), native data leads in early-to-mid training but the translated-data models close the gap and match or exceed the native models in later checkpoints; this is a complementary signal that translated data lacks local cultural anchoring but preserves the structured reasoning patterns of the high-quality English source [91]. Together, the two tasks suggest a content-dependent picture: native data wins on tasks probing culturally or idiomatically local knowledge; on tasks probing transferable reasoning structure, models trained on MultiSynt/MT match or exceed those trained on native data at matched training budget.

6 Discussion and Recommendations↩︎

6.0.0.1 Treat translated data as a complement to native data, not a replacement.

For high-resource languages, where native corpora are already abundant, MultiSynt/MT is most useful as a domain-coverage extension. For medium- and lower-resource languages, translating a high-quality source corpus is the primary vehicle for closing the high-quality data-scale gap, but it should still be paired with whatever native corpus is available to preserve idiomatic and cultural anchoring (Section 5.3).

6.0.0.2 The headline gain may reflect source quality alone, not translation as a beneficial operation.

The Section 4.2 comparison varies both the source corpus (Nemotron-CC HQ vs.native HPLT 2.0 web text) and the translation step, so the gain cannot be attributed to translation per se: it is consistent with the net effect arising entirely from the higher quality of the English source, with translation contributing nothing of its own. Disentangling the two would require a controlled source-corpus comparison that we leave to future work. The practical counterfactual is nonetheless not arbitrary translated versus arbitrary native text, but translated high-quality source text at 100B-token scale versus the much smaller native pool that would survive equivalently stringent quality filtering, which for medium- and lower-resource languages would shrink the already-scarce native data to a small fraction of itself.

6.0.0.3 Pair translation-derived benchmarks with native-authored ones, and complement multiple-choice with quality-sensitive measures.

A large fraction of widely used multilingual benchmarks, including Belebele [57], m-ArenaHard [92], MMMLU [93] and MT-Bench-X [94], are themselves produced by translating English sources, and our embedding-space diagnostic shows that they tend to sit in neighborhoods with more MultiSynt/MT than HPLT 2.0 documents (Section 5.2). A model trained on translated data shares the same source-language frame as these benchmarks and is, in part, being evaluated on how well it has absorbed that frame; standard multiple-choice scoring is additionally blind to fluency-quality differences that the resulting LLMs nevertheless inherit (Section 5.1). Native-authored benchmarks such as NorEval [61] expose gaps that translation-derived benchmarks fail to surface (Section 5.3); free-form quality-sensitive measures such as the LLM-judge protocol of Section 5.1 expose differences that discriminative ones flatten.

7 Conclusion↩︎

MultiSynt/MT provides approximately 4.8T target-language tokens across 36 languages, the largest openly available pre-training corpus to date for many European low- and medium-resource languages, and reference LLMs trained on it match a native-data baseline (HPLT 2.0) using roughly 72% fewer pre-training tokens while exceeding it by approximately 15% relative at a matched 100B-token budget. The translated corpus nonetheless falls short on idiomatic and culturally grounded Norwegian tasks and is best understood as a complement to native data; we release it under CC0 with row-aligned translations from multiple MT systems to support controlled follow-up on translation-system choice, scaling, native-benchmark coverage, and source-quality vs.translation attribution.

Limitations↩︎

7.0.0.1 Corpus and translation biases.

MultiSynt/MT inherits the topical, stylistic, and English-centric biases of its Nemotron-CC HQ source: a quality-filtered Common Crawl pool that favours factual prose and formal registers over oral, vernacular, dialectal, and culturally local text, and that anchors named entities, idioms, and culturally specific knowledge to an English reference frame. The chosen MT systems further introduce systematic translationese artifacts (reduced lexical variety, calque-driven syntax, lower idiomatic density) that the standard multilingual benchmarks of Section 4 flatten but that surface under fluency-sensitive evaluation (Section 5.1) and on natively authored idiomatic and culturally grounded tasks (Section 5.3). The headline comparison against HPLT 2.0 (Section 4.2) additionally varies the source distribution and its quality-filtering pipeline, so the gain cannot be attributed to translation alone; an isolating comparison against an equivalently stringently filtered native baseline is left to future work.

7.0.0.2 Evaluation coverage, scale, and feedback risks.

Our empirical analyses are concentrated on European languages in the high- and medium-resource range and are conducted at a single 1.7B-parameter scale with single-seed pre-training runs (Section 4.1), so behaviour on the lowest-resource and non-European languages in the released corpus, scaling trends to larger models, and seed-level variance are not directly verified by our experiments. The embedding-space alignment diagnostic (Section 5.2) is descriptive rather than causal, the fluency LLM-judge protocol (Section 5.1) inherits the biases of its judge and prompt set, and the human MT evaluation (Section 3.2) covers only seven languages with one annotator each. Finally, releasing MultiSynt/MT openly creates a feedback path into future open MT systems trained on web crawls that will then ingest it, with the attendant long-tail erosion risks of recursive synthetic-data training [95]; we mark the corpus as synthetic in the released metadata and recommend treating it as a complement to native data rather than a replacement (Section 6).

Acknowledgments↩︎

We acknowledge EuroHPC JU for awarding the project ID EHPC-AIF-2025LS01-028 access to the EuroHPC supercomputer LEONARDO hosted by CINECA (Italy) and the LEONARDO consortium. We acknowledge EuroHPC JU for awarding the project ID EHPC-AIF-2025LS16-024 access to the EuroHPC supercomputer MareNostrum 5 hosted by the Barcelona Supercomputing Center (BSC). We acknowledge EuroHPC JU for awarding the project ID HPC-REG-2024R02-167 access to the EuroHPC supercomputer LUMI hosted by CSC (Finland) and the LUMI consortium. Part of the computations were performed on resources provided by Sigma2 - the National Infrastructure for High-Performance Computing and Data Storage in Norway. This research was supported by the OpenEuroLLM project, co-funded by the Digital Europe Programme under GA no. 101195233, and partially by DECOLLAGE (ERC-2022-CoG 101088763). This project is supported by the German Federal Ministry for Economic Affairs and Energy (BMWE) through EU-SAI/SOOFI: Sovereign Open Source Foundation Models for European Intelligence (grant number 13IPC040J).

8 Translation Pipeline Details↩︎

8.0.0.1 Orchestration toolkit (LLM pipeline).

For the LLM-based translation runs of Section 3.3, we use an open-source Slurm-native orchestration toolkit⁸ that scales inference workloads across hundreds to thousands of GPUs while maintaining linear throughput. The toolkit deploys multiple distributed inference servers in parallel: a single YAML configuration file specifies the cluster, inference-server, and data settings, and the toolkit generates the required Slurm job scripts, manages resource allocation, and coordinates distributed execution. It provides error handling with automatic retries and resumption, progress and throughput monitoring, built-in support for large-scale loading of Parquet input files, and a flexible inference server backend. We ran all inference with vLLM [96].

8.0.0.2 LLM runs: per-model throughput.

Across batched production runs (4 GPUs per inference server), Tower+ 9B sustained approximately 1,000 output tokens per second per GPU, and Tower+ 72B approximately 400 output tokens per second per GPU. The 9B variant therefore produces output tokens approximately \(2.6\times\) faster than the 72B variant per GPU-second; the per-request advantage is wider (approximately \(3.2\times\)) because the 72B model supports a longer maximum sequence length, allowing us to feed longer source documents per request and yielding roughly 26% longer translated outputs on average.

8.0.0.3 LLM runs: aggregate compute, energy and carbon footprint.

The LLM-based translation runs were carried out on the Leonardo (CINECA) supercomputer. Aggregate GPU consumption is approximately 3.1 million A100 GPU-hours. We estimate the corresponding energy and carbon footprint following the methodology used in CINECA project reporting: each A100 on Leonardo draws approximately 440 W under load, giving 1,379,638 kWh of consumed energy; applying the 2025 Italian electricity-grid carbon intensity of 358 g CO\(_2\)-eq/kWh yields approximately 493.9 t CO\(_2\)-eq. For reference, this energy budget corresponds to roughly 16% of the annual generation of an average wind turbine (3 MW nameplate capacity, 2,800 equivalent full-load hours per year, \(\approx\)​8.4 GWh/year). These figures cover only the LLM-based translation pipeline; the corresponding energy and carbon estimates for the OPUS-MT/HPLT-MT runs on LUMI (AMD MI250x) are reported alongside the NMT shard statistics below.

8.0.0.4 NMT models: training data and inference setup.

The OPUS-MT and HPLT-MT models we use are decoded with Marian-NMT [40] at beam size 4 on single-node jobs allocating two AMD MI250x GPUs each, on the LUMI supercomputer. All models are trained on OPUS data [39], [97] using the Tatoeba Translation Challenge compilation [71], augmented with back-translations of Wikimedia content. None of the NMT models are fine-tuned for specific tasks; we use them off-the-shelf without further data normalisation or filtering. The final list of selected NMT models is published together with the dataset card. The sentence-splitter pre-processing step greedily merges segments shorter than 256 characters and splits sentences longer than 1024 characters before feeding them to the decoder; this length normalisation matters because NMT batching benefits sharply from uniform input length.

8.0.0.5 NMT runs: sharding, runtimes, energy and per-GPU power.

The corpus was split into 100 shards per language of approximately 30 million sentence-aligned lines (roughly 700 million space-separated tokens each). Per-shard runtime averaged about 23 hours for transformer-base models (\(\sim\)​75M parameters) and about 30 hours for transformer-big models (\(\sim\)​230M parameters), with sustained GPU utilisation between 85% and 95%. Total energy consumption for one shard translated with transformer-base models is roughly 12 kWh, at an average of 260–270 W per GPU. For transformer-big models, total energy consumption is roughly double at 24–26 kWh, at an average of 360–370 W per GPU. The pattern suggests that further batching and inference optimisation could improve GPU utilisation for the smaller models and yield additional speed-ups.

9 Reference Model Hyperparameters↩︎

9.0.0.1 Compute setup.

All reference-model training and evaluation runs were carried out on the LUMI supercomputer (16 nodes of 4 AMD MI250x GPUs per node), with one exception: the Nemotron-CC decontamination control experiment of Appendix 10 was run on Leonardo (CINECA) with NVIDIA A100 nodes.

10 Decontamination: Methodology and Full Tables↩︎

10.0.0.1 Motivation and prior work.

Large web-derived corpora can overlap with evaluation benchmarks through exact duplicates, near-duplicates, paraphrases, or translated variants. The GPT-3 family established n-gram-based decontamination as a standard pre-training practice [98], and deduplication has been shown to reduce memorization and improve downstream model quality [99]. Recent work emphasizes that contamination is difficult to define uniformly and that its impact depends on task, model, and training recipe [100]. For multilingual settings the problem is sharper: contamination can cross language boundaries, and translated benchmark items can evade simple text-overlap detection while still inflating evaluation scores [101]. We therefore decontaminate the English source documents before translation, report overlap statistics per benchmark, and run a controlled training comparison to verify that removing flagged documents has no measurable downstream effect.

10.0.0.2 Released corpus.

The released MultiSynt/MT corpus itself is not decontaminated; the decontamination described here is applied to the English source documents used as input to the controlled training comparison below.

10.0.0.3 Pipeline stages.

We use the NeMo Curator decontamination pipeline, which constructs n-gram representations of downstream task datasets and removes any matching content from the training corpus in three stages. Indexing: each benchmark dataset is normalised and tokenised, and n-grams of size 8–13 are extracted from the test split (or the validation split, where no test split is available), following the text formatting in lm-evaluation-harness, to build a compact n-gram index. Matching: each source document is processed through the same normalisation/tokenisation/n-gram-extraction pipeline and its n-grams are compared against the benchmark index to identify overlaps. Removal: source documents whose matched n-grams occur fewer than 10 times in the corpus are treated as contamination signals and removed in full.

The indexed evaluation suite (see Table 2 for details) spans code, common sense reasoning, instruction following, language understanding, linguistic competence, mathematics, reasoning, translation, and world knowledge.

10.0.0.4 Match counts and rates.

After applying the pipeline to the source Nemotron-CC documents used as input to translation, a total of 34,969 documents per language (0.0002%) are flagged as contaminated. Per-benchmark match counts and contamination rates are reported in Table 2. Contamination is unevenly distributed: the largest absolute contribution comes from BoolQ, followed at lower rates by HellaSwag, MMLU, and MTBench. Even where the absolute number of matches is large, the corresponding percentages remain very low, indicating that contamination is sparse relative to corpus size.

10.0.0.5 Controlled training experiment.

To verify that the residual contamination has no measurable downstream effect, we train a 0.6B-parameter reference model on the chosen Nemotron-CC source slice with and without decontamination, under identical hyperparameters and a 100B-token budget, and evaluate 0-shot on the Finetasks suite (ARC-easy, ARC-challenge, PIQA, HellaSwag, OpenBookQA). The two learning curves track each other throughout training (Figure 7), with no consistent advantage for either setting.

11 Per-Language Resource Statistics↩︎

Table 3 reports per-language Gemma-3 token counts for each MultiSynt/MT translation system and for the HPLT 3.0 native baseline, covering all 36 languages of the released corpus.⁹ Figure 1 in Section 3.4 visualizes a 24-language representative subset drawn from this table; the table itself gives the full picture, including languages where Tower+ 9B or 72B was not run and languages without an HPLT 3.0 baseline. For consistency with Figure 1, the MultiSynt/MT-to-HPLT 3.0 ratio is computed against the per-language maximum across MultiSynt/MT systems (effectively OPUS-MT/HPLT-MT, whose output exceeds Tower+ on every language). Sums of the per-system columns recover the corpus totals reported in Section 3.4: 1.7T from Tower+ 9B (16 languages), 0.5T from Tower+ 72B (5 languages), 4.8T from OPUS-MT/HPLT-MT (36 languages), for approximately 7.1T tokens across all system outputs and approximately 4.8T unique target-language tokens.

12 Human MT evaluation: per-language breakdown↩︎

Figure 8 shows the per-language breakdown of the human evaluation summarized in Section 3.2 and Figure 3.

12.0.0.1 Protocol.

For each language, a single native speaker of the target language rated translations of the same 10 English source documents from each of the six candidate systems, blind to which system produced each translation; because the source documents are held constant across systems within a language, the within-language comparison among systems is paired. Annotators were instructed to focus primarily on target-language fluency and to additionally penalize specific model failures (hallucinations, repetitions, extra text not present in the source, and missing text); machine-translation outputs that were truncated due to inference length constraints were flagged in the guidelines and were exempted from penalty. Annotators recorded each judgment on a continuous slider mapped to a 0–100 scale, with the endpoints anchored as very poor translation (0) and excellent translation (100); the recorded scores are aggregated into five 20-point bins (0–20, 21–40, 41–60, 61–80, 81–100) and reported on a 1–5 Likert-style scale (1 = bad, 5 = excellent).

12.0.0.2 Results.

Tower+ is rated highest in each of the seven evaluated languages; the relative ordering of the other systems varies by language. Czech and Finnish, the two morphologically richest languages in the evaluated set, receive the lowest scores across all systems, including Tower+; on the higher-resource Romance languages (French, Italian, Spanish) the gap between Tower+ and the general-purpose LLMs narrows substantially, while OPUS-MT remains a clear step below the LLM-based systems.

12.0.0.3 Comparison to WMT25.

Recent WMT25 rankings [102] place Tower+ 9B and 72B mid-pack against frontier proprietary systems (e.g., Gemini-2.5-Pro, GPT-4.1, DeepSeek-V3) and against the strongest constrained submissions. That evaluation, however, operates on paragraph-sized segments of approximately 100 words drawn from news, social media, speech transcripts, and literary text, rather than on the full web documents that constitute the Nemotron-CC HQ source distribution; several of the higher-ranked systems are also closed-weights or otherwise infeasible to deploy at the 100B-token source scale on the open compute available for this work.

13 Fluency LLM-judge: sanity check↩︎

Figure 9 reports the sanity check referenced in Section 5.1: the off-the-shelf Qwen 2.5 series at five sizes from 0.5B to 14B parameters, evaluated by the same LLM-judge protocol against the monolingual HPLT 2.0 1.7B baseline. Winrate against the baseline rises monotonically with Qwen model size across all evaluated languages, confirming that the judge tracks generation quality and that the MT-variant ranking reported in Figure 5 reflects real differences rather than judge noise.

14 Embedding-space coverage analysis↩︎

This appendix reports the embedding-space benchmark-alignment diagnostic summarized in Section 5.2. We use it as a descriptive check on benchmark-adjacent coverage rather than as a causal test of the gains in Section 4.2. The analysis measures which data source aligns more closely with benchmark regions in a shared embedding space.

14.0.0.1 Embedding space.

We map each document into an embedding space using intfloat/multilingual-e5-large-instruct [103], producing one vector per document.

Figure 10 illustrates benchmark-adjacent coverage for Spanish ARC-Challenge – the benchmark that deviates most from parity in the main kNN figure (Figure 11) – by partitioning the joint pre-training embedding space into \(k=2{,}000\) clusters using k-means and assigning each benchmark example to its nearest cluster. The two distributions partially overlap, but MultiSynt/MT covers several regions that are essentially unpopulated by HPLT 2.0 (and vice versa); clusters that contain a benchmark item tend to lie in MultiSynt/MT-denser regions, with a mean MultiSynt/MT share of \(0.72\) versus \(0.47\) for clusters that do not. This is consistent with the per-language kNN ratio of \(0.80\) for Spanish ARC-Challenge in Figure 11.

14.0.0.2 kNN coverage.

To quantify this effect across benchmarks and languages, for each benchmark example, we retrieve its top-\(k\) nearest neighbors from the union of native (HPLT 2.0) and synthetic (MultiSynt/MT) pre-training documents under cosine similarity, and report the average fraction \(k_{\mathrm{synthetic}}/k\) across benchmark items for \(k=20\). A value of \(0.5\) indicates equal local density from the two sources; values above \(0.5\) indicate denser synthetic coverage of benchmark-adjacent regions.

The detailed results are shown in Figure 11. Across all benchmarks, the mean values over the available languages indicate that the nearest documents are more often from MultiSynt/MT than from the native HPLT 2.0 data. HellaSwag is the only benchmark for which some languages show a higher fraction of adjacent native documents, though its mean MultiSynt/MT fraction is still 0.53. The other five benchmarks show a higher MultiSynt/MT fraction for all available languages. ARC-Challenge deviates most from the parity line, with a MultiSynt/MT fraction of 0.82. These results suggest that translated data increases the coverage of benchmark-adjacent regions, potentially bringing in content that is otherwise underrepresented in the native datasets. However, we only conducted a descriptive study and did not account for causality of any potential gains.

References↩︎