1 Background↩︎

The popularity and versatility of Large Language Models (LLMs) have introduced a wide spectrum of budget, memory, and hardware constraints for their deployment. To accommodate these varying requirements, it has become crucial to provide LLMs in multiple sizes and formats. Releasing a family of models allows practitioners to select the optimal trade-off between computational cost and predictive performance for their specific deployment scenarios, democratizing access to advanced AI capabilities across different hardware tiers.

However, training an entire family of models from scratch requires prohibitive amounts of compute. Knowledge Distillation (KD) in the pre-training phase, or Pre-training Distillation (PD), offers a powerful solution to dramatically cut these costs [1]. By transferring knowledge from a large, capable teacher model to a smaller student model using the teacher’s generated logits, the student benefits from richer information and implicit label smoothing. This allows the student to converge faster and achieve higher downstream performance with significantly fewer training tokens and compute resources. Consequently, pre-training distillation enables the cost-effective expansion of a model family without the computational burden of standard pre-training.

An orthogonal direction for addressing cost requirements (e.g., disk space or latency) is quantization. While reducing numerical precision significantly lowers the memory footprint and accelerates inference, it inherently introduces a cost-accuracy trade-off. As we show here, by carefully balancing this trade-off around the Pareto frontier of compression methods, practitioners gain finer control over the model’s performance and hardware profile. This fine-grained control allows for further expansion of the model family, bridging the gaps between pre-trained sizes at a cost significantly less than even pre-training distillation.

Our work builds upon the foundation of the Apertus [2] project, which sets a new standard for fully open and compliant LLMs. Unlike many open-weight models that withhold training data and pipelines, the Apertus recipe emphasizes complete transparency, data compliance, and global multilingual representation. By grounding our distillation and quantization pipeline in the Apertus ecosystem, we inherit its rigorous openness and reproducibility.

2 Pre-Training Distillation↩︎

2.1 Recipe↩︎

2.1.0.1 Data.

To produce the highest-quality models, we gathered the data corresponding to Phase 5 (the final phase) of the original Apertus pre-training, which consists of documents and code and instruction samples with the highest level of quality filtering for a total yield of approximately 1.7T tokens. Similar to Apertus, we cut and pack these documents into chunks of 4096 tokens and train with cross-document attention masked.

2.1.0.2 Logits generation.

To be able to efficiently re-use the logits for multiple models, we generated the entire training set in advance. We ran the collected documents through the Apertus-8B-2509 model to obtain \(\approx\)​131k logits per token. After calculating the probability distributions from these logits, top-256 highest probabilities were identified per token. These probabilities, along with corresponding token indices in model vocabulary, were represented in 32-bit precision for a total of \(\approx\)​2KB of data per token. The tensors were batched in groups of \(\approx\)​131k tokens, compressed with gzip and stored in long-term storage for a total footprint of \(\approx\)​1.5PB of disk space. We applied sequences permutation on the logits generation stage to only have to do sequential disk loads when using them for training later.

2.1.0.3 Training objective.

As shown to perform well by [1], we utilize a 90%/10% mix between the KL-Divergence and the label cross-entropy. Since the computed KL-Divergence is sparse, it introduces close to no computational or memory overhead relative to the basic cross-entropy calculation.

2.1.0.4 Model Architecture.

Apertus-v1.1 models follow the same architecture as Apertus: Dense transformer models with grouped-query attention and xIELU [3] activation in the MLP. Table 1 details the architectural configurations, parameter counts, and the resulting memory and computational footprints for the Apertus-v1.1 models. Notably, we used tied embeddings and thinner and deeper architecture for the smallest Apertus-v1.1 model to maximize performance while minimizing memory footprint [4].

2.1.0.5 Training dynamics.

Similar to Apertus, we use the AdEMAMix [5] optimizer with WSD schedule and weight decay. Next-token prediction (NTP) loss shown in Figure 1. Multilingual macro downstream average shown in Figure 2. We observed no training instabilities and consistent improvement in downstream performance, especially during the learning rate annealing stage (highlighted in gray).

2.1.0.6 SFT and alignment.

The supervised fine-tuning (SFT) stage followed immediately after pre-training. For it, we exactly reused the original Apertus SFT recipe, only adjusting the LR to match the post-annealing LR of Apertus-v1.1 models. For the subsequent alignment stage, we utilized a simplified DPO [6] setup.

2.1.0.7 Evaluations.

Following the Apertus evaluation setup, we report multilingual benchmarks average during training in Figure 2, selected final pre-training metrics in Table 3, multilingual post-training evaluations in Table 4 and broader post-training evaluations in Appendix 7. Unsurprisingly, the performance profile of Apertus-v1.1 models is extremely similar to Apertus-8B-2509, demonstrating great multilingual performance for the base models and good multilingual chat performance but lacking in certain capabilities like instruction following and math.

2.2 Cost Analysis↩︎

As seen from Table 2, Apertus-v1.1 models used significantly less compute than similar-sized models, being trained on just 1.7T tokens, in contrast to the 15T tokens of Apertus. The cost of producing the logits from the 8B model is relatively small because one only needs to perform the forward pass to produce logits and the same logits only have to be computed once for the entire family of distilled models, dramatically cutting the teacher cost per-model. The total compute cost of the entire Apertus-v1.1 model family is 2.4E22 FLOPs. This is less than, for example, the cost of standalone pre-training for SmolLM2-1.7B and less than 12% of the original Apertus 8B pre-training cost.

3 Quantization↩︎

While pre-training distillation successfully generated the core Apertus-v1.1models at a fraction of the cost, adapting these models for highly constrained environments requires further optimization for specific hardware profiles. In this section, we consider the problem of adapting Apertus-v1.1 models to NVIDIA GPUs and mobile devices, demonstrating how quantization yields a wider range of optimal, specialized models at close to no cost.

3.1 Apertus-v1.1 Quantization Recipe↩︎

3.1.0.1 Baseline.

We use GPTQ [7], the most widely-used 1-shot LLM quantization method as our baseline. We gauge our improvement over it differently for base and instruction-tuned models:

• For base models, we measure the loss increase over the corresponding unquantized models on the validation set of \(\approx\)​17M tokens from the original pre-training mixture (Apertus Phase 5 data). We test weight+activation (FP8, NVFP4) quantization for base models with focus on NVIDIA Blackwell GPUs, as we foresee their main usage in high-throughput scenarios such as data annotation and embedding.

• For instruction-tuned models, we measure the recovery of macro average over normalized few-shot accuracies on Arc [8], HellaSwag [9], MMLU [10] and WinoGrande [11]. We test weight-only quantization (INT2, INT3, INT4, INT6) for instruction-tuned models with focus on Apple devices (MLX) inference, as we foresee their main usage in memory-limited scenarios such as mobile and edge deployment.

3.1.0.2 Quantization-aware distillation (QAD).

QAD is applied as a short recovery stage on a fully-trained model by treating the entire model as trainable parameters, quantizing its weights every forward pass and updating them with standard gradient-based method via straight-through estimation [12], bridging the gap between full quantization-aware training and PTQ methods. Similar to pre-training distillation, teacher model logits (usually the corresponding unquantized model or a larger model from the same family) provide much richer signal for this phase, making it preferable to quantization-aware supervised fine-tuning. QAD has been shown to yield consistent improvement over 0-shot and 1-shot post-training quantization (PTQ) methods [13]–[15].

The open access to the original pre-training set and SFT mixture utilized for both Apertus and Apertus-v1.1 pre- and post-training allows us to use it for QAD of these models with the highest degree of confidence that the distillation curriculum captures close to the entirety of the models’ capability. We test QAD for both base and instruction-tuned models, using \(\approx\)​100M tokens (we see only marginal improvement beyond that) of the pre-training or the SFT mixture accordingly. We use Apertus-8B-2509 and Apertus-8B-Instruct-2509 as a teacher in this scenario. Additional implementation details and hyper-parameters are described in Appendix 8.2.

3.1.0.3 Norm fusion.

To further improve quantization quality, we propose the following zero-cost static model optimization: We scale attention’s QKV and MLP’s up projection matrices’ columns (input dimension) to have the same norm, multiplicatively fusing the reciprocal scales into the preceding layer-normalization layers’ weights. The idea behind this is to normalize the magnitudes of weight values to prevent flush-to-zero of small-magnitude but important weights and weights adjacent to outlier channels.

The loss measurements for compressed base models and few-shot recovery measurements for the instruction-tuned models show that this yields the most improvement for smaller models. Additionally, although this normalization is mainly designed to assist with weight quantization, we find that it also improves weight+activation quantization (NVFP4), indicating that offloading these scales to activations doesn’t hurt their compressibility.

3.1.0.4 Weight averaging.

Weight averaging (arithmetic averaging of model weight tensors) of the last few checkpoints during the annealing stage has been shown to improve LLMs’ resilience to post-training quantization [16]. To validate it, we tested weight averaging for the Apertus-v1.1 0.5B base model combined with various quantization formats and methods, including RTN, GPTQ [7] and QAD. The results, shown in Figure 5, demonstrate that weight averaging reduces validation loss gap to BF16 by up to 10% for RTN, up to 2% for GPTQ and has close to no discernible effect on QAD. As a result, we did not include it in our final quantization pipeline.

3.1.0.5 Final quantization recipe.

Our final recipe combines QAD with norm fusion to achieve just 0.1-0.2 validation loss increase for base and 90-104% few-shot accuracy recovery for instruction-tuned Apertus and Apertus-v1.1 models, as seen in Figure 4.

3.2 Pareto Optimality↩︎

As mentioned in the beginning of this section, we analyze base model quantization in the context of high-throughput applications and instruction-tuned model quantization in the context of memory-constrained deployment. Naturally, the corresponding cost can be measured for every model we trained (quantized or otherwise), along with a representative measure of it’s capability, quantifying the cost-accuracy trade-off. Covering a larger range of costs is what drove the demand for smaller models in the first place, and in Figure 3 one can see this trade-off visualized.

What is interesting, is that quantized models not only shift the Pareto front (i.e., the enveloping curve) towards more efficient solutions (as seen, for example, by BF16 models almost never being optimal), but also adds more points on the frontier, allowing for more fine-grained control over cost. Without quantization, adding new points would have meant pre-training new models of intermediate sizes, which would have entailed spending compute in the order of trillions tokens. QAD, on the other hand, achieves high recovery after only a few tens of millions of tokens, cutting the cost by more than four orders of magnitude.

4 Released Checkpoints↩︎

We provide a comprehensive suite of pre-trained and instruction-tuned models across multiple quantization formats to support various hardware constraints and deployment scenarios. Table 6 summarizes all the checkpoints released as part of the Apertus and Apertus-v1.1 model families.

5 Conclusion↩︎

We validated pre-training distillation for multi-billion parameter models and multi-trillion token budgets, demonstrating how such model family expansion can be done at a tiny cost (less than 20%) of the teacher model training and far more cheaply than pre-training from scratch. In total, we release 24 new model checkpoints, including the 3 pre-trained base models, 3 instruction-tuned models, 8 quantized checkpoints for NVIDIA devices, 10 quantized checkpoints for Apple devices, as well as all the code to reproduce training, post-training and quantization pipelines.

We hope our open-source, open-data and compliant recipe to be of use for LLM practitioners interested in producing and using small language models.

6 Codebases↩︎

The full codebases for the pre-training distillation, post-training, evaluations and quantization stages of the pipeline are available on GitHub.

• Megatron-LM-Distill: A fork of Megatron-LM with added functionality for teacher logits generation and saving as well as pre-training distillation.

• posttraining: The original post-training codebase from Apertus that was reused for this project.

• qat-suite: A lightweight quantization suite with support for vLLM and MLX data formats and various quantization algorithms, including QAD.

• evals: The Apertus pre-training evaluation suite.

• evals-post-train: The Apertus post-training evaluation suite.

For the data preparation scripts, please refer to the original Apertus report [2].

7 Evaluation Suite Details↩︎

For the evaluations reported in Tables 3 and 4, we used the publicly-available Apertus evaluation suite. The multilingual macro average shown in Figure 2 includes INCLUDE [17], XCOPA [18], XNLI [19], XWinograd [20], PAWS-X [21], Multilingual Arc [22], Global MMLU [23] and Multilingual HellaSwag [22].

8 Additional Hyper-Parameters↩︎

8.1 Pre-Training Details↩︎

Additional per-model pre-training hyper-parameters are shown in Table 5.

8.2 QAT Details↩︎

For the base models, we sample \(\approx\)​130M tokens uniformly from the unused remainder of the gathered pre-training data. For the instruction-tuned models, we sample \(\approx\)​60M uniformly from the Apertus SFT mixture. We train with AdamW [24] with cosine LR schedule. For base models, we use the same sequence length and batch size as in pre-training. For instruction-tuned models, we use slightly larger batch size of 512-2048 to compensate for smaller length of some post-training sequences. Similar to pre-training distillation, we pre-compute and store the sparse logits from the teacher model (Apertus-8B-2509 for base models and Apertus-8B-Instruct-2509 for instruction-tuned models) once and re-use them for all student model and quantization format combinations.

References↩︎