1 Introduction↩︎

Vision-language models (VLMs) have emerged as powerful tools for understanding and processing multimodal information, enabling tasks such as image captioning and visual question answering [@tan2019lxmert; @blip]. Recent advancements in VLMs [@alayrac2022flamingo; @blip2; @liu2023improved] have leveraged large-scale language models (LLMs) to enhance reasoning and generation capabilities by incorporating extensive textual knowledge [@2019t5; @soldaini-etal-2024-dolma]. In pursuit of superior performance, state-of-the-art VLMs [@li2024llava; @wang2024qwen2vl; @deitke2024molmo; @nvlm2024; @chen2024expanding] now integrate scaled-up LLMs with up to 72B parameters, achieving results comparable to proprietary models such as GPT-4V [@gptsyscard] and Claude-3.5 Sonnet [@claude3series2024].

However, the increasing scale of recent VLMs introduces substantial computational overhead, limiting their practicality in real-world scenarios—particularly for on-device deployment. To mitigate this challenge, recent work has explored distillation techniques [@cai2024llava; @shu2024llava; @feng2024align] that transfer knowledge from large (teacher) VLMs to smaller (student) ones. Yet, existing distillation methods suffer from a fundamental limitation: they typically assume that teacher and student produce sequences of equal token length, enabling token-level distance metrics such as KL divergence. This assumption breaks down when the two models use different tokenizers or adopt different image-splitting mechanisms (see Appendix [[app:A]](#app:A){reference-type=“ref” reference=“app:A”}), making the distillation process infeasible. Consequently, the range of viable teacher-student VLM pairs becomes severely restricted (see Appendix [[app:B]](#app:B){reference-type=“ref” reference=“app:B”}). In other words, current distillation approaches only operate when the teacher and student share identical vocabulary sizes, token splits, and token-index-ordering schemes (collectively referred to as token types), as illustrated on the left side of [fig:1]. These differences naturally lead to mismatched input or output token lengths across VLMs, preventing effective distillation. Note that even VLMs within the same family may employ different token types, further hindering compatibility.

To overcome this limitation, we propose Generation after Recalibration (GenRecal), a general-purpose distillation framework for token types-agnostic VLM distillation. At its core, GenRecal introduces a Recalibrator that aligns and adapts the feature representations of small VLMs with those of large VLMs. This design is inspired by prior studies demonstrating that word embeddings from different models can be linearly mapped into a shared embedding space [@mikolov2013exploiting; @smith2017offline; @conneau2017word]. Unlike approaches that align only word embeddings, Recalibrator performs joint visual–linguistic alignment on the hidden representations before the language head. By projecting small VLM features into large VLMs’ feature space as shared representation, Recalibrator effectively bridges the gap between the large and small VLMs, enabling general-purpose knowledge transfer.

To demonstrate the necessity of GenRecal, we first compare its performance with that of traditional distillation, as shown in experiment types C and D of [fig:1]. Although both experiments employ the same token types of VLMs—Qwen2-VL-72B as the teacher and Qwen2-VL-7B as the student—GenRecal consistently outperforms the traditional distillation method implemented by LLaVA-KD [@cai2024llava]. This result highlights that aligning feature representations between large and small VLMs is crucial for minimizing information loss during distillation, even when the models are homogeneous. Moreover, when we replace the large VLM from Qwen2-VL-72B with the more powerful InternVL2.5-78B, GenRecal achieves enhanced performance. Subsequently, replacing the small VLM with InternVL2.5-8B leads to an even stronger model, as summarized on the right side of [fig:1]. These experiments highlight the significance of GenRecal because traditional methods cannot handle VLM pairs with different token types.

Our contributions can be summarized as follows:

• Token Types-agnostic Recalibration: GenRecal employs the Recalibrator to align and adapt the feature representations of large and small VLMs, enabling general-purpose distillation across token types that differ in vocabulary size, token splits, and token-index ordering.

• Broad Applicability: GenRecal is compatible with a wide range of VLM architectures across different model sizes, overcoming the challenge of selecting large VLMs that have different token types and demonstrating its practicality for real-world deployment in resource-constrained settings.

2 Related Work↩︎

Large-scale VLMs such as NVLM-72B [@nvlm2024], Qwen2-VL-72B [@wang2024qwen2vl], and InternVL2.5-78B [@chen2024expanding] have recently approached the performance of GPT-4V [@gptsyscard] and Claude-3.5 Sonnet [@claude3series2024] (see Appendix [[app:C]](#app:C){reference-type=“ref” reference=“app:C”} for the evolution of VLMs). However, they impose a significant computational burden in real-world applications, such as on-device processing. Hence, it is necessary to develop small VLMs for deployment on lightweight devices, and this demand has led to two types of distillation approaches. The first constructs visual instruction tuning datasets [@li2024llava; @gu2024infinity; @zhang2024beyond; @li2024cvlm; @chen2023can; @hu2024mplug; @wang2024enhancing; @Sujet-Finance-QA-Vision-100k] from large-scale VLMs and trains small VLMs on these curated datasets. The second performs feature distillation: LLaVA-MoD [@shu2024llava] conducts logit-based distillation on a mixture-of-experts (MoE) [@shazeer2017; @riquelme2021scaling] architecture. LLaVA-KD [@cai2024llava] also uses logit-based distillation but adopts a three-stage training process in which the trainable parameters differ at each stage to effectively warm up and distill the knowledge. Align-KD [@feng2024align] distills both vision-encoder features and decoder logits, while encouraging the student VLM’s first decoder layer’s attention map to resemble that of the large VLM. MoVE-KD [@cao2025move] employs multiple vision encoders for the large VLM and applies a mixture of LoRA [@hu2021lora] to the small VLM, distilling visual information through visual attention maps. Besides, there are earlier distillation studies (see Appendix [[app:C]](#app:C){reference-type=“ref” reference=“app:C”}), but they consider neither token-derived nor different-token-type distillation.

However, existing works are largely constrained to VLMs sharing identical token types, including vocabulary size, token segmentation, and token index ordering. When the large and small VLMs employ different tokenizers or image embedding strategies, distillation becomes problematic—KL divergence cannot be computed due to mismatched output token counts, and token indices may no longer correspond to semantically equivalent meaning. Overcoming this limitation would represent a significant advancement, broadening the applicability and robustness of distillation techniques for diverse VLM architectures.

3 GenRecal: Generation after Recalibration↩︎

3.1 Model Architecture and Components↩︎

We briefly illustrate the overall model architecture and training procedure in 2. GenRecal consists of three main components: a large (teacher) VLM, a small (student) VLM, and a Recalibrator module. For the large VLMs, we select models with over 72B parameters to ensure competitive performance: NVLM-72B (Qwen2-72B) [@nvlm2024], Qwen2-VL-72B (Qwen2-72B) [@wang2024qwen2vl], InternVL2-76B (Llama3-70B) [@chen2024far], and InternVL2.5-78B (Qwen2.5-72B) [@chen2024expanding]. For the small VLMs, we adopt various model sizes, including Qwen2-VL-2B/7B (Qwen2-2B/7B) and InternVL2.5-1B/2B/4B/8B (InternLM2.5-1.8B/7B and Qwen2.5-0.5B/3B/72B). Note that the parentheses denote the LLMs used within each VLM. Throughout this section, we divide a VLM architecture into four key modules: the vision encoder, vision projector, VLM-body, and VLM-head (language head). Recalibrator is designed to acquire shared feature representations within a common latent space. It consists of two decoder blocks and two projection layers, as depicted in 2 (c). We denote the decoder blocks as Rec-body, and the projection layers as Proj-pre and Proj-post. The decoder blocks, Rec-body, adopt the same structural design as the student VLM’s decoder, while both Proj-pre and Proj-post are implemented as single linear layers.

3.2 Design and Propagation of Recalibrator↩︎

To construct a shared representation space, we first extract and align hidden features from both the large and small VLMs. Since the hidden space of a large VLM typically encompasses a richer vocabulary and more expressive representations, we regard it as the shared latent space which the small VLM is projected into. However, direct alignment of the two VLMs’ tokens within this space is infeasible due to discrepancies in vocabulary size, token split, and token index ordering. To overcome this challenge, we draw inspiration from the principle of autoregressive modeling, wherein a model predicts answer tokens for the given question tokens.

Specifically, we feed the same question-answer pair into both the large and small VLMs, obtaining their VLM-body outputs as: \[\begin{align} [z_{q_l}, z_{a_l}] &= \textit{VLM-body}_l([q_l, a_l]), \quad&(l:\text{large})\\ [z_{q_s}, z_{a_s}] &= \textit{VLM-body}_s([q_s, a_s]), \quad&(l:\text{small}) \end{align}\] where \(q\) and \(a\) denote the question and answer tokens produced by each model’s vision encoder, vision projector, and word embedding [@mikolov2013efficient]. We then extract the question features \(z_{q_s}\) from the small VLM and the answer features \(z_{a_l}\) from the large VLM, and concatenate them to form a joint sequence: \([z_{q_s}, z_{a_l}]\), which is then passed through Recalibrator. Because directly embedding both features into a shared space is infeasible due to different token types, we instead design an autoregressive loss that predicts the answer token index of the large VLM given the question token index of the small VLM. This loss encourages Recalibrator to effectively project the small VLM’s features into the large VLM’s latent space.

However, the hidden dimensions of large and small VLMs often differ. To address this, in Recalibrator, we first apply Proj-pre to \(z_{a_l}\), aligning its dimensionality to that of \(z_{q_s}\). The concatenated sequence \([z_{q_s}, \textit{Proj-pre}(z_{a_l})]\) is then propagated through Rec-body decoder blocks. The output of Rec-body is subsequently passed through Proj-post to restore the hidden dimensionality of the large VLM. This forward rule of Recalibrator is illustrated in 2 (c).

Finally, the answer part of the resulting features \([r_{q_s}, r_{a_l}]=\textit{Recalibrator}([z_{q_s}, z_{a_l}])\) is fed into the VLM-head of the large VLM to compute the autoregressive loss: \[\mathcal{L}_{ar} = \text{CE}(\textit{VLM-head}_l(r_{a_l}), gt_l),\] where \(gt_l\) denotes the answer token index from the large VLM. Additionally, we employ a KL divergence loss to further enhance the Recalibrator’s projection capability in the first training stage and to distill knowledge from the teacher in the second training stage. This process enables the large VLM to interpret the hidden features of the small VLM, thus establishing a bridge for general-purpose distillation across heterogeneous token types.

3.3 Training Process↩︎

To achieve general-purpose distillation, we adopt a three-stage training process. In the first stage, only Recalibrator is trained while keeping all parameters of both the large and small VLMs frozen, as illustrated in 2 (a). During this stage, we compute both the autoregressive loss and the KL divergence, described in 4 where the components indicate the trainable parameters. This step primarily serves to align the feature representations between the large and small VLMs. For the first stage, we introduce regularization loss, as illustrated in 3, which is designed to prevent the feature representations of Recalibrator from drifting too far from those of the large VLM. We use both 4 and 5 together in the first stage.

In the second stage, we keep using the same loss functions of the first stage and additionally incorporate the small VLM’s own autoregressive loss in 6. As depicted in 2 (b), this stage enables knowledge distillation from the large VLM to the small one by training the small VLM’s VLM-body. In the final stage, we remove both Recalibrator and the large VLM, and fine-tune the student VLM. Specifically, we train all parameters of the student VLM except for the vision encoder through supervised fine-tuning (SFT) to further enhance its instruction-following capability.

4 Experiments↩︎

4.1 Implementation Detail↩︎

To ensure reproducibility, we outline three key technical aspects of GenRecal: (a) the structure of the Recalibrator, (b) the projection layers, and (c) training and evaluation details.

(a) Structure Detail of Recalibrator. This is composed of two transformer decoder blocks (Rec-body) and two projectors (Proj-pre and Proj-post). The decoder blocks follow all configurations of the small VLM’s decoder block, such as the causal mask, hidden dimensions, number of heads, and FFN structure. We use transformer decoder blocks for Rec-body because they naturally process the concatenated student and teacher features in 2 as a sequence. Moreover, these features must preserve token ordering, so we introduce new positional embeddings (NPE). This design enables effective sequential modeling and representation alignment between heterogeneous VLMs. To this end, we employ an additional RoPE [@su2024roformer] to realign their positional embeddings, and their position IDs are reassigned accordingly. Lastly, we apply an additional layer norm [@ba2016layernormalization] to the output features of the Recalibrator for stable adaptation.

(b) Projection Layer Design. Two projection layers (Proj-pre and Proj-post) are linear layers introduced for hidden dimension matching. Similar to the role of vision projector that matches the hidden dimensions between a CLIP-like vision encoder and an LLM, Proj-pre matches the teacher features to the student hidden dimension, while Proj-post restores the dimension back to the teacher hidden dimension so that the large VLM-head can be utilized for distillation. This explanation is technically depicted in 2 (c) and 3.

(c) Details of Training and Evaluation. By using DeepSpeed engine with ZeRO-3 [@rajbhandari2020zero], we train and evaluate GenRecal on 128 NVIDIA A100 80GB GPUs. We use AdamW optimizer [@loshchilov2018decoupled] and apply a linearly decayed learning rate from 1e-4 to 1e-5 at each training stage. The first and second stages use the entire 9M dataset (see Appendix [[app:E]](#app:E){reference-type=“ref” reference=“app:E”} for dataset composition and analysis), taking approximately 2–4 hours and 8–11 hours, respectively, depending on model size. The last stage takes 4–6 hours on a 6M dataset obtained by removing general visual question answering samples [@li2024llava]. For stable training, we handle large batch sizes via gradient accumulation with 16 steps. We use 4 (first and third stages) or 2 (second stage) samples per GPU, leading to a total batch size of 8192 (128\(\times\)​16\(\times\)​4) or 4096 (128\(\times\)​16\(\times\)​2). For evaluation, we remove the Recalibrator and the large VLM, and let the small VLM generate answers using the default generation hyperparameters.

(d) Dataset Composition. Since the 9M corpus has a decisive impact on the results, we summarize it here. It covers general visual question answering, dense captioning, chart/diagram/document understanding, commonsense knowledge, science & math, and multi-dimensional reasoning, curated from public sources such as LLaVA-OneVision [@li2024llava], Cambrian [@tong2024cambrian], Infinity-MM [@gu2024infinity], and DenseFusion [@li2024densefusion], among others. For analysis we group these into three domains—Knowledge, Science & Math, and Chart & Document—which align with benchmarks such as MMMU [@yue2023mmmu], MathVista [@lu2023mathvista], and ChartQA [@masry2022chartqa], respectively. A full source list and per-domain breakdown are provided in Appendix [[app:E]](#app:E){reference-type=“ref” reference=“app:E”}.

4.2 Validation and Analysis for GenRecal↩︎

Higher Size, Higher Performance. 1 demonstrates that GenRecal outperforms not only standard-size VLMs but also smaller ones, highlighting that GenRecal generalizes well across model sizes. This table also shows that choosing a higher-performing small VLM such as InternVL2.5-8B [@chen2024expanding] substantially improves the overall distillation results. Furthermore, for a fixed large VLM, selecting a smaller student VLM leads to lower distillation performance, implying that employing more capable small VLMs is crucial for achieving higher distillation performance.

Next, we examine the effect of changing large VLMs to determine whether the aforementioned property holds consistently across different large VLMs. As shown in 1 and 2, selecting more powerful large VLMs leads to greater performance improvements. To investigate this further, we use the InternVL2.5 series [@chen2024expanding] and experiment with various combinations of large and small VLM sizes, as illustrated in 7. The results consistently indicate that choosing larger teacher and student VLMs leads to higher distillation performance.

Analysis of Recalibrator. We now focus on the Recalibrator to explore its role in general-purpose distillation. First, we show that the loss curves for training the Recalibrator converge stably across various combinations of large and small VLMs (see Appendix [[app:D]](#app:D){reference-type=“ref” reference=“app:D”}). Since the Recalibrator losses (‘Recalib\((q_s, a_l)\)’ and ‘Recalib\((q_l, a_l)\)’) are minimized well relative to the perplexity of ‘SmallVLM\((q_s, a_s)\)’ and ‘LargeVLM\((q_l, a_l)\)’, we infer that the Recalibrator successfully aligns the feature representations of large and small VLMs, and that it generalizes well across teacher–student combinations. Furthermore, 8 visualizes multiple feature spaces to assess whether the large- and small-VLM features after the Recalibrator are truly matched and can be regarded as a shared representation. The small-VLM features before the Recalibrator are initially close to those after it, but by the end of training they become matched to the large-VLM features after the Recalibrator. This indicates that the Recalibrator is effective at forming a shared feature representation for large and small VLMs.

Importance of Regularization. We conduct an ablation study that investigates how important the regularization term is for the final distillation performances. We first remove the regularization term and analyze the differences before and after its removal. To illustrate this in detail, we simply prepare 10 different question-answer pairs and input them through VLM-body of both small and large VLMs. Afterwards, we propagate their output features into Recalibrator, as described in 9 (a). We then compute a \(10 \times 10\) cosine similarity matrix between Recalibrator’s output features from the small and large VLMs. To ensure reliable cosine similarity values, we use all training dataset samples and compute the average of the cosine similarity values for all the samples.

With the regularization term included, as shown in 9 (c), we observe that the diagonal values in the cosine similarity matrix are significantly higher than the off-diagonal values. This implies that the features of small and large VLMs are explicitly shared through Recalibrator, aligned with the results in 8 as well. In contrast, when the regularization term is removed, as depicted in 9 (b), the diagonal values are comparable with the off-diagonal values. This indicates that Recalibrator fails to explicitly align large and small VLMs’ features, preventing feature matching and sharing. As a result, this leads to a significant drop in final distillation performances, as reported in ¿tbl:tab:3?. Insufficient alignment of large and student VLMs reduces efficient knowledge transfer during distillation. These findings highlight the crucial role of the regularization term in feature alignment and knowledge transfer for general-purpose distillation.

Autoregressive Loss for Representation Alignment. To align the student representations with the teacher’s latent space, GenRecal uses two objectives: KL divergence loss (\(\mathcal{L}_{kl}\)) and autoregressive loss (\(\mathcal{L}_{ar}\)). The KL divergence loss performs standard distillation by matching the logit-level distribution of teacher and student. However, matching distributions alone does not guarantee precise alignment at the token level. The autoregressive loss provides explicit hard-target supervision, allowing it to predict the exact ground-truth token sequence. In short, \(\mathcal{L}_{kl}\) transfers knowledge at the distribution level, whereas \(\mathcal{L}_{ar}\) ensures exact alignment at the token level. As shown in 3, removing \(\mathcal{L}_{ar}\) consistently degrades performance across benchmarks, confirming that autoregressive supervision is essential for effective representation alignment.

4.3 Ablation Studies: Training Computation and Configurations↩︎

In ¿tbl:tab:5? (a), we compare the model sizes and training FLOPs of the small VLM, large VLM, and Recalibrator to assess whether Recalibrator introduces additional computational overhead during training. The results show that Recalibrator’s FLOPs are substantially lower than those of both the large and small VLMs. Note that Recalibrator is removed at inference time, thus incurring no additional computational cost during inference. We further conduct a series of ablation studies to examine various configurations affecting the distillation performance of GenRecal. As shown in ¿tbl:tab:5? (b), we vary the decoder depth of Recalibrator from 1 to 20 and select two depths for layer number of Rec-body based on a trade-off between computational efficiency and performance. Additionally, ¿tbl:tab:5? (c) demonstrates the importance of employing a new positional embedding (NPE) to handle newly introduced position IDs and to integrate RoPE [@su2024roformer] from both large and small VLMs, which have different token types.

To analyze the effect of dataset scale, we control the dataset size from 1M to 9M samples, as shown in ¿tbl:tab:5? (d). The performance improvements between 5M and 9M samples are marginal, suggesting that a 5M dataset is a practical choice for users with limited training resources. Furthermore, ¿tbl:tab:5? (e) shows that fine-tuning the large VLM on our dataset yields further gains, indicating that the small VLM trained under the same setting has the potential to match or even surpass the large VLM’s performance.

We also compare GenRecal with recent cross-tokenizer distillation methods in ¿tbl:tab:5? (f). These baselines perform distillation via cross-token matching in the logit space using the classical Wasserstein distance [@boizard2025towards] or optimal transport [@cui2025multi], whereas GenRecal introduces the Recalibrator to align hidden features for semantic correspondence. GenRecal achieves a substantial improvement over these approaches, as discussed in the next section. Lastly, ¿tbl:tab:5? (g)–(i) demonstrate GenRecal’s strong compatibility across different LLMs, VLM families, and recently released VLMs. In all cases, GenRecal consistently delivers superior performance, validating its general applicability to diverse model architectures.

4.4 Discussion↩︎

Representation Mismatch Beyond Tokenizer Mismatch. Even within a family the tokenizer can differ—InternVL2.5-8B uses InternLM2.5 [@cai2024internlm2] while InternVL2.5-78B uses Qwen2.5 [@yang2024qwen2], whereas Qwen2-VL-7B/72B [@wang2024qwen2vl] share the Qwen2 tokenizer ([fig:1]). More importantly, even with an identical tokenizer, a large teacher–student gap causes a representation mismatch: conventional distillation routes knowledge through the low-dimensional student head, creating a bottleneck. GenRecal instead distills through the high-capacity teacher head, with the Recalibrator warm-starting student features into the teacher space (2). This is why it helps even under the same tokenizer: in [tbl:tab:4] (Qwen2-VL-72B teacher, 7B/2B students), GenRecal beats both SFT and traditional KD—MiniLLM [@gu2024minillm] (reverse KL), DistiLLM [@ko2024distillm] (skewed KL), and LLaVA-KD [@cai2024llava] (3-stage KL)—by a clear margin, showing explicit feature alignment matters even with shared token types.

Cross-Token Matching. Cross-token matching takes two forms: (a) word-embedding matching, infeasible without retraining the decoder, and (b) semantic feature matching. For (b), ULD [@boizard2025towards] and MOT [@cui2025multi] align logit-space probability mass via Wasserstein distance or optimal transport (¿tbl:tab:5? (f)), but cannot handle token-split indices and resort to zero-padding or truncation, losing information. GenRecal avoids logit-level matching altogether: the Recalibrator aligns features before the head so the teacher VLM-head reads student features directly, needing no token correspondence, probability sorting, padding, or transport cost. It adds only a small FLOPs fraction (¿tbl:tab:5? (a)), so GenRecal’s total cost is comparable to ULD/MOT yet reaches higher accuracy (4; same data, InternVL2.5-8B student and -78B teacher).

Why a Non-Linear Recalibrator, and Where the Novelty Lies. Prior shared-space results rely on linear word-embedding maps [@mikolov2013exploiting; @smith2017offline; @conneau2017word], which act on input embeddings, not on the answer-conditioned hidden states after the VLM-body. Aligning these post-body features while preserving token order exceeds a linear map, so we use shallow transformer decoder blocks. The novelty is thus not the module itself—depth beyond two layers saturates (¿tbl:tab:5? (b))—but the stable recipe that makes cross-tokenizer teacher-head distillation work.

Isolating the Recalibrator from the Recipe. To check whether the gains come from the Recalibrator or merely from the multi-stage recipe and final SFT, we disentangle them on Qwen2-VL-7B with the same-token-type teacher Qwen2-VL-72B (so logit-KD baselines stay valid), all on the same 9M data. The recipe alone (SFT\(+\)logit-KD, no Recalibrator) gives \(+3.4\) Avg, while adding the Recalibrator adds a further \(+4.6\) (5). Nor is the gain from Stage-3 SFT: after Stage 2, before any final SFT, GenRecal already reaches \(74.3\) Avg vs.\(69.2\) for SFT-only on the same 9M (6). Nor is it mere data scale: GenRecal@1M (\(\sim\)​11% data) already beats SFT@9M, and @3M nears the full result (7). Further analyses are provided in the appendix: open-weight teacher dependence and the SFT fallback (Appendix [[app:F]](#app:F){reference-type=“ref” reference=“app:F”}), component-level ablations (Appendix [[app:G]](#app:G){reference-type=“ref” reference=“app:G”}), the regularization formulation (Appendix [[app:H]](#app:H){reference-type=“ref” reference=“app:H”}), error-profile and calibration (Appendix [[app:I]](#app:I){reference-type=“ref” reference=“app:I”}), and a compute/fairness comparison against baselines (Appendix [[app:J]](#app:J){reference-type=“ref” reference=“app:J”}). Note that GenRecal operates under open-weight teacher access, where the teacher’s parameters and hidden features are available for training.

5 Conclusion↩︎

We present Generation after Recalibration (GenRecal), a general-purpose distillation framework that transfers knowledge from large to small vision-language models (VLMs) even when they adopt different token types. Existing distillation methods assume that the teacher and student share the same token types, while GenRecal removes this constraint through a Recalibrator that aligns the student’s hidden features into the teacher’s representation space. Since the Recalibrator is discarded after training, inference preserves the student’s original architecture and cost. Our analyses further indicate that these gains stem primarily from the recalibration mechanism rather than from the multi-stage recipe, and that feature alignment remains beneficial even when the teacher and student already share a tokenizer. For future work, we plan to extend the Recalibrator to intermediate layers to capture fine-grained, sequential knowledge, and to explore distillation from multiple VLM sources. We envision GenRecal as a pivotal and versatile framework for knowledge distillation.

Acknowledgements↩︎

We thank our colleagues at NVIDIA, in particular Sharath Turuvekere Sreenivas, for valuable discussions on cross-tokenizer and cross-model knowledge distillation. We further thank Sepehr Sameni and Daniel Korzekwa for experimenting with this approach on Minitron and Cosmos families; subsequently, Sharath Turuvekere Sreenivas and Pavlo Molchanov developed the follow-up X-Token [@sreenivas2026x], extending it to multi-teacher cross-tokenizer distillation on the logit space.

6 Token Type Examples↩︎

7 Possible Distillation Pair for VLM Combinations: Traditional Distillation versus GenRecal↩︎

8 Extended Related Works↩︎

Since the emergence of visual instruction tuning [@liu2023visual], many variations of VLMs have emerged, such as LLaVA-1.5 [@liu2023improved], InstructBLIP [@dai2023instructblip], ShareGPT-4V [@chen2023sharegpt4v], and MiniGPT-v2 [@chen2023minigpt; @zhu2023minigpt], typically starting at a standard model size of 7B. After that period, numerous VLMs [@liu2024llavanext; @mckinzie2024mm1; @li2024mini; @dong2024internlm4khd] have divided an image into multiple sub-regions to focus on the image’s details so that visual perception is enhanced. CoLLaVO [@lee-etal-2024-collavo] and MoAI [@10.1007/978-3-031-72967-6_16] utilize computer vision models directly for visual capability, and Mini-Gemini [@li2024mini], MoVA [@zong2024mova], and Eagle [@shi2024eagle] employ multiple vision encoders such as CLIP [@clip], ConvNext [@woo2023convnext], DINO-v2 [@oquab2023dinov2], and SAM [@kirillov2023segment]. In parallel, Meteor [@NEURIPS2024_473a9a75] explores the efficient way of learning complex reasoning abilities, and TroL [@lee-etal-2024-trol] and Phantom [@lee2024phantom] investigate propagation modification for how we can embed vision-language knowledge as much as possible despite using the same architectures. More recently, many VLMs like Molmo-72B [@deitke2024molmo], LLaVA-OneVision-72B [@li2024llava], NVLM-72B [@nvlm2024], Qwen2-VL-72B [@wang2024qwen2vl] and InternVL2.5-78B [@chen2024expanding] have employed large-scale language models such as Qwen2/Qwen2.5-72B [@yang2024qwen2]. Thanks to scaling laws [@kaplan2020scaling; @chung2022scaling], they have closely reached the performances of GPT-4V [@gptsyscard] and Claude-3.5 Sonnet [@claude3series2024]. Therefore, developing VLMs with large-scale language models is getting to a standard these days. Beyond architectural design, recent efforts further explore training and evaluating such VLMs, including unified reinforcement and imitation learning [@NEURIPS2025_e5849736], recursive think–answer reasoning [@Lee_2026_CVPR_Recursive], multimodal agentic reasoning [@kang2026agent; @cho2026spatialclawrethinkingactioninterface], and multi-turn conversation benchmarking [@Lee_2025_ICCV].

This provides an effective framework [@hinton2015distilling] for transferring the rich representations of large, high-performing models to smaller, efficient counterparts while preserving performance. Early studies primarily focused on aligning the output logits of a “teacher” model with those of a “student” [@sanh2019distilbert; @turc2019well]. Subsequently, the FitNet framework [@romero2014fitnets] extended this idea by supervising the student using intermediate feature representations rather than final outputs. In this setup, convolutional layers project the student’s features to resemble those of the teacher, and discrepancies are minimized using an L2 loss.

Following this feature-based paradigm, later methods explored more sophisticated ways of leveraging internal representations for effective knowledge transfer. For instance, [@sun2019patient; @chen2021distilling; @wang2024crosskd; @ben2022s] demonstrated that incorporating multiple intermediate layers from the teacher significantly enhances distillation effectiveness. Probabilistic approaches such as PKT [@passalis2020probabilistic] reformulated the teacher’s knowledge as a probability distribution, aligning it with the student’s outputs via KL divergence. Similarly, RKD [@park2019relational] captured inter-sample relationships, while CRD [@tian2019contrastive] combined contrastive learning with traditional distillation.

Multi-stage and relational strategies further broadened the field. AT [@zagoruyko2016paying] utilized attention maps from several teacher layers, and FSP [@yim2017gift] employed matrices derived from feature maps to guide the student. Later, SP [@tung2019similarity] evaluated similarity among input samples, and OFD [@heo2019comprehensive] introduced a margin-based distance metric derived from ReLU activations to capture essential information.

As the methodology matured, two main paradigms emerged. On-policy approaches [@austin2021program; @ko2024distillm; @agarwal2024policy; @gu2024minillm] dynamically sample data during training, whereas off-policy methods [@ko2024distillm; @sun2019patient; @turc2019well; @zhang2024dual; @boizard2025towards] rely on pre-collected datasets. While early research mostly adopted single-teacher frameworks [@agarwal2024policy; @gu2024minillm; @shu2024llava; @muralidharan2024compact], multi-teacher distillation has gained increasing attention despite challenges such as architectural differences, vocabulary mismatches, and task misalignment [@timiryasov2023baby; @lee2023ensemble; @wan2024fusechat]. Representative examples include DKMF [@wang2021distilling] and FNKD [@xu2020feature], which utilize teacher feature representations, while DGKD [@son2021densely] aggregates knowledge from multiple teachers to boost student performance.

Recent work has also emphasized refining loss functions used in distillation. MiniLLM [@gu2024minillm] and DistiLLM [@ko2024distillm] propose modified KL divergences—such as reverse or skewed variants—to reduce overfitting, particularly for long-tail predictions. Similarly, [@wu2024rethinking] introduced a dynamic loss-balancing scheme that adaptively combines conventional and reverse KL divergences during training. Finally, to enhance reasoning capabilities, recent studies integrate chain-of-thought (CoT) supervision. Methods such as [@hsieh2023distilling; @tian2024beyond] leverage detailed reasoning traces from larger models, while TinyLLM [@tian2024beyond] aggregates multi-teacher reasoning to enrich training signals and improve generalization.

In the vision-language setting, recent works distill or compress large VLMs into smaller, deployable ones: VLsI [@Lee_2025_CVPR] transfers knowledge through verbalized layer-to-layer interactions, masking-teacher with student reinforcement [@Lee_2026_CVPR_Masters] and reasoning-prefix masking [@yu2026hide] refine the teacher–student signal, visual-token pruning [@kim2026and] removes redundant visual computation, and broader recipes merge, modify, and distill VLMs [@lee2025building]. For language models, related efforts study prompt-based teaching [@lee2026zone], skill-of-mind conversational modeling [@leeenhancing], and refinement-capability evaluation [@lee2026refinebench]. The authors’ broader research additionally spans robustness [@lee2021towards; @NEURIPS2021_8e5e15c4; @Lee_2022_CVPR; @Lee_2023_ICCV; @lee2020training] and causal representation learning [@Kim_2023_CVPR; @KIM2026112173; @10222502].

9 Loss Graphs for Recalibrator↩︎

10 Dataset Composition and Analysis↩︎

We gather a 9M visual instruction tuning dataset covering a wide range of vision-language capabilities, such as general visual question answering, dense image captioning, chart/diagram/document understanding, commonsense knowledge, science and math understanding, and multi-dimensional reasoning. Our dataset includes LLaVA-OneVision [@li2024llava], MMC [@liu2023mmc], DenseFusion [@li2024densefusion], Cambrian [@tong2024cambrian], GPT-4V-filtered synthetic data of SA-1B [@kirillov2023segment] and Infinity-MM [@gu2024infinity], Finance-QA [@Sujet-Finance-QA-Vision-100k], Wikipedia knowledge [@li2024cvlm], InfoSeek [@chen2023can], science & mathematical reasoning (SMR) [@zhang2024beyond], document-downstream/reasoning [@hu2024mplug], WildVision [@lu2024wildvision], SROIE [@huang2019icdar2019], RLAI-F [@yu2024rlaif], M3CoT [@chen-etal-2024-m3cot], LLaVAR [@zhang2023llavar], KonIQ [@hosu2020koniq], iNaturalist2018 [@van2018inaturalist]. For further analysis, we particularly categorize the visual instruction tuning dataset that we use to build GenRecal into three domains: ‘Knowledge’, ‘Science & Math’, and ‘Chart & Document’. Based on these categories, we conduct dataset composition analysis by removing each of the three domains. This implies that MMMU [@yue2023mmmu] requires VLMs’ ability for the domain of ‘Knowledge’ more than the others, while MathVista [@lu2023mathvista] requires their capabilities for ‘Science & Math’.

• Knowledge: GPT-4V filtered synthetic of SA-1B [@kirillov2023segment] and Infinity-MM [@gu2024infinity], Wikipedia Knowledge [@li2024cvlm], Wild-Vision, InfoSeek [@chen2023can], WildVision [@lu2024wildvision], and iNaturalist2018 [@van2018inaturalist]

• Science & Math: LLaVA-OneVision (subset) [@li2024llava], Cambrian (subset) [@tong2024cambrian], Finance-QA [@Sujet-Finance-QA-Vision-100k], SMR [@zhang2024beyond], M3CoT [@chen-etal-2024-m3cot], and KonIQ [@hosu2020koniq]

• Chart & Document: LLaVA-OneVision (subset) [@li2024llava], Cambrian (subset) [@tong2024cambrian], MMC [@liu2023mmc], RLAI-F [@yu2024rlaif], document-downstream/reasoning [@hu2024mplug], SROIE [@huang2019icdar2019], and LLAVAR [@zhang2023llavar]

11 Open-Weight Teacher Dependence and SFT Fallback↩︎

GenRecal trains with the teacher’s hidden states and language head, so it assumes an open-weight teacher. This matches our central motivation—distilling heterogeneous open VLMs with different token types—rather than closed/API models. When only a closed/API teacher is available, the natural fallback is supervised fine-tuning (SFT) on teacher-generated responses. 8 compares this fallback with GenRecal on two students. Notably, a substantial portion of our 9M corpus already consists of closed/API-generated samples (, Infinity-MM [@gu2024infinity], LLaVA-OneVision [@li2024llava], Cambrian [@tong2024cambrian], and RLAI-F [@yu2024rlaif]; see Appendix E). Thus the “SFT only” row is effectively trained on closed/API-generated responses, yet GenRecal still outperforms it by \(+6.0\) and \(+7.2\) Avg. We therefore scope the general-purpose claim to open-weight teachers and state the teacher-access assumption explicitly in the revision.

We additionally clarify our response-collection setup. Some 9M samples contain very short answers (, “A” or “32”). To provide a richer distillation signal, we collect correct large-teacher responses with more detailed reasoning before training, and all experiments are based on this setup.

12 Component-Level Ablation of GenRecal↩︎

We complement the stage-wise analysis (6) with a component-level view in 9, where each row removes a single component of GenRecal and the corresponding source ablation in the main text is cited. Removing the autoregressive loss \(\mathcal{L}_{ar}\) is by far the most damaging, as it collapses feature alignment; removing the regularization term is the next most harmful. The new positional embedding (NPE) and a deeper Rec-body provide smaller but consistent improvements. This confirms that the Recalibrator and its training objectives—not the multi-stage SFT recipe—are responsible for the bulk of the gain.

13 Regularization Formulation Ablation↩︎

The regularization loss \(\mathcal{L}_{reg}\) passes only the teacher’s own tokens \([z_{q_l}, z_{a_l}]\) through the Recalibrator and constrains the result to remain readable by the teacher head (5). To motivate this specific formulation, 10 ablates alternative constraints on the same teacher input. A pointwise feature constraint (Feature MSE) that simply keeps the Recalibrator near identity helps only marginally, because matching features in \(\ell_2\) does not guarantee they remain interpretable by the language head. Enforcing the constraint through the teacher head—either as a KL or a CE term—is substantially more effective, and combining both (our choice) is best. This explains why the teacher-only pass anchors the Recalibrator on the teacher manifold and prevents drift, complementing the cosine-diagonal analysis in 9.

14 Error Profile and Calibration Analysis↩︎

Beyond aggregate benchmark scores, we analyze how GenRecal errs relative to the teacher, pooled over MathVista [@lu2023mathvista], MM-Vet [@yu2023mm], MMMU [@yue2023mmmu], and AI2D [@kembhavi2016diagram]. In 11, “Err.” is the number of wrong answers; the category columns (Perception/OCR, Spatial, Calculation, Reasoning, Hallucination) report the % distribution among errors; JSD is the Jensen–Shannon divergence between each model’s error distribution and the teacher’s; ECE is the expected calibration error; and WConf.is the average confidence on wrong answers. GenRecal is markedly closer to the teacher than SFT-only in both what it gets wrong (lower JSD) and how confident it is when wrong (lower ECE and WConf.), indicating that distillation transfers the teacher’s error structure and calibration, not merely its accuracy.

15 Compute and Baseline-Fairness Comparison↩︎

To address baseline fairness, we emphasize that all distillation baselines in [tbl:tab:4] use the identical 9M corpus and the same optimization setup within each comparable stage; differences in the number of stages follow the original recipe of each method. ¿tbl:tab:fairness? summarizes per-method compute (per-step model-pass FLOPs) and approximate wall-clock time on the Qwen2-VL-7B student with the Qwen2-VL-72B teacher. Stage 1 of GenRecal is lightweight because only the \(\sim\)​525M Recalibrator (¿tbl:tab:5? (a)) receives gradients while the rest is a forward pass. As a result, despite its three-stage recipe and extra module, GenRecal’s total cost is comparable to—or lower than—the distillation baselines, while achieving substantially higher average performance.