1 Introduction↩︎

The advent of vision foundation models has revolutionized vision-language models (VLMs) by enabling efficient multimodal integration without extensive retraining. Flamingo pioneered this by leveraging frozen vision encoders like CLIP [1], resampling their features via a Perceiver module, and prefixing them into a frozen large language model for few-shot multimodal tasks such as visual question answering and captioning. Building on this, BLIP-2 [2] introduced a lightweight Querying Transformer to bridge frozen CLIP-like vision encoders with frozen LLMs, achieving state-of-the-art zero-shot performance through two-stage pre-training. Kosmos-1 [3] further advanced the paradigm by projecting CLIP features and concatenating them with text tokens in a unified stream for perception-language alignment, supporting tasks like nonverbal reasoning. LLaVA [4] simplified this with direct linear projection of CLIP features into pre-trained LLMs, combined with visual instruction tuning using GPT-4-generated data, yielding versatile multimodal assistants.

Subsequent advancements saw SigLIP [5] and its successor SigLIP2 [6] emerge as popular vision foundation models, offering improved efficiency and multilingual capabilities through sigmoid-based losses and enhanced pre-training recipes. Several top VLMs adopt SigLIP for their vision backbones. For instance, PaliGemma [7] integrates it for multimodal generation, while Idefics2 [8] leverages it for vision-language understanding alongside Llama-based decoders. Cambrian-1 [9] employs an ensemble containing SigLIP for vision-centric tasks. Furthermore, NVILA [10] utilizes it for high-resolution image and video processing, as does Qwen3 VL [11]. Beyond SigLIP, other foundation models like RADIO [12] have been adopted by the Nemotron VL [13] family to power document intelligence, video understanding, and multi-image reasoning.

The vision encoders CLIP, SigLIP, and SigLIP 2, are predominantly trained using contrastive global objectives for matching entire images with their corresponding captions, facilitating high-level semantic alignment between visual and textual modalities. An exception is C-RADIOv2 [12], which employs an agglomerative objective by distilling knowledge from multiple foundation models, enabling broader representational capabilities across scales. This raises the question of why popular vision foundation models trained with dense, local objectives such as DINOv3 [14]’s self-supervised patch-level distillation or SAM3 [15]’s promptable segmentation are rarely adopted as backbones in VLMs. A common hypothesis posits image/text contrastive pretraining yields models more suitable for VLMs owing to already being text aligned [9], [16]–[19], even though the spatial features are not directly supervised with any language objective. If true, it may leave VLMs reliant on encoders with inherently poor dense spatial coherence and explainability, limiting their performance on vision-centric tasks requiring localization or fine-grained reasoning. In this work, we explore this trade-off further by proposing that summarization capabilities rather than raw vision-language alignment may be the most critical factor for effective VLM performance, and we investigate this through novel architectural and training interventions.

Our contributions may be summarized as follows:

• Analysis of vision features: We analyze how vision encoders evolve during VLM fine-tuning, showing a shift toward more abstract, less spatially organized representations. We further identify a small set of specialized tokens in image–text–aligned encoders that capture global image semantics.

• RADIO1D: We propose a method that compresses images into flexible, variable-length 1D token sequences using multi-teacher distillation, hierarchical encoding (with early tokens capturing global information), and an autoencoder-like decoder used only during training.

• Strong summarization: RADIO1D enables accurate scene understanding with very few tokens—often a single token—demonstrating effective compression of essential visual content.

• Composition-aware retrieval: We introduce an evaluation metric that assesses object presence as well as spatial arrangement for retrieved images, and show that compact RADIO1D tokens outperform standard vision encoders.

• VLM integration: Integrating RADIO1D into vision-language models yields flexible accuracy–efficiency trade-offs via variable token counts, competitive performance across multimodal tasks, and reduced computational cost.

• Release: We release model checkpoints under a permissive license.

The authors are employed by NVIDIA, which leads the development of RADIO.

2 Preamble↩︎

2.1 Analysis of two specialists: SigLIP2 and DINOv3↩︎

We analyze two popular vision foundation models: SigLIP2 and DINOv3. SigLIP2 is widely used in VLMs (PaliGemma, Idefics2, Cambrian-1, NVILA, Qwen3-VL) due to strong image-text alignment. DINOv3 excels in dense tasks, achieving state-of-the-art 63.0 mIoU on ADE20K [20] with Mask2Former [21], and extends to detection, depth, and 3D applications [22], [23].

Despite limited crossover—SigLIP2 rarely used for dense tasks, DINOv3 underperforming on OCR—we examine their representations via a PCA projection to 3 channels, then directly map these to RGB (Figure 3). DINOv3 features show clean spatial coherence with color gradients aligned to semantic structures (e.g., fur, leaf). SigLIP2 features are noisier and fragmented, consistent with its global contrastive objective, and suggesting that their masked reconstruction objective doesn’t fully recover DINO-like features.

To probe global summarization, we compute per-token ImageNet Top-1 accuracy via k-NN (k=20) on 256×256 images (Figure 4). Center tokens perform better overall (positional bias). DINOv3 outperforms SigLIP2 on average; however, SigLIP2 has three standout tokens (matching PCA anomalies in Figure 3), acting as powerful global summarizers.

We quantify spatial coherence with Centered Kernel Alignment (CKA) [24] on MS-COCO [25] validation embeddings, utilizing the Hilbert-Schmidt Independence Criterion (HSIC): \[\mathrm{CKA}(\mathbf{X},\mathbf{Y}) = \frac{\mathrm{HSIC}(\mathbf{K}, \mathbf{L})}{\sqrt{\mathrm{HSIC}(\mathbf{K}, \mathbf{K}) \cdot \mathrm{HSIC}(\mathbf{L}, \mathbf{L})}}\]

where \(\mathbf{K} = \mathbf{HXX}^\top \mathbf{H}\), \(\mathbf{L} = \mathbf{HYY}^\top \mathbf{H}\), and \(\mathbf{H} = \mathbf{I} - \frac{1}{n}\mathbf{1}\mathbf{1}^\top\). Figure 5 shows DINOv3 with bright off-diagonals (strong locality: horizontal/vertical adjacencies). SigLIP2 has weaker correlations and three standout tokens (indices 13, 176, 242). DINOv3 has the highest \(\overline{\mathrm{CKA}}_{\mathrm{off}}\); SigLIP2 the lowest. C-RADIOv4 features resemble DINOv3 (strong spatial correlation).

2.2 Analysis of a Fine-tuned Vision Encoder↩︎

We examine VLM fine-tuning effects (Figure 6). Pre-trained C-RADIOv4 [26] features resemble DINOv3 (localized). Post-fine-tuning features become dispersed and noisy, resembling SigLIP2—suggesting VLM optimization discards spatial coherence for cross-modal alignment. Corresponding CKA matrices (Figure 5) show fading off-diagonals post-fine-tuning, with \(\overline{\mathrm{CKA}}_{\mathrm{off}}\) dropping from 0.281 to 0.035.

3 Method↩︎

From an information-theoretic perspective, patch-based image representations impose a fixed 2D grid that is not well matched to the statistics of natural images or the constraints of VLMs. Let an input image \(x\) be a realization of a random variable \(X \sim p(X)\), with task-relevant information captured by a target variable \(Y\). A vision encoder aims to produce a representation \(Z\) that maximizes mutual information \(I(Z; Y)\) while minimizing representational complexity, commonly quantified by the entropy \(H(Z)\). In patch-based encoders, \(Z\) consists of spatially localized tokens arranged on a grid. However, natural images exhibit strong spatial correlations, inducing conditional dependencies between neighboring tokens, i.e., \(I(Z_i; Z_j \mid X) > 0\). This redundancy increases \(H(Z)\), since \(H(Z) = \sum_i H(Z_i) - \sum_{i<j} I(Z_i; Z_j) + \text{higher-order terms}\), where the mutual information terms reflect overlapping content across patches. As a result, patch-based representations fail to efficiently compress spatial structure, leading to a suboptimal rate–distortion trade-off under sequence-length and compute constraints.

In contrast, the 1D sequence representation used in RADIO1D enables a more flexible and compressible encoding by treating tokens as an unstructured, elastic sequence. This design aligns with the Information Bottleneck (IB) principle [27], which seeks a representation maximizing \(I(Z; Y) - \beta I(Z; X)\). The RADIO1D autoencoder can be viewed as a variational approximation to the IB, mapping \(X\) to a variable-length 1D sequence that aggregates global and hierarchical information while reducing redundancy through stochastic slicing and multi-teacher distillation. This enables adaptive compression: visually simpler images require fewer tokens (lower \(H(Z)\)), while preserving task-relevant information comparable to or exceeding that of rigid 2D grids. Empirically, this manifests as reduced sequence length without degradation in downstream performance, and theoretically corresponds to convergence toward a lower-entropy representation manifold consistent with the minimum description length principle [28]. In the following paragraphs, we provide more details about our method, which is also depicted in figure 2.

3.1 Agglomerative Training↩︎

Agglomerative training [29] builds on the idea that diverse foundation models can provide complementary representations from large-scale image data, and their knowledge can be distilled into one unified student model. Given an input image \(x\), the student’s shared backbone generates a summary token \(\mathbf{z}^s \in \mathbb{R}^d\) and a set of patch tokens \(\mathbf{z}_{(i)}^p \in \mathbb{R}^d\) (for \(i = 1, \dots, N\)). For each teacher model \(t\) (with its own embedding dimension \(d_t\)), we attach lightweight adaptor heads—typically small MLPs—to project the student’s features into the teacher’s space:

A summary adaptor \(g_{(t)}^s: \mathbb{R}^d \to \mathbb{R}^{d_t}\) applied to \(\mathbf{z}^s\), yielding \(\hat{\mathbf{z}}_{(t)}^s = g_{(t)}^s(\mathbf{z}^s)\). A patch adaptor \(g_{(t)}^p: \mathbb{R}^d \to \mathbb{R}^{d_t}\) applied to each \(\mathbf{z}_{(i)}^p\), yielding \(\hat{\mathbf{z}}_{(t,i)}^p = g_{(t)}^p(\mathbf{z}_{(i)}^p)\).

Knowledge distillation then aligns these projected student features with the corresponding summary \(\mathbf{z}_{(t)}^s\) and patch features \(\mathbf{z}_{(t,i)}^p\) from teacher \(t\). The per-teacher loss is \[\mathcal{L}_t = \ell_s \left( \hat{\mathbf{z}}_{(t)}^s, \mathbf{z}_{(t)}^s \right) + \frac{1}{N} \sum_{i=1}^N \ell_p \left( \hat{\mathbf{z}}_{(t,i)}^p, \mathbf{z}_{(t,i)}^p \right),\] where \(\ell_s\) and \(\ell_p\) are similarity metrics (e.g., MSE or cosine-based). The total training objective combines losses across all teachers: \(\mathcal{L} = \sum_t \lambda_t \mathcal{L}_t,\) with weights \(\lambda_t\) controlling each teacher’s influence. This process enables the student to integrate diverse visual knowledge without modifying its core backbone. We adopt the training recipe of C-RADIOv4 [26] to produce RADIO1D.

3.2 Student Backbone for 1D Sequence Modeling↩︎

Most teacher models process images by dividing them into non-overlapping 2D patches (e.g., 16×16 pixels for patch size 16) and treating the resulting grid as a flattened 1D sequence of patch tokens. This implicit 2D spatial structure poses a challenge when distilling into a student designed under a strict 1D paradigm, where tokens lack predefined spatial arrangement. To bridge this gap while preserving the 1D modeling objective, RADIO1D employs an encoder-decoder architecture within the student backbone. The encoder operates purely in 1D mode: it takes the input image (patched in a standard 2D grid and flattened) and produces a variable-length sequence of 1D tokens with no enforced spatial meaning. These 1D tokens form the core representation that carries all image information forward. The decoder, active only during RADIO training, reconstructs a 2D-compatible grid of patch tokens suitable for alignment with teachers. It is a lightweight Vision Transformer [30] that receives as input:

• A sequence of duplicated learnable query tokens, whose length matches the number of spatial patches expected by the teachers (e.g., \(\frac{H}{P} \times \frac{W}{P}\) for input resolution \(H \times W\) and patch size \(P\)).

• The 1D tokens from the encoder concatenated as additional register tokens.

Cross-attention within the decoder allows the query tokens to attend to the 1D register tokens, effectively rearranging unstructured 1D information into structured 2D patch features. No direct image or raw patch information bypasses the encoder; the 1D tokens are the sole conduit, ensuring the core model remains 1D-centric.

3.3 Elastic 1D Sequence Generation↩︎

Inspired by FlexTok [31], the 1D tokens are obtained by stochastically slicing a variable number of tokens from the encoder’s output sequence during training. Specifically, we sample a length \(\ell\) from some prior distribution and retain only the first \(\ell\) encoder tokens, discarding the rest. This yields an elastic 1D sequence whose length varies across iterations. This mechanism resembles a form of nested dropout: earlier tokens (always preserved) are encouraged to capture global, high-level semantics, while later tokens (subject to higher dropout probability) specialize in finer details. The resulting variability promotes robust hierarchical encoding and prevents overfitting to fixed sequence lengths. Minimizing the objective ensures that \(I(T_i:Y) > I(T_j:Y)\quad \forall i<j\). We provide empirical evidence in 4.2, 4.3.

3.4 Hierarchical Sequence Downscaling↩︎

In many VLMs, vision encoder patch tokens are downscaled via a pixel unshuffle operation, which groups \(2 \times 2\) adjacent spatial tokens and concatenates them along the channel dimension, reducing the sequence length by a factor of four before integration with the LLM. This operation is also central to hierarchical vision transformers such as Swin [32], where it is referred to as Patch Merging. There, pixel unshuffling expands the embedding dimension from \(d\) to \(4d\), followed by a linear projection to \(\rho d\) channels, with \(\rho\) controlling the capacity increase.

Reducing the sequence length from \(N\) to \(N/4\) yields substantial efficiency gains, particularly at high resolutions (e.g., \(1024 \times 1024\) inputs with \(N = 4096\) for \(16 \times 16\) patches). Transformer complexity consists of a self-attention term \(\mathcal{O}(N^2 d)\) and an FFN term \(\mathcal{O}(N d^2)\). In regimes where \(N \gg d\), the quadratic self-attention term dominates, and downscaling reduces its cost from \(\mathcal{O}(N^2 d)\) to \(\mathcal{O}(N^2 d / 16)\). The FFN cost is reduced from \(\mathcal{O}(N d^2)\) to \(\mathcal{O}(N d^2 / 4)\).

In standard Patch Merging configurations (e.g., Swin), the embedding dimension is doubled after unshuffling (\(\rho = 2\)). Under this setting, self-attention benefits from a net \(\sim\)​8\(\times\) reduction in compute: the \(4\times\) reduction in \(N\) yields a \(16\times\) speedup, partially offset by a \(2\times\) slowdown from increased \(d\). In contrast, FFN compute remains approximately unchanged, as the \(4\times\) reduction in \(N\) is exactly offset by the \(4\times\) increase in \(d^2\). Overall, Patch Merging substantially reduces per-layer compute, driven primarily by self-attention savings, while enabling higher-capacity representations in later stages.

To internalize this efficiency gain, the RADIO1D encoder integrates Patch Merging blocks directly into its backbone, improving both training and inference efficiency while allowing controlled increases in representational capacity. The decoder, used only during training, applies the corresponding reverse operation (Patch Splitting) to upsample the sequence and restore spatial structure for teacher alignment. We detail the initialization of post-downscaling layers from pre-trained models in Section 10.

4 Experiments↩︎

We fine-tuned the C-RADIOv4-H model using the proposed RADIO1D method, distilling knowledge from teachers SigLIP2-g, DINOv3-7B, and SAM3 with a royalty-free subset of the DataComp1B [33] dataset. Training spanned 300k steps without completing a full epoch, employing a global batch size of 512 low-resolution images (stochastically sampled from resolutions of 128, 192, 224, 256, 384, or 432 pixels) plus 64 high-resolution images (from 512, 768, 1024, or 1152 pixels), resulting in approximately 172M total training samples. This replicates C-RADIOv4’s final stage of training, but with half the batch size, to save compute resources.

4.1 Ablation Studies↩︎

We conduct ablation studies to identify optimal design choices for RADIO1D, focusing on (i) the placement of the downscaling operation, (ii) the embedding expansion factor \(\rho\), and (iii) the sampling distribution for the elastic sequence length \(\ell\) during training. All ablations are evaluated using ADE20K mIoU as a proxy for semantic segmentation performance. Experiments use a shortened training schedule of 100k iterations (57M samples) and a lightweight SO400M backbone [34].

We first perform a static analysis of the design space by comparing parameter count and image throughput across configurations that vary the downscaling position \(\kappa\) and expansion factor \(\rho\). The downscaling block consists of a \(2\times2\) pixel unshuffle (expanding patch embeddings to \(4d\)), layer normalization (\(2 \times 4d\) parameters), a linear projection to \(\rho d\) channels (\((4d)\times(\rho d)\) parameters), and a projection for the CLS and register tokens (\(d\times(\rho d)\) parameters). As shown in Table 5, earlier downscaling improves throughput, while large expansion factors (\(\rho > 2\)) incur prohibitive parameter and compute costs. We therefore fix \(\rho = 2\) for all subsequent experiments.

We next ablate two dynamic hyperparameters: the downscaling position \(\kappa\) within the encoder backbone and the training-time sampling distribution for the elastic sequence length \(\ell\). For evaluation, we freeze the trained encoders and apply a linear probe to the reconstructed 2D feature grid produced by the decoder, reporting ADE20K mIoU. Although the decoder is used only for teacher alignment during training, this protocol allows us to assess preservation of spatial-semantic information in a manner consistent with standard encoder evaluations. We vary \(\ell\) from 1 to 256 in powers of two.

We first compare sampling distributions for \(\ell\) (uniform vs.triangular with Probability Density Function (PDF) \(p(x)=2-2x\), see figure 12), fixing \(\kappa=24\). As shown in Figure [fig:ablation95group], the triangular distribution more effectively concentrates information into fewer tokens while maintaining strong accuracy at higher token counts. Using this distribution, we then ablate the downscaling position and find \(\kappa=24\) yields the best overall mIoU.

Finally, we evaluate two alternative design choices commonly used in VLMs: (i) removing downscaling entirely, and (ii) applying pixel unshuffling only at the model output, on the 1D token sequence. Both perform substantially worse than configurations with learnable Patch Merging. We attribute this degradation to the reliance of pixel unshuffling on strong spatial locality between neighboring tokens—an assumption that holds for 2D patch grids but breaks down in our 1D tokenization regime, where adjacent tokens exhibit weak or no spatial coherence. Refer to appendix 9 for more details.

4.2 Summarization Capabilities↩︎

The RADIO1D model demonstrates strong summarization capabilities by learning to encode a robust global token at index 0, as evidenced by its high per-token k-KNN classification accuracy on ImageNet (figure 11). Unlike figure 4, where the x-axis represents distance to the center, this figure plots accuracy against token index, highlighting that global image information is concentrated in the initial token.

We also show in appendix 17 that RADIO1D exhibits strong summarization of semantic content even with a single token in its bottleneck layer, achieving a high mIoU score of 40.23 on semantic segmentation linear probing using the fully trained version. Notably, the predictions closely align with the ground truth in terms of class identification and spatial positioning, and even finds some elements (cushion, painting) that are missing from the ground truth. We highlight the fact that the mIoU achieved with the first token is over four times higher than the mIoU of 8.05 obtained using only the last token. This disparity persists across scales: the first 16 tokens yield 54.11 mIoU compared to 20.84 for the last 16. These results underscore the hierarchical ordering inherent in RADIO1D’s tokenization, where early tokens capture globally informative features essential for broad semantic understanding, while later tokens encode finer-grained details, enabling efficient performance scaling with minimal computational overhead.

4.3 Ties To Information Theory↩︎

To empirically validate the information-theoretic motivation underlying RADIO1D, we adopt the classical rate-distortion (R-D) framework from information theory [35]. In this framework, the rate \(R\) corresponds to the complexity of the representation (approximated here by the number of tokens \(L\)), while the distortion \(D\) measures the loss of task-relevant information (quantified as the relative performance degradation \((mIoU_{\max} - mIoU)/mIoU_{\max}\), where \(mIoU_{\max}\) is each model’s full-resolution performance). This formulation directly aligns with the Information Bottleneck principle [27], which seeks to maximize mutual information \(I(Z; Y)\) between the representation \(Z\) and the target variable \(Y\) (e.g., semantic labels) while minimizing the entropy \(H(Z)\). By plotting R-D curves on ADE20K semantic segmentation using a linear probe decoder, we demonstrate that RADIO1D achieves superior trade-offs–particularly in the low-rate regime–compared to fixed-grid baselines such as SigLIP2, DINOv3, and C-RADIOv4. These baselines are evaluated by subsampling their patch tokens via spatial pooling, allowing a fair comparison of compression efficiency under equivalent representational budgets. As can be seen on figure 8, RADIO1D exhibits consistently lower distortion at all rates. See Appendix 11 for more details.

4.4 Vision-Language Modeling↩︎

To assess the utility of RADIO1D in VLMs, we conducted a controlled study within the multimodal Nemotron VL framework (details in appendix 14). We isolated the impact of the vision encoder by varying only this component while holding all other hyperparameters, architectural elements, and training configurations constant. Vision encoders were paired with the Nemotron-Nano-9B-v2 [36] LLM as the text decoder. Training was performed through the first Supervised Fine-Tuning (SFT) stage on a dataset of 17M multimodal samples drawn from the full Nemotron VL dataset, which comprises diverse image–text pairs curated for multimodal alignment.

We compare RADIO1D at varying token counts against SigLIP2-SO400m, SigLIP2-g, and C-RADIOv4-H baselines. To ensure a fair comparison at an equivalent token budget, the 2D baselines start with 1024 spatial tokens and apply a standard \(2\times2\) pixel unshuffle to reduce the sequence to exactly 256 tokens before integration with the LLM.

Results are reported in Table 1, demonstrating that RADIO1D outperforms these baselines even with fewer tokens per frame and higher throughput, while offering a continuous spectrum of accuracy–latency trade-offs and greater flexibility in deployment.

Furthermore, evaluating RADIO1D across this elastic spectrum reveals distinct failure modes at extreme compression. Tasks requiring holistic scene understanding such as video reasoning (LongVideoBench) and complex multimodal reasoning (MMMU) are remarkably robust to heavy compression. For instance, MMMU performance drops by less than 10% from its peak score even when the visual representation is reduced to a single token. Conversely, the primary failure mode emerges in dense, reading-heavy tasks. Performance on OCR-centric benchmarks (e.g., DocVQA, InfoVQA, OCRBench) drops sharply as the token budget decreases below 128 tokens, indicating that fine-grained text resolution strictly requires the higher capacity preserved by the full 256-token sequence.

4.5 Comparison with Inference-Time Token Pruning↩︎

To validate the effectiveness of our learned 1D elastic sequence, we compare RADIO1D against state-of-the-art inference-time token reduction methods. We evaluate two orthogonal approaches applied post-hoc to our dense 2D baseline (C-RADIOv4): Token Merging (ToMe) [37], which combines similar tokens, and CDPruner [38], an MLLM-specific pruning method that utilizes a greedy Determinantal Point Process (DPP) to select a diverse subset of tokens. For CDPruner, we extract vision tokens from the output of the vision projector and text query embeddings from the language model, applying the selection independently on each image tile.

As shown in Table 2, RADIO1D consistently outperforms both ToMe and CDPruner at equivalent token budgets. Notably, at 128 tokens, RADIO1D (71.64% average accuracy) surpasses all CDPruner configurations and even outperforms C-RADIOv4+ToMe operating at 192 tokens (71.49%). We additionally apply CDPruner on top of the full 256-token RADIO1D sequence, but find that natively generating the desired sequence length via our elastic bottleneck is superior.

Crucially, post-hoc pruning and merging methods struggle significantly with dense reading and OCR tasks (e.g., DocVQA, InfoVQA, OCRBench). In failure case analysis for DocVQA, we observed that merging distant tokens degrades spatial awareness; the VLM frequently identifies the correct text element but places it in the wrong layout context (e.g., returning an incorrect cell from a table). These results suggest that learning to compress visual information dynamically during vision backbone training preserves essential spatial grounding far better than training-free token selection or merging.

4.6 Composition-Based Retrieval↩︎

Noticing that RADIO1D is capable of a 40.23 mIoU on ADE20k with just a single token, it stands to reason that not only are typical semantics of an image encoded, such as “dog”, or “chair”, but further, the entire scene composition must be roughly encoded in this token. In a typical image retrieval setting, we would look for the embeddings in the database nearest to our query embedding. This is exactly how k-NN is computed. However, we could also test how closely the query and key image scenes are aligned, and thus we present a “composition-based retrieval” metric that aims to identify whether similar objects are present in query and key image, and also whether they’re similar in size and location within the image.

Given a query image \(q\), we compute an embedding \(f(q) \in \mathbb{R}^d\) (e.g. a pooled token, CLS token, or summary token) and retrieve the top-\(K\) nearest database images \(\{r_1,...,r_K\}\) using cosine similarity between query and database embedding. Using an object detection dataset, each image is associated with a set of object bounding boxes and categories:

\[\begin{align} \mathcal{B}(q) = \left\{(c_i, b_i) \right\}_{i=1}^{n_q}, \quad \mathcal{B}(r) = \left\{(c'_j, b'_j) \right\}_{j=1}^{n_r} \end{align}\]

To quantify composition similarity between \(q\) and \(r\), we compute the pairwise score matrix \(S \in \mathbb{R}^{n_q \times n_r}\) with \(R_{ij} = \mathbb{1}[c_i = c'_j]\), \(G_{ij} = \frac{\mathrm{gIoU}(b_i, b'_j) + 1}{2}\), and \(S_{ij} = R_{ij} \odot G_{ij}\), where \(\mathbb{1}[\cdot]\) is the indicator function, \(\mathrm{gIoU}(x, y) \in [-1, 1]\) the generalized IoU [39], and \(\odot\) the hadamard product. The matrix \(\mathbf{R}\) can be interpreted as the recall matrix, which we also track separately. \(\mathrm{gIoU}\) is a useful choice because it allows us to encode the difference in object position and size in a single metric. It is preferable compared to \(IoU\) because non-overlapping boxes get a score in the range \([-1, 0]\), with increasing values indicating increased alignment. Entries in the score matrix \(\mathbf{S}\) assign no credit when categories mismatch between objects, and partial credit for object match but spatial mismatch, varying smoothly based on the degree of misalignment. We then compute the optimal bipartite assignment using the Hungarian algorithm [40]: \(\mathcal{M}^*= \mathrm{arg\;max}_\mathcal{M} \sum_{(i,j) \in \mathcal{M}} S_{ij}\), yielding a composition score for the pair \((q,r)\) by normalizing the matched sum:

\[\begin{align} \mathrm{Comp}_{\mathrm{raw}}(q, r) &= \sum_{(i,j) \in \mathcal{M}^*} S_{ij} \label{eq:comp95raw} \\ r^*_{\mathrm{C}}(q) &= \arg\max_{r \in \mathcal{D}} \mathrm{Comp}_{\mathrm{raw}}(q,r) \\ \mathrm{Comp}(q,r) &= \frac{\mathrm{Comp}_{\mathrm{raw}}(q,r)}{\mathrm{Comp}_{\mathrm{raw}}(q,r^*_{\mathrm{C}}(q))} \end{align}\tag{1}\]

With \(r^*_{\mathrm{C}}(q)\) operating as the oracle result in the database, which maximize the raw score. Then, for each query \(q\), we compute the composition score (resp. recall) for the top-\(K\) retrieved neighbors and summarize via a max-over-\(K\) operator: \(\mathrm{Comp}@K(q)=\max_{1 \le t \le K} \mathrm{Comp}(q,r_t)\). Essentially, within the top-\(K\) retrieved results, we find the image that maximizes the composition score. Finally, the full benchmark score:

\[\mathrm{Comp}@K = \frac{1}{|Q|} \sum_{q \in Q} \mathrm{Comp}@K(q)\]

We also calculate \(\mathrm{Recall}@K\) with the same formulation, but replace the score matrix \(\mathbf{S}\) in 1 with \(\mathbf{R}\). We then calculate the composition score for the vision foundation models SigLIP2, DINOv3, C-RADIOv4-H, and finally our proposed RADIO1D. We use MS-COCO and nuImages [41] as evaluation datasets. We show these results in table 3 for \(K=1\), where it can be seen that RADIO1D improves over all other foundation models. We provide more results for each model, as well as recall scores, in section 12 of the appendix.

5 Related Work↩︎

Our analysis of specialized tokens builds upon recent observations of token behaviors in Vision Transformers. [42] identified high-norm "artifact" tokens that degrade 2D spatial representations and proposed appending "register" tokens to absorb this noise and clean the spatial grid. While we similarly observe outlier tokens, our findings indicate that in image-text aligned models like SigLIP2, these form as low-norm tokens that act as highly effective global summarizers. Consequently, rather than using registers to preserve a 2D grid for dense prediction tasks, RADIO1D leverages this summarization capability to abandon the 2D grid entirely, encoding global and hierarchical information into a compact, elastic 1D sequence.

Recent work on vision foundation models has explored alternatives to fixed 2D patch grids to improve efficiency and flexibility in downstream tasks. Methods such as TiTok [43] and Flextok [31] convert images into compact, variable-length 1D token sequences, enabling more adaptive representations. Like RADIO1D, Flextok employs elastic 1D tokenization and mechanisms such as nested dropout to induce hierarchical ordering, where early tokens capture coarse semantics and later tokens encode finer details. This emphasis on variable-length sequences supports scalable compression while preserving task-relevant information.

RADIO1D differs in architecture and training objectives. TiTok and Flextok introduce learnable latent or register tokens concatenated with image patches at the transformer input, whereas RADIO1D processes flattened patches through a pure 1D encoder without input augmentation. Moreover, TiTok relies on vector quantization and Flextok on finite scalar quantization, while RADIO1D operates entirely in continuous embedding space to avoid information loss from discretization. Most importantly, TiTok and Flextok are optimized for image reconstruction and generative fidelity (e.g., class-conditional or text-to-image synthesis), whereas RADIO1D uses multi-teacher distillation from diverse foundation models to agglomerate semantically rich features, prioritizing multimodal alignment and vision-centric understanding.

This design enables direct integration of RADIO1D into VLMs, improving efficiency and performance on tasks such as visual question answering and captioning through abstract, spatially decoupled embeddings. In contrast, TiTok and FlexTok do not evaluate their tokenizers within full VLM pipelines for image understanding. RADIO1D therefore bridges dense vision specialists and language-aligned encoders, offering a complementary paradigm for efficient VLM backbones.

6 Conclusion↩︎

This work challenges the reliance on fixed 2D patch features in vision-language models. Analysis reveals that fine-tuning produces more abstract representations, while image-text aligned encoders like SigLIP2 learn specialized tokens for effective global summarization. We propose RADIO1D, which compresses images into a variable-length 1D token sequence via multi-teacher distillation and autoencoder design. The representations enable strong hierarchical summarization, supporting accurate scene understanding with as few as one token. We introduce a composition-aware image-to-image retrieval benchmark assessing object presence, size, and position, where RADIO1D outperforms strong baselines. In VLMs, it provides flexible accuracy-efficiency tradeoffs via adjustable token counts, competitive multimodal performance, reduced overhead, and improved alignment. Models are released under a permissive license. Future work includes content-aware selection of 1D representation length to adapt token count dynamically to image complexity or task needs, along with extensions to video and further compression techniques. In summary, RADIO1D demonstrates that elastic, summarization-focused representations are the primary alignment that VLMs desire, as opposed to detailed 2D local representations.

Impact Statement↩︎

This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.

7 RADIO1D Architecture Details↩︎

In table 4, we show a detailed view of the RADIO1D architecture, with the input shape, parameter count and Floating-Point Operations (FLOPs) associated with each block, assuming an input image size of 512px. RADIO1D blocks match those of the reference RADIO-H model, up to the downsampling block, after which the sequence length is divided by 4, and the embedding dimension multiplied by \(\rho\). The net effect is a speed-up and reduction in FLOPs, in favor of the 1D model.

8 Static Analysis of the RADIO1D Design Space↩︎

In table 5, we show the parameters of the RADIO1D design space affect throughput and parameter counts. We vary the downscaling position \(\kappa\) and expansion factor \(\rho\). The throughput is measured using vLLM on an H100 GPU with a batch size of 32 and input image size of 1024px. As described in section 3.4, self-attention scales according to \(\mathcal{O}(N^2 d)\), and FFN layers as \(\mathcal{O}(N d^2)\). At the downscaling position \(\kappa\) the number of tokens changes as \(N \rightarrow N/4\), and the feature dimension as \(d \rightarrow \rho d\). When \(\rho=2\), this reduces the self-attention cost by \(\sim8\times\) from \(\mathcal{O}(N^2 d) \rightarrow \mathcal{O}(2N^2 d/16)\), with no change in the FFN cost from \(\mathcal{O}(N d^2) \rightarrow \mathcal{O}(4Nd^2/4)\), which combined leads to fewer flops and higher throughput as \(\kappa\) decreases. For \(\rho=4\), the self-attention cost is reduced by \(\sim4\times\) from \(\mathcal{O}(N^2 d) \rightarrow \mathcal{O}(4N^2 d/16)\), but the FFN cost increases by \(\sim4\times\) from \(\mathcal{O}(N d^2) \rightarrow \mathcal{O}(16Nd^2/4)\), which empirically leads to a reduction in throughput as \(\kappa\) decreases. Thus from both a parameter and throughput perspective, using \(\rho=2\) is preferred over \(\rho=4\).

9 Additional Ablations on Downscaling Strategies↩︎

To further investigate the importance of learnable Patch Merging integrated early in the RADIO1D encoder backbone, we conducted additional experiments on alternative downscaling strategies commonly employed in VLMs. Before incorporating downscaling directly into the model, we evaluated applying pixel unshuffling operations externally on top of the initial RADIO1D token sequence (which lacked integrated downscaling at the time).

We first attempted the standard 2×2 pixel unshuffling operation (grouping spatially adjacent tokens and concatenating along the channel dimension to reduce sequence length by a factor of 4). The model failed to converge, with training loss remaining persistently high and the model exhibiting severe underfitting. We then explored a flattened one-dimensional variant of unshuffling, for example pivoting every group of 4 consecutive tokens into the channel dimension (4× unshuffling). This approach also failed to produce stable training. Even randomizing the order of the 1D tokens prior to applying the one-dimensional unshuffling marginally improved the training loss but yielded no meaningful convergence gains. These failures highlight that pixel unshuffling critically depends on strong spatial locality among neighboring tokens—a property inherent to 2D patch grids but absent (or very weak) in the 1D tokenization regime of RADIO1D, where adjacent tokens lack coherent spatial relationships.

We additionally compared the initial RADIO1D model (without integrated downscaling) against the final design (with integrated learnable Patch Merging) while fixing the output to 256 1D tokens in both cases. Both variants were evaluated after a short VLM training run on a compact evaluation suite. The model with integrated downscaling achieved superior accuracy across metrics while also offering faster inference (due to reduced sequence lengths in downstream transformer layers, as analyzed in Section 3.4).

These results confirm that learnable Patch Merging integrated into the encoder provides both performance gains and efficiency benefits compared to post-hoc or absent downscaling, consistent with the ablation findings in Section 4.1.

10 Initialization from Pre-trained Models↩︎

We initialize RADIO1D from a pre-trained standard 2D Vision Transformer checkpoint (e.g., C-RADIOv4) using a dedicated conversion procedure. For layers with matching shapes, weights are copied directly. For layers after downscaling (where embedding dimensions or sequence lengths differ), we follow the Net2WiderNet [44] approach: we expand source weights by repetition along the expanded dimension, followed by a small amount of random noise to break symmetry. Special handling is applied for stability: output projection weights are divided by the input expansion factor to preserve output magnitude, while LayerNorm parameters are scaled by \(\frac{1}{\sqrt{\rho}}\) to maintain variance.

11 Rate-Distortion Analysis↩︎

11.1 Rate-Distortion Framework for Lossy Compression Analysis↩︎

We evaluate RADIO1D’s information-theoretic motivation using the classical rate-distortion (R-D) framework [35], [45]. Here, the rate \(R\) approximates the complexity or entropy \(H(Z)\) of the learned representation \(Z\) (proxied by token count \(L\)). The distortion \(D\) quantifies loss of task-relevant information, defined as the normalized performance degradation \(D = \frac{mIoU_{\max} - mIoU}{mIoU_{\max}},\) where \(mIoU_{\max}\) is the model’s performance at full token count (\(L=256\) or equivalent). This formulation directly aligns with the Information Bottleneck principle [27], which seeks to maximize mutual information \(I(Z; Y)\) between representation \(Z\) and target variable \(Y\) (semantic labels) while minimizing \(H(Z)\). RADIO1D’s nested dropout and hierarchical design are intended to produce a superior R-D frontier by concentrating task-relevant information in early tokens and discarding spatial redundancy.

11.2 Experimental Setup↩︎

We use the ADE20K dataset and a frozen linear-probe decoder. For RADIO1D, we vary the slice length \(L \in \{1, 4, 16, 64, 256\}\). For fixed-grid baselines (DINOv3-H+, SigLIP2-SO400m, C-RADIOv4-H), we reduce the original 32×32 (1024-token) grid to the target effective token count via spatial average pooling (preserving 2D locality). We show these results in table 7. RADIO1D consistently exhibits the lowest relative distortion across all rates, with a particularly pronounced advantage in the low-rate regime (e.g., at \(L=1\), distortion 27.9% vs. 72-76.7% for baselines). This supports the efficacy of hierarchical encoding: early tokens capture most semantic information, enabling high accuracy with minimal representational budget.

12 Composition-Based Retrieval (Extended)↩︎

For the models that support it, we tried both square center crops, as well as aspect preserving resizing modes, and found, also intuitively, that aspect preserving processing improves the results for all models except for DINOv3-7B on MS-COCO. We can also see that the composition score isn’t strictly dominated by improved recall scores, as SigLIP2-SO400M-NaFlex often achieves the best recall scores, while having lower composition scores than RADIO1D. Also, owing to the fact that RADIO1D produces an ordered set of tokens, we study whether using the first two tokens, by concatenating them in the channel dimension, works better than using solely the first, the “512min_T2” setting. We observe that there is a slightly positive effect for MS-COCO, but negative effect on nuImages, suggesting that this technique is unlikely to be beneficial. We show these results in table 8. We also show qualitative results in figure 9 for MS-COCO, and figure 10 for nuImages. nuImages contains a large degree more examples of roughly the same scene, with variations in objects present.

13 Per-Token k-NN ImageNet-1k Top-1 Accuracy↩︎

In the above figure, we show the summarization capabilities of the first RADIO1D tokens: the first two tokens achieve a k-NN Top-1 ImageNet-1k classification accuracy of 85.0% and 84.5% respectively. We note that the following tokens still exhibit mild spatial correlations: tokens that map to near-center locations of the image show slightly better accuracy, and the oscillations are still visible with a period of 16 tokens, corresponding to the width of the 512px input image.

14 Nemotron VL Framework↩︎

We adopted key elements of the training protocol from Nemotron VL 2 [13], leveraging the Megatron ¹ framework for distributed training, Transformer Engine ² for optimized mixed-precision computations, and the Megatron Energon ³ dataloader for efficient data loading and preprocessing.

Similar to InternVL [46], we employ image tiling to support variable input resolutions while approximately preserving aspect ratio. We use a tile size of 512px with a maximum of 12 tiles, in addition to a “thumbnail” tile obtained by resizing the original image to a square of 512px. For SigLIP2-G-384, originally trained at 384px resolution, we apply positional encoding interpolation to 512px.

For 2D vision encoders, we apply a \(2\times2\) pixel unshuffle to reduce the number of vision tokens by a factor of 4. Since RADIO1D already integrates a similar downscaling operation internally, we skip the additional pixel unshuffle for this vision encoder.

Training proceeds in two stages. In the first pre-training stage (Nemotron VL Stage 0), we train only the vision-to-language projector while keeping the vision encoder and language model frozen. In the second stage, we unfreeze all components and perform supervised fine-tuning (SFT) on a dataset of 17M samples (image–text subset of the full 25M Nemotron VL dataset, which additionally includes text-only and video data).

We train with a maximum sequence length of 16,384 tokens and apply data packing during SFT to improve efficiency, achieving \(\sim\)​5 samples per batch per data-parallel rank. For vision encoders that produce fewer output tokens (e.g., RADIO1D with 32 tokens instead of the full 256), we reduce the packing size to maintain \(\sim\)​5 samples per batch. This enables a fair comparison across vision encoders with varying token counts without modifying hyperparameters such as the learning rate or training schedule.

We omit Nemotron VL’s Stage 2 (49k context extension), Stage 3 (49k text recovery), and Stage 4 (300k context extension).

15 Distribution Visualizations↩︎

In figure 12 we plot the PDFs of the uniform and triangular distribution (\(2-2x\)) studied in the ablations. We also plot the survival function \(f(y)=P(X\ge y)\), which is: \[\begin{align} f_{uni}(y) &= 1 - y \\ f_{tri}(y) &= 1 - 2y + y^2 \end{align}\]

for the uniform and our triangular distribution respectively. The triangular distribution allocates much more probability density toward lower token counts, clearly having an effect on semantic segmentation at lower token counts, as seen in figure [fig:ablation95group]. Interestingly, the likelihood of high token counts is much lower for the triangular distribution, likely explaining why mIoU stops improving after 32 tokens, however, it still is able to match the quality of the uniform distribution model at these higher counts.

16 SigLIP2 Feature Visualizations↩︎

In figure 13 we present visualizations of the SigLIP2-SO400m model’s token embeddings for four diverse input images, each resized to 512\(\times\)​512 pixels, resulting in a 32\(\times\)​32 grid of tokens. The middle column displays heatmaps of the token norms, where darker regions indicate lower-norm values. Notably, across all images (regardless of content) three consistent groups of tokens exhibit markedly lower norms: one positioned near the top, slightly inset from the right border; another along the left border, approximately two-thirds down vertically; and a third at the bottom, roughly one-quarter of the way from the left horizontally. These same token groups also stand out prominently in the right column’s PCA projections of the embeddings into RGB space, appearing as distinct outliers in color and pattern compared to the surrounding tokens. As discussed in the main text, these invariant positions correspond to the "global" tokens we identified, which appear to capture global image information rather than localized visual features.

17 Semantic Segmentation Visualizations↩︎

The visualizations in figure 14 demonstrate the effectiveness of our semantic segmentation model on ADE20K validation scenes, comparing ground truth annotations (left) against predictions using a single token per 512\(\times\)​512 crop in the bottleneck (middle) and 256 tokens per crop (right). Remarkably, even with just one token, the model reconstitutes a substantial portion of the scene’s semantic content, accurately identifying major elements such as cars, buildings, trees, and furniture, though with vague, blob-like outlines that approximate shapes without fine boundaries. For instance, in the urban street scene, parked cars appear as broad blue masses merging into the road, while in the indoor living room, armchairs and tables form rough cyan and pink clusters. This resembles a semantic segmentation of low-resolution version of the input image, despite the model never being trained to explicitly capture such information. Scaling to 256 tokens significantly enhances fidelity, yielding highly detailed shapes that closely mirror the ground truth; standout improvements include precise delineations of slender objects like streetlights and flags in outdoor views, intricate furniture contours such as individual cushions on armchairs or curved lamp shades in the living room, and subtle bedroom elements like pillows, fans, and windowpanes, enabling near-photorealistic semantic reconstructions that capture nuanced spatial relationships and object intricacies.

18 CKA Regularization↩︎

Following from the analysis in section 2.1, with DINOv3 having a much higher off-diagonal CKA versus SigLIP2 (figure 5), along with each patch generally operating well as a classifier (figure 4), it implies that the representations of DINOv3 are highly redundant. For dense spatial tasks, especially with a frozen backbone, this is a feature, not a bug, because it means that each token is encoding information that is local and spatially coherent. Given the bottleneck structure of RADIO1D, it gives us the flexibility to encourage the encoder representations to behave a certain way directly. For this, we formulate the off-diagonal mean CKA as a regularization term directly:

\[\begin{align} \text{CKA}_\ell(\mathbf{X},\mathbf{Y}) &= \frac{\left\|\mathbf{X}^\intercal \mathbf{Y} \right\|_F^2}{\left\|\mathbf{X}^\intercal \mathbf{X} \right\|_F^2 \left\|\mathbf{Y}^\intercal \mathbf{Y} \right\|_F^2} \tag{2} \\ \text{CKA}_\text{reg}(\boldsymbol{\mathcal{X}}) &= \frac{\gamma}{\ell(\ell-1)} \sum_{\substack{i,j \in \ell \\ i \neq j}} \text{CKA}_\ell(\boldsymbol{\mathcal{X}}_i, \boldsymbol{\mathcal{X}}_j) \tag{3} \end{align}\]

with \(\text{CKA}_\ell\) being the linear CKA kernel, and \(\text{CKA}_\text{reg}\) being the regularization formula, with \(\gamma\) being the loss weight. Given that \(\mathbf{X},\mathbf{Y} \in \mathbb{R}^{B \times D}\) are matrices in 2 , then \(\boldsymbol{\mathcal{X}} \in \mathbb{R}^{B \times \ell \times D}\) is the output of the encoder, with batch size \(B\), tokens \(\ell\), and dimension \(D\). So, we construct the \(\ell \times \ell\) cartesian product of CKAs between position \(i\) and \(j\) s.t. \(i \neq j\), and then average. Given that \(B=576\), \(\ell \in \{1, 2, ..., 1296\}\), and \(D=1152,1280\) for SO400M and H respectively, this becomes a very computationally heavy operation: \(O(B^2 \ell^2 D^2)\). To make this something can be tractably computed online during training, we stochastically reduce the size of \(\boldsymbol{\mathcal{X}}\) in the following way:

• We reduce \(D \rightarrow \hat{D}\) with a random \(D \times \hat{D}\) projection matrix, relying on the JL lemma for approximate distance preservation.

• We reduce \(B \rightarrow \hat{B}\) by randomly sampling a subset from \(B\).

• We reduce \(\ell \rightarrow \hat{\ell}\) by randomly sampling a subset from \(\ell\).

We provide the pseudocode in section 18.1. In figure 15 we show the CKA visuals for the baseline 2D model, as well as without, and with, the CKA regularization. It is immediately clear that the regularizer works as intended, as the off-diagonal relationships are nearly 0. It’s also apparent that without CKA, the model retains a lot of 2D structure, albeit with much less energy. It appears to be the case that a lot of the correlations in 2D are between consecutive rows in the prefix, and much lower correlation between consecutive tokens horizontally.

To see if this structure is important, with a new training run we continually adjusted whether the CKA was applied at various points in training, starting off with \(\gamma = 0.05\) to “break” the 2D structure, and then decay to \(\gamma = 0\), allowing unconstrained correlations to emerge. In figure 18 we show the CKA plots for this model. Notably, these 2D structures are actually quite sticky, as [18 (b)] and [18 (e)] are nearly the same. Because we were able to reduce it to nearly 0 before epoch 30, relaxed it by 30, constrained again by epoch 50, and fully relaxed again by epoch 140, the learning process appears to specifically prefer this structure.

In figure 16 we plot the ADE20k mIoU as a function of the number of tokens preserved in the encoder. We can see that while the representations must have very different structure, it has an insignificant effect on the decoder’s ability to reconstruct a semantic image. Notably, the model without CKA achieves an mIoU of 55.66 with just 32 tokens, nearly matching DINOv3-7B at \(<\frac{1}{10}\) the parameters, and surpassing the baseline C-RADIOv4-SO400m 2D model, which achieved 55.14 using 1024 tokens. This appears to lend credence to the conjecture in [43] that an image is worth 32 tokens.

In figure 17 we study two different reconstruction modes, where we measure the MSE between the decoder given some subset of tokens, versus the full number of tokens. In [17 (a)] we use a prefix of length K, where we’d expect the reconstruction error to decrease as the number of tokens increases. We can see that initially the “CKA 0.05” model does marginally better at self-reconstruction, up to token 16, and then does not improve on the error at nearly the same rate as the “CKA 0” model. In [17 (b)] we instead use only the token at index M to perform the reconstruction. Unsurprisingly, by construction, M=0 has the lowest error. For \(M>0\), we are no longer dealing with a prefix, and thus the most informative tokens are missing. Instead, what we see is that the “CKA 0.05” model has a consistently lower reconstruction error for any given token. While the CKA loss term was successful at strongly reducing these correlations, it failed to lead to any measurable improvement on downstream metrics (e.g. figure 16), and curiously leading to potentially increased redundancy, seen in figure 17 (b), and so we don’t include it during regular model training.

18.1 CKA Regularization Pseudocode↩︎

In algorithm 19 we show the pseudocode for computing the off-diagonal CKA regularization. Given that we are doing distributed training, and that different ranks will have a different sampled number of tokens, as well as different batch sizes, we need to deal with gathering data across the ranks. First, we find the minimum sampled \(T\) across all ranks, and truncate the second dimension to match. This allows us to all-gather the tensors across the ranks, concatenating into \(B\). Next, we stochastically subsample from \(B\) and \(T\) to reduce the tensor size. Each rank samples a different subset. Finally, each rank independently projects the dimension \(C\) down to \(d\) using a random projection. Finally, we compute \(\text{CKA}_\text{reg}\) as defined in 3 .

References↩︎