1 Introduction↩︎

From supervised pre-training [1], to vision-centric self-supervised learning [2], [3], to vision-language contrastive learning [4], further to hybrid training paradigms [5], the computer vision community keeps pursuing more transferable visual representations. High-quality image embeddings [6], [7] have fundamentally advanced the development of a wide range of perception and generation tasks [8], [9]. In recent years, witnessing the power of LLMs [10], finding better visual tokens for multimodal LLMs (MLLMs) [11] is receiving growing attention [12], [13]. The quality of these tokens is critical for MLLMs to accurately perceive and reason over visual signals.

There are three mainstream approaches to improving the visual representations in MLLMs. The first is to directly pre-train a better vision encoder from scratch with better algorithms [6], [14], more data [15], [16], or larger models [7]. This line of work requires tremendous resources (e.g., million GPU hours, billion data), which are unaffordable for most researchers. The second approach is leveraging multiple well-trained encoders [12], [13], [17]. Different encoders have their own strengths. CLIPs [4] are good at modeling vision-language correspondence, while vision-only models [14] excel at capturing detailed visual correlations. Incorporating them may amplify their distinct advantages while suppressing their drawbacks. Despite being intuitively promising, the actual gain is limited or even negative [13].

The last approach is simply to forward higher-resolution input [18], [22], [23]. Higher-resolution images can yield more fine-grained visual tokens, making MLLMs see more clearly. This principle is gradually strengthened by state-of-the-art MLLMs [21], [24], [25], from resizing to a fixed larger size [18], to fully preserving the native resolution [19], [20], steadily contributing to better OCR capability. Our work is inspired by this observation, but with fundamentally different roadmaps. We highlight that, although increased image size improves visual perception, it brings significant extra complexity in both compute and design. The image has to be sliced into small tiles [19], [22] to match the pre-trained vision resolution. This not only requires time-consuming multiple forward passes, but also incurs more visual tokens. To alleviate the LLM’s burden, further post-processing, e.g., resampler [26] and pixel unshuffle [27], is necessary for token compression, making the framework even more complicated.

We reflect on the scaling trend. Before increasing the input resolution, have we fully unlocked the model’s perception capability at a standard (medium) resolution (e.g., 512²px)? Can we achieve fine-grained visual perception under lower cost without larger images?

At least our human visual system can. As shown in Figure 1, when downsampling the image from its native resolution (1722px in height) to a medium size (512px), although the characters are blurred, we can still accurately understand the content. Therefore, if an AI visual system is developed capable enough to match human performance, it should be able to perceive at such a medium resolution, without requiring a larger image. It will avoid unnecessary computation at high resolutions, reducing token redundancy and improving system efficiency.

So the remaining problem is, how to make vision models acquire such promising perception capability. Our solution is highly simple yet effective. We design a fine-to-coarse supervision mechanism to transfer high-quality visual representation at a high resolution to coarse visual tokens at a standard resolution. Concretely, given a pre-trained vision encoder [6], we further fine-tune it at a standard resolution by enforcing its output features to mimic the fine-grained features of the corresponding high-resolution images. Through this mechanism, the encoder gradually bootstraps its representations from coarse level to fine level. Our method does not rely on any human or synthetic labels for fine-grained perception [28]–[30], purely exploring cheap raw images for self-improvement. Without using any auxiliary upsamplers [31], [32], our final model inherits the same structure and IO as our pre-trained model, making it very easy and efficient to deploy in complex systems. Users only need to modify the checkpoint path to ours, and then enjoy better perception capability.

Our main contributions are summarized below:

• We reflect on the scaling trend of image resolution in MLLMs. Beyond using high-resolution images for high-quality visual tokens, we study an intriguing problem: how to achieve fine-grained perception for MLLMs under lower cost without larger images.

• We present a highly simple fine-to-coarse supervision mechanism to strengthen perception capability at a standard resolution by imitating “teacher” tokens of high-resolution images.

• We build our method on the competitive SigLIP 2 encoder [6]. With the same inference cost, our SigLIP-HD delivers better results across diverse MLLM benchmarks under various protocols, especially for scenarios that favor fine-grained perception.

Following the philosophy of our fine-to-coarse supervision, in Section 2, we first investigate what good fine-grained representation to learn. Then in Section 3, we present our framework on how to learn from the fine-grained representation.

2 What Is Good Representation to Learn?↩︎

In this work, we build our fine-to-coarse supervision methodology on the SigLIP 2 encoder [6], or more specifically, its So400m/16-512px version. This model of 429M parameters takes a 512²px image as its input, and produces 32² visual tokens under patch size 16. As compared in Table ¿tbl:tab:pilot95encoders?, its So400m/16-512px version is much better than other counterparts in important OCR scenarios [33], [34], due to its larger resolution. So we mainly use this version for validation. Furthermore, to be more convincing, we also provide our results built on the legacy model OpenAI-CLIP [4] (Table ¿tbl:tab:exp95compare95clip?).

Prior to fine-to-coarse supervision, a primary step is to find out what optimal fine-grained features to learn. Therefore, this section presents necessary pilot investigations.

Which image scales should be used to obtain fine-grained features? Common practices [22], [27] suggest that it is beneficial to incorporate fine-grained but local features from high-resolution images with coarse but global features from base-scale images (i.e., thumbnails). We feed pre-trained SigLIP 2-512px with a single-scale image or multi-scale images: 1) 512² scale, 2) 512²+1024² scales, 3) 512²+1024²+1536² scales, 4) 512²+1024²+1536²+2048² scales, 5) 1024² scale, and 6) 1536² scale. By default, for high-resolution images, we split them into non-overlapping base-scale images for inference. Then, we interpolate the re-assembled high-resolution features to the same shape as base-scale features (32² tokens). Finally, we average multi-scale features (if any) to ensure the same number of visual tokens across all settings.

As shown in Table ¿tbl:tab:pilot95scales?, even with the same number of tokens, introducing high-resolution images (e.g., 512² \(\rightarrow\) 512²+1024²) facilitates most benchmarks (10 out of 12), especially for OCR and chart tasks. But such improvement saturates or disappears when further adding the 4\(\times\) (2048²) scale, showcasing it is suboptimal to blindly scale up the resolution. The single high-resolution view is generally inferior to multi-scale views, highlighting the necessity of maintaining a global view. Lastly, we can observe some benchmarks, e.g., MMBench [35], even do not prefer fine-grained features.

How to infer a model on high-resolution images? To apply a model trained on base scale (e.g., 512px) to higher-resolution input, existing works [22], [27] typically slide a base-scale window across the entire image with a non-overlapping stride. However, alternative strategies exist. For example, following practices in dense prediction tasks [36], using overlapping sliding windows can enhance local feature coherence and reduce boundary artifacts. Besides, we can directly interpolate the pre-trained positional embeddings to match the high-resolution input [37], though this may introduce distortion in fine-grained spatial relationships.

We compare these inference approaches in Table ¿tbl:tab:pilot95infer?. As expected, interpolating positional embeddings underperforms the basic sliding window strategy. Surprisingly, overlapping windows also degrade performance, contrary to lessons from dense prediction tasks. While the exact cause remains unclear, we hypothesize two potential factors: 1) inconsistent token distributions: overlapping regions receive ensemble treatment while others do not, or 2) positional embedding conflicts for tokens appearing in multiple windows. Finally, the best practice is still using non-overlapping sliding window.

How to fuse features from multiple resolutions? For multi-scale features, some works [21], [38] first interpolate high-resolution features to the same scale as base-resolution features and then concatenate them along the channel dimension. Other than concatenation, we can simply average them. Besides, pixel unshuffle [39] is another choice [40] to downsample features.

As compared in Table ¿tbl:tab:pilot95fusion?, the simplest “interpolate + average” practice stands out as the best. Beyond delivering better performance, this strategy produces ensembled features with the same dimension as base-scale features, eliminating the need for a projection head [41] in later learning. Indeed, there are more sophisticated fusion options, e.g., using weighted average instead of a simple mean. We leave these investigations to our final main experiments (Table ¿tbl:tab:exp95compare95fusion95weight?).

Summary. Our pilot studies demonstrate that the optimal multi-scale configuration is two or three scales (512²+1024² or 512²+1024²+1536²). For high-resolution inference, non-overlapping sliding window yields the best results. Finally, high-resolution features should be interpolated and then averaged with the base-scale features to produce the final features. Based on these findings, we will next present our fine-to-coarse supervision mechanism.

3 SigLIP-HD↩︎

Our core methodology is fine-to-coarse supervision. As illustrated in Figure 2, our framework is very simple. It comprises two branches, one inference branch to produce high-quality visual features, and one trainable branch to learn towards the better features.

Inference branch. As studied in our pilot experiments (Table ¿tbl:tab:pilot95scales?), multi-scale input is better than single-scale input. It can harvest both advantages of high-resolution images (fine-grained details) and base-resolution images (global information), while suppressing their weaknesses (local information and coarse details). Therefore, we feed our pre-trained SigLIP 2-So400m/16-512px encoder with two scales: base scale (512²) and 2\(\times\) scale (1024²). We do not introduce more scales for inference, because we empirically find that more scales fail to consistently yield stronger results (see Table ¿tbl:tab:exp95compare95scale?). In addition, too many scales significantly increase the computational burden, e.g., requiring nine more feedforward passes if adding a 3\(\times\) scale.

The pre-trained SigLIP 2 model is frozen and it is inferred on the multi-scale input images by non-overlapping sliding window if needed (supported by Table ¿tbl:tab:pilot95infer?). We obtain a base-resolution feature \(F^b \in \mathbb{R}^{C\times H\times W}\) and a high-resolution feature \(F^h \in \mathbb{R}^{C\times 2H\times 2W}\) from the two scales, respectively. To fuse the feature maps of different scales, we first downsample \(F^h\) to the same size as \(F^b\) via bilinear interpolation. We then produce the final high-quality feature \(F^t\) by directly averaging \(F^b\) and the downsampled \(F^h\). This simple strategy is more effective than other counterparts, such as pixel unshuffle and concatenation, as validated in Table ¿tbl:tab:pilot95fusion?.

Trainable branch. Our SigLIP-HD in the trainable branch is initialized with the pre-trained SigLIP 2 parameters. It shares exactly the same architecture and input/output as the pre-trained model, without adding any projection modules [41]. It takes a base-scale image of 512² pixels as input and produces 32² tokens. We denote its feature map as \(F^s\). The optimization target of this branch is enforcing \(F^s\) to align with the better feature \(F^t\) at the patch level.

There are various options in the loss function for feature alignment, such as cosine similarity loss [42], or a combination of smooth L1 and cosine similarity loss [41]. In practice, we observe the strictest and simplest L1 loss performs the best, as compared in Table ¿tbl:tab:exp95compare95loss?.

Application scope. Although our method can polish the visual representation at a standard resolution, we do not anticipate it will entirely bypass the native-resolution practice in MLLMs. Some visual details may be completely lost after down-sampling. Fortunately, our method is fully compatible with existing practices, regardless of using down-sampled resolution (Table ¿tbl:tab:exp95compare95siglip2?) or operating on native resolution (Table ¿tbl:tab:exp95compare95anyres?), as they both rely on a pre-trained vision encoder.

4 Experiment↩︎

Implementation details. We train our fine-to-coarse supervision framework on the 4.5M raw images from the Cambrian-1 collected data [13], since they cover diverse scenarios, including natural images, scene text images, documents, etc. We train our SigLIP-HD with an AdamW optimizer [43], with an initial learning rate of 5e-5 and weight decay of 0.04. The model is trained for 90K iterations with a total batch size of 512. We use the cosine learning rate scheduler with a warm-up period of 4K iterations. We exactly follow the image pre-processing pipeline of SigLIP 2 [6], except when producing the high-resolution image (size changed from 512 to 1024). Finally, as aforementioned, we adopt the strict L1 loss to optimize our features. It takes only 34 hours on 32 A100 GPUs with BFloat16 training.

We evaluate on diverse MLLM datasets [44]: including DocVQA [33], ChartQA [45], TextVQA [34], InfoVQA [46], TextCaps [47], HRBench [48], RealWorldQA [49], GQA [50], MME Perception [51], POPE [52], MMBench [35], ScienceQA-IMG [53], and AI2D [54].

4.1 Comparison with SigLIP 2↩︎

Our framework is built on the highly capable SigLIP 2 model. Through our fine-to-coarse supervision, we aim to further enhance its perception capability in MLLMs. Therefore, we systematically compare our SigLIP-HD encoder with the original SigLIP 2-So400m/16-512px encoder. By default, we adopt the two-stage training pipeline of LLaVA [18]. The input image is resized to 512² pixels and encoded into 32² visual tokens to be sent into the LLM.

As compared in Table ¿tbl:tab:exp95compare95siglip2?, with Vicuna-1.5-7B [55] as the LLM, we try different supervised fine-tuning (SFT) data, including LLaVA-1.5 [18] and LLaVA-NeXT [22] that contains more OCR-related data. We also attempt different training configurations, including freezing or unfreezing the vision encoder during the fine-tuning stage. Across all settings, our SigLIP-HD consistently outperforms our SigLIP 2 baseline. Notably, on OCR-related benchmarks, such as DocVQA and ChartQA, we improve SigLIP 2 from 56.0 \(\rightarrow\) 59.6 (+3.6) and from 61.6 \(\rightarrow\) 65.2 (+3.6), respectively. On general VQA benchmarks that require fine-grained perception, e.g., HRBench, our SigLIP-HD outperforms its baseline by +4.8 (43.5 \(\rightarrow\) 48.3). Qualitative comparisons are provided in Figure 3. Our SigLIP-HD exhibits better capability in perceiving fine-grained content.

Using AnyRes. Although we primarily focus on unleashing the perception capability at a standard (medium) resolution, our encoder is indeed compatible with the AnyRes [19] strategy (i.e., operating on native resolution). We report the results in Table ¿tbl:tab:exp95compare95anyres?. We choose the closest resolution from \(512\times\{\{1\times 1\},\cdots, \{3\times 3\}\}\). The max tokens are set as 32²\(\times\)​4. While the baseline method has been significantly enhanced by such an AnyRes strategy, our SigLIP-HD can further outperform it by aN even larger margin on OCR tasks, e.g., improving ChartQA from 63.9 to 67.4 (+3.5).

More LLMs. In addition to the widely used Vicuna-1.5 [55], which is fine-tuned from Llama 2 [56], we further extend our vision encoder to other LLMs. As shown in Table ¿tbl:tab:exp95compare95siglip295morellm?, with a smaller Llama-3.2-3B LLM [57], our SigLIP-HD still showcases clear advantages. It surpasses SigLIP 2 by +4.6 (45.2 \(\rightarrow\) 49.8) on ChartQA, +2.6 (47.3 \(\rightarrow\) 49.9) on DocVQA, and +1.6 (59.2 \(\rightarrow\) 60.8) on TextVQA. Beyond the Llama series, with the latest Qwen2.5-7B [58] as the LLM, our SigLIP-HD is also consistently superior to our baseline encoder, e.g., boosting it from 62.5 \(\rightarrow\) 64.2 (+1.7) on DocVQA.

4.2 Ablation Study↩︎

We use the LLaVA-1.5 data [18] for instruction tuning and freeze the vision encoder.

Feature alignment loss. Existing works have provided several candidate choices on the loss function used for feature alignment, such as the cosine similarity loss adopted by EVA [42] and the cosine similarity + smooth L1 loss adopted by AM-RADIO [41]. In Table ¿tbl:tab:exp95compare95loss?, we compare these losses, as well as the plain L1 loss. These three losses deliver very close results on average, all surpassing our SigLIP 2 baseline. Nevertheless, the simplest and strictest L1 loss is slightly better than the other two losses. So we choose it as our final feature alignment loss.

Image scale. From our pilot studies (Table ¿tbl:tab:pilot95scales?), we can conclude that feeding multi-scale images of two scales or three scales to the MLLM performs the best. Here, we further examine the final performance of our SigLIP-HD encoder when trained under different configurations of multiple scales. As shown in Table ¿tbl:tab:exp95compare95scale?, similar to our observations in pilot studies, the two-scale configuration (one base-scale 512²px image + one high-resolution 1024²px image) achieves the best results, providing the most suitable features for our SigLIP-HD to learn. We believe there may be more sophisticated and better configurations, e.g., searching for the optimal fusion weight for the three-scale or even four-scale features, but they are out of the scope of this paper. We aim to provide a neat, universal, and efficient training framework without too many hand-crafted hyper-parameters.

Multi-scale fusion weight. We find the simplest “interpolate + average” practice delivers better results than pixel unshuffle and channel-wise concatenation in our pilot studies (Table ¿tbl:tab:pilot95fusion?). Here we further validate whether mean average is better than weighted average. As compared in Table ¿tbl:tab:exp95compare95fusion95weight?, we attempt to increase the weight of high-resolution features or the weight of base-scale features. As a result, neither of them outperforms the default mean average strategy. Slightly to our surprise, even on benchmarks that require fine-grained details, e.g., DocVQA, increasing the weight of high-resolution features still deteriorates the final result. This further proves the indispensable role of base-scale images in capturing the global content.

4.3 Comparison with Legacy Model OpenAI-CLIP↩︎

Until now, we have provided comprehensive results to demonstrate the superiority of our SigLIP-HD over SigLIP 2. To be more convincing, we here further validate our fine-to-coarse supervision methodology on the legacy model OpenAI-CLIP-L/14-336px [4]. However, as illustrated in Figure 1, it is impossible even for humans to recognize detailed image content under the 336px low resolution. Therefore, our model is trained at 518² pixels (37² visual tokens) by interpolating the pre-trained positional embeddings. The high-quality features used for learning are produced under three image scales (336²+672²+1008²). We interpolate the three-scale features to the same shape (37²) and then average them to supervise our CLIP-HD.

To ensure a fair comparison due to our larger resolution and more visual tokens, when using the pre-trained CLIP in MLLMs, we feed a multi-scale input (336²+672²) following LLaVA-NeXT [22]. The obtained two-scale features are both interpolated to the same size as ours and then averaged to be sent into LLM. As compared in Table ¿tbl:tab:exp95compare95clip?, although multi-scale inputs and increased visual tokens have significantly boosted the results on OCR-related benchmarks, our CLIP-HD can further enhance the performance at the same cost, e.g., 22.4 \(\rightarrow\) 31.1 \(\rightarrow\) 33.2 on DocVQA. Our superior results based on the legacy model CLIP and the latest SOTA model SigLIP 2 demonstrate the universality of our proposed fine-to-coarse supervision mechanism.

5 Related Work↩︎

Visual representation learning. Learning robust visual representation has been a fundamental goal in computer vision for decades. The rise of deep learning revolutionized the field [59], starting with supervised learning [60]. But human labels are inherently biased [61], not enough to produce transferable features. Therefore, CLIPs [4] leverage web captions to learn more informative representations. However, such alt-text data is often noisy, prompting continued interest in vision-centric self-supervised learning [62]. This line of work, popularized by contrastive learning [63] and masked image modeling [3], remains highly active. Thanks to large-scale curated data [64] and hybrid supervision signals [14], latest works [6], [16] demonstrate the potential of a universal representation for diverse downstream tasks.

This work does not aim to propose a new pre-training strategy, but to further enhance the capability of a pre-trained encoder with a simple and efficient framework.

Multi-modality large language model (MLLM). There are two mainstream roadmaps to deal with visual inputs: encoder-based and encoder-free. Encoder-based MLLMs [18], [27], [65] use a pre-trained vision encoder [4], [66] to extract visual tokens for LLMs. They can leverage rich pre-trained visual knowledge. But they are not unified, and the visual inputs are constrained by the pre-trained resolution. To bypass these limitations, encoder-free MLLMs [67]–[70] are attracting growing research interest. Unfortunately, until now, they require more training budget and still lag behind encoder-based models. Therefore, our work on seeking better visual representations for MLLMs is still of high value.

Visual representation in MLLMs. There are broadly three approaches to enhancing visual representation in MLLMs. The first involves directly pre-training more powerful vision encoders [15], [16]. Despite the success, their high training costs remain prohibitive for most researchers. The second approach combines multiple pre-trained encoders [12], aiming for mutual gains. In practice, unfortunately, they rarely yield substantial gains over a single well-optimized encoder [17]. The third and most prevalent strategy is scaling up image resolution [20], [40]. Early works resize images to a fixed larger size [18], but recent techniques preserve native resolutions [20]. Beyond global resizing, local zoom-in methods [71], [72] can also amplify details.

In contrast to this scaling trend, our work takes a step back, demonstrating that even at a standard resolution, fine-grained perception can be achieved effectively.

Knowledge distillation. Our work shares core spirit with knowledge distillation [73], in that we both learn from better “teacher” tokens. However, we do not rely on any external knowledge of an auxiliary model. Instead, we unleash the inherent potential of the current model. We eliminate the need to align multiple models into a shared space [41], [74]. CLIPSelf [75] similarly refines CLIP features. But it uses region-level coarse supervision and targets at the open-vocabulary dense prediction, while we adopt patch-wise fine-grained supervision and address the key challenge of MLLMs. Lastly, our work is slightly related to a recent work [76]. But we address fundamentally different problems (MLLM fine-grained perception vs. depth estimation) and operate on different spaces (feature vs. label).

While knowledge distillation or teacher-student training is well-established techniques, what to distill is critical for effective learning. Our key insight lies in presenting a multi-resolution to standard-resolution distillation framework that enhances representation quality without increasing inference cost. This is orthogonal to existing multi-teacher distillation works [41], [74]. We demonstrate that multi-resolution distillation can serve as an effective, lightweight post-training stage for enhancing fine-grained understanding without architectural changes or inference overhead. It can be a practical solution for resource-constrained deployment scenarios.

6 Conclusion↩︎

In this work, we reflect on the scaling trend of image resolution in MLLMs. Motivated by the amazing capability of the human visual system, we investigate how to enhance the perception capability of AI systems at a standard (medium) resolution, without using larger images. We present a highly simple yet effective fine-to-coarse supervision mechanism to address this. Features of the base-scale image are enforced to mimic the high-quality ensembled features of multi-scale images. Built on the latest SigLIP 2 encoder, our fine-tuned SigLIP-HD encoder delivers stronger results across extensive MLLM benchmarks under various training protocols, especially for OCR-related tasks.

Acknowledgment. This work is supported by the National Natural Science Foundation of China (No. 62422606, 62441615) and Hong Kong Research Grant Council General Research Fund (No. 17213925).

References↩︎

---

1. Corresponding author↩︎