0.0.1 Introduction↩︎

The performance of foundation models is fundamentally shaped by the composition and quality of their pretraining¹data [5]–[14]. This has led to a rise of systematic studies of pretraining data curation, including DataComp [5] for contrastive vision-language models, DCLM [6], Nemotron-CC [15], and FineWeb [8] for language models, and OlmoASR [16] in the speech domain. The core design principle of these works is to fix model architecture and pretraining procedure while varying only the data, enabling isolated measurement of data-centric interventions. However, progress in autoregressive vision-language models (VLMs) has mostly focused on novel architectures [17]–[27], training recipes [2]–[4], [28]–[38], or evaluation protocols [39]–[42], treating data as a second-class citizen. The data curation strategies behind their success (which datasets to include, how to filter them, what ratios to mix them in) remain poorly understood and largely irreproducible [2], [3], [20], [28]–[30], [35], [43]. Our goal is precisely to fill this gap and enable open data curation research for the latest class of modern autoregressive VLMs.

Several factors make VLM data curation more challenging compared to other domains. First, unlike early text or vision-language models that often train directly on raw web crawls (e.g., CommonCrawl), modern VLMs are typically trained by aggregating existing datasets from a wide variety of data types—web-crawled image-caption pairs, interleaved multimodal documents, text-only corpora, and multimodal instruction-tuning data—that differ in quality and downstream utility. Because these datasets have already undergone varying degrees of upstream curation, what actually drives quality under this aggregation-based regime—filtering, mixing ratios, or something else—remains an open question. Indeed, existing models largely sidestep it, drawing on a single data type [4] or, at most, an ad-hoc subset [45], [46]. Second, existing open training datasets [17], [22], [28], [29], [44], [47] operate at the scale of millions of samples, far below the trillions of tokens used by state-of-the-art (SoTA) models [3], [21], [48], [49]. This limits the scope of curation experiments that can be conducted. Third, the interaction between data types, model scale, and training budget creates a design space that is too large for exhaustive experimentation. Fourth, VLM evaluation lacks standardization [41]: different papers use different benchmark suites, making fair comparisons across datasets difficult.

To mitigate these challenges and enable controlled comparisons, we introduce DataComp for VLMs (DCVLM), the first benchmark designed to systematically study data curation strategies within a realistic VLM-practitioner’s paradigm. DCVLM provides the following:

1. A standardized data pool of 160 existing datasets spanning four data types: image-caption pairs, multimodal documents, text-only data, and (multimodal) instruction-tuning data. Our pool contains 6T multimodal tokens, enabling a diverse range of data-centric experiments.

2. A principled scaling ladder spanning 1B–8B model parameters and 6.25B–200B training tokens. This enables researchers to test curation strategies across a wide range of compute scales.

3. A comprehensive evaluation protocol with 52 downstream benchmarks organized across 9 domains, split into validation, core, and extended tiers, filtered for stability and reliability.

Using our benchmark, we conduct more than 1,000 experiments yielding multiple findings, including:

Mixing, not filtering, is the dominant lever. Recent VLM technical reports often apply additional downstream quality filters (e.g., CLIP-score or image quality) on top of existing public datasets [48], [50], [51]. Yet, through controlled experiments with common quality filters, we find that such downstream filtering provides diminishing, and sometimes negative, returns (0.0.4.1). We trace this to the modern data landscape: unlike models trained directly on raw web crawls (e.g., CommonCrawl), today’s VLM datasets have already undergone moderate to significant upstream curation. While curating VLM training data directly from raw pools remains an important direction for future work, our results show that applying additional filters to already-curated data is largely ineffective. In contrast, optimizing mixture ratios, which specifically interpolate instruction-tuning and image-caption proportions, yields significant, scale-dependent gains: instruction-heavy mixtures scale better than caption-heavy ones, and this gap widens with model size and token budget (0.0.4.2).

Pretraining decisions reliably transfer after supervised fine-tuning and across backbones. We show that pretraining performance predicts post-SFT performance with near-perfect fidelity (Pearson \(r = 0.99\) across 54 SFT runs), and that our findings are robust to the choice of LLM initialization, i.e., initializing from Qwen2.5-Base or Qwen2.5-Instruct [52] produces similar data rankings. This validates the use of pretraining-only metrics for data curation research with DCVLM (0.0.4.3).

Our controlled experiments yield DCVLM-Baseline, a new state-of-the-art open VLM training dataset (1). At the x-large scale (8B model, 200B tokens), a DCVLM-Baseline-trained model achieves \(63.6\%\) on our core set, outperforming FineVision [44], the previous best open dataset, by \({+5.4}\) pp (0.0.5). We release the full data pool, evaluation suite, model checkpoints at four scales, and all experimental infrastructure to serve as a reproducible testbed for future research.

0.0.2 Related Work↩︎

Vision-Language Pretraining. Modern VLMs adopt a modular architecture consisting of a pretrained vision encoder, a language model backbone, and a connector [2]–[4], [29], [46], [49], [53], [54]. Originally, “pretraining” involved training the connector on large-scale data, predominantly image-caption pairs [29]. In contrast, recent SoTA models train all parameters [2], [3] and incorporate diverse data types. However, the precise mixture ratios, filtering criteria, and formatting choices largely remain proprietary and poorly documented across leading VLMs, motivating our benchmark.

Benchmarking Data Curation. DataComp [5] and DataPerf [55] established the paradigm of fixing model architecture and training procedure while varying only data. DCLM [6] extended this paradigm to language models, demonstrating that a fasttext classifier trained on high-quality samples can substantially boost performance. FineWeb [8] and its educational-quality variant, FineWeb-Edu, showed similar gains through filtering. Generally, quality-based filtering has shown strong results for text [6], [8] and image-text pairs [5], [56]. Common approaches include CLIP-score filtering [57], image quality assessment [58], [59], text quality classifiers [15], [22], and multimodal quality estimators [56]. Beyond filtering, prior works have also explored data mixing approaches such as domain weighting [60], [61], mixture optimization [62]–[67], and temperature-sampling [17], [68]. Despite recent released datasets (e.g., FineVision [44], Cauldron [69]), there exists no systematic study on filtering and mixing strategies in the VLM setting. Our work fills this gap by providing the first scale-aware study of data curation for VLMs.

0.0.3 The DCVLM Benchmark↩︎

DCVLM provides a controlled framework (2) for constructing VLM training sets. We fix the model and training recipe, and participants propose ways to filter and mix data from our pool. We next describe the pool (0.0.3.1), training recipe (0.0.3.2), scales (0.0.3.3), and evaluation (0.0.3.4).

0.0.3.1 Data Pool Construction

Our data pool aggregates 160 publicly available datasets organized into four data types. For the full list of source datasets, pool composition, visualizations, and sample and token counts, see 1.0.5.

Image-caption pairs form the largest component, with great variability in their constituent datasets. At one end, sources like DataComp-1B [5] and ReLAION-2B [14] provide abundant, CLIP-score-filtered image-alt-text pairs from web crawls. At the other, datasets like the synthetic ShareGPT-4o [70] and the human-annotated Pixmo-Cap [17] offer fewer yet higher quality samples.

Multimodal interleaved documents consist of web-crawled interleaved image-text sequences as they appear on websites, PDF documents, and academic papers. Sources include MINT-1T-HTML [71], MINT-1T-PDF [71], WanJuan [72], OmniCorpus [73], and Multimodal-Textbook [74]. These are the least curated sources in our pool. Most are scraped directly from the web with minimal URL-based and heuristic filtering, and thus tend to have lower quality scores on average.

Text-only data preserve the language model’s capabilities during multimodal training, following recent VLMs [2], [49]. Examples include FLAN [75], SlimOrca [76], and Dolly [77], alongside image-free science and knowledge sources such as Numina-Math-1.5 [78] and xCoder80k [79].

Multimodal instruction-tuning data comprise single or multi-turn instruction-tuning datasets, typically with human-written or model-generated question-answer pairs grounded in one or multiple images. We manually categorize these into eight capabilities following [44]: knowledge, chart & table understanding, general-QA, grounding & counting, math, naive OCR, OCR-QA, and science. For a complete breakdown of the capability distribution of instruction-tuning data, see 1.0.5.1.

Our DCVLM pool contains 6T multimodal tokens (measured using the InternVL-2.5 [1] tokenizer). It is highly heterogeneous in source quality, data types, instruction-tuning capabilities (e.g., grounding, OCR, chart and table understanding, captioning), visual and textual domains (e.g., natural, synthetic, tabular images), and languages (over \(20\) including English and Chinese, see 1.0.5.5). This heterogeneity is deliberate: it lets participants study curation recipes in a realistic setup with several confounders to control for. To avoid train-test overlap, we decontaminate our entire pool against our Extended eval suite of 52 benchmarks (0.0.3.4): multimodal samples are filtered with a ResNet-50 SSCD embedding model [80] (cosine-sim \(>0.75\) to any test image) and text-only samples with MinHash [81] Jaccard similarity (\(>0.55\)). The exact details of decontamination are in 1.0.7.

0.0.3.2 Model Architecture and Training Recipe

To ensure DCVLM employs a state-of-the-art training recipe, we use an architecture that mimics InternVL3 models [2]: an InternViT-300M vision encoder [1], a 2-layer MLP projector, and a Qwen2.5-Base language model [52] (we show that our central findings transfer to Instruct backbones as well in 0.0.4.3). We adopt AnyRes [32] tiling, where images are dynamically split into \(448{\times}448\) tiles, each encoded into 256 visual tokens after pixel shuffling [1], [82]. We use the AdamW [83] optimizer, a linear \({3}{\%}\) warmup, and a cosine decay with peak learning rate of \(2\times10^{-5}\), identified after an initial sweep to ensure optimal hyperparameters. For more details, refer to [app:architecture] [app:hparams].

0.0.3.3 Competition Scales and Design Principles

width=.9

A key principle of DCVLM is to evaluate data curation strategies across scales, because findings at small scales may not transfer to larger ones [84]–[87]. To simultaneously (i) approach the scale of foundation models like InternVL-3 [2] and (ii) ensure accessibility for researchers with fewer resources, we define four scales: small, medium, large, and x-large. Model sizes and token budgets are illustrated in ¿tbl:tab:scales?. We design the small, medium, and large scales such that a step corresponds to an \(8\times\) compute increase: models become \(2\times\) larger and tokens increase by \(4\times\). At the x-large scale, our entire pool of 6T tokens is the candidate for dataset construction. We design all scales to fix pool-to-training token ratio at \({30}{\times}\), i.e, the pool always contains \(30\times\) more tokens than the training budget. The primary reason for keeping this ratio constant is to enable participants to experiment with aggressive filtering at all scales while hitting a constant number of data repetitions.

0.0.3.4 Evaluation Protocol

Participants in DCVLM can evaluate models on up to 52 benchmarks. To get reliable signal, we start from a candidate set of 65 benchmarks, which we categorize across 9 domains based on the majority consensus of prior work [1], [2], [28], [35], [88]: General Understanding, Knowledge-Centric, OCR & Charts, Vision-Centric, Multilingual, Text-Only, Safety, Hallucination, and Reasoning benchmarks. We then filter them for (i) stability, removing those with high seed variance [89], [90], and (ii) monotonicity, removing those that do not improve from small to medium scales [8], [91]. We organize benchmarks into three nested tiers, each a superset of the previous: a Validation set, used for rapid iteration (13 benchmarks), a Core set, the primary tier used for main results (33 benchmarks), and an Extended set (52 benchmarks), including all benchmarks for comprehensive analysis. Safety, Hallucination, and Reasoning are deferred to the extended tier, as they are typically targeted by (and thus most relevant for) post-training methods. Unless otherwise specified, we report the average accuracy across all benchmarks in a given tier. For full details of benchmark selection, see 1.0.6.

0.0.4 Towards a Strong Baseline on DCVLM↩︎

We now present a suite of controlled experiments showing how to obtain a strong baseline dataset on DCVLM, along two primary axes: data filtering (0.0.4.1) and data mixing (0.0.4.2). For additional axes (including data formatting, synthetic captions, and temperature sampling), refer to 1.0.8. We also run control experiments validating the generality of our results (0.0.4.3). Unless specified, all filtering experiments use a base mixture of \(75\)% image-caption, \(18\)% text-only, \(4\)% multimodal documents, and \(3\)% instruction-tuning data, derived by length-proportional sampling across the pool. In this section, we always report results on our 33-task Core evaluation suite.

0.0.4.1 Data Filtering

Quality-based filtering has been central to pretraining strong language [6], [8], [61], [92] and CLIP models [5], [89], [93], [94], hence a natural question is whether these gains transfer to VLMs as well. We answer this question in the negative by testing more than 60 filter configurations at both small and medium scales (for an exhaustive report, refer to 1.0.8.1). To illustrate our findings, here we report and discuss medium scale results for filters shown to be successful in prior work (in 1.0.8.1, we describe several other variants across scales, yielding the same conclusions):

• CLIP-score. We experiment with filtering image-caption pairs according to three different CLIP models: OpenAI’s CLIP ViT-L/14 [95], DFN-CLIP [93], and SigLIP-2-B/16@384 [96].

• Text quality classifiers. We experiment with filtering samples according to the quality of their constituent text snippet(s), as judged by three classifiers: DCLM’s fasttext classifier [6], as well as NVIDIA’s Nemotron and Mixtral educational-quality classifiers [15].

• Multimodal filters. We additionally experiment with (i) filtering with two UniFilter models [56] (Qwen2.5-1.5B and Qwen3-0.6B), and (ii) filters grounded in perplexity [97]: text-only perplexity (computed on text tokens by excluding image tokens), multimodal perplexity (computed on text tokens by including image tokens), and Conditional Mutual Information [98], which measures their difference (i.e., the reduction in perplexity with and without image tokens).

Importantly, we study two different filtering paradigms to isolate the impact of data filtering and that of implicit data mixing: Local filtering, which computes filtering percentile thresholds independently within each source dataset. This preserves the global mixture by construction: every dataset loses the same sample fraction, and Global filtering, which computes a single filtering threshold across the entire pool of samples to which the filter can be applied. Because different data sources have systematically different score distributions, a global cut implicitly reshapes the data mixture. Following prior evidence that smaller models benefit from more aggressive filtering [85], [99], we retain the top-10% of samples at the small scale, and the top-40% at the medium scale.

3 illustrates the results. We make two key observations: (i) regardless of whether the mixture is held fixed, no quality filter we tested produces a robust and significant improvement over a no-filter baseline; and (ii) local and global filtering yield notably different results. We expand on each below.

Filtering rarely helps, but why? The best filtering outcome is given by SigLIP-2 when globally filtering image-text pairs (rightmost bar in 3), yet this result is defined by a marginal \(+0.8\)pp improvement, far below the gains one would apriori expect from quality-based filtering [5], [6], [14], [15], [89], [94]. Other filters either leave the baseline mostly unchanged or actively hurt performance. This observation holds across both small and medium scales (see 17 for the small figure).

This failure is surprising, especially in light of strong results from prior works. We hypothesize this is because there is no significant noise to remove from our base pool: unlike raw Common Crawl (used in DCLM [6]) or raw web-crawled image-text pairs (used in DataComp-CLIP [5]), existing VLM training sets aggregate datasets that have already undergone a level of upstream filtering (e.g. CLIP-score filtering) by their original creators, and our pool follows this data collection process. To validate our hypothesis, we create three sub-pools from our original pool, varying the effective percentage (25%, 65%, and 100%) of “pre-filtered” data samples in the mixture (see 1.0.8.3 for more details). From each of these sub-pools, we create three more training sets by applying further CLIP-score filtering to the image-caption data. For each pair of datasets, we then train small scale models and measure the performance gain due to downstream filtering (4). For 25% pre-filtering (i.e., when the sub-pool is dominated by unfiltered data), the gain is significant (\(+2.4\)pp). However, this decreases the more the sub-pool is pre-filtered (dropping to \(+1.3\)pp at 65% and \(+0.6\)pp at 100%). In other words, additional filtering on top of already-curated data operates in a regime of diminishing returns.

Interaction between filtering and implicit mixing. The second takeaway from 3 is local and global filtering produce very different results. The inconsistent trends suggests that global filtering is not a reliable strategy. However, given significant performance fluctuations between local and global filtering, we hypothesize the underlying mixture distribution is the lever that dictates performance.

0.0.4.2 Data Mixing

Having established that filtering over the base mixture provides negligible gains on our pool, we turn to data mixing, i.e., the allocation of training samples across data types, as our primary curation lever.

Setup. We optimize the mix along an important axis based on prior work [4], [28], [35], [100]–[103]: the ratio of image-caption pairs to instruction-tuning data. Text-only samples and multimodal documents are fixed at \(15\)% and \(5\)%, respectively, as supporting components. Here, we study three ratios along the image-caption \(\leftrightarrow\) instruction-tuning axis: (i) a Caption-heavy mixture with \(65\)% image-caption pairs and \(15\)% instruction-tuning data; a (ii) Balanced mixture with \(40\)% image-caption and \(40\)% instruction; and (iii) an Instruction-heavy mixture with \(10\)% image-caption and \(70\)% instruction-tuning data (see 1.0.9 for finer sweeps across scales). Each mixture is evaluated across a scaling grid of 3 model sizes (1B, 2B, 4B) \(\times\) 3 token budgets (6.25B, 12.5B, 25B).

r0.4

Data mixing cannot be scale agnostic. 5 reveals a striking interaction between data mixture and compute scale: as both model size and token budget increase, the Instruction-heavy mix exhibits a markedly steeper scaling slope. It starts as the worst mixture at 1B\(\times\)​6.25B (small scale) but becomes the best at 2B\(\times\)​25B (medium scale), and remains so at 4B\(\times\)​25B. This crossover pattern has an important practical implication: mixture rankings established at small scale do not transfer reliably to larger scales. In our setting, optimizing the data mix at the small scale (1B\(\times\)​6.25B) would select the Caption-heavy mix and miss the Instruction-heavy configuration that ultimately performs best. This underscores the need for scale-aware data curation that validates mixture choices across multiple points on the scaling ladder, rather than at a single small-scale proxy [84]–[86], [102], [104]–[106].

Repeatability of instruction-tuning data. Given our previous finding that Instruction-heavy mixes scale better, a natural concern about scalability arises: instruction-tuning datasets are typically orders of magnitude smaller in size than web-crawled image-caption pairs. A 70% allocation might require extreme data repetitions to fill the token budget, a known cause of performance degradation [107]–[111]. We test this effect by holding all non-instruction data sources fixed and randomly subsampling instruction-tuning data to induce up to 2\(\times\), 4\(\times\), and 8\(\times\) repetitions at the medium scale of our benchmark. From 1, we find that performance degrades gracefully: each doubling of repetition factor costs roughly 0.5–1.0% in performance. Notably, the Instruction-heavy mix with 2\(\times\) repetitions (50.2%) still matches the Caption-heavy mix with fully unique data (50.3%), and at 4\(\times\) repetitions it remains above the base mix (49.8% vs 48.8%). The mix ultimately degrades at \({\sim}8\times\) repetitions. This result has a practical takeaway: the benefits of a good mixture outweigh the costs of moderate data repetition. Our results corroborate similar findings from the language domain regarding the benefits of including instruction-like data during pretraining [112]–[118].

0.0.4.3 Control Experiments

We now verify the generality of our findings before scaling up. Specifically, we ask: (i) Does the effectiveness of pretraining data curation hold after supervised fine-tuning (SFT)?, and (ii) Are our findings tied to the LM backbone (Qwen2.5-Base) used for initialization? We provide answers next.

Pretraining results transfer reliably post-SFT. A common concern with pretraining-only evaluations is the worry that SFT will overwrite differences induced by pretraining data choices [119]. In particular, given the findings in 0.0.4.2, it is natural to hypothesize that SFT (which also uses instruction-tuning data by definition) may interfere with or diminish the effect of using an Instruction-heavy pretraining mixture. We study this by SFT-ing all 27 pretrained checkpoints from our previous scaling grid (3 mixes \(\times\) 3 model sizes \(\times\) 3 token budgets) using two different SFT datasets: LLaVA-665K [4] and Mammoth-VL-12M [120], for a total of 54 SFT runs. We set the total SFT tokens to \(0.29\times\) the pretraining tokens by estimating InternVL3’s [2] SFT-to-pretraining token ratio. 6 (left) shows the results with LLaVA-665K (refer to 1.0.12 for identical results with Mammoth-VL-12M). We observe that pretraining and post-SFT scores are near-perfectly correlated (Pearson \({r}{=}{0.99}\); Spearman \({\rho}{=}{0.99}\)), and the pretraining ordering is preserved across all runs.

Our findings are robust to LM initialization. So far, we’ve used Qwen2.5-Base as the language model backbone. To verify that our findings are not specific to this particular choice, we repeat the full 2B-model sweep (3 mixes \(\times\) 3 token budgets) with Qwen2.5-Instruct-2B as the LM. This allows us to verify whether instruction-heavy mixes are better even when the LM has been “unimodally” instruction-tuned already. As shown in 6 (right), it produces nearly identical mixture rankings to Qwen2.5-Base (Pearson \(r{=}0.97\)), especially at larger token budgets (denoted by larger markers). These results provide some evidence of generality of our results—particularly, the advantage of instruction-heavy mixes after training at scale may be agnostic to the LM initialization choice.

0.0.5 Scaling Up Our Findings↩︎

Building on our findings, we propose DCVLM-Baseline—a simple data recipe that forgoes filtering and instead focuses on carefully tuned data mixtures. Accordingly, we use the Instruction-heavy mix: 10% Image-Caption data, 5% Multimodal Documents, 15% Text-Only data, and 70% Instruction-Tuning data (see the full mix in 1), which was found to be optimal for medium and large scales (0.0.4.2). For simplicity, we use this as well for the small scale—however, we reiterate the scale-aware nature of data curation and the fact that the “optimal” mixture for the small scale was in fact the Caption-heavy one (5). For each data type, we fill the token budget by drawing samples from its constituent sources via simple length-proportional sampling.

We compare DCVLM-Baseline to three open VLM pretraining datasets: LLaVA-OneVision-1.5-Midtraining-85M, used to pretrain the LLaVA-OneVision-1.5 family [29]; the public data released with Nemotron-VL-2 [22]; and FineVision [44], the prior largest effort to unify existing sources into a single open dataset. As upper-bound reference points, we also report results from the pretrained InternVL 8B models (InternVL-2.5, InternVL-3, InternVL-3.5).

We train models on DCVLM-Baseline and the FineVision baseline at all compute scales of our benchmark (small, medium, large, and x-large). For the other baselines (LLaVA-OneVision-1.5 and Nemotron-VL-2), we observe that performance is quite poor at smaller scales and hence did not use those datasets for training at the larger scales. We report all results in 2. These results, across all scales, confirm that DCVLM-Baseline outperforms open pretraining dataset for VLMs. Specifically, compared with FineVision (the previous best open pretraining dataset), we observe consistent gains that increase with scale: DCVLM-Baseline achieves progressive gains of \(+0.3\)pp (small), \(+1.1\)pp (medium), \(+4.7\)pp (large), and \(+5.4\)pp (x-large) on our 33-task Core evaluations. Remarkably, a 4B model trained for 100B tokens on DCVLM-Baseline (our large scale) outperforms an 8B model trained for 200B tokens on FineVision (x-large).

52-task Extended results. These trends are further confirmed by our 52-task Extended evaluation suite (1.0.13), where a DCVLM-Baseline-trained model at the x-large scale, scores \(60.5\%\) vs. \(56.6\%\) for the corresponding x-large scale FineVision-trained model (an absolute improvement of \(+3.9\)pp). In fact, with a score of \(56.0\), the DCVLM-Baseline-trained model at the large scale nearly achieves the same performance as the FineVision x-large model.

0.0.6 Conclusion↩︎

We introduced DCVLM, a systematic benchmark for studying data curation strategies for VLM pretraining. Through extensive experimentation across a principled scaling ladder, we established two central findings: (i) individual quality filters provide negligible benefits when the source pools are pre-filtered, which is typical for VLMs, and (ii) data mixture optimization (specifically, instruction-heavy mixtures) is the most effective curation lever, providing gains that scale reliably with model size and compute; at our largest scale (8B model, 200B tokens), DCVLM-Baseline outperforms FineVision by \(+5.4\)pp on our comprehensive 33-benchmark core set. We release the full data pool (160 datasets), evaluation suite (52 benchmarks in total), model checkpoints at 4 scales, and all experimental infrastructure to serve as a reproducible testbed for future data research.

Acknowledgements↩︎

The authors would like to thank (in no particular order): Jeffrey Li, Etash Guha, Alex Fang, Pratyush Maini, Hritik Bansal, Moreno D’Incá, Songlong Xing, Olivier Henaff, Matthew Leavitt, Siddharth Joshi, Wieland Brendel, Samuel Albanie, Francesco Tonini, and Evgenia Rusak, for thoughtful feedback and comments throughout the project.

VU, AH, AG, SD and JS thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS). VU, SK and SD also thank the European Laboratory for Learning and Intelligent Systems (ELLIS) PhD program for support. AH acknowledges funding by the Federal Ministry of Research, Technology and Space (BMFTR), FKZ: 16IS24079A. SD acknowledges support by the Tübingen AI Center. AP acknowledges funding by the Federal Ministry of Research, Technology and Space (BMFTR), FKZ: 16IS24085B. VU was supported by a Google PhD Fellowship in Machine Intelligence. This work was supported by the German Research Foundation (DFG): SFB 1233, Robust Vision: Inference Principles and Neural Mechanisms, TP4, project number: 276693517 and the UKRI grant: Turing AI Fellowship EP/W002981/1. MB is a member of the Machine Learning Cluster of Excellence, funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany’s Excellence Strategy – EXC number 2064/1 – Project number 390727645. The authors gratefully acknowledge LAION and the Gauss Centre for Supercomputing e.V. for funding this work by providing computing time on the JUWELS Booster at Jülich Supercomputing Centre (JSC). AG, MM, ER, JJ and HK receive funding from the European Union’s Horizon Europe research and innovation program under ELLIOT - Grant Agreement No 101214398. SJ is funded by the European Research Council (ERC) under the Starting Grant GraViLa (101117556). MF acknowledges travel support from ELIAS (GA no 101120237). MB acknowledge financial support by the Federal Ministry of Education and Research (BMBF), FKZ: 011524085B and Open Philanthropy Foundation funded by the Good Ventures Foundation. The authors acknowledge the CINECA award under the ISCRA initiative for the availability of high-performance computing resources and support, and the projects EU Horizon projects ELIAS (No. 101120237) and ELLIOT (No. 101214398). The authors also acknowledge the Leonardo supercomputing hours awarded by through the project EHPC-AIF-2025SC03-174. TN and SO acknowledge NSF grants 2505865, 2229876, 2229876, and 2502281. SK is supported by the CS at Max Planck Doctoral Program, VIA Center and Saarland Informatics Campus. The authors acknowledge the GCP Credit Award Program by Google with award GCP444206605 for supporting the project with computational credits on GCP. MBö is supported by the Swiss National Science Foundation (project number 200021_204620).

1 Appendix↩︎

1.0.1 Contributions↩︎

This was a large collaborative effort, and the work spanned data curation, infrastructure, experimentation, and analysis. Below we summarize the main contribution areas. Within each area, contributors are listed in a random shuffled order. An author may appear under multiple areas.

Project coordination. Matteo Farina, Vishaal Udandarao, Thao Nguyen, Nikhil Parthasarathy, Ludwig Schmidt

Data pool construction. Thao Nguyen, Vishaal Udandarao, Matteo Farina, Selim Kuzucu, Andreas Hochlehnert, Adhiraj Ghosh, Mehdi Cherti, Karsten Roth, Joschka Struber, Yuhui Zhang, Sebastian Dziadzio, Elaine Sui, Dhruba Ghosh, Hasan Hammoud, Thomas De Min, Simone Caldarella, Sedrick Keh

Data filtering. Vishaal Udandarao, Matteo Farina, Selim Kuzucu, Thao Nguyen, Maximilan Böther, Andreas Hochlehnert, Marianna Nezhurina, Adhiraj Ghosh, Soumya Jahagirdar, Elaine Sui, Jehanzeb Mirza

Train–test decontamination. Matteo Farina, Vishaal Udandarao, Maximilian Böther, Adhiraj Ghosh, Marianna Nezhurina

Annotation infrastructure. Maximilian Böther, Matteo Farina, Marianna Nezhurina

Data mixing. Vishaal Udandarao, Matteo Farina

Training infrastructure and scaling. Matteo Farina, Marianna Nezhurina, Vishaal Udandarao

Evaluation suite. Matteo Farina, Sebastian Dziadzio, Karsten Roth

Transfer and controlled experiments. Vishaal Udandarao, Matteo Farina, Thao Nguyen, Andreas Hochlehnert, Adhiraj Ghosh

Writing. Vishaal Udandarao, Matteo Farina, Adhiraj Ghosh, Soumya Jahagirdar, Massimiliano Mancini, Nikhil Parthasarathy, Ludwig Schmidt

Advising and supervision. Nikhil Parthasarathy, Ludwig Schmidt, Massimiliano Mancini, Jenia Jitsev, Alessio Tonioni, Ameya Prabhu, Sewoong Oh, Matthias Bethge, Elisa Ricci, Ana Klimovic, Federico Tombari, Muhammad Ferjad Naeem, Serena Yeung-Levy, Bernt Schiele, Hilde Kuehne

1.0.2 Extended Related Work↩︎

In the main paper, we provided a brief overview of the most relevant recent papers for our work. Here, we provide a deeper dive into these related papers.

Vision-Language Model Training Regimes. The development of modern autoregressive VLMs has converged on a modular architecture, consisting of a pretrained vision encoder, a language model backbone, and a lightweight connector between the two. Early methods differed in how this connection was implemented. Notable works include BLIP-2 [121] which used a Q-former to compress visual tokens and Flamingo [45], which inserted cross-attention layers between frozen vision and language features. The dominant blueprint can be attributed to LLaVA [4], [31], [32] which popularized the simpler recipe of “pretraining" the connector on predominantly image-text pairs  [29], before conducting supervised fine-tuning (SFT) on curated instruction data.

In contrast, recent works have considerably relaxed these constraints. First, frontier works like InternVL3[2], InternVL3.5[3] and LLaVA-OneVision-1.5 [29] fine-tune all model parameters from scratch. The relationship between these training choices and model scale, image resolution and data composition have been studied too [35], [101]. Concurrently, the focus has also shifted to making the data composition more heterogeneous while training VLMs, moving away from an over-reliance on image-text pairs. Idefics [122], [123] trains on interleaved image-text sequences, UReader uses multimodal documents [124], PaliGemma [27] combines image-text pairs with generated VQA, multi-object detection and OCR, Cambrian [28] includes text-only corpora for preserving language capabilities, etc. Most notably,  [2], [44] advocate for using instruction-tuning data during pretraining itself. However, the precise mixture ratios, filtering criteria, and formatting choices that drive these systems remain proprietary or only coarsely documented, motivating our systematic benchmark.

Benchmarking Data Curation. Controlled data-curation benchmarks keep model architectures and training pipelines fixed and only vary the data distribution fed to the model. DataPerf [55] established this paradigm, while DataComp [5] brought it to CLIP pretraining, enabling principled comparison of curation strategies at scale. DCLM [6] extended this to language model pretraining, demonstrating that a simple fasttext classifier trained on high-quality text can substantially improve downstream performance. FineWeb [8] and its educational-quality variant, FineWeb-Edu, showed similar gains through quality-based filtering of Common Crawl. In general, quality-based data filtering has shown strong results for text [6], [8] and image-text pairs [5], [56].

Existing data curation methods can be categorized into two groups: filtering and mixing. Common filtering approaches include CLIP-score filtering [57], image quality assessment [125], text quality classifiers [15], [22], and learned multimodal quality estimators [56]. These filtering methods have proved to be quite effective in driving downstream performance for single data-type (image-text or text-only) datasets, training better CLIP models being an example [5]. Using pretrained data-selector models or multimodal quality scores are more recent approaches to quantify whether a data sample is likely to improve pretraining [103], [126]–[129].

Beyond model-based filters, offline curation also comprises deduplication, recaptioning and concept-aware selection. Deduplication ranges from general pruning [130] to semantic deduplication [131], [132]. Recaptioning methods aims to replace weak web-scale alt-text with synthetic or fused captions using VLMs or caption augmentation  [12], [133]–[137]. Concept-aware methods control the training distribution through concept filtering or balancing  [12], [138], [139]. Put together, it has specifically been shown that the offline curation of noisy web-scale data results in large pretraining efficiency gains [5], [93], [140]–[143].

Prior works have also explored data mixing instead of filtering, with standard approaches relying on strategies such as domain weighting [60], [61], mixture optimization [62]–[67] and temperature-scaled sampling [17], [68]. Despite the efforts in releasing curated datasets (e.g., FineVision [44]), there exists no systematic study ablating filtering or mixing strategies in the VLM setting. Our work fills this gap by providing a controlled testbed for multimodal data curation, providing the first scale-aware study of data-type mixing for VLMs.

Train-Test (De)contamination. Train-test overlap (contamination) is a well-documented concern, especially in language model evaluation, where several works have demonstrated how benchmark scores can be inflated when test-set or their near-duplicate samples appear in the pretraining data pool  [144]–[148]. This is a problem for VLM training as well as the contamination can stem from many sources: text, duplicate or near-duplicate images or documents, etc. To mitigate such concerns, and also to ensure that models do not degrade to rote memorization of the training sets, several works conduct robust decontamination procedures on their training sets [27], [85], [99], [139], [149]–[151], i.e, they attempt to remove training examples too similar (or, at worst, identical) to evaluation examples. Some canonical methods include embedding-based similarity search [44], [139], MinHash signatures for approximate text-matching  [6], [8], [81] and direct string-search using suffix arrays [152]. In our work, we employ two-way decontamination: a form of embedding-based decontamination for multimodal samples and MinHash signatures for text-only samples.

Scaling Laws and Scale-Aware Curation. An important consequence of scaling-law studies is that a data curation strategy chosen at one scale may not remain optimal at others. A growing body of evidence suggests that the effectiveness of these filters is scale-dependent: [84] and [85] show that optimal filtering aggressiveness decreases with compute budget. Our work successfully extends this finding to the multimodal setting. showing that at sufficient scale and with optimized mixtures, no individual quality filter provides reliable and consistent gains.

1.0.3 Model Architecture Details↩︎

All our experiments use a single VLM architecture template, parameterised across the four scales of the scaling ladder (¿tbl:tab:scales?). The template follows the InternVL-3 [2] family: a vision encoder \(\rightarrow\) a randomly-initialised MLP projector \(\rightarrow\) an autoregressive language-model backbone, with all three components trained jointly from the start (single-stage pretraining, no frozen components). Across our four scales, only the language-model backbone changes; the vision encoder and the projector recipe are held fixed. We document each component in turn.

1.0.3.1 Vision Encoder

We use InternViT-300M-448px-V2.5 [1] for all experiments. It is a Vision Transformer [153] with the modifications introduced in the InternVL series [1], [2], [103], kept identical across our four scales. 3 reports its key structural choices.

The tile-based tokenisation produces \(32 \times 32 = 1024\) patches per \(448 \times 448\) tile. After the projector’s pixel-shuffle reduction (1.0.3.2), this becomes \(16 \times 16 = 256\) visual tokens per tile (\(4{\times}\) reduction) that are fed to the language model. Multiple tiles and a thumbnail image are concatenated into the LM input, following the dynamic high-resolution scheme of InternVL-2.5 [1].

1.0.3.2 Projector

The vision encoder and the language model are bridged by a small randomly-initialised MLP-style projector (often called the “connector” in VLM literature [4], [35]). It is the only module that is randomly initialised at the start of training, everything else is loaded from pretrained checkpoints. 4 gives the exact structure.

Following InternVL-2.5/3, we keep depth and activation fixed across scales; only the projector’s hidden width tracks the LM.

1.0.3.3 Language Model Backbones

For the four points on our scaling ladder we use four different sizes from the Qwen2.5 family [52]: \(0.5\)B, \(1.5\)B, \(3\)B, and \(7\)B parameters. All four share the Qwen2 transformer architecture [155]—SwiGLU FFN [156], RMSNorm [157], RoPE position embeddings [158], grouped-query attention (GQA) [159], no QK-normalisation. They differ only in their depth/width/head budget and in two minor configuration knobs (max position length and embedding tying), summarised in 5. Unless specified, we always initialise from the Base (non-Instruct) checkpoints by default.

4pt

A few cross-scale observations are worth flagging because they surface in our scaling experiments:

• Head dimension is not constant. The \(0.5\)B model uses \(64\)-dim heads, while \(1.5\)B/\(3\)B/\(7\)B all use \(128\)-dim heads. Practitioners scaling pretraining recipes should be aware that the small scale therefore has a slightly different attention behaviour than the rest of the ladder, even though the rest of the architecture is uniform.

• KV-head count is heavily compressed. GQA ratios are \(14{:}2\), \(12{:}2\), \(16{:}2\), and \(28{:}4\) from Small to X-Large—all sub-7B models share the same minimal \(2\) KV heads.

• Tied embeddings only at sub-7B. The \(7\)B model is the only scale where input/output embeddings are not tied. This costs \(\sim\)​50M extra LM parameters at the X-Large scale.

• Layer count is non-monotonic. The \(3\)B model is deeper (36 layers) than the \(7\)B model (28 layers); \(7\)B grows primarily by widening (hidden size \(2048 \rightarrow 3584\)) rather than deepening.

These idiosyncrasies are inherited from the Qwen2.5 release and we deliberately do not smooth them out, since our purpose is to produce a benchmark whose scaling axis can be reproduced from publicly-released checkpoints rather than to study clean architectural scaling.

1.0.3.4 End-to-end Parameter Accounting

6 sums the three components per scale, giving the total trainable-parameter count behind the “1B / 2B / 4B / 8B” labels used throughout the paper. The vision encoder and projector together contribute roughly \(5\)–\(60\)% of parameters at the small scale and \(\sim\)​5% at the X-Large scale.

All parameters are trained jointly—there is no frozen-encoder pretraining stage, no LoRA [160] adapter, and no separate connector-warmup phase. We refer the reader to 7 for the optimizer, schedule, and packing settings used during this joint training.

1.0.4 Training and hyperparameter details↩︎

We provide the exact hyperparameters we use for all our training runs in 7. For the most part, these were derived from the InternVL-2.5 [1] and InternVL-3 [2] configurations. However, we did run a small learning rate (LR) sweep of our own to confirm that these were indeed the best performing on a subset of downstream evaluations.

1.0.4.1 Learning Rate Selection

To ensure that the InternVL LR configurations were optimal, we conducted a small LR-sweep ourselves². We select the learning rate by sweeping five values (\(2 \times 10^{-4}\), \(4 \times 10^{-5}\), \(2 \times 10^{-5}\), \(8.91 \times 10^{-6}\), \(2 \times 10^{-6}\)) at each model scale using 10B training tokens with the base mixture. All other hyperparameters are held fixed (global batch size 1024, cosine schedule, 3% warmup) according to those specified in 7. As shown in 8, \(\text{lr} = 2 \times 10^{-5}\) achieves the best or second-best performance at every model scale and LM backbone setting. Learning rates above \(4 \times 10^{-5}\) cause training instability—particularly at the 1B scale, where \(\text{lr} = 2 \times 10^{-4}\) collapses to near-chance performance. This behaviour has also been observed in prior works [164]–[166]. Learning rates below \(10^{-5}\) underfit, with the gap widening at larger model sizes. We therefore adopt \(\text{lr} = 2 \times 10^{-5}\) for all our experiments. This hence also corroborates the LR order-of-magnitude used in InternVL-2.5 and InternVL-3.

1.0.5 DCVLM Pool Details↩︎

Our DCVLM pool aggregates 160 publicly available datasets across four data types: image-caption pairs (13 datasets), multimodal interleaved documents (5), text-only (33), and multimodal instruction-tuning (109, spanning 8 capability categories following [44]: Captioning & Knowledge, Chart & Table, General QA, Grounding & Counting, Math, Naive OCR, OCR QA, and Science). Our full pool contains 3.9B samples and 6.0T multimodal tokens, averaging 1.5K tokens per sample. All token counts are measured using the InternVL-2.5 [1] tokenizer over the full pool. The complete per-dataset breakdown of our DCVLM pool is given in ¿tbl:tab:dcvlm-datamix-samples? (showing number of samples per dataset) and ¿tbl:tab:dcvlm-datamix-tokens? (showing number of multimodal tokens per dataset).

1.0.5.1 Pool Composition

1.2pt

1.2pt

7 summarises how our pool is split across the four data types, in terms of both samples and multimodal tokens. The detailed per-category sample and token totals are given in ¿tbl:tab:dcvlm-datamix-samples? ¿tbl:tab:dcvlm-datamix-tokens?. The two views differ in informative ways:

• Image-caption pairs dominate both axes (74% of samples, 83% of tokens). The token share exceeds the sample share because every image-caption sample contributes \(None\)​256 visual tokens per-tile on top of a short caption, inflating the per-sample token count relative to the text-only data.

• Text-only data shows the opposite asymmetry: 19% of samples but only 5% of tokens, because individual text samples are short (440 tok/sample on average) and contribute no visual tokens.

• Multimodal documents, despite making up just 4% of samples, contribute 7% of tokens. They are the densest data type, having \(\sim\)​3K tok/sample on average, since each sample typically interleaves several images with surrounding text.

• Instruction-tuning data spans 8 capability categories and sits between image-caption pairs and multimodal documents in density (\(\sim\)​2.2K tok/sample). Also in this case, the presence of multi-image samples contributes to a greater per-sample token average.

1.0.5.2 Per-Dataset Variation

The 160 datasets in the pool span six orders of magnitude in sample count—from \(\sim\)​1K (e.g.ChartLlama, LIMA, ViQuAE) to \(\sim\)​1.5B (ReLAION-2B-en, DataComp-1B)—and four orders of magnitude in tokens-per-sample (8). We dig into the per-data-type statistics below:

• Image-caption datasets cluster tightly along the 1–2K tok/sample diagonal (8). The narrow band reflects that image-caption tokens are dominated by the fixed-cost visual-token contribution (\(\sim\)​256 tokens for a single \(448{\times}448\) tile after pixel shuffling), with caption length contributing only secondary variation.

• Multimodal documents sit a decade higher, in the 3–6K tok/sample regime, because each sample carries multiple images and longer interleaved text spans.

• Text-only datasets occupy the widest tokens-per-sample band of any data type (100–15K), with two distinct clusters: short-form instruction data (Dolly, Unnatural-Instructions, MathQA, GSM8K-Socratic) at \(\sim\)​200–400 tok/sample, and long-context corpora (LongCite-45K, LongInstruct-Para., LongAlpaca, Long-QLoRA) above 5K tok/sample.

• Instruction-tuning datasets span the largest dynamic range in size but a relatively narrow tokens-per-sample band (\(\sim\)​1–3K). The handful of high-density outliers (MMDU, MapQA, MathWriting) correspond to multi-turn or multi-image conversations.

1.0.5.3 Data Sources and Licensing

We source our DCVLM pool from four different data-types, each with a distinct sourcing strategy. Image-caption pairs are primarily sourced from web-crawled image-alt-text corpora (DataComp-1B [5], ReLAION-2B [14]) and synthetic/human-curated caption datasets (PixMo-Cap [17], ShareGPT-4o [167], GRIT [168]). Multimodal documents come from web-crawled interleaved sources (MINT-1T-HTML [71], OmniCC [73], WanJuan [72]) and curated PDF corpora (Multimodal-Textbook [74], MINT-1T-PDF [71]). Instruction-tuning data is aggregated from academic benchmarks with train splits, synthetic generation pipelines, and existing curated datasets across 8 capability categories. Text-only data combines general instruction sets (Dolly, FLAN/FLAN-v2, SlimOrca) with long-context (LongAlpaca, LongCite-45K) and code/math reasoning (Numina-Math, MetaMathQA, Glaive-Code) corpora. In [tab:dcvlm-datamix-licenses], we provide the licensing information and original sources from which we collected each sub-dataset in our DCVLM pool.

1.0.5.4 Sample visualizations

To enable quick and easy visualization of random samples from each of our 160 source datasets, we provide a visualization tool built with Streamlit here: https://dcvlm-dataset-browser.streamlit.app/. The tool enables browsing samples from datasets, categorized by data type and afforded capability (for multimodal instruction tuning datasets).

1.0.5.5 Multilingual Nature

As can be seen from [tab:dcvlm-datamix-licenses], the DCVLM pool is sourced primarily from English-centric web corpora (DataComp [5], ReLAION-2B-en [14]) and English instruction-tuning datasets, but it inherits a non-trivial multilingual tail from datasets that target other languages explicitly (e.g.Chinese instruction corpora, multilingual OCR generators) and from the natural multilingual contamination present in any large web crawl. To quantify this, we annotate every sample in the pool with two independent language detectors and take their top-1 prediction.

1.0.5.5.0.1 Annotation methodology.

For each sample we run two language detectors over the per-sample text (caption, instruction, or document content):

• Lingua³: a lightweight character-\(n\)-gram detector covering 75 languages with good coverage of European tongues but limited support for languages with non-Latin scripts.

• NLLB-218 [169]: the multilingual classifier shipped with the No-Language-Left-Behind translation model, covering 218 languages and substantially better than Lingua on Cyrillic, Devanagari, and Southeast-Asian scripts.

Each annotation stores the top-3 language hypotheses with calibrated scores. For the analysis below we report only the top-1 prediction from each detector. Both detectors operate on text with images stripped, so they describe the linguistic distribution of the textual content of the pool, not of any visual content.

1.0.5.5.0.2 Pool-wide distribution.

Figure 9 shows the top-15 languages by sample count under each detector, over our small pool (each dataset is capped at 100,000 samples as a representative subset). We notice that the distribution is heavily tailed—English alone accounts for \(91.1\%\) (Lingua) / \(92.8\%\) (NLLB) of all samples. The single largest non-English language is Chinese at \(3.9\%\) / \(3.5\%\). Beyond Chinese, the remaining languages are spread thinly: French, German, Japanese, Korean, and Russian each contribute \(0.2\%\)–\(0.8\%\), and the rest of the long tail (a further 70-110 languages depending on detector) sums to under \(1\%\). Despite the long tail Lingua and NLLB largely agree—the two detectors return the same top-1 language on \(96.6\%\) of samples—so our statistics are robust to the choice of detector.

1.0.5.5.0.3 Where does the non-English content come from?

Roughly half of the pool’s non-English samples trace to a small number of datasets that were either built for a specific non-English language or rely on synthetic multilingual generation:

• Chinese: firefly (a general-purpose Chinese instruction corpus, \(\sim\)​91% non-English, contributing \(\sim\)​33% of all Chinese samples), wanjuan (curated Chinese multimodal interleaved documents, \(\sim\)​84% non-English), internvl_sa1b_caption (Chinese-translated SA-1B captions), and magpie_qwen2_pro (Chinese instruction synthesis).

• French / German / Russian: dominated by flanv2, whose translation tasks bundle large numbers of parallel corpora—alone responsible for \(72.4\%\) of the pool’s French, \(78.3\%\) of its German, and \(64.6\%\) of its Russian samples.

• Japanese / Korean: synthdog, a synthetic OCR-style image-text generator, supplies \(76\%\) of Japanese and \(88\%\) of Korean samples; the remaining mass comes from eaten, mavis_geometry, and small instruction sets such as kopen-hq-hermes-25-60K (\(100\%\) non-English).

• Vietnamese / Arabic / Indonesian: largely supplied by sea-vl, a Southeast-Asian vision-language dataset (\(93\%\) Vietnamese, with smaller contributions to Arabic and Indonesian).

The remaining non-English content is the natural multilingual residue of the web crawls—although relaion-2b-en and datacomp_1b are English-filtered upstream, \(\sim\)​5% of their captions are still flagged as non-English by NLLB, predominantly European languages (German, French, Italian, Dutch). This also corroborates the findings from [137] and [170].

1.0.5.5.0.4 Implications.

The pool is overwhelmingly English by design: this matches the practice of most recent open VLM pretraining corpus we are aware of and reflects the asymmetry of available web data [171], [172]. The Chinese sub-pool is large enough to plausibly support Chinese-capable downstream models, as we show in our experiments in 0.0.4. Languages outside the top five are too sparse to sustain meaningful pretraining signal on their own—this is a limitation of the underlying source data rather than of DCVLM per se, and addressing it would require deliberate ingestion of non-English vision-language datasets.

1.0.5.5.0.5 How much does English data drive downstream performance?

To test how dominant the English pretraining data is in our pool, we run a set of language ablation experiments at the medium scale (2B model, 25B tokens). Using the per-sample NLLB top-1 language annotations described above, we partition the pool into English and non-English samples and construct three training sets, holding the data mixture fixed: (i) All languages, our default pool with no language-based selection; (ii) English-only, which retains only samples whose detected language is English; and (iii) Non-English-only, which retains only the complementary non-English tail. Since English accounts for \({\sim}92\%\) of the pool (9), the English-only set is close in size to the full pool, whereas the non-English-only set is far smaller and is consequently seen with more repetitions to fill the same 25B-token budget. We report core-set results in 9.

Three observations stand out. First, English data carries almost all of the signal: the English-only model trails the full pool by only \(0.6\)pp, despite discarding \({\sim}8\%\) of the pool. Second, the non-English tail alone is insufficient: training exclusively on non-English samples drops performance by \(3.0\)pp relative to the full pool, with the largest regressions on General (\(-5.3\)pp), OCR (\(-3.3\)pp), and Text-Only (\(-2.8\)pp). Third, and most notably, multilingual benchmark performance is essentially flat across all three pools (\(39.8\) / \(39.9\) / \(39.1\)): neither removing non-English data nor training solely on it meaningfully moves the Multilingual score.

We caution, however, that two confounders complicate the interpretation of this experiment: (i) since English makes up \({\sim}92\%\) of the pool, the non-English-only run draws from far fewer unique samples and incurs substantially more repetition than the English-only run to fill the same token budget, so its performance drop cannot be cleanly attributed to language as opposed to repetition; and (ii) our per-sample language annotations are noisy—especially for multilingual samples, where the detector is biased toward the head (high-resource) languages—making the English/non-English partition imperfect.

1.0.6 Evaluation Suite Details↩︎

In this section, we highlight our protocol to select and categorize benchmarks for evaluating models. We start with a candidate pool of 65 benchmarks, which we categorize according to the majority consensus of previous work: Cambrian-1 [28], InternVL2.5 & InternVL-3 [1], [2], MM-1 [35], Qwen2-VL & Qwen2.5-VL [46], [88] into 9 distinct categories:

• General Understanding: this category includes established multimodal benchmarks, as well as comprehensive, all-round benchmarks assessing several capabilities at once.

• Knowledge-Centric benchmarks: these test factual, scientific and/or educational knowledge.

• OCR & Charts, testing the ability of models to read and understand visually rendered characters, diagrams, layout, and PDFs.

• Vision-Centric benchmarks, requiring models to perceive spatially, perform fine-grained visual processing such as counting, and compare items across multiple images.

• Multilingual benchmarks, which assess general knowledge in less common languages.

• Text-Only benchmarks, testing whether the unimodal, language-only capabilities of the language model are preserved;

• Safety benchmarks, which measure the ability of models to detect and, potentially, refuse to respond to, unsafe queries (e.g., violent, harmful and/or explicit content).

• Hallucination benchmarks, stress-testing the capabilities of models to assess evident facts such as the existence of explicit objects in images;

• Reasoning benchmarks, with a math- and logic-centric focus (including visual tasks such as rendered geometry problems and language-only tasks such as GSM-8K [173]).

Prior works are far from unanimous in how they organize these benchmarks: the same task is frequently placed in different categories by different works. For instance, AI2D is treated as a Knowledge benchmark by Cambrian-1 but as an OCR/document task by Qwen2-VL and InternVL, and MMMU is variously labelled as Knowledge, Mathematics, or Reasoning. To obtain a single consistent taxonomy, we collect the category each prior work assigns to every benchmark and resolve disagreements by majority vote wherever a consensus exists, adjudicating the remaining ambiguous cases ourselves. 1 reports, for each benchmark, the verbatim category given by every prior work alongside our final DCVLM assignment. 11 then lists the resulting mapping from each DCVLM category to its benchmarks. Whenever possible, we use benchmark implementations in VLMEvalKit [39] and OpenCompass [174], which we integrate into the toolkit we plan to release.

Monotonicity selection. We filter 12 out of 65 benchmarks because they do not pass our first selection criterion: monotonicity. To ensure reliable evaluation of data-centric decisions across scales, we filter our initial benchmark pool to retain only tasks that exhibit consistent improvement with scale. Concretely, we train models at two scales (small and medium) for three different random seeds, for a total of six runs. We evaluate all resulting checkpoints and compute the per-scale average. Since DCVLM is a scale-centric benchmark, we rule out benchmarks for which performance does not monotonically improve from small to medium scale, as such benchmarks would risk obscuring the signal attributable to data curation choices. This yields a curated subset of benchmarks (shown in 10 as points above the diagonal) that exhibit reliable scale-monotonic behavior.

Stability selection. In parallel to assessing monotonicity, we also search for stable evaluations. For this reason, we further inspect the seed variance of benchmarks using the same six runs. We perform stability filtering in two steps: globally and locally within each category.

To provide an initial global picture, we display the standard deviation for all the initial 65 benchmarks at both scales in 11. We find striking instability with POPE [175] compared to all other benchmarks (\(16\%\) seed variance over three runs at the small scale), and we therefore discard it from our pool. This decreases the total from 53 to 52 remaining benchmarks, which we jointly call our Extended set.

Extended, Core, and Validation suites. Since we are primarily interested in evaluating pretrained models (according to our definition in 0.0.1), we create a smaller subset of tasks by excluding verticals that are mostly of interest for post-training strategies than they are for pretraining [209], [231]: Safety, Hallucination, and Reasoning benchmarks. We then apply a second iteration of stability filtering that removes the most unreliable tasks per category according to the average of their standard deviation across small and medium scale runs. This corresponds to removing: (i) MME [177] from the General category, (ii) TextVQA and DocVQA from the OCR & Charts category, (iii) C3 and HellaSwag from the Text-Only category, (iv) MMVP and MantisEval from the Vision-Centric category. We do not remove additional tasks from the Knowledge-Centric and the Multilingual categories, as they already contain a more modest number of tasks compared to other categories. This additional selection yields a 33-task suite, which we call our Core set. As highlighted in the main body, we use this set to report all major results. We additionally create a 13-task Validation set, which we use for faster experimentation, by picking the most established subset of benchmarks per category.

To enumerate benchmarks and illustrate tier organization, we list Extended benchmarks in 12, Core in 13, and Validation in 14.

1.0.6.1 Issues with Grounding Benchmarks

We initially considered including grounding and dense recognition benchmarks, such as RefCOCO [232], RefCOCO+ [233], RefCOCOg [234], and related COCO-derived tasks, in the DCVLM evaluation suite. We ultimately excluded them, not because grounding is unimportant, but because the standard grounding benchmarks introduce unusually severe contamination and split ambiguities. These issues make it difficult to interpret them as reliable held-out evaluations in a data-centric benchmark at the scale of DCVLM.

The main issue is that many grounding, VQA, and captioning benchmarks are built on overlapping subsets of MS COCO images [235]. The RefCOCO family uses COCO images, as do COCO Captions [182], [236], OK-VQA [237], VSR [238], and TolokaVQA [239]. As a result, the same underlying image can appear as a training example for one task and as an evaluation example for another, possibly with a different annotation format, e.g., a caption, a VQA pair, or a referring expression. This makes image-level decontamination across the full training pool difficult, especially when downstream datasets rename files, construct custom splits, or mix COCO2014 and COCO2017 images in non-standard ways.

The RefCOCO variants further complicate this picture because they are not independent held-out datasets. RefCOCO and RefCOCO+ were collected through ReferItGame and commonly use the unc split, where train/test separation is performed at the image level [232], [233]. RefCOCOg was collected through AMT and commonly uses the Google split, where the split is defined at the object or annotation level [234]. The variants also differ in their linguistic supervision: RefCOCO contains relatively short expressions, RefCOCOg contains longer and more detailed expressions, and RefCOCO+ restricts expressions to appearance-based descriptions by excluding certain spatial phrases such as “person on the right” [233], [234]. Thus, a model trained on one variant may have already seen the same image-object pair that appears in another variant, only paired with a different referring expression.

This concern is amplified by common VLM pretraining practices. In several open training recipes, grounding data is not treated as a single isolated benchmark split, but appears to group annotations from RefCOCO and RefCOCOg under the same image ⁴. In such cases, evaluation on a nominally separate RefCOCO-style split may partially measure familiarity with recurring COCO image-object pairs rather than genuine grounding generalization.

For these reasons, we exclude grounding benchmarks from the main DCVLM evaluation suite. Including them would make it difficult to attribute performance differences cleanly to data mixing, filtering and other curation choices, since gains could arise from uncontrolled overlap between COCO-derived training and evaluation data.

1.0.7 Train-Test Decontamination↩︎

To ensure that conclusions from our data-centric experiments are not confounded by train-test overlap, we perform strict two-way decontamination of all training sources against the DCVLM-Extended evaluation suite. We apply separate procedures for multimodal and text-only samples.

1.0.7.1 Image-Based Decontamination

For the multimodal portion of our pool, we decontaminate at the image level using the self-supervised copy-detection descriptor SSCD model [80] (ResNet-50 backbone). We embed every image in both the training pool and the DCVLM-Extended evaluation suite under an identical transform: we resize the shorter edge to \(288\)px while preserving aspect ratio, then apply ImageNet normalization. We deliberately favor fidelity over speed here [240]—the \(288\)px input and the absence of any center crop avoid truncating charts, tables, and documents.

Matching procedure. We build a single exact (brute-force, flat \(L_2\)) FAISS index [241] over all evaluation-image descriptors. For each training image we retrieve its top-1 nearest evaluation image and convert the squared-\(L_2\) distance \(d\) to a cosine similarity \(s = 1 - d/2\) (valid because SSCD descriptors are unit-normalized). A training sample is discarded if any of its images attains \(s \geq 0.75\) to any evaluation image. For multi-image samples (e.g.multimodal documents) we thus aggregate by the maximum similarity over the sample’s images. Because the evaluation index is small, exact search resolves millions of queries per second on a single GPU, so we use no approximate indexing. Our pipeline relies on a distributed implementation that builds on top of exact FAISS [241] search.

Choosing the threshold. We use a similarity threshold of \(0.75\), deliberately more conservative than the \(0.95\) adopted by FineVision [44]. 13 shows representative train/test matches across the SSCD similarity scale, drawn from four pool sources that together span natural photographs (ViQuAE, DataComp-1B), diagrams (AI2D), and mixed document/chart-style imagery (MMDU). Only near-pixel-identical copies score \(\geq 0.95\). The \(0.75\)–\(0.95\) range is instead dominated by genuine duplicates that have undergone benign transformations—diagrams with labels stripped or re-typeset, re-crops with added text overlays, and re-scans of the same source image. A \(0.95\) threshold retains all of these, i.e.it admits many false negatives. We found this effect to be systematic for non-natural image domains—charts, tables, documents, and UI screenshots—where genuine train/test near-duplicates receive lower SSCD scores than natural photographs do. Our stricter \(0.75\) cutoff removes this band at the cost of some false positives: we knowingly discard a modest amount of clean training data in exchange for substantially stronger decontamination guarantees. Below \(0.75\), matches are visually or semantically similar but distinct images (different photographs of the same landmark, same-layout documents with different content) and are correctly retained. Note that even below the \(0.75\) threshold, there are indeed some false negatives, however we found \(0.75\) to be a stable threshold that balanced both precision and recall, enabling us to perform decontamination in a faithful manner while minimizing discard rates.

Comparison to FineVision’s threshold. 12 quantifies the gap between the two protocols: for each of the four sources of 13 we plot the fraction of pool images removed as a function of the threshold. The region between \(0.75\) and \(0.95\) is overlap that FineVision’s protocol leaves in the pool—for AI2D, the \(0.75\) threshold removes \(16.9\%\) of pool images versus only \(10.9\%\) at \(0.95\), i.e.over a third of the detected overlap survives the lenient threshold.

Removal rates on our DCVLM pool. The fraction of training images flagged varies sharply by source and is concentrated in datasets whose training and evaluation splits are drawn from the same underlying image distribution. At the \(0.75\) threshold, notable per-source removal rates on our DCVLM-pool large scale include ScienceQA (\(60.4\%\)), ChartQA (\(25.6\%\)), AI2D (\(25.0\%\)), OCR-VQA (\(2.8\%\)), and InfoVQA (\(100\%\), as its training and evaluation images entirely coincide).

1.0.7.2 Text-Based Decontamination

For the text-only portion of our pool, we decontaminate at the prompt/question level, following the protocol of Tulu-3 [242]: for each training sample we concatenate its human turns into a single query string and match it against the prompts of every text-only evaluation. We decontaminate against all text-only benchmarks in the InternVL-3 evaluation pool (\(16\) benchmarks, namely MMLU, CMMLU, C-Eval, GaoKao, TriviaQA, NaturalQuestions, RACE, WinoGrande, HellaSwag, BBH, GSM8K, MATH, HumanEval, MBPP, MBPP-CN, TheoremQA). We use a two-stage MinHash + exact-substring pipeline, detailed below. 14 15 justify our hyperparameter choices, and 15 shows representative detected overlaps.

Matching procedure. We tokenize text with the Tulu-3 regular-expression tokenizer (lowercasing and splitting on whitespace and punctuation) and represent each sample by the set of its word-level \(5\)-grams.⁵ We summarize each set with a \(128\)-permutation MinHash signature [81] and estimate the Jaccard similarity between a training sample and its nearest evaluation sample by exact (brute-force) signature comparison. Because the evaluation index is small (\({\sim}86\)k samples), exact top-1 search is fast enough that we do not require LSH-based approximate matching.

Two-stage filtering. In the first stage, we discard every training sample whose top-1 Jaccard similarity to any evaluation prompt exceeds \(0.55\). In the second stage, training samples falling in the ambiguous band (\(0.3 \leq \text{Jaccard} < 0.55\)) are subjected to a bi-directional exact-substring check: a sample is discarded if either the training query is contained verbatim in the matched evaluation prompt or vice versa. The second stage recovers true positives whose Jaccard score is diluted below \(0.55\)—for example, a short evaluation question embedded inside a longer training prompt. This combination balances recall (fuzzy near-duplicates caught by MinHash) with precision (the substring check filters out spurious matches in the intermediate band). Training samples too short to form a single \(5\)-gram are always retained.

Choosing the hyperparameters. We sweep the three free parameters of the MinHash stage and report per-dataset match rates over \(10\) instruction datasets in 14. (i) Signature size: increasing the number of permutations from \(128\) to \(1024\) leaves match rates essentially unchanged (14 (a)), so we use \(128\) for faster processing. (ii) \(n\)-gram length: \(5\)-grams detect overlaps at meaningful rates, whereas \(8\)- and \(13\)-grams detect almost nothing (14 (c)), because many evaluation prompts are shorter than \(8\) (let alone \(13\)) tokens and therefore never form a matching \(n\)-gram. We therefore use \(5\)-grams for higher recall. (iii) Threshold: across all datasets the match rate falls steeply with the similarity threshold and is already below \({\sim}0.1\%\) by \(0.5\) (14 (b)). At our operating point of \(0.55\) fewer than \(0.02\%\) of text samples are removed, so a conservative threshold costs almost no data.

Selecting the threshold via human annotation. The match-rate sweep does not by itself tell us which threshold separates genuine contamination from coincidental overlap. To answer this quantitatively, we ran a small human-annotation study: we drew candidate matches spanning all similarity bins from the \(10\) datasets above, and seven of the authors independently labeled each candidate as a true overlap (the two prompts ask the same question) or a false positive. Aggregating the true-positive (TPR) and false-positive (FPR) rates per similarity bin (15), we find that below \({\sim}0.55\) the FPR is high (most matches are spurious, e.g.questions that share a template but have different answers), while the TPR rises steeply just above it and the two curves cross around \(0.55\)–\(0.6\). We therefore adopt \(0.55\) as the first-stage threshold, and rely on the second-stage substring check to recover the genuine matches that fall just below it.

1.0.7.3 Overall removal rates

16 reports per-dataset removal rates of the full decontamination protocol (image- and text-based combined) over our medium DCVLM-pool as a representative. Removal is concentrated in datasets whose training and evaluation splits are drawn from the same underlying distribution—InfoVQA loses \(100\%\) of its samples (its training and evaluation images almost entirely coincide), ScienceQA \(66.4\%\), TabMWP \(63.4\%\), FigureQA \(56.6\%\), ChartQA \(41.7\%\), and AI2D \(21.9\%\). At the pool level, however, decontamination is cheap—only \(0.29\%\) of all training samples are removed. Strict decontamination therefore costs almost no data while removing precisely the samples most likely to inflate benchmark scores.

1.0.8 Additional Experiments and Details↩︎

1.0.8.1 Filtering rarely helps

17 reports the filtering results at the small scale (1B model, 6.25B tokens), complementing the medium-scale results in 0.0.4.1. The conclusions are fully consistent: local filtering uniformly fails to improve over the no-filter baseline. The only positive outcomes still arise from global CLIP-score filtering (SigLIP-2 \(+1.1\)pp and OpenAI-CLIP \(+0.6\)pp), in line with 3, yet, once again, these gains are lower than what one would expect from prior work.

Consistently, global filtering and local filtering produce different results also at this scale. Overall, these results reinforce the central finding of 0.0.4.1 that filtering is not a reliable curation strategy, and lead to hypotheses that observed performance variations might be primarily driven by the underlying mixture changes.

Additional filtering experiments. For completeness, we report the full benchmarking we conducted with a variety of additional configurations for the filters in 0.0.4.1. Specifically, we tested several local filtering variants of the common filters presented in 0.0.4.1:

• for CLIP-score, we experiment with filtering not only image-caption pairs alone, but also multimodal documents and instruction tuning data. For the former, we score a document according to the average CLIP-score of all its adjacent image-text snippets. For the latter, we employ image-to-question similarity. In case of multi-image examples, we use an identical strategy to multimodal documents and score the closest image to the question in the conversation history. For multi-turn samples, we average across all adjacent pairs as we do for multimodal documents. We use the same three models described in 0.0.4.1

• for text quality classifiers, we additionally experiment with filtering only the language portion of our pool. This experiment is motivated by the hypothesis that text-only sources might be more in-distribution with respect to the classifiers themselves. Identically to CLIP-score, we report all three classifiers of the main body.

• For Text-Only Perplexity, Multimodal Perplexity, and Conditional Mutual Information filters we experiment with applying them to all multimodal data types jointly (image-caption pairs, instruction-tuning data, multimodal documents), as well as subset combinations.

We additionally experimented with basic filtering strategies:

• Length heuristics, that remove the shortest and the longest samples (removes bottom-10% shortest and top-90% longest at both scales). We experiment with several interpretations of “length”: (i) the count of all tokens, (ii) the count of answer tokens, on top of which the loss is ultimately computed during training; and (iii) the count of language tokens (i.e., question + answer tokens);

• Image Quality Assessment (IQA) models, to remove low-quality images. We test with the ARNIQA [125] suite of image quality models. Specifically, we use the regressors trained on the Kadid-10k [243] and Tid-2013 [244] datasets, since they are one of the highest performing image quality assessment classifiers on standard IQA benchmarks.

Except for length heuristics, which use the same (bottom-10%, top-90%) percentiles on both scales, all experiments are conducted with an aggressive top-10% filtering threshold on the small scale and a more lenient top-40% threshold on the medium scale, following 0.0.4.1.

Given the large number of experiments, we opted for faster iterations on our 13-task Validation suite. As anticipated in the main body, no configuration provides significant gains.

For completeness, we report Validation results for global filtering in 18 as well (we did not test all variants here, but rather only those presented in 3).

1.0.8.2 Data formatting

We use the Chat Markup Language (ChatML) format [245] for training, which assigns a role (“system”, “user”, or “assistant”) to all messages in a conversation. ChatML is a structured format explicitly separating conversation roles (e.g., system, user, assistant) and demarcating message boundaries using specific tokens (e.g., <|im_start|> and <|im_end|>). Text-only and instruction-tuning data typically come in this format already, while image-caption pairs can be easily formatted using templates such as “Describe the image” to serve as user messages. However, there is no consensus on how to format multimodal documents, with prior works exploring contrasting methods [71], [73], [122], [246].

r0.27

We thus conduct an experiment at the small scale to compare two strategies: in-context, which uses the last text chunk as assistant response, and concatenates everything except the last text chunk into the user message vs multi-turn, which interleaves images and text chunks into multiple turns, so that images are user messages and all text chunks are assistant responses To ensure reliable signal, we use a uniform mixture across data types (\(25\)% for all data types). From 19, we find multi-turn formatting superior to in-context formatting. We thus adopt multi-turn formatting for all our experiments.

1.0.8.3 Diminishing returns of downstream filtering

This section provides further details on the upstream vs.downstream filtering experiment in 0.0.4.1.

We start from the base DCVLM small-pool of 160 source datasets. By virtue of our collection procedure (0.0.3.1), this pool has already been “upstream-filtered” by the source dataset providers, so we treat it as the “Upstream-filtered=100%” setting. It follows our pool’s base mixture ratio: \(75\)% image-caption pairs, \(18\)% text-only, \(4\)% multimodal documents, and \(3\)% multimodal instruction-tuning data. The datacomp_1b subset alone accounts for roughly \(47\)% of the image-caption portion.

To construct the other two settings, we progressively replace pre-filtered image-caption data with raw, unfiltered pairs crawled directly from the released CommonPool URLs:

• Upstream-filtered=65%: replace pre-filtered datacomp_1b with its unfiltered counterpart, yielding \(0.47 \times 75 \approx 35\)% unfiltered data.

• Upstream-filtered=25%: replace the entire image-caption portion with raw unfiltered datacomp_1b, yielding \(75\)% unfiltered data.

For each setting, we apply CLIP-score downstream filtering on image-caption pairs using OpenAI-CLIP-ViT-L/14 [95], and train small-scale models on the resulting datasets.

The gain from downstream filtering shrinks monotonically with the level of upstream curation: \(+2.4\)pp at Upstream-filtered=25%, \(+1.3\)pp at \(65\)%, and only \(+0.6\)pp at \(100\)%. The first result confirms that filtering does help on genuinely noisy data, but the trend establishes that additional filtering on top of already-curated data operates in a regime of diminishing returns.

1.0.8.4 Temperature-scaled sampling

Several of our most task-relevant datasets are already present in the pretraining pool: the training splits of ScienceQA, AI2D, ChartQA, InfoVQA and OCR-VQA all live in our instruction-tuning sources. It is therefore natural to ask why the corresponding benchmarks do not benefit more, and a simple hypothesis presents itself—these datasets may be severely undersampled during training. By default we sample sources within each data type in proportion to their size, \(p(d) \propto \mathrm{len}(d)\) (where \(\mathrm{len}(d)\) is the number of samples \(d\) contributes to the pool), so a handful of large web-scale sources can dominate the mixture and drown out small but high-quality or diverse datasets.

Prior work tackles this imbalance in two ways: by hand-setting mixture ratios based on intuition (e.g.the Nemotron [64] and OLMo-2 [61] series, and FineWeb [8]), or by principled downweighting of large sources—most notably the square-root sampling used in the Molmo line of work [17], [68]. We study this finer, within-type sampling in a principled manner via temperature-scaled sampling, \[p(d) \;\propto\; \mathrm{len}(d)^{1/T}\,,\] where \(T=1\) recovers length-proportional sampling, \(T=2\) corresponds exactly to the square-root sampling of Molmo, and \(T\rightarrow\infty\) approaches uniform sampling over datasets (flattening, which upweights small sources). We additionally explore \(T<1\), which sharpens the distribution toward the largest sources. We apply the sweep \(T \in \{0.5, 0.8, 1.0, 2.0, 4.0\}\) to our balanced mixture (\(40\)% image-caption, \(5\)% multimodal documents, \(15\)% text, \(40\)% instruction-tuning, see 0.0.4.2), holding all other settings fixed, and report results on our 13-task Validation suite. We run these experiments at the small scale of our benchmark.

As shown in 18, the undersampling hypothesis does not hold. The default \(T=1\) is already near-optimal, and flattening the distribution—which upweights exactly the small, task-relevant datasets we suspected of being undersampled—consistently hurts: square-root sampling (\(T=2\)) drops to \(42.4\)% (\(-1.7\)pp) and the near-uniform \(T=4\) falls to \(40.0\)% (\(-4.1\)pp), below even the FineVision baseline. Sharpening offers no benefit either (\(T=0.8\): \(43.6\)%, \(-0.5\)pp; \(T=0.5\): \(43.2\)%, \(-0.9\)pp). We hypothesize the contradicting results between ours and the Molmo series are due to the differing mixture ratios and pool composition—we believe there might be some non-trivial effects between the broader data-type mixing ratios and the within-data-type source sampling, which would be an interesting exploration for future work. Given our results, we hence retain length-proportional (\(T=1\)) sampling throughout—noting that the cross-type mixing ratios of 0.0.4.2 are a far more impactful lever than the within-type temperature.

1.0.8.5 Synthetic recaptioning

Replacing noisy web-crawled alt-text with synthetic captions is a well-established lever for improving contrastive (CLIP-style) pretraining [12], [133], [134]. We test whether the same holds for generative VLM pretraining. We hold the images and their training order fixed and vary only the caption text on the DataComp [5] subset of our image-caption pool, comparing three variants:

• Alt-text (original): the raw web alt-text from the original DataComp-1B image-caption pairs.

• Synthetic short: concise captions generated by Qwen2-VL-7B [88], conditioned on the image, its alt-text, and a set of detected concepts [12].

• Synthetic spatial: the short captions further augmented with explicit 2D/3D spatial-relationship descriptions, to probe whether richer, layout-aware captions help the more vision-centric benchmarks.

We source all caption variants using the released datasets of [12]. We run these experiments on a mixture of \(65\)% image-caption, \(5\)% multimodal documents, \(15\)% text, and \(15\)% instruction-tuning data, at the small scale of our benchmark. Crucially, the caption intervention is applied only to DataComp-1B, not to the entire image-caption portion: since DataComp-1B accounts for roughly \(47\)% of the image-caption data, the captions are rewritten for only \(\approx\!47\%\times65\% \approx 30\%\) of the overall training mixture, while every other source (including image-caption datasets that already come with synthetic captions) retains its original text. The images, mixture ratios, and data order are identical across the three runs, so any difference is attributable solely to DataComp-1B captions. As with other experiments, we report on the 13-task Validation suite.

20 shows that neither synthetic captioning scheme provides a meaningful improvement: original alt-text reaches \(44.1\)% average, synthetic-short \(43.2\)% (\(-0.9\)pp), and synthetic-spatial \(44.0\)% (\(-0.1\)pp). Per-category, the changes are small and inconsistent in sign—spatial captions help General (\(+0.7\)) and MTL (\(+1.2\)) but hurt OCR (\(-0.9\)) and Vision (\(-0.9\)), netting essentially zero overall. This contrasts with the consistent gains recaptioning brings to CLIP training, and we hypothesize that the difference is in our pretraining setup: the instruction-tuning portion of the mixture already supplies abundant high-quality, densely-described image-text pairs, so perhaps the marginal value of rewriting the image-caption captions is low. In other words, at the VLM pretraining stage the text content of image-caption pairs matters far less than their proportion in the overall mixture (0.0.4.2)—given the substantial cost of running a 7B captioner over the full pool, we use the original alt-text throughout.

We caveat that this null result is specific to our setup, and a different configuration could plausibly surface gains. Two factors likely dampen the effect here. First, the intervention touches only \(\approx\!30\%\) of the training mixture (the DataComp subset)—recaptioning a larger fraction of the pool, or starting from a pool that is entirely alt-text, may tell a different story. Second, and more importantly, our image-caption pool already contains several sources beyond DataComp-1B and ReLAION that come with high-quality synthetic or re-written captions. In their presence, rewriting the captions of one additional subset has little marginal effect—much as additional filtering yields diminishing returns once a pool is already well-curated (1.0.8.3). The benefit of recaptioning may thus be largely realized already, and washed out, by the synthetic captions present elsewhere in the pool.

1.0.8.6 Online Filtering

Prior data curation benchmarks such as DataComp [5] and DCLM [6] employ offline resharding: given a raw pool and a set of metadata annotations, the entire dataset is filtered according to a predicate and written to disk as a new, smaller dataset, which is then used for training. While this makes training efficient—the dataloader never encounters rejected samples—it incurs substantial storage costs, particularly when sweeping over filtering configurations. For example, determining the optimal CLIP-score threshold requires training runs at several percentiles, each of which produces a separate copy of the filtered pool on disk. With dozens of filters and multiple threshold values, storage requirements can grow by an order of magnitude.

Both DataComp and DCLM cite the computational cost of online filtering—applying predicate conditions during training—as the reason for preferring offline resharding. We revisit this assumption and find that, in our setting, online filtering matches the throughput of unfiltered training, effectively eliminating the storage bottleneck. We describe our implementation in detail below.

1.0.8.6.1 Metadata Annotation

Each sample in our pool is annotated with scores from every filter we consider. Concretely, for a given filter \(f\) (e.g., oai-clip-vit-b-32), we run a batch inference job over the entire pool that computes a scalar quality score \(s_f(x) \in \mathbb{R}\) for each sample \(x\). These scores are stored as sidecar metadata files alongside the original WebDataset shards: for each shard xxxxx.tar, we produce a corresponding xxxxx.json file containing a dictionary mapping sample keys to their scores across all filters. A single sample’s metadata entry looks like:

  "sample_00042_017": {
      ...
      "clip_score_clip_vitl14_224_standard": 31.984,
      "clip_score_siglip2_b16_224_standard": 5.421,
      "text_quality_dclmbaseline_fasttext_score": 0.091,
      "text_quality_nvidia_edu_mixtral_score": 0.122,
      "iqa_kadid10k": 0.574
      ...
  }

This annotation step is performed once per pool and is fully parallelizable across shards. Adding a new filter (e.g., a new CLIP model variant) requires scoring the entire pool only with the new filter; existing annotations are preserved and do not need to be recomputed. In total, our small pool ships with over 100 precomputed metadata annotations per sample.

1.0.8.6.2 Threshold Pre-Computation

Given a filter \(f\) and a desired retention rate \(p\) (e.g., “keep the top 60%”), we need a threshold \(\tau_{f,d}\) for each sub-dataset \(d\) (in the local filtering setting) or a single global threshold \(\tau_f\) (in the global filtering setting). We pre-compute these thresholds for all filters at percentile increments of 5 (i.e., \(p \in \{0.1, 0.15, 0.2, 0.25, \ldots, 0.9, 0.95\}\)) using torch.quantile() with linear interpolation over the full score distribution of each sub-dataset (for local filtering) or the full merged score distribution across all sub-datasets (for global filtering). The output is a lightweight JSON lookup table that maps each filter to the score value corresponding to each percentile, which is then used as the filtering threshold:

    filter_name: {
      "0.10": 0.052,   # 10th percentile
      "0.15": 0.095,   # 15th percentile
      "0.20": 0.118,   # 20th percentile
      ...
      "0.90": 0.891    # 90th percentile
      "0.95": 0.945    # 95th percentile
    }

1.0.8.6.3 Runtime Integration

During training, the dataloader reads samples from WebDataset shards as usual, but before a sample is yielded to the training loop, it passes through a lightweight predicate function registered via wds.select(). The predicate performs the following steps for each sample:

1. Look up the sample’s pre-computed filter scores from the sidecar metadata (loaded into a hash map at initialization).

2. For each active filter, compare the sample’s score against the corresponding threshold for that particular sub-dataset.

3. If the score falls below the threshold (i.e., the sample is in the rejected percentile), return False; the sample is silently skipped and the dataloader advances to the next sample in the shard.

4. If all active filters pass, return True; the sample proceeds to tokenization, packing, and batching as normal.

For multi-filter configurations (e.g., applying both a CLIP-score filter and a text-quality filter simultaneously), the predicates are composed with logical AND: a sample must pass all active filters to be retained. Since the predicate evaluation involves only a dictionary lookup and a scalar comparison per filter, the per-sample cost is on the order of microseconds—negligible compared to the millisecond-scale cost of image decoding, tokenization, and GPU transfer.

Switching between filtering configurations requires changing only the filter names and percentile values in a configuration file; no data needs to be resharded or copied. This enables rapid iteration: practitioners can launch dozens of filtering experiments against the same on-disk pool, varying filters, thresholds, and combinations, without any storage overhead beyond the original pool and its sidecar metadata.

1.0.8.6.4 Overhead Measurement

A natural concern is that high rejection rates will cause the dataloader to “spin”—reading and discarding many samples before finding one that passes the filter, thereby starving the GPUs. We benchmark this directly on 2 nodes (8 A100 GPUs), with per-device batch size 4 and effective sequence length 8192, across two representative datasets with different characteristics: ShareGPT4V (high-quality synthetic captions, relatively uniform scores) and MINT-HTML (web-crawled multimodal documents, highly variable scores).

19 summarizes the results across three experimental axes.

First, when varying the rejection percentile of a single filter from 0% to 90% (19, left), runtime at 50M tokens remains within 3.5% of the unfiltered baseline, and this spread shrinks to just 0.3% at 500M tokens (19, center)—well within normal run-to-run variance. The lack of degradation even at 90% rejection (where 9 out of 10 samples are discarded) can be attributed to two factors: (i) the predicate check is cheap relative to downstream processing, allowing the dataloader to skip rejected samples faster than the GPU can consume accepted ones; and (ii) our models are more compute-bound than the CLIP models trained in DataComp, since every sample passes through both a vision encoder and a full language model—this leaves more headroom in the data-loading pipeline, making it naturally tolerant of the additional filtering overhead.

Second, when scaling the number of simultaneous filters at a fixed 80% rejection rate per filter (19, right), applying 1–2 filters actually runs faster than the unfiltered baseline (\(0.98\times\)), likely because rejected samples bypass the more expensive downstream stages (image decoding, pixel-shuffle, tokenization, packing). Even with 3 concurrent filters—the most aggressive configuration we tested, where the compound rejection rate exceeds 99%—the overhead is only \(1.10\times\) the unfiltered baseline.

These results demonstrate that our online filtering is a practical and storage-efficient alternative to offline resharding for VLM pretraining at our scale, enabling rapid iteration over filtering configurations without materializing each filtered subset on disk.

1.0.9 Fine-grained sweep of mixture optimization experiments↩︎

In 0.0.4.2 of the main paper, we conducted a full scaling grid across \(3\) model sizes (1B, 2B, 4B), \(3\) token budgets (6.25B, 12.5B, 25B) and \(3\) mixing ratios (Instruction-heavy, Balanced, Caption-heavy). Here, we aim to do a finer-grained sweep over \(9\) total mixtures. To that end, we conduct a dense line search between the orpotion of image-caption data and instruction-tuning data, which means that we gradually decrease the proportion of crawled Image-Text data in favour of Instruction Tuning data. We keep the multimodal docs and text proportion fixed at \(5\)% and \(15\)% respectively, and only vary the image-caption to instruction-tuning data proportion from \(65\)% captioning-\(15\)% instruction to \(10\)% captioning-\(70\)% instruction.

In 0.0.4.2 of the main paper, we conducted a full scaling grid across \(3\) model sizes (1B, 2B, 4B), \(3\) token budgets (6.25B, 12.5B, 25B), and \(3\) mixing ratios (Instruction-heavy, Balanced, Caption-heavy). Here we complement that grid with a finer-grained sweep over \(9\) mixtures at two scales: small (1B model, 6.25B tokens) and medium (2B model, 25B tokens). Concretely, we conduct a dense line search between image-caption and instruction-tuning data gradually trading off image-text data for instruction-tuning data, while keeping the multimodal-document and text-only shares fixed at \(5\%\) and \(15\%\) respectively. The image-caption-to-instruction-tuning split is varied from \(65\%\) caption / \(15\%\) instruction (Caption-heavy mix in 0.0.4.2) down to \(10\%\) caption / \(70\%\) instruction (Instruction-heavy mix in 0.0.4.2) in 8 increments, yielding the \(9\) operating points reported in 20.

We find that the optimal data mixture at the small scale is the \(65\)% caption-\(15\)% instruction mix, and performance gracefully degrades as we increase the proportion of instruction-tuning data in the mixture. Conversely, the trend reverses for the medium experiment, wherein the best mixture ratio is exactly at the opposite operating point of \(10\)% caption-\(70\)% instruction. This highlights the importance of scale-aware data-curation—strategies that are optimal at one scale need not transfer to larger ones, and hence one must be prudent while making such transfer decisions, which has also been discussed in prior works [84], [85], [87].

1.0.10 Packing Implementation Details↩︎

Our pretraining corpus mixes data types whose sequence lengths span several orders of magnitude—from single image-caption pairs of a few hundred tokens to multimodal documents and multi-turn instruction conversations that approach the full context window. Training on such data without packing forces every example to be padded to a common length, wasting a large fraction of every batch on padding tokens that carry no gradient. The standard remedy is sequence packing: concatenating several short samples into one fixed-length training example so that almost every token is a real token. For vision-language models, however, packing must respect a constraint that text-only packing does not: each image is split by dynamic tiling into a variable number of tiles, and every tile expands into a fixed block of visual tokens that the vision encoder must process. A pack is therefore bounded by two budgets at once, and we pack against both.

1.0.10.1 Dual constraints

Each emitted training example must simultaneously satisfy:

• Token budget \(L = 8192\): the total number of (text \(+\) visual) tokens may not exceed the LLM context length we train at.

• Image-tile budget \(M = 24\): the total number of image tiles—summed over all images of all samples in the pack—may not exceed \(M\). This caps the vision-encoder footprint of a single forward pass and is the constraint unique to the multimodal setting.

A pack is considered complete as soon as it reaches either budget exactly, since a single text-heavy multimodal document can saturate \(L\) with few images, while a gallery of single-tile thumbnails can saturate \(M\) while leaving \(L\) far from full.

1.0.10.2 Streaming best-fit bin-packing

Samples are produced one at a time by the WebDataset mixing stream described in 1.0.8.6—already filtered, templated, and tokenized—so we never see the full dataset and cannot solve the offline (optimal) bin-packing problem. We instead maintain a pool of open buffers, each a partially-filled pack, and assign samples greedily as they arrive (21a). The pool is kept sorted in order of decreasing image-tile occupancy (fullest first), an invariant we exploit to make assignment a single linear scan. For each incoming sample \(S_i\):

1. Best-fit search. Scan the pool from fullest to emptiest and select the first buffer \(b\) for which adding \(S_i\) would violate neither budget, i.e.\(\mathrm{tiles}(b) + \mathrm{tiles}(S_i) \le M\) and \(\mathrm{tokens}(b) + \mathrm{tokens}(S_i) \le L\). Because the pool is fullest-first, this returns the most-occupied pack that can still host \(S_i\)—a best-fit choice that drives buffers toward completion rather than leaving many half-full.

2. Assignment. If such a buffer is found, \(S_i\) is concatenated into it. Otherwise \(S_i\) seeds a fresh singleton buffer. Each token of \(S_i\) is tagged with a within-pack sample index (data_index), recording which original sample it belongs to (used below to prevent cross-sample attention).

3. Re-insertion. The (possibly grown) buffer is re-inserted at the position that restores the fullest-first ordering.

A buffer is emitted as a finished training example under one of two conditions:

• Exactly full: it reaches either budget exactly (\(\mathrm{tiles}=M\) or \(\mathrm{tokens}=L\)). If this happens, these are the fully well-packed examples and are emitted immediately, fullest first.

• Pool overflow: if the number of open buffers exceeds a cap \(B = 20\), the fullest buffer is force-emitted even if not exactly full. This bounds the memory and latency cost of holding many partially-filled packs and is the only source of imperfectly-filled examples in practice.

No explicit padding is introduced during packing itself. The only padding is whatever the collator adds to align tensors within a micro-batch, and is compensated for by per-sample loss weighting (see details below).

1.0.10.3 Preventing cross-sample contamination

A single pack contains several independent samples that must not attend to one another, or the model would condition predictions on unrelated context. We enforce this at the attention level rather than by re-padding. The per-token data_index tags are converted at collation time into cumulative segment boundaries (cu_seqlens) and passed—in place of a dense attention mask—into a patched attention kernel (flash_attn_varlen_func) for the LM backbone. This yields exact block-diagonal attention (21b): every sample attends only within its own segment, and position ids restart at zero at each segment boundary, so a packed sample is numerically identical to the same sample trained in isolation. Finally, because samples within a pack have very different lengths, we re-weight the per-token loss by a function of each segment’s effective (loss-bearing) token count, so that a single long sample does not dominate the pack’s gradient relative to the other short samples in a pack.

1.0.10.4 Stateful and resumable packing

Packing is both streaming and stateful: at any instant a worker holds a pool of partially-filled buffers that are not derivable from the dataloader position alone. To make training resumable without re-seeing or dropping data at a checkpoint boundary, each (rank, worker) pair persists its open buffer pool to disk (rank<r>_worker<w>_buffers.pt) alongside the WebDataset shard/sample/epoch cursors that record exactly how far each source has been consumed. On resume, the pool and cursors are restored and packing continues from precisely the state it was checkpointed in. This buffer state is written as part of the same dataset_state mechanism used for dataloader resumption.

In all our runs we pack to \(L = 8192\) tokens and \(M = 24\) image tiles, with a pool cap of \(B = 20\) open buffers.

1.0.11 Annotation Infrastructure↩︎

To run our filtering experiments, we need to annotate all samples in our data pool with the respective filtering metric—for every sample, and for every filter, we compute a scalar score for each filter (e.g., compute the CLIP score or the decontamination score according to a set of reference images). At our scale, this amounts to scoring trillions of tokens under many filters whose resource profiles range from cheap CPU-only text-based classifiers (e.g., length) to GPU-bound encoders (e.g., CLIP score). We build a Ray-based⁶ annotation framework that runs inside a Slurm allocation, parallelizes across nodes, and tolerates common failures (decode errors, OOMs, transient file-system issues, Slurm preemption) that any multi-day multi-node annotation job encounters. We plan to release the framework as part of the DCVLM benchmark.

1.0.11.1 Pipeline and Actor Model

A run is parameterised by an input directory of WebDataset [247] shards and a list of filter names. Each filter declares its per-actor CPU, GPU, and memory requirements via a BaseFilter interface. Because a worker runs the selected filters sequentially on a shared decoded batch, the framework sizes the actor pool from the maximum declared requirement across filters on each axis (CPU, GPU, DRAM). A run proceeds in three overlapping phases:

1. Extraction. One Ray task per tar reads the raw bytes (without decoding any sample components like images, json files etc.), groups samples into batches of \(B\) samples, and places each batch in the Ray object store. Decoding is deferred so that PIL images never round-trip through the object store, which keeps inter-node traffic to compressed bytes only.

2. Processing. A fixed pool of FilterExecutionWorker actors is created once at the start of the run. For each batch, the actor decodes once and runs every filter on the decoded samples in sequence, returning per-sample results and per-filter statistics. The actor model where each actor has some form of state is important. Loading the scoring model dominates per-batch cost for most filters, so we want to pay for it once per actor rather than once per batch, and decoding once per batch lets multi-filter runs share work.

3. Streaming writes. As soon as all batches for a tar resolve, we schedule a write task. This decouples write IO from active extraction and inference, and enables continuation after a Slurm interruption.

We support two output modes. In --only-write-metadata mode, each input tar xxxxx.tar creates a sibling xxxxx.json mapping every sample key to a flat dict of filter scores. The original shards are left untouched. In the alternative full-rewrite mode, the same scores are embedded as a filters.json field inside a freshly written tar. This is convenient when a downstream consumer wants a self-contained WebDataset.

1.0.11.2 Backpressure and Fault Tolerance

A max_batches_in_flight cap bounds the number of outstanding processing tasks, which in turn bounds the working set of the Ray object store. Without it, extraction outruns the GPU pool by orders of magnitude and PIL-decoded images blow up RAM (and trigger costly object-store spilling).

We tolerate failures at four levels with intentionally minimal recovery logic. (i) Sample decode errors are retried \(N\) times then skipped or raised per a configurable handler. (ii) A batch that raises during filtering is resubmitted up to max_resubmits times (default \(5\)); the actor is health-checked via a ping before being returned to the idle pool. (iii) If an actor dies (RayActorError, e.g.kernel OOM-kill or a CUDA fault), the in-flight task is resubmitted and a best-effort replacement actor is spawned, so the pool does not shrink monotonically over the run. (iv) At the job level, atomic .tmp+rename writes are combined with a startup pass that skips tars whose final output already exists, so an interrupted job can be resumed by simply re-running it. The core assumption we rely on is that filters are deterministic, which makes duplicate computation under retry safe.

1.0.12 Additional SFT Results↩︎

The SFT control experiments in 0.0.4.3 rely on two design choices that we fix once and reuse across all 54 runs: the SFT token budget and the peak learning rate. We first describe how we set both (1.0.12.1), and then verify that our conclusions are robust to the choice of SFT dataset (1.0.12.2).

1.0.12.1 SFT Compute Budget and Learning-Rate Selection

SFT token budget. Rather than fine-tuning each checkpoint with an arbitrary fixed schedule, we scale the SFT budget proportionally to each checkpoint’s pretraining budget. We estimate the SFT-to-pretraining token ratio of InternVL3 [2] to be \(\approx\!0.29\) and apply this multiplier to each of our three pretraining budgets, yielding SFT budgets of \(1.8\)B, \(3.6\)B, and \(7.2\)B tokens for checkpoints pretrained on \(6.25\)B, \(12.5\)B, and \(25\)B tokens, respectively. Holding this ratio fixed ensures that larger-compute checkpoints receive proportionally more fine-tuning, rather than coupling our conclusions to a single absolute SFT schedule.

Hyperparameter considerations. For pretraining we use a fixed peak LR of \(2\times10^{-5}\) across all scales, chosen via a sweep spanning three orders of magnitude. For SFT, common practice across recent VLMs is to keep the LR within the same order of magnitude as pretraining: InternVL-2.5 [1] and Nemotron Nano-V2-VL [22] reuse the pretraining LR unchanged, MM1 [35] and MiMo-VL [21] stay within the same order of magnitude (the latter increasing it slightly), and PerceptionLM [30] reuses the LR but halves the global batch size, effectively doubling it under the linear-scaling rule. Only Molmo [17] drops by a full order of magnitude (from \(2\times10^{-4}\) to \(1\times10^{-5}\)), and additionally uses separate LRs for the language model, projector, and vision encoder.

Learning-rate sweep. Guided by this, we keep the global batch size fixed at \(1024\) (matching pretraining) and sweep the SFT peak LR over \(\{4\times10^{-6},\, 2\times10^{-5},\, 4\times10^{-5}\}\), i.e.one order of magnitude around the pretraining value. All runs use a cosine-decay schedule with \(3\%\) warmup and weight decay \(0.01\). We sweep on two representative \(2\)B checkpoints—both pretrained on the Instruction-heavy mix that is optimal at this scale—one pretrained for \(6.25\)B tokens (fine-tuned for \(1.8\)B SFT tokens) and one for \(12.5\)B tokens (fine-tuned for \(3.6\)B), using LLaVA-665K [4] as the SFT dataset.

21 reports the results. The lowest LR, \(4\times10^{-6}\), is best on Core Avg at both checkpoints, while the highest LR, \(4\times10^{-5}\), is consistently worst—most visibly on Text (\(-4.4\)pp and \(-4.5\)pp relative to \(4\times10^{-6}\) at the \(6.25\)B and \(12.5\)B checkpoints), indicating that an overly aggressive SFT LR erodes the language capabilities established during pretraining. The pretraining LR (\(2\times10^{-5}\)) is a close runner-up. We therefore adopt a peak LR of \(4\times10^{-6}\) for all SFT runs in this work. Reassuringly, SFT lifts every checkpoint above its pretrained starting point at the chosen LR (\(+2.4\) and \(+2.8\)), without altering the underlying pretraining ranking.

1.0.12.2 Robustness to the SFT Dataset

22 replicates the analysis of 0.0.4.3 using Mammoth-VL-12M [120] as the SFT dataset in place of LLaVA-665K [53]. Results are fully consistent: pretraining and post-SFT scores remain near-perfectly correlated (Pearson \(r{=}0.99\); Spearman \(\rho{=}0.99\)), and the pretraining ranking is preserved across all 27 checkpoints. These results confirm that the conclusion is robust to the choice of SFT dataset: a stronger pretraining checkpoint consistently yields a stronger fine-tuned model. In particular, the advantage of Instruction-heavy pretraining mixtures is not diminished by the additional instruction-tuning signal introduced during SFT.

1.0.13 Extended Suite Evaluations↩︎

22 reports results on our 52-task Extended evaluation suite, which augments the Core suite with Hallucination (Hall), Safety (Safety), and Reasoning (Reason) categories.

DCVLM-Baseline maintains its overall advantage over FineVision at both scales, achieving \(+3.0\)pp and \(+3.9\)pp gains at large and x-large, respectively. Gains are most pronounced in Vision-Centric (\(+23.1\)pp / \(+21.3\)pp) and General Understanding (\(+8.4\)pp / \(+8.5\)pp) tasks, consistent with the Core suite trends.

Regarding newly added categories, Hallucination scores also favor DCVLM-Baseline at both scales (\(+3.4\)pp / \(+3.0\)pp), suggesting our data mixture yields better-calibrated visual grounding. However, two notable weaknesses emerge. First, DCVLM-Baseline substantially underperforms on Safety (\(-17.7\)pp / \(-19.2\)pp across both scales). Second, FineVision retains an advantage on OCR (\(+6.5\)pp / \(+5.2\)pp) and Reasoning (\(+4.3\)pp / \(+4.9\)pp), suggesting its document-rich sources benefit reading and understanding details, text, figures, and equations in images. We leave the exploration of how to address these gaps, particularly regarding safety alignment, for future work.

1.0.14 Safety Preprocessing: Harmful, Toxic and Biased Content↩︎

As with all large uncurated data-pools (largely sourced from the web), especially for ones that are fully open-sourced, there can be concerns regarding releasing harmful and unsafe content openly. We invested some efforts into ensuring that our raw data-pool has as little unsafe content as possible, via two main analysis methods—(1) Text-based removal and (2) Image-level Safety Analysis.

1.0.14.1 Unsafe, Biased and Toxic Text Removal

To ensure data quality and safety, we filter unsafe, biased, and toxic content in DCVLM pool using Detoxify [248], a transformer-based multilingual toxicity classifier. The classifier produces scores across multiple harm categories which include toxicity, severe toxicity, obscenity, threat, insult and identity attack. We retain a sample only if every category score falls at or below a threshold of \(0.1\), following the filtering criteria used in DataComp-CLIP [5].

1.0.14.2 Image-level Safety Analysis

We analyze a subset of \(20,000\) image–text pairs to assess image safety. For this purpose, we employ ShieldGemma2 [249], a 4B-parameter image content moderation model built on Gemma 3, which provides robust safety risk predictions across multiple categories, including sexually explicit content, violence and gore, and dangerous content. Within the sampled dataset, the model classifies \(19\) images as dangerous, \(117\) as sexually explicit, and \(4\) as containing violence and gore, each with a confidence score of at least \(90\%\). This suggests that potentially unsafe content represents only a small fraction of the dataset, indicating a relatively low overall presence of harmful imagery and further supports using the analyzed subset as a representative estimate of the safety characteristics of the full dataset.

1.0.15 Per-Dataset Core Results↩︎

For completeness and ease of inspection, we report the per-dataset scores underlying the per-category Core averages of 2. Each table below corresponds to one of our six Core categories and lists every constituent benchmark for the same set of models and scales as 2. Averaging a row’s datasets within a table recovers that row’s per-category number in 2, and averaging across all 33 datasets (six tables) recovers its Core Avg.

1.0.16 Limitations and Future Directions↩︎

Here, we highlight some limitations and open questions of our work which may be useful for exciting future research.

1.0.16.1 Limitations

• Scaling to longer token budgets. In this work, we see clear scaling trends in downstream performance across our model and token budget scales from small (1B, 6.25B tokens) to x-large (8B, 200B tokens), however we remain compute-limited to a 200B token pretraining budget. Competitive open-weight alternatives such as Qwen2-VL and Qwen3-VL train on 1.4T and 2.2T tokens during pretraining respectively.

• Scaling to larger model sizes. In addition to scaling token budgets during the pretraining stage, stress-testing our our data pool and data mixture conclusions at model sizes \(\geq\)​20B is an important research question not empirically addressed in this work.

• Testing DCVLM data on native multimodal pretraining. The pretraining strategy adopted in this work starts with already pretrained vision encoders and language models. However, recent works have advocated for native pretraining paradigms to enable training VLMs from scratch, which is not something we have implemented in this work. Effectively the dominance of instruction-heavy data mixtures may not exist when the inductive biases of separately pretrained encoders are absent, and is an important research question worth addressing.

• More principled scaling ladder. Our four-scale ladder is not tied to the joint optimization of model size and data token budgets, currently set up as practical defaults instead of theoretical optimal estimates. Designing a more principled scaling ladder, especially for vision-language models that use pretrained components, is a research problem of its own and we consider this out-of-scope in this work.

• Capability Coverage. While DCVLM spans a broad range of capabilities in both the extended and core set of evaluations, certain tasks like grounding and detection are absent. While we observe issues with these benchmarks and tasks and highlighted them in 1.0.6.1, incorporating these tasks in a faithful evaluation setup is a natural and critical followup.

• Poor OCR performance. We conclude that our OCR performance lags behind FineVision due to their better coverage and priority given to high-resolution scanned documents and pdf-derived sources. Whether simply scaling up OCR data or curating extremely high-quality and information-dense samples can bridge this gap is not adequately resolved in this work.

1.0.16.2 Future Work

• Towards instructional multimodal documents. Our results suggest that instruction-heavy mixtures scale more favorably with compute, raising an important question: how can we better exploit and transform the vast supply of raw multimodal documents? Such transformations may preserve the breadth and diversity of naturally occurring multimodal data while providing a denser and more targeted learning signal for instruction-tuned multimodal models.

• Fine-grained mixture optimization. Our mixing experiments only involved dense line-search sweeps over the proportion of image-caption pairs and instruction-tuning data in the mixture. A direction worth exploring is to use swarm-based methods [60], [62], [67] using proxy models and verify if they discover similar findings of instruction-heavy mixes being more scalable.

• Token-level mixing. Our mixing experiments operationalized data mixing at the sample level, i.e. we fix the data type ratios based on the number of samples in each dataset. However, a good direction to explore would be to operationalize mixing at the token-level. As seen from 1.0.5.2, the distribution of token-counts and sample-counts per-dataset can vary, and hence it would be an interesting avenue to compare and contrast different mixing strategies across.

• Quality-aware upsampling. In our work, we only used all the quality-scores for data filtering, i.e. to discard data samples below a certain threshold. This is exclusionary by nature, and might induce harmful biases into models [250], [251]. Another potential use for such quality-scores which does not involved discarding data is using those scores for upsampling data samples rather than filtering.

NeurIPS Paper Checklist↩︎

1. Claims

2. Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

3. Answer:

4. Justification: This work proposes a benchmark for controlled data-centric experiments to improve VLM training. Additionally, we show that data mixing is the key to assembling high-quality training data. Finally, the baseline dataset (DCVLM-Baseline) obtained from our results enables training models to achieve an accuracy improvement of +5.4pp. The benchmark is described in 0.0.3, results on data mixing in 0.0.4.2, and improvements over existing pretraining datasets in 0.0.5.

5. Guidelines:

   • The answer means that the abstract and introduction do not include the claims made in the paper.

   • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A or answer to this question will not be perceived well by the reviewers.

   • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

   • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

6. Limitations

7. Question: Does the paper discuss the limitations of the work performed by the authors?

8. Answer:

9. Justification: The limitations are discussed in the “Limitations and Future Directions” section (1.0.16.1).

10. Guidelines:

    • The answer means that the paper has no limitation while the answer means that the paper has limitations, but those are not discussed in the paper.

    • The authors are encouraged to create a separate “Limitations” section in their paper.

    • The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

    • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

    • The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

    • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

    • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

    • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren’t acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

11. Theory assumptions and proofs

12. Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

13. Answer:

14. Justification: The paper does not include theoretical results and proofs.

15. Guidelines:

    • The answer means that the paper does not include theoretical results.

    • All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.

    • All assumptions should be clearly stated or referenced in the statement of any theorems.

    • The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

    • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

    • Theorems and Lemmas that the proof relies upon should be properly referenced.

16. Experimental result reproducibility

17. Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

18. Answer:

19. Justification: While general details regarding data collection, curation, and training are provided in the main paper, the Appendix fully complements these, reporting trained model architecture details (1.0.3), the training hyperparameters used (1.0.4), the annotation infrastructure (1.0.11), train-test decontamination (1.0.7), DCVLM data pool details (1.0.5), our evaluation suite (1.0.6), and safety preprocessing details (1.0.14).

20. Guidelines:

    • The answer means that the paper does not include experiments.

    • If the paper includes experiments, a answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

    • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

    • Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

    • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

      1. If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

      2. If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

      3. If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

      4. We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

21. Open access to data and code

22. Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

23. Answer:

24. Justification: The submission comes with a hyperlink to where the data is stored, while the Appendix fully reports the necessary details to reproduce our results.

25. Guidelines:

    • The answer means that paper does not include experiments requiring code.

    • Please see the NeurIPS code and data submission guidelines (https://neurips.cc/public/guides/CodeSubmissionPolicy) for more details.

    • While we encourage the release of code and data, we understand that this might not be possible, so is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

    • The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://neurips.cc/public/guides/CodeSubmissionPolicy) for more details.

    • The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

    • The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

    • At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

    • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

26. Experimental setting/details

27. Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer) necessary to understand the results?

28. Answer:

29. Justification: The main experimental procedure is described in [sec:methodology] [sec:experiments] [sec:results], and more details are provided in the Appendix ([app:architecture] [app:hparams] [app:decontamination] [app:dataset_pool] [app:eval_details]).

30. Guidelines:

    • The answer means that the paper does not include experiments.

    • The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

    • The full details can be provided either with the code, in appendix, or as supplemental material.

31. Experiment statistical significance

32. Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

33. Answer:

34. Justification: Error bars are not reported as they are too computationally expensive to compute.

35. Guidelines:

    • The answer means that the paper does not include experiments.

    • The authors should answer if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

    • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

    • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

    • The assumptions made should be given (e.g., Normally distributed errors).

    • It should be clear whether the error bar is the standard deviation or the standard error of the mean.

    • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

    • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g., negative error rates).

    • If error bars are reported in tables or plots, the authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

36. Experiments compute resources

37. Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

38. Answer:

39. Justification: For each experiment “scale” (small, medium, large, x-large), ¿tbl:tab:scales? reports the number of ‘H100 hrs’ required to reproduce our results.

40. Guidelines:

    • The answer means that the paper does not include experiments.

    • The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

    • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

    • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn’t make it into the paper).

41. Code of ethics

42. Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?

43. Answer:

44. Justification: We carefully reviewed the NeurIPS Code of Ethics and believe our work complies with it.

45. Guidelines:

    • The answer means that the authors have not reviewed the NeurIPS Code of Ethics.

    • If the authors answer , they should explain the special circumstances that require a deviation from the Code of Ethics.

    • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

46. Broader impacts

47. Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

48. Answer:

49. Justification: Since DCVLM is a benchmark for controlled data-centric experiments with the goal of improving VLM training, our work fits as fundamental research and, therefore, it is not tied to particular applications.

50. Guidelines:

    • The answer means that there is no societal impact of the work performed.

    • If the authors answer or , they should explain why their work has no societal impact or why the paper does not address societal impact.

    • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

    • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate Deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

    • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

    • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

51. Safeguards

52. Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)?

53. Answer:

54. Justification: 1.0.14 reports the safety processing details we used to reduce as much as possible harmful, toxic, and biased content.

55. Guidelines:

    • The answer means that the paper poses no such risks.

    • Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

    • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

    • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

56. Licenses for existing assets

57. Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

58. Answer:

59. Justification: [appsub:licensing,tab:dcvlm-datamix-licenses] report links pointing to the datasets we used in the DCVLM pool, with all available licenses associated. Furthermore, we properly credited authors by citing their work.

60. Guidelines:

    • The answer means that the paper does not use existing assets.

    • The authors should cite the original paper that produced the code package or dataset.

    • The authors should state which version of the asset is used and, if possible, include a URL.

    • The name of the license (e.g., CC-BY 4.0) should be included for each asset.

    • For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

    • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

    • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

    • If this information is not available online, the authors are encouraged to reach out to the asset’s creators.

61. New assets

62. Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

63. Answer:

64. Justification: The link to the data attached to this submission contains a README file explaining how the data is organized and how to navigate it.

65. Guidelines:

    • The answer means that the paper does not release new assets.

    • Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

    • The paper should discuss whether and how consent was obtained from people whose asset is used.

    • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

66. Crowdsourcing and research with human subjects

67. Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

68. Answer:

69. Justification: Our work does not involve crowdsourcing nor research with human subjects.

70. Guidelines:

    • The answer means that the paper does not involve crowdsourcing nor research with human subjects.

    • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

    • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

71. Institutional review board (IRB) approvals or equivalent for research with human subjects

72. Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

73. Answer:

74. Justification: Our work does not involve crowdsourcing nor research with human subjects.

75. Guidelines:

    • The answer means that the paper does not involve crowdsourcing nor research with human subjects.

    • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

    • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

    • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

76. Declaration of LLM usage

77. Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigor, or originality of the research, declaration is not required.

78. Answer:

79. Justification: The paper is about training large vision-language models (rather than using them as components). We describe the architecture and the pretrained LLM we use in Section 0.0.3.2 and Appendix 1.0.3.

80. Guidelines:

    • The answer means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

    • Please refer to our LLM policy in the NeurIPS handbook for what should or should not be described.

References↩︎