1 Introduction↩︎

Large-scale visual-document corpora are becoming increasingly common [1], [2], but visual-document retrieval—the task of retrieving image-based documents for a user’s text query—still lacks a production-scale lexical retriever [3], [4]. The missing operating point is a retriever that serves text queries without neural query encoding while directly indexing visual documents. Such a system would make large-scale visual-document retrieval more cost-effective. However, current approaches satisfy only part of this requirement. End-to-end dense and multi-vector VLM retrievers operate directly on visual documents and achieve strong accuracy, but require neural query encoding, larger backbones, or expensive multi-vector scoring at serving time [3], [5], [6]. Conversely, OCR- or caption-based BM25 [7] provides query-encoding-free lexical retrieval, but only after each visual document is converted into text through a time-consuming OCR pipeline or costly caption generation [3], [4].

This leaves a missing operating point for large-scale visual-document retrieval: a lexical retriever that directly indexes visual-documents and serves queries without neural encoding. Inference-free multimodal learned sparse retrieval (MMLSR) naturally fits this regime. A VLM-based sparse encoder can map visual-documents directly into lexical sparse embeddings without OCR or caption generation, while Bag-of-Words (BoW) queries enable inverted-index retrieval without neural query encoding. Yet MMLSR remains largely underexplored for visual-documents, especially in its inference-free form. Although several notable efforts have developed MMLSR methods for general text-to-visual retrieval [8], [9], existing approaches have not yet matched the performance of comparable-scale dense models while preserving high sparsity. This gap is more pronounced in the ViDoRe leaderboard, a widely used benchmark for visual-document retrieval systems, which contains no MMLSR entry as of writing [10].

We attribute this limitation to a lexical grounding problem in sparse visual-document retrieval. By lexical grounding, we mean the ability of a multimodal learned sparse retriever to map visually presented lexical evidence, in a rendered page image such as words and numbers, to the relevant lexical dimensions in its sparse representation. In text sparse retrieval, this mapping is direct because document words are provided as input tokens, which naturally anchor vocabulary-indexed outputs. In visual sparse retrieval, however, the same lexical content is observed only as pixels. The encoder must therefore infer which vocabulary dimensions to activate without explicit text-token anchors. When this grounding fails, the sparse representation may miss important document terms or activate spurious dimensions, making lexical matching less reliable. We make this gap observable through a diagnostic study on rendered text documents.

To address this problem, we introduce caption-gated token supervision, a training-only signal that uses VLM-generated captions to provide lexical token-level cues for visual sparse representations. As shown in Figure 1, the visual-document and its offline-generated caption are encoded into the same sparse vocabulary space. The caption vector gates the image vector, reinforcing dimensions supported by both sparse views as reliable lexical evidence on the image side. The caption branch is used only during training; at inference, the encoder maps each visual-document directly to an image-side sparse vector.

With this supervision, we introduce V-SPLADE, a lexically grounded sparse retriever for the missing serving regime in visual-document retrieval. V-SPLADE maps each visual-document directly into a vocabulary-indexed sparse representation using a compact 250M visual-to-sparse encoder, without OCR or caption generation. At serving time, sparse document vectors are stored in a standard inverted index, while query-token weights are served by learned token lookup without neural query encoding. Furthermore, the lexical nature of these representations makes V-SPLADE more robust under corpus scaling and complementary to dense retrievers, yielding gains both as a standalone retriever and as a fusion component.

V-SPLADE consistently outperforms the main target-regime baselines in both standard benchmarks and large-scale retrieval. Across six visual-document retrieval benchmarks, V-SPLADE improves average NDCG@5 by +13.8pp over BiModernVBERT, a state-of-the-art compact dense retriever for visual-document retrieval in the same model-size regime, and by +5.7pp over the OCR-based lexical baseline. On an 18.7M-document corpus, V-SPLADE reaches R@5=0.228, compared with 0.090 for the same-backbone dense retriever. It supports sub-10ms exact inverted-index search, approximate inverted-index search with latency comparable to HNSW [11], and document encoding over \(20\times\) faster than caption generation or OCR-based lexical pipelines. As the corpus grows, V-SPLADE retains recall more robustly than the dense baseline. Through score fusion, it further improves competitive dense retrievers by up to +2.4pp R@5 on the 18.7M corpus. Together, these results position V-SPLADE as a scalable lexical retrieval layer: fast to index, efficient to serve, robust under corpus scaling, and complementary to existing retrieval systems.

In summary, we make the following contributions:

1. We diagnose a lexical grounding problem in sparse visual-document retrieval.

2. We propose caption-gated token supervision to lexically ground visual sparse representations.

3. We develop V-SPLADE, a sparse retriever for the missing serving regime in production-scale visual-document retrieval.

2 Related Work↩︎

2.0.0.1 Visual document retrieval.

Visual-document retrieval has traditionally relied on OCR-extracted text for retrieval, but has increasingly shifted toward VLM retrievers that operate directly on rendered pages [3]. The strongest line has largely followed ColBERT-style late interaction, preserving fine-grained page evidence through multi-vector representations and MaxSim-style scoring [3], [5], [6], [12]. While effective for complex pages, this regime is costly as a full-corpus first-stage retriever because it requires neural query encoding and scoring over hundreds to roughly a thousand visual tokens per document. Recent compact backbones and reduced-token late-interaction models improve efficiency [13]–[15], but still differ from the query-encoding-free lexical serving regime targeted by V-SPLADE.

2.0.0.2 Text learned sparse retrieval.

Text learned sparse retrieval provides the closest template for query-encoding-free lexical serving [16]–[21]. Among these methods, SPLADE [16], [17] is the most prominent example: it maps text into sparse vectors over the language-model vocabulary, enabling weighted lexical matching with inverted indexes. Because SPLADE representations live in vocabulary space, they can be paired with BoW-style queries to avoid neural query encoding. Li-LSR [22] further strengthens this inference-free query side by replacing naive BoW weights with learned token-level lookup weights. However, these successes have been largely confined to text, where input tokens directly anchor sparse vocabulary dimensions; rendered visual documents lack such anchors.

2.0.0.3 Multimodal learned sparse retrieval and caption supervision.

Multimodal learned sparse retrieval has seen several notable research efforts, but remains far less established than text learned sparse retrieval [23], [24]. Notable MMLSR work has used dataset-provided captions for stable sparse expansion control [8] or sparse–dense inter-score self-distillation to improve multimodal sparse representations [9]. However, these efforts have not yet achieved performance comparable to same-size dense models while maintaining high sparsity, and have rarely been applied to visual-document retrieval. At the same time, work on generated textual descriptions suggests a promising source of lexical supervision. Caption-based BM25 provides competitive lexical baselines for visual documents [4], [25], and generate-and-encode methods such as SERVAL show that VLM-generated document descriptions can act as strong semantic proxies [26]. MLLM-generated captions have also been used as auxiliary signals beyond visual documents, such as in text-video retrieval [27]. These results suggest that captions provide useful lexical and semantic evidence for retrieval. This evidence motivates V-SPLADE and its core training signal, caption-gated token supervision.

3 Motivation: Lexical Grounding↩︎

This section examines a lexical grounding problem in sparse visual-document retrieval. Text learned sparse retrieval is grounded in text by construction, because the input itself is a sequence of lexical tokens. However, in sparse visual-document retrieval, the same lexical content appears only as visual evidence in a rendered page image. A multimodal sparse encoder must therefore ground visual evidence into lexical vocabulary dimensions without explicit text-token anchors. We refer to the challenge of activating vocabulary dimensions that correspond to lexical evidence observed in the page image as the lexical grounding problem.

To make this problem observable, we conduct a controlled diagnostic experiment on 1,000 RLHN text documents [28], a well-curated text IR dataset. For each document, we create two aligned input views with the same underlying content. First, we keep the original source text as a text input. Second, we render the same text into a PDF-like page image and use the rendered page as a visual-document input. We then pass each view through a vision-language backbone with an LM head, which projects the representation into language-model vocabulary space, and inspect the vocabulary dimensions with the highest activation values. For a given top-\(k\) set of activated dimensions, we measure the fraction of dimensions that overlap with the source-text BoW. In summary, the diagnostic asks the following question: when the same document content is provided either as source text or as a rendered page image, how differently is its lexical content recovered in sparse vocabulary space?

We use ModernVBERT as the diagnostic encoder because its MLM-based modality alignment trains it to recover text tokens from rendered document image inputs. This lets us examine how much of the lexical grounding problem remains under a model explicitly trained to recover lexical information from visual documents.

We compare four views:

1. Source text as text: the original source text, which serves as a text-side upper bound.

2. Rendered image as image: the PDF-like page image rendered from the same source text.

3. VLM-generated caption as caption: a caption of the rendered image, produced by Qwen3-VL-30B [29] using the same prompt as ColPali [3] to extract visual-document information.

4. Image–caption gate as gate: the element-wise product between the image sparse representation from (2) and the caption sparse representation from (3).

Table 1 shows a large gap between the source-text upper bound and the rendered-image representation. At top-30, the overlap drops from 0.974 for source text to 0.560 for the rendered image. This indicates that the lexical content remains visually present in the page, but is only partially recovered by the image-side sparse encoder. As a result, query matching must rely on a smaller and noisier set of lexical dimensions, which may explain why high-sparsity multimodal learned sparse retrieval is difficult to train.

Caption text partially improves lexical overlap, but remains insufficient on its own: at top-30, it reaches 0.574, slightly above the image representation at 0.560. Although prompt tuning optimized for this specific task may further improve the caption-only result, this setting also reflects a realistic condition where captions can be noisy or imperfectly aligned with the original document content.

In contrast, a product gate between the image sparse representation and the caption sparse representation yields a much stronger overlap, reaching 0.794 at top-30. This suggests that the two views provide complementary evidence: the image representation carries meaningful but noisy lexical evidence from the rendered page while the caption representation provides a sharper lexical prior. This motivates caption-gated token supervision, which reinforces image-side vocabulary dimensions supported by the caption signal only during training.

4 Method↩︎

4.1 Overview↩︎

Before describing how caption-gated token supervision addresses the lexical grounding problem, we first define the basic architecture of V-SPLADE. This architecture combines SPLADE-based [17] lexical sparse representations with Li-LSR-style [22] token lookup, forming an inference-free MMLSR retriever for the missing operating point of visual-document retrieval: direct visual-document indexing with query-encoding-free serving. We then introduce caption-gated token supervision as a training-only signal that lexically grounds image-side sparse activations produced by V-SPLADE.

4.2 V-SPLADE↩︎

V-SPLADE extends SPLADE for direct visual-document indexing with a backbone choice tailored to the visual-document setting. Prior MMLSR methods [8], [9] often start from CLIP-style image–text encoders and learn a sparse output space on top of them. In contrast, V-SPLADE uses a vision-language backbone with an LM head, which better matches the SPLADE formulation because its hidden states are already aligned with vocabulary prediction.

4.2.0.1 Image-side sparse representation.

Following SPLADE [17], we map a rendered page image into a vocabulary-indexed sparse vector by applying an LM-head projection and SPLADE activation to the visual hidden states produced by the vision-language backbone. Given visual hidden states \(\{\mathbf{h}_t\}_{t=1}^{L}\), the image-side sparse representation is \[\mathbf{w}_p = \max_{t=1}^{L} \log \left(1 + \mathrm{ReLU}(\mathrm{LMHead}(\mathbf{h}_{t}))\right) \in \mathbb{R}^{|V|}.\] Here, \(|V|\) is the vocabulary size and \(\mathbf{w}_p[v]\) is the sparse weight assigned to vocabulary token \(v\). The transformation applies ReLU activation followed by max pooling over visual tokens.

4.2.0.2 Query-encoding-free representation.

V-SPLADE avoids neural query encoding by adapting Li-LSR-style [22] learned query-token lookup to MMLSR. The query is represented as a weighted BoW vector, where each query token receives a learned vocabulary-level weight from a lookup table. For each vocabulary token \(v\), we compute \[a_v = \mathrm{softplus}(\mathbf{e}_v^\top \mathbf{u} + b),\] where \(\mathbf{e}_v \in \mathbb{R}^{d}\) is the frozen token embedding, and \(\mathbf{u} \in \mathbb{R}^{d}\) and \(b\) are learned parameters. Given the BoW mask \(m_q[v] \in \{0,1\}\) of query \(q\), the query representation is \[\mathbf{w}_q[v] = m_q[v] \cdot a_v.\] After training, all \(a_v\) values are stored in a lookup table, so inference only requires tokenization and weight lookup.

We make one important adaptation for the visual sparse setting. The original ReLU-based lookup in Li-LSR is prone to all-zero activation collapse when combined with visual sparse training and sparsity regularization, as shown in Section [sec:design95choices]. We therefore replace ReLU with softplus. This adaptation preserves the simplicity of Li-LSR-style lookup while improving training stability in the visual sparse setting.

4.2.0.3 Retrieval scoring.

Retrieval is performed by sparse lexical matching. Given a query representation \(\mathbf{w}_q \in \mathbb{R}^{|V|}\) and a document representation \(\mathbf{w}_p \in \mathbb{R}^{|V|}\), the score is \[\label{eq:scoring} s(q,p) = \mathbf{w}_q^\top \mathbf{w}_p = \sum_{v \in V} \mathbf{w}_q[v]\mathbf{w}_p[v].\tag{1}\] The ranking loss based on Eq. 1 and the sparsity regularizer for controlling active vocabulary dimensions are defined in Section 4.4.

4.3 Lexical Grounding through Caption-Gated Token Supervision↩︎

The lexical grounding problem makes visual sparse training difficult: the top image-side activations provide only partial coverage of the document’s actual lexical content. Query–document matching can then rely on generic or spurious dimensions, producing coarse ranking feedback. Caption-gated token supervision addresses this by using captions to selectively reinforce content-bearing lexical dimensions on the image side. As supported by the token-level analysis in Section [sec:design95choices], this allows ranking loss to further activate retrieval-relevant dimensions that contain lexical information in the document, while generic or unsupported activations are left to be suppressed by the sparsity regularizer.

4.3.0.1 Training-time captions.

We generate one offline caption for each training document and use captions only during training. Although caption quality may vary with alternative captioning prompts or pipelines, our focus is not caption generation itself, but how to use available captions as sparse supervision. We therefore follow the standard ColPali [3] setup: captions are generated by a multimodal LLM with the ColPali prompt without modification; full prompt and generation details are provided in Section 5.2.0.1. Because raw captions are not directly optimized for retrieval, we next convert them into retrieval-aware sparse representations.

4.3.0.2 Retrieval-aware caption representation.

To obtain a retrieval-aware caption representation, we encode the caption with the same SPLADE-style sparse encoder used for the image side and apply the caption BoW mask to produce \(\mathbf{w}_c \in \mathbb{R}^{|V|}\).

We train the caption representation with a query–caption ranking loss: \[\label{eq:cap95rank} \mathcal{L}_{\mathrm{cap\_rank}} = -\frac{1}{B} \sum_{i=1}^{B} \log \frac{ \exp(s(q_i,c_i)/\tau) }{ \sum_{j=1}^{B} \exp(s(q_i,c_j)/\tau) },\tag{2}\] where \(B\) is the batch size, \(s(q_i,c_j)=\mathbf{w}_{q_i}^{\top}\mathbf{w}_{c_j}\), and \(\tau\) is the ranking temperature. This ranking loss turns the caption sparse representation into a retrieval-aware lexical signal used for gating.

4.3.0.3 Caption-gated token supervision.

Let \(\mathbf{w}^{\mathrm{sg}}_p, \mathbf{w}^{\mathrm{sg}}_c \in \mathbb{R}^{|V|}\) be stop-gradient sparse representations of the page image and caption. For each vocabulary token \(v\), we define the overlap gate: \[o[v] = \mathbf{w}^{\mathrm{sg}}_p[v] \cdot \mathbf{w}^{\mathrm{sg}}_c[v].\]

The gate therefore selects vocabulary dimensions that both the image and caption representations consider useful. We then obtain \(\alpha[v]\) from \(o[v]\) by temperature sharpening and L1 normalization over vocabulary dimensions: \[\alpha[v] = \frac{o[v]^{\,1/\tau_{\mathrm{cap}}}}{\sum_{v'} o[v']^{\,1/\tau_{\mathrm{cap}}}},\] where \(\tau_{\mathrm{cap}}\) controls the sharpness of the caption-gated supervision.

Let \(z_p[v]\) be the image-side vocabulary logit before the SPLADE ReLU and log-saturation. We apply caption-gated token supervision to this pre-activation logit so that gated dimensions receive direct gradients. The caption-gated token supervision loss is defined as \[\label{eq:cap95gated} \mathcal{L}_{\mathrm{cap\_gated}} = -\frac{1}{B} \sum_{i=1}^{B} \sum_v \alpha_i[v]\, \log \sigma(z_{p_i}[v]).\tag{3}\]

3pt

4.4 Training Objective and Deployment Path↩︎

We train V-SPLADE with three loss groups: ranking, caption-gated token supervision, and sparsity regularization.

For image-side ranking, we use an in-batch InfoNCE loss on the query–page score \(s(q,p)\): \[\mathcal{L}_{\mathrm{rank}} = -\frac{1}{B} \sum_{i=1}^{B} \log \frac{ \exp(s(q_i,p_i)/\tau) }{ \sum_{j=1}^{B} \exp(s(q_i,p_j)/\tau) }.\] Here, \(p_i\) is the positive page for \(q_i\), other pages in the batch are negatives, and \(\tau\) is the ranking temperature. The caption-side ranking loss \(\mathcal{L}_{\mathrm{cap\_rank}}\) has the same form with the query–caption score \(s(q,c)\).

We control sparsity with the FLOPS regularizer [17]. For a batch of sparse representations, it is \[\ell_{\mathrm{FLOPS}} = \sum_{j \in V} \left( \frac{1}{B} \sum_{i=1}^{B} \mathbf{w}_i[j] \right)^2 .\] This penalizes vocabulary dimensions that are active across many examples. We apply it separately to the image-side and caption-side batches, yielding \(\ell_{\mathrm{FLOPS}}^{p}\) and \(\ell_{\mathrm{FLOPS}}^{c}\), respectively.

We group the losses as \[\begin{align} \mathcal{L}_{\mathrm{R}} &= \mathcal{L}_{\mathrm{rank}} + \lambda_{\mathrm{cap\_rank}} \mathcal{L}_{\mathrm{cap\_rank}},\\[6pt] \mathcal{L}_{\mathrm{S}} &= \lambda_p \ell_{\mathrm{FLOPS}}^{p} + \lambda_c \ell_{\mathrm{FLOPS}}^{c},\\[6pt] \mathcal{L} &= \mathcal{L}_{\mathrm{R}} + \lambda_{\mathrm{cap\_gated}} \mathcal{L}_{\mathrm{cap\_gated}} + \mathcal{L}_{\mathrm{S}}. \end{align}\] All \(\lambda\) values are tunable hyperparameters.

At deployment, all caption-side components are removed. Retrieval uses the sparse dot product between query and image-side document representations.

5 Experiments↩︎

5.1 Overview↩︎

We evaluate V-SPLADE as a scalable lexical retriever for visual-documents. We first report benchmark retrieval quality with indexing throughput, then test production-scale retrieval on an 18.7M-document corpus, measuring serving latency, scaling robustness, and dense-retriever complementarity. Finally, we ablate caption-gated token supervision and analyze its token-level effects.

5.2 Setup↩︎

5.2.0.1 Backbone and training data.

We use ModernVBERT [13] (\(\sim\)​250M parameters), a vision-language backbone with a SigLIP-2 vision encoder, ModernBERT text components, and an LM head. We choose this backbone for its compact size (suitable for large-scale indexing) and its built-in LM head that naturally projects to vocabulary space. We use the same training data as BiModernVBERT [13], the visual-document retrieval adaptation of ModernVBERT. The training mixture consists of ColPali training data (118K image–query pairs) mixed with RLHN [28] (300K text retrieval pairs, 2 hard negatives per query) at a 3:1 text-to-image ratio. BiModernVBERT additionally uses hard negatives on the image side, but the mined hard negatives are not publicly available, so for image–query pairs we rely on in-batch negatives only. We adopt the BiModernVBERT [13] training recipe, which mixes image and text retrieval data. Captions are generated offline only for the ColPali image–query pairs (text retrieval pairs are unchanged) using Qwen3-VL-30B (228 median words per caption) with the ColPali prompt without modification. The full prompt is:

You are an assistant specialized in document analysis. Given a table or a figure, provide a detailed summary (maximum 3000 characters). Your summary should be qualitative and not quantitative. Here is the table/figure: Answer ONLY with the caption.

5.2.0.2 Hyperparameters.

We train for 3 epochs on 4\(\times\)H100 GPUs with batch size 42 per GPU using AdamW [42] with a WSD schedule (5% warmup, 20% decay) and learning rate \(5{\times}10^{-4}\). LoRA [43] (\(r{=}32\)) is applied to both the encoder and the LM head; document tokens are aggregated via max-pooling. The ranking softmax temperature \(\tau\) and caption-gated focus temperature \(\tau_{\mathrm{cap}}\) are set to 0.1 and 0.5, respectively. Sparsity-regularizer warmup is 500 steps, and the caption-gated token supervision weight is \(\lambda_{\mathrm{cap\_gated}}{=}5\) with caption-side FLOPs weight \(\lambda_c{=}0.005\). The quality and efficient operating points share all settings above and differ only in two regularizers: passage FLOPs weight (\(\lambda_p{=}0.01\) for quality vs.\(0.05\) for efficient) and caption sparse-rank weight (\(\lambda_{\mathrm{cap\_rank}}{=}1.0\) vs.\(0.5\)).

5.2.0.3 Evaluation.

We evaluate on standard visual-document retrieval benchmarks, focusing on English-language settings: ViDoRe v1 [3], v2 [30], and v3 [31] (ViD-v1/v2/v3), VisRAG (VRG) [32], VisDoc OOD (VOD) [33], and IRPAPERS (IRP) [4]. Together, these six benchmarks cover a comprehensive range of domains—from arXiv papers [3], [4] to financial and enterprise reports [3], [33]—and retrieval situations—from short keyword queries (“two factor authentication vs single factor security mechanisms”) [3] to figure-grounded queries with LaTeX symbols (“value of \(E_{\alpha}(k)\)”) [32]. We report NDCG@5 [44] unless stated otherwise. For lexical sparse retrievers, including MMLSR and BM25-based systems, we report FLOPs [16] as an efficiency proxy for query–document sparse matching. It estimates the average number of floating-point operations from overlapping active vocabulary dimensions in query–document sparse matching. We abbreviate this metric as FL in tables. Retrieval wall time is measured on a 2-socket Intel Xeon Platinum 8462Y+ server; each latency table reports the CPU thread count used for the corresponding measurement. Document and query encoding are measured on a single NVIDIA H100 GPU.

5.3 Benchmark Retrieval Quality↩︎

Table 2 compares V-SPLADE with baselines in our target serving regime, along with high-capacity reference models.

5.3.0.1 Main result.

V-SPLADE outperforms the two most relevant baselines in this regime. BiModernVBERT serves as the same-scale state-of-the-art compact dense baseline for visual-document retrieval, with fast query encoding and retrieval. BM25 over OCR text or VLM-generated captions represents the query-encoding-free lexical regime. The OCR baseline uses unstructured [36], a widely used document parsing pipeline, with Tesseract [45] in the hi_res setting, following ColPali. The caption baseline uses Qwen3-VL-30B-A3B with the ColPali prompt. For both text sources, we build BM25 indexes with Pyserini [46]. V-SPLADE offers two operating points for inverted-index retrieval: the quality variant maximizes retrieval accuracy at FLOPs = 1.9, while the efficient variant reduces retrieval cost to FLOPs = 1.1. On the six-benchmark average, they reach 60.1 and 56.4 NDCG@5, respectively, and both outperform the main target-regime baselines. Compared with BiModernVBERT’s 46.3—trained on the same backbone and the same training data—they improve by +13.8pp and +10.1pp; they also improve over OCR-based BM25 (54.4) and caption-based BM25 (53.8). We include prior MMLSR methods, D2S [8] and JSDO-Sparse [9], as reference points rather than directly comparable baselines, since they were not trained on visual-document retrieval data. Their results illustrate that existing MMLSR methods do not yet cover the visual-document retrieval task.

5.3.0.2 Complementarity with dense retrieval.

V-SPLADE is not only a standalone lexical retriever; it can also be added to an existing dense retriever or BM25 system as a complementary signal. We use Relative Score Fusion (RSF) [47]: scores from each retriever are min–max normalized per query and then combined with fusion weights that sum to 1, where \(w_{\mathrm{V}}\), \(w_{\mathrm{B}}\), and \(w_{\mathrm{M}}\) denote the weights for V-SPLADE, BiModernVBERT, and BM25, respectively.

With RSF, V-SPLADE quality plus BiModernVBERT, with \(w_{\mathrm{V}}=0.6\) and \(w_{\mathrm{B}}=0.4\), raises the average NDCG@5 from 60.1 to 62.2, indicating that the sparse and dense representations capture complementary evidence. V-SPLADE also complements text-derived lexical retrieval: fusing it with BM25 over unstructured text, with \(w_{\mathrm{V}}=0.7\) and \(w_{\mathrm{M}}=0.3\), improves the average NDCG@5 to 63.0. This two-way lexical fusion requires no neural query encoding, making it useful when serving-time compute is highly constrained. A three-way fusion of V-SPLADE, BiModernVBERT, and BM25 reaches 64.5, with \(w_{\mathrm{V}}=0.5\), \(w_{\mathrm{B}}=0.3\), and \(w_{\mathrm{M}}=0.2\). These results suggest that V-SPLADE can be added as a complementary lexical layer on top of either existing dense visual retrievers or OCR-based BM25 retrieval systems.

The fused same-scale systems also narrow the gap to stronger neural retrievers. In particular, the three-way fusion comes within roughly one NDCG@5 point of ColModernVBERT, the same-scale late-interaction model, while avoiding multi-vector scoring in the first-stage retrievers.

5.3.0.3 Efficiency:

V-SPLADE has substantially lower serving cost than high-capacity models from other retrieval regimes. Although these models serve as references for the current accuracy upper bound, they do not operate in the same serving regime as V-SPLADE. To quantify this gap, we approximate query-time serving cost as the FLOPs required to score 1,000 documents for a single query, averaged over 50 sampled queries and 1,000 sampled documents from the six benchmarks. We measure query-encoding FLOPs with fvcore [48] under each model’s actual evaluation template. We compute scoring FLOPs analytically from each retrieval form: sparse dot products for V-SPLADE, dense vector products for single-vector retrievers, and MaxSim operations for multi-vector retrievers. Figure 2 shows that the strongest models in the competing dense and late-interaction regimes require higher online compute than V-SPLADE.

5.3.0.4 Document encoding throughput.

We measure document encoding throughput to test whether V-SPLADE scales efficiently compared with other query-encoding-free lexical candidates. We compare V-SPLADE against two text-generation/extraction pipelines used to build BM25-style lexical indexes: a Qwen3-VL-30B-A3B MoE caption generator (3B effective active parameters at inference, served with vLLM [49]) and an OCR pipeline (unstructured [36] with Tesseract [45] hi_res). For V-SPLADE, we run inference with a standard PyTorch DataLoader; the caption and OCR baselines are run with their respective serving stacks. All systems are measured on a single H100 GPU with 4 CPU cores, using 1,000 sampled documents from the six benchmarks. As shown in Table 3, V-SPLADE encodes page images at 20.19 pages/sec, compared with 0.83 pages/sec for caption generation and 0.90 pages/sec for the OCR pipeline, making it over \(20\times\) faster than both text-extraction alternatives.

4pt

Taken together, these results show that V-SPLADE is competitive standalone, complementary to OCR-based BM25 and BiModernVBERT, and efficient in both online serving and document encoding.

5.4 Production-Scale Retrieval↩︎

The standard benchmarks in Section 5.3 contain under 25,000 pages per benchmark, which is still far from production-scale retrieval. We therefore evaluate V-SPLADE on an 18.7M-page corpus built from PDFA [50] and DocMatix [1]. PDFA is a large open-source PDF corpus derived from SafeDocs [51], while DocMatix is a labeled subset of PDFA with generated question–answer pairs. We render PDFA into page images and treat each page image as a retrievable item. Each DocMatix question is then converted into a recall query whose relevant items are the page images from the corresponding source document.

We compare V-SPLADE with BiModernVBERT, the same-scale compact state-of-the-art dense retriever trained for visual-document retrieval. For compactness, we denote BiModernVBERT as vbert in the tables. Dense retrieval uses FAISS [52] Flat and HNSW [11], while V-SPLADE uses PISA [53] inverted-index retrieval. For HNSW indexing, we use \(M{=}32\) and \(\texttt{efConstruction}{=}128\). We report the best latency–recall trade-off at \(\texttt{efSearch}{=}256\). All recall and latency measurements in Table 4 are computed on a fixed set of 1,000 randomly sampled queries. For V-SPLADE, we evaluate both full-index search and a two-stage sparse search. Full search scores the complete inverted index. The two-stage variant adopts the idea of Two-Step SPLADE [54]: a pruned index is used for fast candidate generation, and the selected candidates are rescored with the full index. In our implementation, the first stage uses a pruned top-50 index. Unless stated otherwise, we use the quality variant of V-SPLADE in the production-scale retrieval experiment; the efficient variant is used to show the latency–accuracy trade-off.

4pt

5.4.0.1 Retrieval quality and latency at 18.7M scale.

V-SPLADE consistently outperforms the same-backbone dense baseline on the 18.7M-page corpus without neural query encoding. Table 4 shows a clear gap at the top ranks: full-index V-SPLADE reaches R@5=0.228, compared with 0.090 for FAISS Flat BiModernVBERT. The same trend holds at a broader cutoff, where full-index V-SPLADE reaches R@100=0.396, compared with 0.228 for Flat. The two-stage sparse variant, which uses a pruned first-stage index before rescoring, reaches R@5=0.183 and R@100=0.278, well above the dense HNSW baseline at R@5=0.071 and R@100=0.191.

Table 4 shows that V-SPLADE reaches the low-latency regime without neural query encoding. In the low-latency setting, two-stage sparse retrieval reaches 0.45 ms/query for the quality model and 0.38 ms/query for the efficient model with 20 CPU threads, which is comparable to HNSW retrieval at 0.31 ms/query. Unlike the dense baselines, these sparse variants require no neural query encoding, avoiding the additional 85.87 ms CPU or 0.08 ms batched query-encoding cost on an H100 GPU. When recall is prioritized, full-index sparse retrieval provides a higher-recall operating point. Although slower than HNSW, the quality variant still runs in the sub-10 ms regime at 5.89 ms/query with 20 CPU threads and achieves the highest recall among the same-backbone systems.

5.4.0.2 Larger gains on lexically specific queries.

V-SPLADE shows its largest gains on queries when exact lexical evidence is likely to matter. We split queries by two heuristics—whether they contain digits or uppercase characters beyond the first character—which approximate numerals and proper-noun-like expressions. As shown in Table 5, the sparse-over-dense gap is larger on these subsets than on all queries, peaking when both features are present: V-SPLADE reaches R@5=0.363 versus 0.135 for the dense baseline. This suggests that the production-scale gains are concentrated in lexically specific queries.

5.4.0.3 Robustness under corpus scaling.

Sparse lexical representations may be more robust under corpus scaling because they operate over a much larger vocabulary-indexed space than fixed-dimensional dense embeddings. Motivated by recent work on the dimensional limits of dense retrieval capacity [55], we evaluate this hypothesis by scaling the corpus from 500K to 18.7M pages and report R@5/R@100 in Table 6. At R@5, the dense retriever drops from 0.260 to 0.090, retaining only 35% of its 500K performance, whereas V-SPLADE drops from 0.424 to 0.228, retaining 54%. Figure 3 visualizes the same trend with recall normalized to the 500K setting: the dense–sparse gap widens monotonically as the corpus grows. This suggests that the lexical sparse representations are more robust to performance degradation under corpus scaling.

5pt

5.4.0.4 Complementarity with existing neural retrievers at large scale.

V-SPLADE remains complementary to existing neural retrievers at production scale, including a billion-scale dense model and a late-interaction retriever. Dense and late-interaction retrievers can also be practical when sufficient GPU resources are available, especially for compact models such as BiModernVBERT. We therefore ask whether V-SPLADE adds value beyond replacing dense retrieval. On the 18.7M corpus, we evaluate two integration scenarios: score fusion with dense retrievers and first-stage retrieval for multi-vector reranking.

For score fusion, we use Relative Score Fusion (RSF) [47], where scores from each retriever are min–max normalized per query and combined as \(s_{\mathrm{fused}} = w_d \cdot s_{\mathrm{sparse}} + (1 - w_d) \cdot s_{\mathrm{dense}}\), where \(w_d\) denotes the mixing weight assigned to V-SPLADE. We test fusion with two representative dense retrievers at different scales: Qwen3-VL-Embedding-2B [41], a billion-scale SOTA dense retriever, and BiModernVBERT [13], a compact same-backbone dense retriever. We denote them as qwen and vbert in Table 7. Table 7 shows that V-SPLADE improves both dense systems. With Qwen3-VL-Embedding-2B, fusion improves R@5 from 0.327 to 0.343. With BiModernVBERT, fusion improves R@5 from the better standalone system by +2.4pp. By contrast, RSF between the two dense retrievers does not improve over the stronger dense retriever, suggesting that the gains are unlikely to be explained solely by ensembling retrievers. These gains suggest that the lexical signal from V-SPLADE can complement dense retrievers across model scales.

We also test whether V-SPLADE can improve a two-stage pipeline with a stronger multi-vector reranker. Multi-vector visual retrievers are accurate but too expensive to scan the full 18.7M corpus, so they require a cheap first-stage retriever. Figure 4 compares two top-100 first-stage pipelines before ColModernVBERT reranking: BiModernVBERT \(\rightarrow\) ColModernVBERT and V-SPLADE \(\rightarrow\) ColModernVBERT.

After reranking, the V-SPLADE first-stage pipeline reaches 0.311 R@5, up from 0.228 before reranking, whereas the dense first-stage pipeline reaches 0.194, up from 0.090. The difference is governed by the first-stage recall ceiling: V-SPLADE retrieves a stronger top-100 candidate set than BiModernVBERT, with R@100 of 0.396 versus 0.228. Thus, V-SPLADE is not only an efficient standalone retriever, but also a complementary lexical component that boosts dense retrievers through score fusion and provides stronger candidates for multi-vector reranking.

Overall, these results show that V-SPLADE is effective at large corpus scale, especially for lexically specific queries. V-SPLADE also scales robustly and complements stronger dense and late-interaction systems as a lexical layer.

5.5 Ablations and Analysis↩︎

We analyze caption-gated token supervision from three perspectives: whether the signal is necessary, whether our design choices are effective, and how the signal acts on the image sparse representation during training. Through component ablations and alternative-design studies, we first examine the contribution of caption-gated token supervision to the quality–efficiency trade-off. We then analyze token-level training dynamics to show how this signal complements and reshapes the visual passage sparse embedding. We use the efficient V-SPLADE variant (\(\lambda_p{=}0.05\)) and run quantitative ablations across all six benchmarks.

5.5.0.1 Ablating the training signal.

Table 8 shows that caption-gated token supervision is the key component behind the improved quality–efficiency trade-off. Li-LSR alone improves over the binary BoW baseline (.537\(\rightarrow\).549), but increases FLOPs from 1.88 to 3.06. With caption-gated supervision, NDCG@5 rises to .564 and FLOPs drops to 1.10. Compared with caption-based BM25 baselines, Li-LSR alone reaches a similar quality range; caption-gated supervision opens a clear gap, improving NDCG@5 by more than two points while reducing FLOPs.

The alternative-design variants in Table 8 further support our design choices. Treating captions only as additional ranking positives provides a modest improvement over Li-LSR alone (.549\(\rightarrow\).556), while also slightly reducing FLOPs (3.06\(\rightarrow\)​2.53). Directly aligning passage and caption sparse vectors with cosine similarity was unstable: it collapsed under the efficient-variant hyperparameters across multiple seeds, and even with the quality-variant hyperparameters it reached only 56.16 NDCG@5 at 2.47 FLOPs, below the original quality model’s 60.1 NDCG@5 at 1.9 FLOPs. Replacing the softplus Li-LSR activation with ReLU consistently collapsed training across multiple seeds, including runs with the quality-variant hyperparameters.

5.5.0.2 How lexical grounding reshapes sparse activations during training.

Finally, we examine what additional lexical evidence the caption sparse embedding provides beyond the visual passage embedding alone. For this analysis, we train V-SPLADE for one epoch on 90% of the ColPali image–query pairs, and then inspect the held-out 10%. On these unseen samples, we compare the top sparse tokens produced by the visual passage encoder and by the corresponding caption encoder.

4pt

The passage and caption sparse embeddings do not activate the same vocabulary dimensions: 64% of top-30 tokens are disjoint on average. Table 9 shows that this complementarity is useful for retrieval: passage-only and caption-only tokens reach Hit@1 of 0.43 and 0.44, while taking their union improves Hit@1 to 0.53. This indicates that passage and caption embeddings provide different views of the same visual-document rather than redundant token sets.

2pt

Table 10 qualitatively illustrates this effect. Q, P, and C denote the top activated tokens from the query, image passage, and caption representations, while P* reports the image-passage activation of the top caption token. Captions often highlight retrieval-relevant tokens that are weak in the passage embedding alone: in Case 1, “SVM” is strongly activated by the query and caption (0.77 and 1.02), but only weakly by the image passage (0.18). Caption-gated token supervision can therefore reinforce such under-activated image-side dimensions that the passage encoder would otherwise miss.

We next examine how this mechanism changes training dynamics. Figure 5 tracks the stopword fraction among query–image matched tokens during training, using the 33 Lucene stopwords with casing variants [46]. After the 0.5-epoch warmup, Li-LSR increasingly matches queries through stopwords, whereas V-SPLADE keeps this fraction much lower. This suggests that caption-gated supervision shifts matching away from generic lexical dimensions toward content-bearing document tokens.

4pt

Table 11 summarizes the resulting token-level behavior. Compared with Li-LSR alone, V-SPLADE is less sensitive to masking high-weight query tokens, yields a larger top-1/top-2 margin, and produces fewer stopword activations with shorter passage token lists. Together, these results suggest that caption-gated token supervision makes sparse representations more selective and less dominated by noisy or brittle lexical matches.

5.5.0.3 Effect of caption-generator scale.

We also test whether the effectiveness of caption-gated token supervision depends on the caption generator scale. We generate captions with Qwen3-VL models of varying sizes [29] and train V-SPLADE with each caption source under the efficient variant hyperparameters. Across different caption generators, caption-gated supervision consistently improves retrieval quality over the Li-LSR only baseline, indicating that the gain is not tied to a single caption model but comes from using captions as lexical supervision for image-side sparse representations.

5pt

In summary, caption-gated token supervision improves lexical grounding by reinforcing retrieval-relevant image-side vocabulary dimensions, outperforming alternative supervision strategies and remaining robust across caption generators.

6 Conclusion↩︎

Visual document retrieval still lacks a deployable lexical retriever for production-scale search. Multimodal learned sparse retrieval naturally fits this missing operating point, but suffers from a lexical grounding challenge: visual pages do not explicitly indicate which vocabulary dimensions should be activated. We address this challenge with V-SPLADE, which uses caption-gated token supervision to guide image-side vocabulary activation during training.

Across standard benchmarks and an 18.7M-document corpus, V-SPLADE outperforms the main baselines in its target regime without neural query encoding, and builds indexes substantially faster than OCR- or caption-based lexical pipelines. It also complements neural retrievers through score fusion and as a first-stage retriever for multi-vector reranking. Taken together, these results position V-SPLADE as a lexical retrieval layer for the missing deployment setting in visual-document retrieval: query-encoding-free serving over directly indexed visual documents, with strong standalone performance and complementarity to dense and multi-vector systems.

6.0.0.1 Limitations and Future Work.

Our study has four main limitations that we view as natural directions for future work. First, we restrict the evaluation to English-language visual-documents; how the same caption-gated supervision behaves under multilingual retrieval remains to be studied. Second, we deliberately focus on a compact, efficient sub-billion-parameter sparse model, leaving open how the approach scales when applied on top of substantially larger backbones. Third, our analysis is limited to visual-document retrieval; whether the same lexical grounding mechanism transfers to broader multimodal tasks (e.g., natural-image retrieval or video) is an open question. Fourth, we use a fixed captioning setup; studying prompt-controlled captions that target specific document regions, structures, or domain cues is a promising direction for specialized visual-document tasks.

GenAI Usage Disclosure↩︎

Generative AI tools were used to improve the clarity and style of the writing and to assist with code drafting and debugging. All technical content, experiments, code, and results were reviewed and validated by the authors.

References↩︎