Eddy-VL 1.9B: Structural Pruning and Layered Distillation
for Edge-Deployable Multimodal Embedding
July 15, 2026
Abstract
In this report, we introduce Eddy-VL 1.9B, a compressed multimodal embedding model built on Qwen3-VL-Embedding-2B [1], [2] for offline, edge-deployable vision–language retrieval. Eddy-VL targets air-gapped forensic and investigative settings where cloud APIs are unavailable and latency matters. Compression combines (i) probe-driven structural pruning that removes four redundant text-decoder layers (28\(\rightarrow\)24) ranked by adjacent-layer linear CKA [3], and (ii) layered knowledge distillation [4] with hole-covering teacher–student mappings, mid-layer attention-map \(1{-}\mathrm{CKA}\), and final-layer MSE+cosine with Matryoshka dims [5] \(\{128,\ldots,2048\}\). The release model contains 1,926,188,032 parameters (3.85 GB bf16), a \(\sim\)9.5% reduction from the 2.13B teacher.1 Empirical evaluations on MMEB-V2 [6] (78 tasks, VLM2Vec [7]) show that Eddy-VL scores 63.2 overall vs.68.9 for the teacher under our protocol (91.7% retained), recovering 6.4 of 12.1 points lost by pruning alone (56.8). Compositional probes remain near teacher quality on SugarCrepe [8] (86.1 vs.86.4), MR2-Bench [9] (24.5 vs.24.7), and ARO [10] (59.5 vs.60.4); Winoground [11] group score is the main weakness (6.8 vs.8.5). Depth pruning also reduces forward latency by \(\sim\)10% (150.0\(\rightarrow\)136.4 ms/image on NVIDIA DGX Spark; FlashAttention-2 [12]). This report presents the architecture, compression methodology, training procedures, and evaluation results, demonstrating effectiveness for multimodal retrieval under constrained edge deployment. Weights and inference code are released on Hugging Face [13].
1 Introduction↩︎
Multimodal embedders map images, text, video, and documents into a shared vector space for nearest-neighbor retrieval. Recent VLM-based embedders [2], [7] achieve strong MMEB-V2 [6] scores via contrastive pre-training and reranker distillation on Qwen3-VL backbones [14]. Qwen3-VL-Embedding-2B is a leading open teacher, but its 28-layer text decoder and 4.26 GB checkpoint remain heavy for forensic edge settings: air-gapped networks, seized-media processing, and strict no-cloud policies.
We study depth compression of an existing strong teacher—preserving retrieval behavior, not only parameter count—through probe-driven pruning and layered distillation. Unlike training a new foundation embedder from scratch, Eddy-VL removes four decoder blocks and recovers quality with teacher-aligned mid- and final-layer objectives on an explicit teacher–student layer map that brackets each removed block.
All reported scores use the official MMEB-V2 benchmark via VLM2Vec (78 tasks) and standard compositional probes. We report relative compression gains under a fixed evaluation protocol rather than claiming parity with every published leaderboard configuration (§6.1).
In the following sections, we describe the model (§2), the training data (§3), training strategy (§4), training objectives (§5), evaluation (§6), analysis (§7), and conclude with limitations and deployment guidance (§8).
2 Model↩︎
Eddy-VL and its teacher follow the bi-encoder embedding paradigm introduced in Qwen3-VL-Embedding [2]. Each multimodal instance is encoded independently; relevance is cosine similarity in a shared 2048-dimensional space. Eddy-VL 1.9B is derived from Qwen3-VL-Embedding-2B [1] by probe-driven structural pruning of the text decoder (28\(\rightarrow\)24 layers), followed by layered knowledge distillation from the unchanged teacher checkpoint.
2.1 Architecture↩︎
Qwen3-VL-Embedding-2B [1], [2] uses a
24-block vision encoder (\(d{=}1024\), patch 16, merge 2, proj 2048) and a 28-layer text decoder (\(d{=}2048\), 16 Q-heads / 8 KV-heads GQA, FFN 6144). Eddy-VL removes four decoder blocks
(\(\mathcal{D}=\{3,22,24,25\}\), 0-based HuggingFace layers.* indices), yielding a 24-layer student with remapped indices. The vision encoder is frozen during distillation; only the compressed text decoder is
trained. Table 1 summarizes the release checkpoint.
| Teacher | Eddy-VL 1.9B | |
|---|---|---|
| Parameters | 2,127,532,032 | 1,926,188,032 |
| Checkpoint (bf16) | 4.26 GB | 3.85 GB |
| Vision encoder blocks | 24 | 24 (frozen) |
| Text decoder layers | 28 | 24 |
| Embedding dimension | 2048 | 2048 |
| MRL dims | \(\{128,\ldots,2048\}\) | \(\{128,\ldots,2048\}\) |
2.2 Embedding method↩︎
The input format follows the Qwen3-VL-Embedding chat template: a system message carries the retrieval instruction (default: “Represent the user’s input.”), and the multimodal instance is passed as a user message (text, image, video, or mixed
modalities). A final <|endoftext|> token is appended; the last hidden state at this token is L2-normalized to produce the retrieval vector. The released embedder exposes this pipeline through a self-contained API on Hugging Face.
Unless otherwise stated, we use the full 2048-dimensional embedding for retrieval and evaluation (MRL enables computing lower-dimensional prefixes with the same encoder when needed).
3 Data↩︎
Distillation uses two proprietary corpora with complementary supervision (Table 2): Korean COCO supplies image–caption pairs—each sample includes a Korean text caption alongside the image (§3.1); OurDataset is image-only—roughly 45k RGB images with no accompanying text (§3.2). Both streams share the default retrieval instruction and last-token pooling; only Korean COCO instances pass caption tokens through the text decoder. Ablation and release checkpoints train on the combined mixture.
| Corpus | Supervision | Sample format | Text? | Approx.\(N\) |
|---|---|---|---|---|
| Korean COCO | Caption | image + Korean caption | yes | \({\sim}\)410k |
| OurDataset | None | image only | no | \({\sim}\)45k |
| Combined | Mixed | both formats above | — | \({\sim}\)455k |
3.1 Korean COCO captions↩︎
Korean COCO denotes the Korean Image Captioning Dataset released on AI Hub [15], [16] (dataset no.): MS COCO [17] images paired with Korean text captions
obtained by machine-translating the original English COCO captions, followed by AI Hub quality correction. Each record provides one or more caption_ko strings per image (JSON metadata with UTF-8 text, as distributed by AI Hub). We use a
processed training split with on the order of 80k unique images; with multiple captions per image this expands to roughly 410k distillation instances. Every instance is a multimodal forward pass: image tokens plus a Korean
caption under the default retrieval instruction. The corpus is licensed through AI Hub and is not redistributed with Eddy-VL.
3.2 OurDataset↩︎
OurDataset is a proprietary synthesized image corpus for forensic and investigative retrieval, built by merging curated subsets from multiple sources: licensed AI Hub resources [16] (office-document generation, Korean visual understanding, multimodal information retrieval, and academic/paper understanding), public benchmarks (MS COCO [17], Korean COCO [15], SUN [18], RVL-CDIP [19], CORD v2 [20]), and internally collected or synthetically generated media. Unlike Korean COCO, no text captions are provided: each sample is a single image with the default retrieval instruction only (image-only distillation).
The corpus spans forensic-relevant domains including scenes and places, people and animals, email, academic papers, presentations and slides, handwriting, screenshots, administrative and financial documents, road imagery, device and PC/phone screens, identity documents, technical drawings, logistics, receipts, GPS/geolocation imagery, and whiteboard captures. Overall volume is roughly 45k images. Neither OurDataset nor AI Hub–derived subsets (including Korean COCO [15]) are publicly redistributed. Source-level attributions (AI Hub, Korean COCO, MS COCO, SUN, RVL-CDIP, CORD) appear in the prose above; Table 2 reports only corpus scale and supervision type.
4 Training strategy↩︎
Training proceeds in two stages: probe-driven structural pruning (§4.1), then distillation from the unchanged teacher checkpoint on Korean COCO + OurDataset. Figure 3 shows the training loss curve; the best checkpoint is used for all reported Eddy-VL results.
4.1 Probe-driven structural pruning↩︎
On \(N\) calibration image–caption pairs, extract last-token hiddens after each of the 28 teacher decoder blocks. We use 0-based block indices \(\ell\in\{0,\ldots,27\}\)
(HuggingFace language_model.layers[\(\ell\)]), form \(X_\ell\in\mathbb{R}^{N\times H}\), and score adjacent-layer redundancy with linear CKA [3]: \(s(\ell)=\mathrm{CKA}(X_{\ell-1},X_\ell)+\mathrm{CKA}(X_\ell,X_{\ell+1})\) (middle blocks only). We rank blocks on \(N{=}1{,}000\) calibration pairs (Korean COCO + OurDataset mix). Greedy top-4 CKA would remove \(\{21,22,23,24\}\); we use \(\mathcal{D}=\{3,22,24,25\}\) instead
(block 3 replaces block 23 for early-depth coverage). Figure 2 plots \(s(\ell)\) across all decoder blocks.
4.1.0.1 Beyond the \(s(\ell)\) sum.
Ranking only by \(s(\ell)\) also suggests an alternative four-block set \(\{2,21,22,24\}\) (blocks 2, 21, 22, and 24 rank 7th, 3rd, 1st, and 4th among middle blocks). We therefore inspect the decomposition \(s(\ell)=\mathrm{CKA}(X_{\ell-1},X_\ell)+\mathrm{CKA}(X_\ell,X_{\ell+1})\) rather than the sum alone (Table 3).
| Block | \(\mathrm{CKA}_{\ell-1,\ell}\) | \(\mathrm{CKA}_{\ell,\ell+1}\) | \(s(\ell)\) | Balance |
|---|---|---|---|---|
| 2 | 0.746 | 0.668 | 1.414 | 0.944 |
| 3\(^\ast\) | 0.668 | 0.702 | 1.370 | 0.975 |
| 21 | 0.803 | 0.825 | 1.629 | 0.986 |
| 22\(^\ast\) | 0.825 | 0.865 | 1.691 | 0.976 |
| 23 | 0.865 | 0.786 | 1.652 | 0.952 |
| 24\(^\ast\) | 0.786 | 0.792 | 1.579 | 0.996 |
| 25\(^\ast\) | 0.792 | 0.735 | 1.528 | 0.963 |
\(^\ast\)Removed in \(\mathcal{D}=\{3,22,24,25\}\)
.
Block 2 has a high \(s(\ell)\) because both adjacent CKAs are elevated (\(0.746\) below, \(0.668\) above), not because one side dominates; its balance score (\(1 - |\mathrm{CKA}_{\ell-1,\ell}-\mathrm{CKA}_{\ell,\ell+1}|/s(\ell)=0.944\)) is only moderately asymmetric. However, block 2 is more similar to block 1 than to block 3 (\(\mathrm{CKA}_{1,2}>\mathrm{CKA}_{2,3}\)), whereas block 3 is more balanced (balance \(0.975\)) and better preserves the early-depth transition. The neighbor block 1 shows the largest local imbalance in the stack (\(0.368\) vs.\(0.746\)), indicating a sharp early representation ramp that makes shallow-layer removal risky even when \(s(\ell)\) is high.
We validate the layer choice with a structural-pruning ablation (weight remapping only, no distillation): export a 24-layer student with \(\{2,21,22,24\}\) removed and score it on MMEB-Image (batch size 128; Table 4).
| Task | \(\{2,21,22,24\}\) | \(\{3,22,24,25\}\) | \(\Delta\) |
|---|---|---|---|
| HatefulMemes | 49.2 | 59.2 | -10.0 |
| ImageNet-1K | 0.1 | 44.3 | -44.2 |
| ImageNet-A | 1.6 | 30.3 | -28.7 |
| ImageNet-R | 0.9 | 69.7 | -68.8 |
| N24News | 4.5 | 25.3 | -20.8 |
| Place365 | 0.3 | 28.1 | -27.8 |
| SUN397 | 0.3 | 52.1 | -51.8 |
| VOC2007 | 3.0 | 80.0 | -77.0 |
| Macro-avg (8 tasks) | 7.5 | 48.6 | -41.1 |
The alternative set collapses on classification-heavy tasks (e.g., ImageNet-1K \(0.1\) vs.\(44.3\) hit@1), confirming that probe sum alone is insufficient and supporting \(\mathcal{D}=\{3,22,24,25\}\) for the release pruned student.
4.2 Pipeline summary↩︎
Rank teacher layers by \(s(\ell)\) on \(N{=}1{,}000\) calibration pairs.
Export a 24-layer pruned student by weight remapping (no fine-tuning).
Distill the pruned student from the frozen-vision teacher on Korean COCO + OurDataset.
Select the best checkpoint and release it as Eddy-VL 1.9B.
| Setting | Value |
|---|---|
| Distillation | Layered (attention-map mid, MRL final) |
| Mid / final loss | \(1{-}\mathrm{CKA}\) / MSE+cosine (0.3/0.7) |
| MRL dims | 128, 256, 512, 1024, 2048 |
| Layer mapping | \(3{:}4,21{:}23,22{:}26,23{:}27,24{:}28\) |
| Batch / LR / warmup | 32 / \(3.63{\times}10^{-5}\) / 0.05 |
| Max length / image pixels | 8192 / 1024–921600 |
| Vision encoder | Frozen (no LoRA) |
| Steps / best checkpoint | 15068 / 12366 |
| Best total loss | 0.889 |
5 Training objective↩︎
All distillation runs start from the same 24-layer pruned student with a frozen vision encoder and a hole-covering teacher–student mapping \(\mathcal{M}=\{(3,4),(21,23),(22,26),(23,27),(24,28)\}\) (student indices 0-based; teacher indices use the probe hidden-state numbering, teacher block \(b\) \(\leftrightarrow\) index \(b{+}1\)) that pairs each removed teacher block with neighboring survivor layers. We compare two distillation families: hidden distillation and layered distillation.
5.0.0.1 Hidden distillation (final hidden only).
Hidden distillation aligns only the final hidden states: teacher and student last-token embeddings are matched with MSE+cosine. No losses are applied on intermediate layers in \(\mathcal{M}\).
5.0.0.2 Layered distillation (mid + final).
Layered distillation adds mid-layer losses on each mapped pair in \(\mathcal{M}\), aligning intermediate teacher and student activations (default: \(1{-}\mathrm{CKA}\) [3] on flattened attention maps), plus a final-layer loss on \((24,28)\) using MSE+cosine with Matryoshka Representation Learning [5]. Dynamic loss scaling targets a 40/60 mid:final ratio. The release model uses attention-map CKA mid-losses. Alternative mid-layer objectives include RCKA [21], STRUCTURE [22], and mutual-kNN [23].
5.0.0.3 Mid-layer objective variants.
All variants below replace only the mid term under layered distillation (the final-layer MRL MSE+cosine term is kept the same). RCKA [21] matches relation-centered similarity by comparing centered Gram matrices of teacher vs.student representations. STRUCTURE [22] aligns neighborhood structure by applying a row-softmax to Gram similarities and minimizing a Jensen–Shannon divergence over the resulting transition distributions. Mutual-kNN [23] builds a mutual \(k\)-nearest-neighbor graph on teacher features within the batch and minimizes MSE over student vs.teacher pairwise cosine similarities on the mutual edges.
6 Evaluation↩︎
6.1 Setup↩︎
We evaluate on the official MMEB-V2 benchmark [6] with the public VLM2Vec harness [7] (78 tasks), following the framework of Qwen3-VL-Embedding [2]. MMEB-V2 overall is the unweighted mean of 78 per-task metrics (36 image + 18 video Hit@1; 24 VisDoc nDCG@5). Fine-grained probes (SugarCrepe, ARO, Winoground, MR2) are a separate compositional suite with cosine retrieval accuracy or nDCG@10; they do not overlap with MMEB-V2 task definitions.
6.1.0.1 Hardware.
Unless noted otherwise, all benchmark scores, fine-grained probes, and forward-pass latency measurements in this report were run on a single NVIDIA DGX Spark GPU (bf16, FlashAttention-2 where applicable).
6.1.0.2 Protocol caveats.
Our VLM2Vec evaluation uses a fixed generic instruction (“Represent the user’s input.”) and 8 video frames per clip. Qwen’s published MMEB numbers use task-tuned prompts and up to 64 frames. Our teacher re-evaluation scores 68.9 overall vs.73.2 on the Qwen model card [1], [2]. We therefore report relative gains (68.9\(\rightarrow\)56.8\(\rightarrow\)63.2), not parity with the public leaderboard.
6.2 MMEB-V2 results↩︎
Pruning costs 12.1 points; layered CKA distillation recovers 6.4 (68.9 \(\rightarrow\) 56.8 \(\rightarrow\) 63.2). Table 6 lists headline scores; the 73.2 teacher figure is the public Qwen3-VL-Embedding-2B leaderboard entry [1], [2].
| Model | Overall | Image | Video | VisDoc |
|---|---|---|---|---|
| Teacher (public leaderboard) | 73.2 | — | — | — |
| Teacher (our eval.) | 68.9 | 71.0 | 55.2 | 76.1 |
| Pruned baseline | 56.8 | 55.7 | 46.2 | 66.4 |
| Eddy-VL 1.9B | 63.2 | 63.4 | 50.0 | 72.7 |
Eddy retains 91.7% of teacher MMEB under our protocol while removing 4 decoder layers. Eddy recovers a substantial portion of the prune-only gap, with especially strong retention on fine-grained compositional probes (§7).
6.3 Fine-grained probes↩︎
| Benchmark | Metric | Teacher | Pruned baseline | Eddy |
|---|---|---|---|---|
| SugarCrepe | Overall (%) | 86.4 | 75.7 | 86.1 |
| ARO | Overall (%) | 60.4 | 61.7 | 59.5 |
| Winoground | Group (%) | 8.5 | 5.0 | 6.8 |
| MR\(^2\) | nDCG@10 | 24.7 | 20.3 | 24.5 |
Eddy is within 0.3 points of teacher on SugarCrepe and within 0.2 on MR\(^2\) (Table 7), indicating near-full recovery on key compositional probes.
6.4 Efficiency↩︎
Forward latency (Table 8) was measured on NVIDIA DGX Spark with FlashAttention-2 [12] on a held-out internal image set (\(\sim\)44k images; batch size 24; single GPU). Eddy-VL matches the pruned baseline architecture; depth pruning yields \(\sim\)10% lower forward time vs.the teacher.
| Model | Decoder layers | Forward mean (ms/image) |
|---|---|---|
| Qwen3-VL-Embedding-2B (teacher) | 28 | 150.0 |
| Pruned baseline | 24 | 136.4 |
| Speedup (pruned vs.teacher) | 1.10\(\times\) (\(\sim\)10% faster) |
7 Analysis↩︎
7.1 Compression and distillation ablations↩︎
| Training recipe | Overall | Image | Video | VisDoc |
|---|---|---|---|---|
| Qwen3-VL-Embedding-2B (teacher) | 68.9 | 71.0 | 55.2 | 76.1 |
| Layered / attention CKA (Eddy release)\(^\star\) | 63.2 | 63.4 | 50.0 | 72.7 |
| Pruned baseline | 56.8 | 55.7 | 46.2 | 66.4 |
| Hidden distill (final hidden only)\(^\ddagger\) | 60.8 | 60.8 | 48.6 | 69.9 |
| Layered / RCKA\(^\dagger\) | 61.7 | 62.1 | 48.9 | 70.8 |
| Layered / STRUCTURE\(^\dagger\) | 60.6 | 60.6 | 48.1 | 70.0 |
| Layered / mutual-kNN\(^\dagger\) | 58.9 | 58.8 | 45.2 | 68.2 |
7.2 Compositional retention↩︎
SugarCrepe and MR2 recover to within \(\sim\)1 point of teacher; Winoground group score is the lowest among the listed probes (6.8 vs.).
7.2.0.1 Interpretation.
Fine-grained probes highlight how pruning and distillation affect compositional alignment separately from MMEB-V2 task mixtures: Eddy recovers to near-teacher on SugarCrepe and MR\(^2\), while Winoground leaves the most headroom for improvement.
8 Conclusion↩︎
Eddy-VL 1.9B combines probe-driven depth pruning and layered CKA distillation to deliver a 1.93B multimodal embedder retaining \(\sim\)92% of teacher MMEB score under our evaluation protocol, with near-parity on compositional probes and \(\sim\)10% lower forward latency. Layered attention-map CKA outperformed hidden-only distillation and other mid-loss variants in ablations on the combined Korean COCO + OurDataset mixture, yielding the release score of 63.2. Eddy-VL is an embedder for candidate surfacing—pair with a reranker for final ordering.
8.0.0.1 Limitations.
\(\sim\)8.3% relative MMEB gap vs.teacher under our protocol (\(-\)5.7 absolute).
Our MMEB re-evaluation \(\neq\) Qwen leaderboard 73.2 (§6.1).
Training corpora are proprietary and not publicly released (AI Hub–licensed and in-house subsets; see §3.2).
Winoground remains the most challenging probe under our compression recipe.
References↩︎
“Eddy-VL 1.9B” is a release name; the exact parameter count is 1.93B.↩︎