1 Introduction↩︎

Current paradigms in visual representation learning struggle to reconcile geometry with semantics. Contrastive methods and large Vision-Language Models (VLMs) [1]–[3] operate primarily in language space, yielding strong semantics but neglecting precise spatio-temporal details. Conversely, pixel-level reconstruction approaches like masked auto-encoding (MAE) [4]–[7] excel at preserving local geometry and motion but struggle to generalise to broad semantic tasks. A single, task-agnostic foundation video model that supports both types of capabilities remains an open problem.

Inspired by the predictive coding theory of the brain, which posits that prediction is key to cognition [8], [9], we investigate how to build general-purpose video models based on generative video diffusion transformers. We argue that generating temporally and geometrically consistent video requires the model to implicitly learn the underlying mechanics of the physical world, such as motion, object permanence, and part-object relationships. However, recent studies probing frozen video diffusion representations concluded that they capture low-level geometry but struggle with high-level semantics, casting doubt on their eligibility as general-purpose perception encoders [10], [11].

In this work, we demonstrate that state-of-the-art video diffusion models do, in fact, overcome this limitation as they become more capable at generating physically and temporally consistent videos. By systematically investigating the intermediate activations of both proprietary (Veo3 [12]) and open-weight (Wan2.2-T2V-A14B [13]) models, we map the hierarchy of their latent spaces using recent zero-shot mutual k-NN alignment metrics [11], [14] and we identify optimal spots for extracting linearly separable global semantics and more entangled, spatially-distributed features that require attention-based pooling to extract effectively.

Based on these observations, we introduce Gen4U (Generation for Understanding), a framework that extracts and repurposes generative representations for a diverse set of downstream visual tasks. Because Gen4U identifies the optimal block and noise level for feature extraction, it requires only a single forward pass through the diffusion backbone. This makes it as computationally efficient as a standard discriminative visual encoder, e.g. [6], [15], circumventing the high inference costs typically associated with iterative denoising. By pairing these frozen representations with lightweight decoders, Gen4U achieves strong perception performance without altering the pre-trained generative weights. Our extensive evaluations demonstrate that frozen video diffusion models can function as general-purpose video encoders.

To summarise, our main contributions are:

1. Latent space analysis: We map the evolution of diffusion features across depth and noise levels using recent alignment metrics, identifying the optimal points to extract representations for understanding.

2. Semantics and temporal dynamics: We show that Gen4U captures rich, high-level understanding, achieving state-of-the-art results on zero-shot alignment and video classification on Something-Something V2 dataset [16], alongside satisfactory results on image and video captioning.

3. Geometry understanding: We demonstrate that these same frozen representations excel at lower level geometry awareness, achieving strong performance on monocular depth estimation and camera pose estimation.

2 Related work↩︎

The most common paradigms for learning visual representations are variations of masked autoencoding (MAE) [4], [5], [17], scaled in 4DS [6] and D4RT [7], and contrastive learning, with or without negative pairs (SigLip [18], SimCLR [19], CLIP [1], BYOL [20], BRAVE [21], DINO [22] to name a few). These methods generally strike a trade-off between low level scene understanding and high level semantics, failing to support both. Even hybrid approaches [15], [23], [24] that combine masked auto-encoding (MAE) with contrastive learning can still accurately span only one category of tasks. More recently, the V-JEPA line of works [25]–[27] focus on predicting the future in latent space to avoid the computational cost of pixel-level prediction, and show strong results on a wide range of understanding tasks. These pre-training methods lead to strong results on downstream tasks, but most of the time they require extensive fine-tuning or have fairly complex pre-training pipelines, balancing different optimisation objectives. Moreover none of these pre-training methods support high-quality video generation.

Driven by the impressive realism and diversity of the outputs generated by diffusion models [12], [13], [28], [29], multiple works have started to investigate their understanding capabilities. In [30] the authors show that a diffusion-based setup can perform zero-shot classification by conditioning the generative model with the possible classes and selecting the class that leads to the best reconstruction of the noise added to the input image. In [31], the authors build diffusion hyperfeatures by combining the activations from different blocks and noise levels to build robust pixel-level descriptors. In [32], the authors show impressive generation results by prompting the Veo3 model with challenging tasks that require geometry and physics understanding, hinting that Veo3 has implicitly learnt a world model. In addition, works like [33], [34], [35], [36] confirm that generative models support learning a world model for simulated environments, with representations that can be used to inform action policies.

The closest work to our approach is [10]. The authors repurpose activations extracted from variants of Walt model [29] adapted for image and video, for understanding tasks and conclude that diffusion models excel at lower level understanding problems like depth or camera pose estimation, but struggle with high-level semantic tasks. Similarly, in [11], the authors measure the alignment of representations produced by different visual encoders amongst themselves and with language embeddings. Interestingly, their findings suggest that diffusion models like Walt produce representations that align with established visual encoders like DINOv2 [37] and VideoMAEv2 [5], but the alignment with text embeddings is significantly lower. In our work, we show that the recent generation of video diffusion models have strong semantic awareness, in addition to geometry and motion understanding, constituting a strong foundation for a complete perception system. We show state-of-the-art results on video classification on the challenging SSv2 dataset [16], compared to strong baselines from the other families of visual pre-training methods (MAE, contrastive, latent prediction). To further the investigation of semantic understanding, we set up captioning experiments, showing satisfactory performance compared to a strong SigLIP-based baseline [18] from the PaliGemma family [2].

The interplay between generation and understanding started to receive more attention recently. For example, detailed point tracking conditioning leads to more consistent generation [38], or detailed language captioning produced by Gemini models lead to improved Veo generation [12]. Transfusion [39] elegantly combines generation and understanding pre-training into a single model, allowing the two objectives to support each other. However, in all these works, the main finding has been that advanced understanding models lead to significant improvement in generative models. The other direction, how can generation help understanding, has so far been explored mainly for low-level geometry tasks. In our work, we show that frozen diffusion generative models can power video understanding and achieve competitive performance on a wide range of visual tasks, while being able to generate high-quality videos. This unifies video generation and understanding paradigms. We estimate that this finding can be very impactful for the community, by encouraging more research into understanding the representation power of diffusion models. In addition, the implications from a practical point of view could be substantial, as it could lead to one visual model shared across generation and understanding.

3 The structure of the generative manifold↩︎

The iterative diffusion generation requires many denoising steps to generate a video. This slow sequential operation mode is not suitable for video encoding. To design efficient video feature extractors based on video diffusion models, we conduct a detailed investigation of the structure of the latent embeddings spawned by these models, in search for the optimal depth and noise level from where to extract representations with a single forward pass through the model. After a brief overview of diffusion models, we detail the zero-shot and trained light probes used for our analysis, and we conclude this section with a summary discussion of the findings.

3.1 Background on latent diffusion models↩︎

Video Latent Diffusion Models (LDMs) like Veo3 operate within a compressed, lower-dimensional representation space. During training, given an input video \(x\in\mathbb{R}^{F\times H\times W\times C}\) (containing \(F\) frames), an encoder \(\textit{E}\) maps the data into a spatiotemporal latent code \(z_0=\textit{E}(x)\). A forward process corrupts this latent representation over \(T\) steps. At an arbitrary noise level \(t\in[0,T]\), the corrupted latent \(z_t\) is generated by adding Gaussian noise to \(z_0\): \(z_t=\sqrt{\bar{\alpha}_t}z_0+\sqrt{1-\bar{\alpha}_t}\epsilon\), where \(\epsilon\sim\mathcal{N}(0,I)\) and \(\bar{\alpha}_t\) dictates the variance schedule. To invert this process, a backbone network, typically a Diffusion Transformer (DiT) or a U-Net, denoted as \(f_\theta\), is trained to predict the injected noise \(\epsilon\). The network is conditioned on the corrupted latent \(z_t\), the noise step \(t\), and contextual information \(c\) (e.g., text embeddings). The model is optimised via the standard reweighted variational lower bound: \(\mathcal{L}_{LDM}=\mathbb{E}_{z_0,\epsilon,t}\left[\|\epsilon-f_\theta(z_t,t,c)\|_2^2\right]\). Note that flow matching models like Wan 2.2, also included in our study, replace this stochastic forward process with a continuous-time deterministic framework based on Optimal Transport. In this setup, for a continuous time step \(t \in [0,1]\), the corrupted latent is a linear interpolation between the data and a base noise distribution: \(z_t = (1 - t)z_0 + t\epsilon\). Rather than predicting the injected noise, the network \(f_\theta\) is optimised to regress the velocity vector field \(f_t = \frac{d z_t}{d t} = \epsilon - z_0\) that transports the data to the noise. The model is trained via the flow matching objective: \(\mathcal{L}_{FM} = \mathbb{E}_{z_0, \epsilon, t} \left[ ||f_\theta(z_t, t, c) - (\epsilon - z_0)||_2^2 \right]\).

The backbone \(f_\theta\) consists of a sequence of \(L\) layers (e.g., transformer blocks in the case of a DiT). We denote the intermediate spatiotemporal activations extracted from the \(l\)-th layer at noise step \(t\) as \(h_t^{(l)}\). During inference, the model iteratively denoises pure Gaussian noise \(z_T\) to recover an estimated clean latent \(\hat{z}_0\). Finally, \(\hat{z}_0\) is mapped back to pixel space via a decoder \(\textit{D}\), yielding the generated video \(\hat{x}=\textit{D}(\hat{z}_0)\).

Figure 1 illustrates our diffusion-based video encoder Gen4U. We condition the diffusion transformer on a generic text embedding: e.g. A video of a scene, to not leak the ground truth (action label or caption) for semantic tasks. We have also experimented with conditioning on an empty text string, without significant difference in results. To deal with the inherent stochasticity of the diffusion process, we use a fixed random seed throughout all our experiments to ensure reproducibility. In all our plots, we report results for blocks and noise levels expressed as percentages obtained as a linear fraction of the total depth or noise steps.

3.2 Zero-shot probes↩︎

PCA visualisation: To qualitatively assess the evolution of the diffusion features along network depth and noise levels, we plot in Figure 2 the PCA visualisation of the activations extracted from different depths and noise levels in Veo3. We can observe that at high levels of noise, the features are simple, low-frequency, encoding the general shapes in the scene. At lower noise levels, the features become more complex, encoding higher-frequency, more refined details. This is in line with the intuition that diffusion is spectral autoregression [40]: at each denoising step, the model learns to produce the features at the next (higher) frequency level.

Mutual k-NN alignment: To quantitatively assess the structure of the diffusion embedding space, we adopt the zero-shot Mutual \(k\)-Nearest Neighbours (M\(k\)NN) alignment metric proposed by [14] and extended to video in [11]. The key idea is to compare the neighbourhood structure induced by the diffusion model’s intermediate representations against that of established, independently trained encoders, without requiring any learned mapping between the two spaces.

Concretely, given a dataset of \(N\) videos, paired with ground truth captions, we extract a video-level embedding from each video using both the diffusion model and a reference encoder. For the diffusion model, we obtain intermediate activations \(h_t^{(l)}\) at a chosen layer \(l\) and noise level \(t\), and aggregate them spatiotemporally to produce a single vector per video (see appendix 6 for details). For the reference encoder, which may operate in a different modality (e.g., a language model) and produce embeddings of a different dimensionality, we similarly obtain one vector per video. In particular, for image models, we follow [11] and average the individual image features across input frames. We then independently construct the \(k\)-nearest-neighbour graph in each embedding space and measure the average overlap between the two sets of neighbours. Intuitively, a high overlap indicates that the diffusion model organises videos in a manner consistent with the reference encoder, despite never having been trained with the same objective. More details are included in the appendix 6. Following prior work [11], [14], we additionally optimise over the choice of intermediate layers in both encoders and report the pair of layers that maximises the alignment score. We use \(k{=}10\) and \(N{=}1024\) videos.

In Figure 3, we show the alignment of Veo3 and Wan 2.2 respectively, against the Gemma-2-9b-it text encoder [41] on the VATEX dataset. This metric has been shown to correlate with downstream performance in semantic and even non-semantic tasks [11]. In addition, we also extract in Figure 4 the alignment of diffusion representations against DINOv2 [37], a powerful image representation model, and VideoMAEv2 [5], a strong video representation model. Note that the features Veo3, especially when probed at the appropriate depth and noise levels align significantly more strongly with text embeddings of their associated captions. We emphasize that the captions are not given as context during generation. This suggests that a powerful video generative model has the capacity of encoding rich semantic information within its intermediate features.

3.3 Linear and attention probes↩︎

We use linear and attention probes to further assess the quality of diffusion representations. The linear head does global pooling over spatiotemporal dimensions followed by a linear projection. The attention probe uses a cross-attention head, with task-specific queries. We show results for video classification probes in Figure 5.

We also experiment with lightweight decoders that reconstruct the input in RGB space given the representations from different blocks and noise levels, to get a sense of the compression and artefacts present at each stage, but the results are less conclusive; see details and visualisations in appendix 8.

3.4 Discussion↩︎

Feature routing across network depth: The evolution of diffusion representations follows a dual-axis trajectory along both depth \(l\) and noise level \(t\). Interestingly, we observe distinct evolution patterns for the two models. The results for Wan 2.2 (Figure 3 right, Figure 4 right) show a unimodal pattern, aligning with broader diffusion literature (e.g., [31]). The alignment peaks early, at around 25% depth, indicating that the model resolves global structure and scene details early before gradually settling finer details. Conversely, Veo3 exhibits a distinct bimodal pattern, a phenomenon not previously reported in the diffusion literature. This mirrors the geometry of representations learnt by large-scale transformers [42], where the representation manifold expands early on (first peak), contracts significantly in the middle layers to route information, and then expands again toward the end (second peak) before the final output projection. We hypothesise a similar behaviour drives Veo3, supported by PCA visualisations (Figure 2) that show somewhat noisier activations halfway through the network, aligning with the dip in the bimodal pattern. We leave a conclusive characterisation of this behaviour for future work.

\(\mathbf{60\%}\)-noise semantic bottleneck: Despite exhibiting divergent topological patterns across depth, both models share a consistent optimal noise level for alignment against both text and visual encoders at \(t = 60\%\). This indicates the emergence of a semantic bottleneck for both models at this specific noise level.

Evolution of feature complexity, linear vs. attention probes: Further analysis of Figure 5 reveals a slight shift between the optimal extraction points for linear probes (\(t \approx 60\%\), \(l \approx 70\%\)) and attention probes (\(t \approx 30\%\), \(l \approx 80\%\)). While both fall within the second mode of Veo3’s bimodal pattern, indicating the sweet spot for semantics, this shift suggests an evolution in feature complexity. At deeper blocks and lower noise levels, linear probes and zero-shot alignment metrics become insufficient possibly because semantic information becomes patch-specific and spatially scattered. Instead, attention probes excel by acting as a dynamic spatiotemporal pooling mechanism, selectively attending to and aggregating these distributed, fine-grained details. This suggests that lower-noise representations do not discard semantic meaning; they simply require more complex probes to decode it. Importantly, the superiority of the attention probe is not merely an artefact of its higher parameter count (6M vs. 1M for the linear probe). If parameter count were the sole driver, the attention head would uniformly dominate across all noise levels. Instead, at extreme noise levels (\(t \approx 90\%\), red lines), its performance is severely constrained (\(\approx\)​15–30%), offering minimal advantage over the linear probe at early-to-mid depths. Finally, all probes experience a sharp performance drop at the deepest layers, indicating that the network discards semantic abstractions at the very end to perform pixel-level signal reconstruction.

Emergence of general-purpose features: Overall, these alignment results are remarkable. Prior evaluations of generative video models, such as WALT [11], demonstrated limited alignment with semantic encoders. In contrast, our findings show that training purely for video generation at scale is sufficient to endow a model with powerful, general-purpose features. While these large-scale diffusion models are extensively trained with text conditioning, they are never explicitly optimised to align their internal states with text representations (unlike contrastive learning). The implicit emergence of this alignment is a significant finding not reported in prior work. Furthermore, both models demonstrate strong alignment with established visual encoders (Figure 4). Veo3, in particular, exhibits high alignment with DINOv2 and VideoMAEv2—models designed for image and video understanding tasks respectively. This confirms that generative representations possess strong potential for diverse downstream tasks requiring a combination of appearance, motion, and semantic understanding. Finally, the consistently higher alignment scores of Veo3 compared to Wan 2.2 confirm that stronger generative models yield correspondingly richer and more powerful representations.

Combination of features: We investigated whether combining features across multiple stages of the diffusion process could yield complementary information. We include details in Appendix 7.

4 Diffusion models as video encoders↩︎

We demonstrate the potential of diffusion models to act as general-purpose video encoders by evaluating frozen diffusion representations on diverse semantic and non-semantic downstream tasks. While our latent space analysis in Section 3 confirmed the presence of optimal extraction points across both open-weight (Wan 2.2) and proprietary (Veo3) models, it also revealed that feature alignment scales directly with generative capability. Therefore, to test our hypothesis and establish the upper bound of the Gen4U framework, we focus our downstream evaluations on the highly scaled Veo3 model. We rely on one-block attention decoders to map diffusion representations to various task embedding spaces, similar to 4DS [6]. For captioning, we use a small language model fine-tuned on top of the frozen visual diffusion tokens. Only these lightweight decoders are trained, the generative model stays frozen, preserving its generation capabilities.

4.1 Semantic understanding↩︎

Video classification: We use the challenging SSv2 dataset [16] for this task and follow the protocol in [6], using simple linear and one-block attention decoders on top of frozen diffusion representations. The SSv2 dataset contains 220,847 shorter videos (2-6s long), sampled at 12fps, representing 174 classes. The videos depict actions that differ in finer motion-related details, requiring a deeper temporal understanding, e.g. pouring something into something vs pretending to pour something into something.

Unlike previous work [6], we do not do data augmentation on the raw videos. Instead, to reduce computation cost, we process only the original videos with Veo3. We then apply data augmentations directly to the intermediate activations: temporal masking (i.e. set to 0) of 16.7% of the frames, dropout in the attention masking of 40% and label smoothing of 0.2.

We report performance with and without these augmentations in Table 1, compared to strong competitor models. We achieve SOTA performance on this task in this evaluation setup. Note that even without the data augmentation, our setup outperforms all baselines, except V-JEPA.

Image and video captioning: In this suite of experiments, we attach the Gemma-2 (2B) ([41]) text-only large language model as a decoder for image and video

captioning. Our captioning setup consists of the following two steps: (1.) A cross-attention adapter, inspired by the decoder design of [43], with three cross-attention blocks, projects the visual representation into \(32\) learnable query tokens; and (2.) we prepend these \(32\) tokens to the text input before the [BOS] token and process the entire sequence with the LLM using teacher forcing during training and next-token prediction during inference.

We train separate decoders for each of the MS-COCO image captioning dataset (COCO) ([44]), the Something-Something V2 video action recognition dataset (SSv2)³ and the Vatex dataset ([45]). We report the CIDEr and BLEU@4 metrics and compare against two baseline models: SigLIP-so400m/14 model from the PaliGemma series ([2], [46]) and the SigLIP2-B/16 model ([47]).

2 presents the results of Veo3 representations compared to the SigLIP baselines on all three datasets. Note that our setup does not involve tuning the adapter and LLM decoder on large vision-language data, so it is not directly comparable to SoTA methods like BLIP-2 ([48]) and PaLI Gemma ([2]). We train our adapter and LLM only on the train splits of the considered datasets to probe the existence of semantic knowledge decodable through an LLM. We observe that video diffusion representations excel on SSv2 but struggle on COCO and Vatex, compared to the SigLIP baselines, which are highly optimised for alignment with text. Nonetheless, the model produces reasonable captions. A couple of qualitative examples are shown in Fig. 7.

Vatex [Frozen LLM]: We freeze the LLM in this experiment and train only the cross-attention adapter on the Vatex train split. Results are reported in the last two columns of Table 2. Model performance decreases only slightly relative to the full fine-tuning experiments. Moreover, we significantly outperform VideoPrism which reports a CIDEr of \(31.7\) on this dataset. Although VideoPrism does not finetune their adapter on captioning, they pre-train it on a large vision-language corpus.

Data augmentation using multiple noise injection levels: As mentioned above, we do not pre-train our adapter and LLM on large datasets. To limit the overfitting caused by small training sets, we augment the data by concatenating the datasets across various noise injection levels \(10\%, 30\%, 60\%~\mathrm{and}~90\%\). We find that for COCO and Vatex using the 10%, 30% and 60% noise injection levels is optimal. The model is evaluated on the 30% level in all cases. The other noise levels purely serve as data augmentation. For SSv2, we do not see any improvements with data augmentation, likely because the caption space is simpler and the training dataset is much larger than with COCO and Vatex. The resulting performance is shown in 2 under ‘+ Noise Aug.’.

Benefits of high-resolution Veo: COCO images contain several small objects and the model needs to mention a few of them to score highly on the CIDEr metric. We ablate doubling the input resolution. We pick the optimal combination of noise levels (‘+ Noise Aug.’) for each resolution on the validation split and report test set numbers in Table 2 under ‘+ Noise Aug. + High res.’. The high res. model achieves gains \(+ 32.7\) CIDEr points. Thus by using multiple noise levels as data augmentation and using higher resolution diffusion representations, one can reduce the gap to discriminative models trained explicitly for the captioning task on large-scale vision-language datasets.

4.2 Geometry understanding↩︎

Depth estimation: We evaluate monocular depth prediction on ScanNet [49]. Following prior work [6], we report the absolute relative error (AbsRel), computed as \(|d^{*} - d|/(d + \epsilon)\) where \(d^{*}\) and \(d\) denote predicted and ground-truth depth, respectively, and the threshold accuracy \(\delta_1\). Target depth values outside the \((0.001, 10)\) m range are masked out. We use a Dense Prediction Transformer (DPT) readout head [50] with approximately 23M trainable parameters, trained with the scale-invariant logarithmic (SiLog) loss [51]. The entire diffusion backbone remains frozen.

Figure 6 (right) presents AbsRel across all blocks and noise levels. The results confirm the sweet spot for extracting representations around (depth 80%, noise 30%), identified using zero-shot alignment and classification probes. The best configuration achieves AbsRel of 0.075 and \(\delta_1{=}0.952\), a 10.7% relative improvement over the frozen-feature baseline of \(0.084\) reported in [6]. To our knowledge, this is the best result on ScanNet depth with frozen video model features.

Camera pose estimation: We are interested in probing the ability to infer the 6DoF relative camera poses from diffusion representations. We use the setup and metric introduced in [6], applied to Scannet dataset. Given diffusion representations for a clip of \(F\) frames, we train a one-block attention decoder to predict the relative pose between the first and last frame of the clip, in the form of a \(12D\) vector encoding an SE(3) pose transformation (\(3\times3\) estimated rotation matrix and a \(3\times1\) translation vector). As evaluation metric, we use end-point-error (EPE), which considers the rotation and translation components jointly; see [6] for more details. We obtain 1.10 EPE, which is on par with a strong DINOv2 baseline obtaining 1.08 EPE. The sweep across blocks and noise levels (Figure 6 left) is consistent with the findings for the other tasks, with performance peaking reliably in the mid-to-late stages of the network (\(\sim\)​75% depth), at medium noise level (60%).

5 Conclusion↩︎

We show that state-of-the-art large-scale video diffusion models can act as competitive video encoders on a diverse set of tasks (video classification, depth estimation, camera pose estimation, video captioning), supporting the first successful attempt to unify video generation and video understanding. Using zero-shot probes and trained lightweight decoders, we investigated the structure of the diffusion latent space and identified the optimal block and noise level to extract representations for understanding tasks. All our results for downstream tasks use representations extracted from a single block and noise level, so we only need to run a single forward pass through the model, being as efficient as large-scale discriminative visual encoders. As future work, we aim to study further the bimodal pattern identified for Veo3 and push the performance to achieve SOTA on a wide range of tasks.

Limitations: Our experiments are conducted mainly on a proprietary video model, Veo3, with analysis replicated on an open-source model (Wan 2.2), limiting the reproducibility of our study. However, we believe that this is still a useful investigation for the community and we hope it will encourage more work in studying large-scale diffusion models for representation learning.

Acknowledgments↩︎

We are deeply grateful to Rahul Sukthankar, Howard Zhou, Daniel Zoran, Mehdi S. M. Sajjadi, Forrester Cole, João Carreira, Shiry Ginosar, and Andrew Zisserman for their support and insightful feedback throughout this project.

6 Mutual \(k\)-NN alignment metric↩︎

We describe the Mutual \(k\)-Nearest Neighbours (M\(k\)NN) alignment metric used throughout the paper, following [11], [14].

6.0.0.1 Notation.

Consider a dataset of \(N\) videos \(\{v_i\}_{i=1}^{N}\), each paired with some text \(c_i\). We compare the representations of two encoders whose embedding dimensionalities may differ.

6.0.0.2 Diffusion encoder.

To use the LDM \(f_\theta\) as a video encoder, we run the model in its forward (training) mode at a chosen noise level \(t\) and extract the intermediate activations \(h_t^{(l)} \in \mathbb{R}^{F' \times H' \times W' \times D}\) from layer \(l\) of the backbone, where \(F'\), \(H'\), \(W'\) are the spatiotemporal dimensions of the latent and \(D\) is the channel dimension. We aggregate these activations via global average pooling over the spatiotemporal axes to obtain a single embedding vector per video: \[\mathbf{x}_i = \mathrm{AvgPool}\!\bigl(h_t^{(l)}(v_i)\bigr) \in \mathbb{R}^{D}.\] We stack all video embeddings into a matrix \(X \in \mathbb{R}^{N \times D}\).

6.0.0.3 Reference encoder.

A reference encoder \(E_{\mathrm{ref}}\) (e.g.a text encoder applied to text descriptions, or an independently trained video encoder) maps each input to an embedding of potentially different dimensionality \(D'\): \[\mathbf{y}_i = E_{\mathrm{ref}}(c_i) \in \mathbb{R}^{D'}, \quad Y \in \mathbb{R}^{N \times D'}.\] When the reference encoder is a pure image model (e.g.DINOv2), we average its frame-level features across the temporal dimension, following [11].

6.0.0.4 Feature preprocessing.

Before computing nearest neighbors, we clip each feature matrix to suppress outliers: for a chosen quantile \(q\) (we use \(q{=}0.95\)), we compute \(\tau = \mathrm{Quantile}_{q}\!\bigl(|X|\bigr)\) and clip all values to \([-\tau, \tau]\). We then \(\ell_2\)-normalise each row so that similarities reduce to dot products.

6.0.0.5 Nearest-neighbour graphs.

Let \(\mathcal{N}_k^X(i)\) denote the set of \(k\) nearest neighbours of sample \(i\) in the embedding space \(X\), computed via dot-product similarity (excluding self-matches). We define an analogous set \(\mathcal{N}_k^Y(i)\) for space \(Y\). Equivalently, we can encode these as binary indicator matrices \(M^X, M^Y \in \{0,1\}^{N \times N}\), where \[M^X_{ij} = \begin{cases} 1 & \text{if } j \in \mathcal{N}_k^X(i), \\ 0 & \text{otherwise.} \end{cases}\]

6.0.0.6 Alignment score.

The M\(k\)NN alignment between \(X\) and \(Y\) is the mean fraction of shared neighbours: \[\mathcal{A}_{\mathrm{M}k\mathrm{NN}}(X, Y) = \frac{1}{kN} \sum_{i=1}^{N} \sum_{j=1}^{N} \bigl(M^X \odot M^Y\bigr)_{ij} = \frac{1}{N} \sum_{i=1}^{N} \frac{\bigl|\mathcal{N}_k^X(i) \cap \mathcal{N}_k^Y(i)\bigr|}{k}, \label{eq:mknn}\tag{1}\] where \(\odot\) denotes element-wise multiplication. This score ranges from \(0\) (no overlap) to \(1\) (perfect agreement) and is invariant to the dimensionality and scale of each space, since it depends only on the rank-order of pairwise similarities.

6.0.0.7 Layer optimisation.

Both the diffusion backbone and multi-layer reference encoders expose representations from multiple intermediate layers. Following [14], we sweep over all pairs of layers \((l, l')\)—where \(l\) indexes a layer in \(f_\theta\) and \(l'\) a layer in \(E_{\mathrm{ref}}\)—and report the pair that maximises \(\mathcal{A}_{\mathrm{M}k\mathrm{NN}}\): \[(l^\star, l'^\star) = \mathop{\mathrm{arg\,max}}_{l, l'} \; \mathcal{A}_{\mathrm{M}k\mathrm{NN}}\!\bigl(X^{(l)}, Y^{(l')}\bigr).\] We use \(k{=}10\) in all experiments.

7 Combining Veo3 latents across blocks and noise levels↩︎

In this section, we provide details and preliminary results regarding the experiments mentioned in Section 3.4 on combining Veo3 representations across different blocks and noise levels.

7.1 Experimental setup and architectures↩︎

To investigate the complementarity of Veo3 representations, we extract features from 12 equally spaced blocks across 4 noise levels (\(t \in \lbrace 10\%, 30\%, 60\%, 90\% \rbrace\)). Features are averaged across space and time. This results in \(4 \times 12 = 48\) distinct D-dimensional feature vectors per video. We evaluate four architectures to map these \(48\) feature vectors into a single global video representation:

• Linear adapter: This computes a simple weighted sum of the 48 feature vectors: \[F_{combined} = \sum_{i=1}^{48} w_i \cdot F_i\] where \(w \in \mathbb{R}^{48}\) is a vector of learnable scalar weights. This architecture has the fewest parameters (48) and demonstrated the best generalization.

• Shared MLP: The input features are transposed to \((D, 48)\) and passed through a series of linear layers with ReLU activations shared across the feature dimension: \[\begin{align} \text{Input}(48, D) &\to \text{Transpose}(D, 48) \to \text{Linear}(48 \to M) \to \text{ReLU} \\ &\to \text{Linear}(M \to H) \to \text{ReLU} \to \text{Linear}(H \to 1) \end{align}\] The output is then squeezed to produce the final \(1 \times D\) vector. We experimented with configurations such as \(M=8, H=8\).

• Self-attention: A learnable CLS token \(c \in \mathbb{R}^{D}\) is concatenated with the 48 features to form a \(49 \times D\) matrix. All 49 tokens attend to each other via standard self-attention, with queries, keys, and values projected to \(d_k=64\). The output corresponding to the CLS token is extracted and projected back to \(D\).

• Cross-attention: A Perceiver-style architecture where a single learnable query token cross-attends to the 48 features (which are first compressed to \(d_k=64\)). This significantly reduces parameters compared to full self-attention while allowing dynamic weighting.

7.2 Training objectives↩︎

We considered two primary training losses to train these adapters:

• Cross-entropy loss (for direct classification on SSv2): \[\mathcal{L}_{CE} = -\frac{1}{B} \sum_{i=1}^{B} \sum_{c=1}^{C} \mathbf{1}[y_i = c] \log \frac{\exp(\hat{y}_{i,c})}{\sum_{c'=1}^{C} \exp(\hat{y}_{i,c'})}\] where \(B\) is the batch size, \(C\) is the number of classes (174 for SSv2), \(y_i\) is the true label, and \(\hat{y}_{i,c}\) are the logits produced by adding a linear classification head after the adapter.

• Multi-positive InfoNCE loss (for cross-modal alignment): \[\mathcal{L}_{InfoNCE} = -\frac{1}{B} \sum_{i=1}^{B} \log \frac{\sum_{j \in \mathcal{P}(i)} \exp(\text{sim}(\mathbf{v}_i, \mathbf{v}_j)/\tau)}{\sum_{k=1}^{B} \exp(\text{sim}(\mathbf{v}_i, \mathbf{v}_k)/\tau)}\] Here, \(\mathbf{v}_i\) is the \(L_2\)-normalized adapter output for sample \(i\), \(\text{sim}(\cdot,\cdot)\) denotes cosine similarity, and \(\tau\) is a temperature parameter. The set \(\mathcal{P}(i)\) contains the indices of the \(k\)-nearest neighbors in the text feature space (e.g., Gemma embeddings), acting as multiple positives.

7.3 Results↩︎

7.3.0.1 Dataset.

We run preliminary experiments using a subset of the SSv2 training set. Specifically, we keep 100 training samples per class, resulting in a training set with 17400 samples. We evaluate on the full SSv2 validation set. The baseline performance of a linear classifier trained on the single best block and noise level using this subset is 17.04%. This is a modest performance, but it is significantly better than chance, providing the necessary signal for our investigation; see Table 3.

Linear adapters consistently outperform non-linear ones in terms of zero-shot generalization across datasets and tasks. For instance, a linear adapter pre-trained purely for text alignment on VATEX generalised effectively to SSv2 out-of-the-box. Conversely, non-linear models—such as the cross-attention and self-attention architectures—were significantly harder to train. In early experiments, they frequently achieved high alignment scores on the training set while generalizing poorly to downstream classification tasks, indicating severe overfitting.

7.3.0.2 Training for non-linear adapters.

To stabilize the training of higher-capacity non-linear adapters, we found that modifying the batch composition was critical. Specifically, increasing the effective batch size and ensuring a robust ratio of anchors, positives, and negatives prevented the cross-attention models from collapsing. Our optimal configuration utilized 16 anchors per batch. For each anchor, we sampled 10 positive matches and 22 negative matches (yielding 32 samples per anchor). This structured batching was the key heuristic that allowed the cross-attention adapters to converge without overfitting to the training distribution.

7.3.0.3 Peak performance via cross-attention.

While linear adapters are robust and easily generalisable, the carefully tuned cross-attention adapter ultimately yielded the highest peak performance when trained directly on the target distribution. When trained specifically for classification on the SSv2 dataset using the aforementioned batching heuristics, the cross-attention adapter achieved a downstream classification accuracy of 24.9% on this SSv2 subset. This represents a substantial improvement over the 17.04% baseline (which utilises a single optimal block and noise level) and also outperforms the best linear adapter’s peak of 21.9% on this SSv2 subset. All of these results were obtained using a simple linear readout head without any data augmentation.

7.3.0.4 Correlation across training objectives.

Throughout our ablation studies across different training objectives (classification, class-vector alignment, and text-embedding alignment), we observed a highly consistent empirical trend: training the adapter for classification utilising a discrete label signal simultaneously improved all proxy metrics for feature quality.

Most notably, adapters trained using a standard cross-entropy classification loss consistently exhibited better zero-shot cross-modal text alignment (e.g., higher mutual k-NN alignment with Gemma text features) than adapters that were explicitly trained to maximize text alignment. Similarly, 1-NN retrieval accuracy also peaked under the classification objective. We hypothesize that the discrete cross-entropy objective provides a cleaner, more stable gradient signal for the adapter to learn the underlying semantic structure of the VEO representations, compared to the noisy gradients inherently present in multi-positive contrastive alignment over text embeddings.

8 Preview decoding↩︎

We train linear and attention preview decoders on the Something-Something-V2 dataset [16], i.e. we attempt to decode early (and efficiently) the final RGB output to get a qualitative glimpse into the features encoded at different blocks and noise levels. We use the mean L2 error on RGB values as the training loss.

Linear readout head: The linear head applies a single learned affine projection from the token embedding space to RGB colour. Given input tokens \(\mathbf{X} \in \mathbb{R}^{B \times F \times N \times C}\) (\(B\) is the batch size, \(F\) is the number of frames, \(N\) is the number of spatial tokens per frame, and \(C\) is the number of channels per token), the head computes \(\hat{\mathbf{Y}}_{\mathrm{lin}} = \mathbf{X} W + b\) with \(W \in \mathbb{R}^{C \times 3}\) and \(b \in \mathbb{R}^{3}\). The output is reshaped to \((B, F, H, W, 3)\), where \(H \cdot W = N\), recovering the native spatial layout of the Veo3 token grid.

Attention readout head: The attention head uses cross-attention with learnable queries to decode a target-resolution RGB video. A bank of \(F \!\cdot\! H \!\cdot\! W\) query vectors is learned, each of dimension \(d\) < \(C\). The input tokens are first flattened to \((B, F{\cdot}N, C)\) and linearly projected to keys and values of dimension \(d=512\). A standard multi-head dot-product attention layer with \(h{=}8\) heads (head dimension 64) then computes cross-attention from the learned queries to the projected keys and values, yielding an output of dimension \(d{=}512\) per query. A final linear projection maps each query output to 3 (RGB) channels, and the result is reshaped to \((B, F, H, W, 3)\).

Both heads are trained with per-pixel L2 loss and evaluated with MSE. The optimizer is Adam with a linear warmup (1k steps) to a peak learning rate of \(10^{-4}\), with gradient accumulation to an effective batch size of 16.

8.0.0.1 Results.

Figure 8 presents frames from the original video and their previews decoded from the outputs at depth 75%, evaluated at noise level 60%. The linear decoder presents frames that are sharper, but temporally misaligned with the ground truth, as compared to the attention head.

Figure 9 shows MSE reconstruction metrics for attaching the preview head at various network depths, and training it on various levels of injected noise. Unlike the zero-shot alignment scores or the classification probes, regressing the denoised video’s pixel values from various heads does not show a clear pattern with an optimal sweet spot for extracting features. The linear head does show generally lower error at higher network depths, reflecting that deeper layers are closer to the final latents to be decoded into RGB values. On average, the attention head outperforms the linear head by having better temporal alignment with the original video.

9 Image and video captioning details↩︎

The cross-attention adapter alternates cross-attention layers from learned queries into input vision tokens with self-attention layers across query tokens. The adapter is designed to output \(\mathbf{R}^{2304}\) features matching Gemma2-2B’s input embedding size. Gemma2 is provided an attention mask that is bi-directional on the vision tokens and causal on the text tokens. The loss is applied to the text tokens after the [BOS] token. During inference we use greedy sampling and discard all tokens after the first [EOS] token is sampled. A maximum sampling sequence length of \(64\) tokens is used for the COCO and SSv2 datasets and \(96\) tokens are sampled for the Vatex dataset. This was adjusted for Vatex to accommodate longer captions in the dataset.

For SigLIP baselines, input frames are sampled for each video such that the total token count roughly aligns with the number of tokens output by Veo3. The SigLIP-so400m/14 model is from the PaliGemma series ([2], [46]) which operates at \(896\times896\) resolution with pre-pooling giving us 256 tokens per frame, and the SigLIP2-B/16 model ([47]) operates at \(512\times512\) resolution without pre-pooling and outputs 1024 tokens per frame.

Learning rate follows the cosine decay schedule with linear warmup over \(1k\) iterations. The peak learning rate is \(5\times10^5\). The Gemma2 weights are initially frozen and gradually thawed with a linear schedule going from \(0\) to \(0.01\) over \(10k\) iterations. It plateaus after \(10k\) steps so that the LLM is still updated very slowly to avoid catastrophic forgetting. We use the ADAM optimizer [52] with batch size of \(64\) for all experiments in this subsection.

Models are trained for \(32k\) iterations in total. In cases where overfitting was observed, early stopping at \(12k\) or \(15k\) iterations was applied. This was especially useful on the SSv2 and Vatex datasets when using a single noise injection level as training data (no ‘Noise Aug.’).

The MS-COCO experiments use the Karpathy splits for training, validation and test with 5000 test images [53].

10 Asset licensing and terms of use↩︎

The open-weight models utilised in this study (Wan 2.2, DINOv2) are distributed under the permissive Apache 2.0 license, while Gemma-2 is used in accordance with the Gemma Terms of Use; all three permit broad research and commercial applications. The evaluation datasets (Something-Something V2, COCO, ScanNet, and Vatex) are utilised strictly for non-commercial academic research and benchmarking, in full compliance with their respective institutional Terms of Use and Creative Commons distributions.

References↩︎