Bridging Video Understanding and Generation in a Unified Framework
January 01, 1970
Abstract
Recently, unified image generation and understanding have been extensively explored. However, extending such unified modeling paradigms to the video domain remains largely underexplored. A central challenge is that video understanding favors compact, discriminative semantic representations, whereas video generation requires dense signals that preserve visual details and temporal coherence. Videos naturally capture both spatial semantics and temporal dynamics, making them a more suitable modality for unified multimodal modeling compared to static images. In this paper, we propose Vega, a unified framework that bridges video understanding and generation. Vega leverages a shared vocabulary to jointly model text and visual representations and employs a hybrid architecture combining autoregressive (AR) prediction with diffusion-based rendering. Specifically, the AR model focuses on predicting semantically meaningful visual tokens for keyframes, providing a structured representation that guides the diffusion module in rendering dense, high-resolution video frames. Extensive experiments demonstrate that Vega achieves strong performance on video generation benchmarks such as VBench and video understanding benchmarks like VideoMME.
1 Introduction↩︎
Recent years have witnessed the remarkable success of unified models in natural language processing, evolving from task-specific models to Large Language Models (LLMs) as unified foundation models. Inspired by this paradigm, researchers have also sought to develop unified models in the vision domain [1]–[3]. In the image domain, unified models for generation and understanding have attracted significant attention [4], [5], leading to the exploration of diverse architectures, including purely autoregressive (AR) [6]–[8], hybrid AR–diffusion [9]–[16], and fully diffusion [17], [18], all aimed at learning a single model with both understanding and generative capabilities. However, current research efforts remain largely confined to static images, with limited exploration of unified modeling for videos.
The success of LLMs reflects a profound idea: in language, to generate is to understand. Extending this notion to vision, video emerges as a more natural form for unified modeling, embodying both spatial semantics and the temporal–causal structure of the real world. Recent studies [19]–[21] have further echoed this philosophy: by generating what it understands, and understanding what it generates, video models—such as Sora2 and Veo3—demonstrate that video generation can act as a universal interface for visual understanding, enabling zero-shot reasoning and segmentation across diverse tasks. Despite this progress, existing research on video understanding and video generation remains largely disjoint. The former is primarily driven by vision–language models (VLMs) [22]–[24] that emphasize visual recognition and reasoning, whereas the latter is advanced by diffusion-based generative paradigms [25]–[29], which focus on producing visually realistic and temporally coherent videos. This disconnect motivates a fundamental question: can we bridge these two paradigms within a single, unified framework?
Bridging these paradigms is inherently challenging, as video understanding and generation demand fundamentally different types of representations. Understanding tasks require compact, discriminative semantic embeddings for recognition and reasoning, while generation necessitates dense, high-fidelity signals to synthesize realistic and temporally coherent visual content. Prior efforts can be broadly categorized into two directions. The first investigates discrete, fully autoregressive (AR) formulations [7], [8], extending image-based modeling to the video domain. However, these extensions often underperform—autoregressive video generation remains substantially inferior to diffusion-based methods in terms of visual fidelity and temporal consistency. This limitation arises from the intrinsically low signal-to-noise ratio (SNR) in video data, which causes pixel-space VQ compression to generate an excessive number of visual tokens and introduces strong temporal redundancy that hinders effective learning. The second line of work explores hybrid AR–diffusion architectures [30], [31], which integrate multimodal large language models (MLLMs) with diffusion decoders, serving as replacements for traditional text encoders such as T5 [32]. While such hybrid designs enhance in-context generation quality, they inherently prioritize generative capabilities. Their understanding components still depend on frozen MLLMs, limiting the potential to achieve a truly unified framework that bridges visual understanding and generation. Despite these efforts, existing approaches either suffer from inferior visual quality or depend on disjoint modules for generation and understanding respectively, underscoring the need for a unified solution that can bridge both paradigms.
To address these challenges, we introduce Vega, a unified framework for video generation and understanding. Our approach harnesses the complementary strengths of autoregressive (AR) and diffusion modeling: the AR module operates in a semantic space, predicting keyframe-level representations that capture high-level scene dynamics, while the diffusion decoder serves purely as a pixel-level renderer, translating these semantic embeddings into continuous and realistic video sequences. At the core of Vega lies the use of semantic vision tokens as a shared representation, bridging the gap between understanding and generation. This not only strengthens the semantic alignment with text tokens but also significantly reduces the number of visual tokens. As for generation tasks, we adopt a unified video-centric modeling approach that encompasses a variety of tasks. In this formulation, image editing can be treated as a special case of a two-frame video [33]. To mitigate the loss of fine-grained details by semantic tokens and ensure temporal consistency, we employ noise-controlled visual conditioning, which enforces coherence in the generated visual content. On the understanding side, motivated by the slowfast concept [34], we introduce a dual-flow selection mechanism to further compress the number of visual tokens, and incorporate a generative visual loss as auxiliary supervision. Our experiments suggest that this auxiliary supervision can provide additional learning signals for video understanding, highlighting the potential benefits of integrating generation and understanding within a single framework.
Our contributions can be summarized as follows:
We propose Vega, a unified framework that bridges video generation and understanding through a shared representation space. By converting both textual and visual signals into a unified set of discrete tokens, Vega enables a consistent multimodal representation that supports both generative and understanding tasks.
Unlike prior autoregressive generation methods that directly model pixel-level dependencies, our framework performs autoregression in the visual semantic space. The AR module models high-level scene dynamics by predicting semantic representations for keyframes, which the diffusion decoder refines into high-fidelity videos. This design establishes a complementary interplay: the AR module learns state transitions in the latent semantic space, while the diffusion decoder renders detailed appearances in pixel space.
We propose a dual-flow selection mechanism for visual tokens in video understanding, comprising complementary pivot and detail flows that capture temporal dynamics across multiple timescales. The detail flow focuses on rapidly changing features under higher spatial compression, while the pivot flow models slowly varying patterns with lower compression. This design enables efficient hierarchical temporal modeling and significantly reduces the number of visual tokens. Furthermore, incorporating generative supervision on vision tokens provides an auxiliary signal for video understanding.
Vega achieves leading performance among unified video understanding-generation models, while showing competitive results against specialized models on both video generation and video understanding benchmarks. Moreover, it unifies image-to-video (I2V) generation and image editing under a single video-centric modeling framework.
2 Related Works↩︎
2.1 Unified Multimodal Understanding and Generation↩︎
Previous VLMs based on autoregressive (AR) modeling and generative models based on diffusion modeling have achieved remarkable success in visual understanding and generation, respectively. Recently, unified multimodal understanding and generation [5] has attracted growing attention, leading to a series of approaches for the image modality, including pure AR [6], [8], [35], hybrid AR-diffusion [9]–[12], [14], [36]–[39], and pure diffusion paradigms [17], [18]. Purely autoregressive approaches, such as Chameleon [6] and Emu3 [8], represent images and videos as discrete tokens using VQ-VAE [40], and train models in an autoregressive manner. Hybrid autoregressive–diffusion models, exemplified by Janus-Pro [37] and Bagel [12], enable generation and understanding to proceed through distinct pathways, while supporting joint training within a unified framework. Pure diffusion-based approaches constitute another emerging research direction. Representative works such as MMaDA [17] leverage diffusion models to achieve unified modeling of images and text.
Extending these efforts beyond the image modality, recent studies [15], [30], [31], [41] have begun to explore unified modeling for videos. However, most existing approaches condition diffusion models solely on text embeddings enhanced by MLLMs, focusing primarily on improving the expressiveness of textual representations. In contrast, our framework emphasizes the joint learning of vision–language representations within the MLLM itself, achieving a tighter integration between the two modalities. Specifically, the autoregressive branch predicts the visual tokens of key frames, which are then decoded by the diffusion module into temporally dense video sequences.
2.2 Autoregressive Video Generation↩︎
Despite the success of diffusion models in video generation, recent studies have increasingly explored autoregressive alternatives [7], [8], [42]. Existing approaches can be broadly grouped into two main paradigms.
, which leverage discrete VQ tokens and autoregressive transformers for video generation (e.g., VideoPoet [43], Emu3 [8], Lonng [44], VILA-U [7], COSMOS [45]). Other works, such as Lumos-1 [46] and Show-o [47], also adopt discrete tokens, but model them through discrete diffusion—often implemented as masked autoregressive denoising—rather than conventional next-token prediction.
, which operate on VAE latents and receive supervision through diffusion heads, draw inspiration from the recent success of diffusion models [25], [48]. For instance, NOVA [42] and VideoMAR [49] extend MAR [50] to the video domain by employing a diffusion head to autoregressively model spatio-temporal latent sequences.
However, prior approaches—whether based on discrete tokens or continuous latents—primarily model videos in the pixel space. In contrast, our method performs autoregressive video prediction in the semantic space [51].
3 Method↩︎
In this section, we introduce the overall framework (Section 3.1), as illustrated in Figure 1. Both textual and visual inputs are first converted into discrete tokens and jointly modeled by an autoregressive transformer with a shared codebook. For video generation (Section 3.2), the model predicts sparse keyframe-level visual tokens, which are then decoded into dense videos via a diffusion-based decoder. For video understanding (Section 3.3), multi-scale visual representations are extracted using a dual-flow selection strategy, with additional supervision applied at the visual-token level. In the following, we provide a detailed description of each component.
3.1 Overall Framework↩︎
3.1.0.1 Autoregressive Transformer
Given a multimodal sequence comprising image/video and text inputs, we first discretize these heterogeneous signals into a unified sequence of discrete tokens, enabling a shared representation space across modalities. The unified sequence is modeled autoregressively as: \[P(x_1, x_2, \cdots, x_T) = \prod_{t=1}^{T} P(x_t \mid x_1, x_2, \cdots, x_{t-1}) \label{eq:autoregressive}\tag{1}\] where the model learns to predict each token conditioned on all its preceding context, regardless of modality.
3.1.0.2 Vision Tokenizer
The Vision Tokenizer converts images into discrete tokens. Unlike conventional methods like VQ-VAE [40], [52], which quantize in pixel space, we use TA-Tok [53] to tokenize in the semantic feature space, yielding more compact and semantically meaningful representations.
3.1.0.3 Diffusion Decoder
The integration of an LLM backbone with an external diffusion module has been extensively explored in the image domain [36], [54]–[56], where the LLM serves as a prompt-driven embedding enhancer to improve the overall generative quality.
In contrast to previous approaches that autoregressively generate text-conditioned signals, our method enables the model to predict visual-conditioned signals. Specifically, for video generation, the model autoregressively predicts keyframe-level visual tokens, which are then provided as visual conditions to the diffusion decoder for dense frame rendering.
3.2 Video Generation↩︎
3.2.0.1 Autoregressive Video Modeling in the Semantic Space.
In the autoregressive transformer, we model video sequences autoregressively in the semantic token space rather than the pixel space. For a given input video, a set of keyframes is uniformly sampled, and each frame is encoded using a visual tokenizer to obtain discrete visual tokens. Unlike pixel-space representations [40], which are difficult to compress and require a large number of tokens per frame, semantic tokens provide a compact and semantically meaningful representation of visual content. This compactness reduces computational cost and mitigates redundancy across consecutive frames, making the representation well-suited for sequential modeling. Autoregressive training follows a standard next-token prediction objective, where the model learns to predict each semantic token conditioned on all previously observed tokens.
3.2.0.2 Noise-Controlled Visual Conditioning.
While the autoregressive transformer can effectively predict semantic tokens for keyframes, these sparse representations alone are insufficient for producing temporally dense videos. Therefore, we employ a diffusion-based decoder to render the predicted semantic tokens into continuous, high-fidelity video sequences. However, the semantic encoding inevitably discards certain low-level appearance details, which are crucial for video generation tasks such as image-to-video (I2V) generation. To mitigate this issue, we incorporate information from the reference frame as an additional conditioning signal. Inspired by Diffusion Forcing [57], we control the visual conditioning by applying different noise levels to the latent representations. Specifically, given video latents \(z \in \mathbb{R}^{C \times T \times H \times W}\) encoded by the VAE, we introduce a binary mask \(M \in \{0,1\}^{T}\) to regulate the conditioning process. For text-to-video generation, all mask values are set to 1, meaning that Gaussian noise is added to all frames during training. In contrast, for image-to-video or video-to-video generation, the mask values corresponding to the conditioned frames are set to 0 (no noise added), while the remaining frames follow the standard diffusion noise schedule. Formally, the noisy latent is defined as: \[\tilde{z}_t = \sqrt{\bar{\alpha}_t} \, z + M \odot \sqrt{1 - \bar{\alpha}_t} \, \epsilon, \quad \epsilon \sim \mathcal{N}(0, I) \label{eq:mask95diffusion}\tag{2}\]
where \(\bar{\alpha}_t\) denotes the cumulative noise coefficient at timestep \(t\), and \(\odot\) represents element-wise multiplication. This unified masking mechanism ensures consistent training across video generation tasks and enables autoregressive generation, allowing video clips to be seamlessly extended or stitched over time.
3.3 Video Understanding↩︎
For video understanding, our core design leverages dual-flow selection for multi-scale visual feature extraction and vision supervision to provide auxiliary guidance on vision tokens.
3.3.0.1 Dual-flow Selection
Video understanding faces the challenge of handling a large number of visual tokens. To address this while avoiding excessive computational cost, we introduce a Dual-flow Selection mechanism. This approach is based on the observation that videos typically consist of multiple coherent semantic segments, or “plots”. We treat the initial frame of each segment (the “pivot” frame) as containing the highest-level semantic information, while the subsequent “detail” frames provide context and fine-grained dynamic information. Accordingly, video frames are processed through two information flows with distinct downsampling scales, as illustrated in Figure 2.
Pivot Flow First, we uniformly sample \(N_p\) frames from the complete video sequence \(V = \{f_t\}_{t=1}^T\) to form the set of pivot frames \(\mathcal{F}_p\). We hypothesize that these frames \(f_p \in \mathcal{F}_p\) contain most of the semantic events. Each frame \(f_p\) is converted into a token sequence \(X_p = \mathcal{E}(f_p)\) via visual tokenizer \(\mathcal{E}\). Given the semantic importance of pivot frames, we adopt a conservative downsampling strategy to preserve critical information. Specifically, we apply a token average pooling operation \(\text{AvgPool}(\cdot, s)\) with a small scale \(s_p\): \[X'_p = \text{AvgPool}(X_p, s_p = 2)\] This smaller pooling scale ensures that the rich spatio-temporal semantic information required for identifying core events is preserved with high fidelity.
Detail Flow To further capture the fine-grained dynamics and contextual information within each semantic segment, we uniformly sample \(N_d\) frames between each consecutive pair of pivot frames \((f_{t_i}, f_{t_{i+1}})\) (where \(f_{t_i}, f_{t_{i+1}} \in \mathcal{F}_p\)), forming the detail frame set \(\mathcal{F}_d^{(i)} \subset \{f_t | t_i < t < t_{i+1}\}\).
These detail frames \(f_d \in \mathcal{F}_d^{(i)}\) are similarly transformed into token sequences \(X_d = \mathcal{E}(f_d)\), which provide supplementary semantic information. To efficiently manage the token budget, we apply a more aggressive downsampling strategy using a larger pooling scale \(s_d\): \[X'_d = \text{AvgPool}(X_d, s_d = 3)\]
Mechanism Rationale By distinguishing between pivot and detail frames and adopting different pooling strategies with \(s_p < s_d\), our dual-flow selection mechanism strategically allocates the model’s computational budget. It preserves key information at critical semantic boundaries (pivots) while efficiently compressing supplementary information within semantic segments (details). This strategy enables the model to process a larger total number of frames \(\mathcal{F} = \mathcal{F}_p \cup (\bigcup_i \mathcal{F}_d^{(i)})\), achieving a more comprehensive understanding of long videos within a manageable computational cost.
3.3.0.2 Vision Supervision
To further enhance the representational power of the visual tokens extracted by our model, we introduce an auxiliary self-supervised task. We term this auxiliary task “Vision Supervision”, drawing inspiration from the masked autoencoding objectives [58] commonly used in transformer-based sequence models.
The core idea is to perform masked token prediction on the selected visual-token sequence. For a given video, we concatenate the processed pivot tokens and detail tokens into a sequence \(\mathcal{X}' = [\mathcal{F}_p', \mathcal{F}_d']\), where \(\mathcal{F}_p' = \{X'_{p,i}\}\) and \(\mathcal{F}_d' = \{X'_{d,j}\}\) denote the pooled tokens from the pivot and detail flows, respectively.
During training, we randomly replace a subset of tokens in \(\mathcal{X}'\) with mask tokens before feeding the sequence into the model. Let \(\mathcal{M}\) be the set of masked positions and \(\mathcal{U}\) be the set of unmasked positions. The model is trained to recover the original discrete visual token IDs at the masked positions \(\mathcal{X}'_{\text{masked}} = \{X'_k | k \in \mathcal{M}\}\), conditioned on the remaining unmasked tokens \(\mathcal{X}'_{\text{unmasked}} = \{X'_k | k \in \mathcal{U}\}\) and the surrounding multimodal context.
Let the model’s predictions for the masked tokens be \(\hat{\mathcal{X}}'_{\text{masked}} = \{\hat{X}'_k | k \in \mathcal{M}\}\). The supervision objective minimizes the cross-entropy loss between the predicted and original token IDs only at the masked positions, encouraging the model to learn context-aware spatio-temporal representations for downstream video understanding.
| Model | #Params | VBench Scores \(\uparrow\) | VBench++ Scores \(\uparrow\) | ||||||
|---|---|---|---|---|---|---|---|---|---|
| 3-5 (lr)6-8 | Total Score | Quality Score | Semantic Score | Total Score | Quality Score | I2V Score | |||
| Diffusion models | |||||||||
| OpenSoraPlan V1.3 [59] | - | 77.23 | 80.14 | 65.62 | - | - | - | ||
| Show-1 [60] | 6B | 78.93 | 80.42 | 72.98 | - | - | - | ||
| LTX-Video [61] | 1.9B | 80.15 | 82.52 | 70.68 | - | - | - | ||
| CogVideoX [26] | 5B | 81.91 | 83.05 | 77.33 | 86.7 | 78.6 | 94.8 | ||
| Step-Video [62] | 30B | 81.83 | 84.46 | 71.28 | 88.4 | 81.2 | 95.5 | ||
| Gen3 [63] | - | 82.32 | 84.11 | 75.17 | - | - | - | ||
| Kling1.6 [27] | - | 83.40 | 85.00 | 76.99 | - | - | - | ||
| HunyuanVideo [28] | 13B | 83.24 | 85.09 | 75.82 | 86.8 | 78.5 | 95.1 | ||
| Wan2.1 [29] | 14B | 84.70 | 85.64 | 80.95 | 86.9 | 80.8 | 92.9 | ||
| Autoregressive models | |||||||||
| VILA-U [7] | 7B | 74.01 | 76.26 | 65.04 | - | - | - | ||
| Lumos-1 [46] | 3.6B | 78.32 | 79.52 | 73.51 | 84.7 | 76.1 | 93.3 | ||
| NOVA [42] | 0.6B | 80.12 | 80.39 | 79.05 | - | - | - | ||
| CausVid [64] | 1.3B | 81.46 | 84.05 | 69.80 | - | - | - | ||
| Pyramid Flow [65] | 2B | 81.72 | 84.74 | 69.62 | - | - | - | ||
| VideoMAR [49] | 1.4B | - | - | - | 84.8 | 75.6 | 94.0 | ||
| Unified models | |||||||||
| HaploOmni [66] | 9B | 78.10 | - | - | - | - | - | ||
| Emu3 [8] | 8B | 80.96 | 84.09 | 68.43 | - | - | - | ||
| Show-o2 [15] | 2B | 81.34 | 82.10 | 78.31 | 85.9 | 77.7 | 94.1 | ||
| (Ours) | 3B | 82.58 | 82.64 | 82.35 | 86.7 | 79.4 | 94.1 | ||
4 Experiment↩︎
4.1 Experiment Setup↩︎
Dataset The training is divided into two stages. The first stage involves joint training on image-text modalities, while the second stage fine-tunes the model for downstream video tasks. The first-stage dataset comprises approximately 100 million text-image samples, including Blip3o [10], JourneyDB [67], DenseFusion [68], FineVision [69], LLaVA-OneVision [70], and Infinity-mm [71]. The second-stage dataset comprises 5 million video samples sourced from Koala-36M [72], OpenVid-1M [73], LLaVA-Video-178K [74], together with image data from ShareGPT-4o-Image [75], Blip3o-60k [10] and Echo-4o [76].
Benchmarks We primarily evaluate our model on both video generation and video understanding benchmarks. For video generation, we use the VBench series [77], [78] benchmarks to assess the quality and fidelity of generated videos. For video understanding, we evaluate performance on established multimodal understanding benchmarks, including VideoMME [79], LongVideo Bench [80], Egoschema [81], Next-QA [82] and MLVU [83].
Training Details We employ the Qwen2.5-3B model [84] as the base autoregressive transformer, while the diffusion decoder is initialized from Wan2.1-1.3B [29]. The vision tokenizer is adopted from Tar [53], yielding a total of 65,536 discrete visual tokens. A lightweight two-layer MLP is further introduced to bridge the hidden states of the autoregressive transformer with the diffusion decoder. Training is conducted in two stages. The first stage involves large-scale image-text autoregressive pretraining to establish strong multimodal alignment. For Stage 1 training, we use a batch size of 256 and train the model for 200,000 steps. For video generation training, we train the AR and diffusion modules jointly with a batch size of 64 for 60,000 steps at a resolution of 49\(\times\)832\(\times\)480. The videos are sampled at 8 FPS, with keyframes extracted every 2 seconds. To enhance semantic understanding, video data are mixed with text-to-image (T2I) samples at a 2:1 ratio during training. Additionally, during video training, there is a 30% probability of conditioning on the first frame to train image-to-video (I2V) generation. For ablation studies, we used a resolution of 33\(\times\)832\(\times\)480 and trained the model for 20,000 steps for faster evaluation. For video understanding training, we set batch size as 16 and train the model for 80,000 steps, with 30 frames sampled from each video in total, including 10 pivot frames and 20 detail frames.
| Model | #Params. | VideoMME\(\uparrow\) | MLVU\(\uparrow\) | LongVideoBench\(\uparrow\) | NextQA\(\uparrow\) | EgoSchema\(\uparrow\) |
|---|---|---|---|---|---|---|
| GPT-4V [85] | - | 57.0 | 43.5 | 61.3 | - | 55.6 |
| GPT-4o [86] | - | 71.9 | 64.6 | 66.7 | - | 72.2 |
| Gemini-1.5-Flash [87] | - | 70.3 | 51.3 | 61.6 | - | - |
| Gemini-1.5-Pro [87] | - | 75.0 | - | 64.0 | - | 63.2 |
| LongVA [88] | 7B | 52.6 | - | - | 68.3 | - |
| VideoLLaMA2 [22] | 7B | 47.9 | - | 58.2 | - | 51.7 |
| VideoLLaMA3 [89] | 7B | 66.2 | 73.0 | 63.3 | 84.5 | 63.3 |
| LLaVA-OV [70] | 7B | 58.2 | 56.3 | 56.4 | 79.4 | 60.1 |
| LLaVA-Video [90] | 7B | 63.3 | 70.8 | 58.2 | 83.2 | 57.3 |
| HaploOmni [66] | 9B | - | - | - | - | 47.1 |
| Show-o2 [15] | 7B | 57.4 | - | 55.5 | 79.0 | - |
| (Ours) | 3B | 69.4 | 71.6 | 60.1 | 76.3 | 52.9 |
4pt
Inference Details For text-to-video generation, we use the original prompt suite provided in VBench, which contains 946 prompts across 16 dimensions, without applying prompt rewriting. For evaluating video understanding performance, we adopt LMMs-Eval [91].
For video generation inference, we configure the autoregressive transformer with top-k = 256 and top-p = 0.95 for visual token sampling, while the diffusion decoder operates with a classifier-free guidance (CFG) scale of 4.0 and performs 50 denoising steps to generate the final video frames.
For video understanding, we evaluate VideoMME without subtitles, LongVideoBench on the validation set, and EgoSchema on the test set. The MLVU accuracy is reported as the mean over all subtasks. We use the same frame selection mechanism from the training stage, with 30 frames sampled in total, 10 and 20 for pivot and detail frames.
4.2 Main Results↩︎
4.2.0.1 Video Generation
We evaluate text-to-video performance on VBench [77] and image-to-video performance on VBench++[78]. As shown in Table 1, Vega achieves leading performance among unified video understanding-generation models, while showing competitive results against specialized video generation models. Furthermore, its strong image-to-video results demonstrate the framework’s versatility for general-purpose video generation.
4.2.0.2 Video Understanding
As shown in Table 2, we evaluate our model across a range of video understanding benchmarks. Our approach achieves competitive performance compared with strong open-source understanding-only video models. Compared with prior unified approaches, our model shows clear gains on VideoMME and LongVideoBench, achieving 69.4 and 60.1, respectively. It is worth noting that competing models, both understanding-only and unified, typically contain over 7B parameters, whereas our 3B-parameter model achieves comparable or superior performance. This underscores the strong parameter efficiency and competitive capability of our architecture.
4.3 Ablation Studies↩︎
In this section, we provide an in-depth analysis of the proposed method.
4.3.0.1 Visual Condition vs. Text Condition
In Table 3, we investigate the benefits of employing visual tokens as conditioning inputs in place of text tokens. The results show that conditioning on visual tokens consistently yields better generation quality compared to text-based conditioning, suggesting that visual representations provide richer and more fine-grained context for guiding video generation. Interestingly, when visual and textual tokens are used jointly, the performance is comparable to or even slightly lower than using visual tokens alone. This finding further highlights that visual tokens alone provide sufficiently expressive conditioning signals for effective modeling, revealing that text embeddings may not serve as the optimal conditioning modality for video generation.
| vision cond. | text cond. | VBench Scores \(\uparrow\) | ||
|---|---|---|---|---|
| 3-5 | Total Score | Quality Score | Semantic Score | |
| ✔ | 82.13 | 83.36 | 77.21 | |
| ✔ | 78.61 | 81.70 | 66.28 | |
| ✔ | ✔ | 81.43 | 82.57 | 76.86 |
| Keyframes | Tokens/Frame | Total | Keyframes Interval | VBench Score \(\uparrow\) |
|---|---|---|---|---|
| 3 | 81 | 243 | 2.0 s | 81.78 |
| 5 | 81 | 405 | 1.0 s | 81.43 |
| 9 | 81 | 729 | 0.5 s | 81.14 |
| 2 | 169 | 338 | 4.0 s | 81.31 |
| 3 | 169 | 507 | 2.0 s | 82.13 |
| 5 | 169 | 845 | 1.0 s | 81.72 |
| 9 | 169 | 1521 | 0.5 s | 81.46 |
4.3.0.2 Analysis of Visual Token Numbers
Table 4 analyzes the impact of varying the number of keyframes and per-frame visual tokens on performance. The results indicate that more visual tokens do not necessarily lead to better performance; we find that predicting keyframes at 2s intervals yields the best results, and moderately increasing the number of tokens per frame can also bring performance gains.
4.3.0.3 Effectiveness of Dual-flow Selection
Table 5 demonstrates that the proposed dual-flow selection significantly enhances temporal modeling. Compared to the baseline, which uniformly samples 20 frames with a single pooling scale of 2, the dual-flow design (Section 3.3.0.1) captures richer temporal cues while avoiding excessive visual tokens, resulting in consistent improvements across all benchmarks.
| dual-flow | gen. sup. | VideoMME \(\uparrow\) | MLVU \(\uparrow\) | LongVideoBench\(\uparrow\) | NextQA \(\uparrow\) |
|---|---|---|---|---|---|
| 60.04 | 64.78 | 53.83 | 49.46 | ||
| ✔ | 60.48 | 65.64 | 54.32 | 58.84 | |
| ✔ | ✔ | 66.89 | 66.46 | 58.74 | 76.41 |
4.3.0.4 Generative Supervision for Understanding
Building on the dual-flow setup, we add the generative supervision loss (Section 3.3.0.2). This objective encourages the model to predict visual tokens from surrounding context, providing an auxiliary training signal in addition to text-only supervision. As shown in Table 5, this additional supervision leads to further improvements beyond dual-flow, suggesting that visual-token prediction can complement standard video understanding training. We hypothesize that these gains may come from the dense supervision provided by autoregressive token prediction over video sequences, which encourages the model to encode temporal and semantic structure more explicitly.
4.4 Qualitative Analysis↩︎
This section provides qualitative results on image-to-video and image editing tasks, as shown in Figure 3.
4.4.0.1 Effectiveness of Noise-controlled Visual Conditioning
Although using semantic visual tokens significantly reduces the number of tokens, projecting them into the semantic space inevitably leads to the loss of fine-grained details. This can cause inconsistency between the generated video and the first-frame condition in tasks such as image-to-video (I2V) generation. To alleviate this issue, we propose noise-controlled visual conditioning (Section 3.2.0.2), which effectively mitigates such inconsistencies and yields visually consistent results, as demonstrated in Figure 3.
4.4.0.2 Reframing Image Editing as Video Generation
Within our unified framework, image editing can be naturally interpreted as a form of dynamic video modeling. Conventional image editing takes a reference image and a text prompt to produce a target image. In our formulation, we treat the reference–target pair as a two-frame video sequence, allowing the model to capture the transition dynamics between the original and edited states. As illustrated in Figure 3, our method generates coherent and faithful editing results, demonstrating the compatibility and unifying potential of our video-centric modeling paradigm.
5 Conclusion↩︎
In this paper, we introduce Vega, a unified framework for video generation and understanding. By transforming multimodal signals into discrete tokens, our method establishes a unified representation space for different modalities. This enables coherent modeling of visual semantics and temporal dynamics through a hybrid architecture that integrates autoregressive semantic prediction with diffusion-based rendering. Extensive experiments on both generation and understanding benchmarks demonstrate the effectiveness of our approach and show that learning from visual tokens can benefit both tasks: offering richer conditioning for generation and providing auxiliary generative supervision for understanding. Overall, this work underscores the promise of video-centric modeling as a foundation for advancing unified multimodal learning.
5.0.0.1 Limitations
Due to computational constraints, the current data scale and model size for video generation and understanding remain limited; thus, our framework represents an initial step in this direction. We believe it is inherently well-suited for multimodal reinforcement learning and in-context learning, which we plan to explore further in future work.
6 Qualitative Results↩︎
This section provides additional examples that further demonstrate the model’s capabilities in both video understanding and video generation.
6.1 Video Generation↩︎
As shown in Figure 4, our model is capable of generating a diverse set of visually appealing video sequences from textual descriptions, while demonstrating strong instruction-following capabilities.
6.2 Video Understanding↩︎
Here we present additional qualitative examples showcasing the capabilities of Vega in responding to various video inputs and corresponding text prompts. Figure 5 provides four examples demonstrating the video understanding capability, including (a) an example illustrating the model’s ability to infer fine-grained content with basic knowledge; (b) an example illustrating the model’s ability to understand hours-long videos; (c) an example illustrating the model’s ability to perform causal reasoning, such as explaining humorous content in the video; and (d) an example illustrating the model’s ability to understand the logical relationship between two long-duration plots.
7 Implementation Details↩︎
7.0.0.1 Training Format
The first-stage model is trained on image–text pairs, and to ensure compatibility with video inputs, the specific training strategy and data format are described as follows. For video understanding tasks, we read videos as byte streams and integrate them with the corresponding text conversations, forming the standard conversation format used by OpenAI.
{
"video": video_byte_stream,
"conversations":
[{'from': 'human',
'value': <video>\n+Question}
{'from': 'gpt',
'value': Answer}]
}Similarly, for video generation tasks, the training format is organized as shown below.
{
"video": video_byte_stream,
"conversations":
[{'from': 'human',
'value': text prompt}
{'from': 'gpt',
'value': <video>}]
}During both training and inference, we read the video as a byte stream and select the corresponding frames according to the designated sampling strategy. The \(<video>\) segment is then replaced with the corresponding visual tokens.
7.0.0.2 Evaluation Details
We conduct evaluations on video understanding benchmarks using the open-source project lmms-eval. For the VideoMME and MLVU benchmarks, we directly report the scores from lmms-eval. However, on the multiple-choice tasks of the NextQA, LongVideoBench, and EgoSchema benchmarks, the model outputs explanatory content beyond the options. This caused the lmms-eval filter to fail parsing the answers. To address this, we implement a custom filter that extracts the first character of the response text as the model’s answer. We then compare this answer with the ground truth to calculate the scores.
7.0.0.3 Failure Analysis
Here, we provide representative failure cases in video understanding and analyze the key factors that prevent our model from correctly handling these scenarios.
The observed cases demonstrate the model’s limitations in fine-grained frame selection and content understanding. Given the high frame count of long videos, sampling only 30 frames is insufficient to capture all critical information, leading to performance gaps on detail-oriented benchmarks. Furthermore, the use of token pooling compromises the model’s ability to perceive fine-grained spatial details, including small visual elements such as subtitles. A detailed analysis of representative cases is provided below: In case (a), the model mistakenly identifies the sliced chilis as tomatoes, leading it to conclude that the ring on the chef’s hand appears in the scene. In reality, the chef is slicing tomatoes with a golden fork, but the relevant frames are not included in the sampled inputs. In case (b), the model fails to correctly interpret the frame in which the soldier is opening a can, resulting in an incorrect ordering of the scenes. (As the original prompt is too long to display, for simplicity we only show the right and wrong answers chosen by our method.) In cases (c) and (d), the model is unable to capture the subtitle or caption information present in the required frames, and thus cannot produce the correct answers.