1 Introduction↩︎

Large Vision-Language Models (LVLMs) have achieved remarkable progress in video understanding by leveraging powerful cross-modal alignment and temporal modeling capabilities[1]–[4]. However, these models typically rely on uniform temporal sampling, which becomes increasingly problematic for long-duration videos: the growing number of visual tokens escalates computational cost and often exceeds the constrained context windows of LVLMs. This forces a suboptimal trade-off between temporal and spatial resolution, inevitably leading to significant information loss and hindering the practical deployment of LVLMs for long video understanding.

Keyframe selection offers an effective paradigm to balance efficiency and accuracy by distilling a concise, task-relevant subset of frames from a long video, thereby minimizing visual token overhead while preserving essential information for downstream LVLMs. Recent studies identify informative keyframes by either developing trainable selection modules [5]–[9] or designing heuristic training-free policies [10]–[14] based on frame-wise query-relevance scores computed by off-the-shelf cross-modal embedding models (e.g., CLIP [15]). However, as illustrated in Fig.1, existing methods treat video frames as isolated units, neglecting intrinsic temporal coherence and higher-level semantic structures. Moreover, they adopt an equal allocation strategy that assigns identical token budgets to all selected keyframes regardless of their semantic significance, wasting computational resources on temporally redundant or marginally informative frames.

To address these issues, we propose GMM-EVA (Gaussian Mixture Modeling for Event-Aware Visual Allocation), a training-free, plug-and-play method that explicitly models event-level structure to adaptively distribute the visual token budget (Fig.1). The core of our approach involves parameterizing the temporal importance distribution via an event-centric Gaussian Mixture Model (GMM), where each component characterizes a latent semantic event. This formulation is motivated by the fact that events are high-level semantic concepts crucial for video understanding tasks [16]–[18], and the inherent smoothness and multi-peak nature of GMM make it well-suited for representing the continuous temporal span of multiple key events [19], [20]. Concretely, frame-wise query-relevance scores serve as discrete observations of latent temporal importance. By applying the Expectation-Maximization (EM) algorithm [21], GMM-EVA transforms these noisy observations into a structured, continuous representation where latent events are explicitly defined by individual Gaussian components. By operating purely on scalar relevance scores, GMM-EVA decouples the event discovery process from any specific embedding model or downstream LVLM, enabling seamless integration across diverse model combinations.

Building on this event-level representation, we further propose an event-aware visual allocation strategy. Each keyframe is assigned to its most probable latent event via GMM posterior probabilities, and within each event the most salient frame is designated as the primary keyframe while the rest serve as secondary keyframes. By allocating high-resolution budgets to primary keyframes as semantic anchors and low-resolution budgets to secondary keyframes for temporal context, this design prunes intra-event redundancy and optimizes the token budget without compromising performance.

To summarize, our contributions are three-fold: 1) We employ a Gaussian Mixture Model fitted via the EM algorithm to transform noisy frame-wise relevance scores into a structured representation of latent semantic events, enabling explicit temporal structuring of long videos. 2) Building on the discovered events, we design an event-aware allocation strategy that concentrates high-resolution budgets on primary keyframes as semantic anchors while using low-resolution secondary keyframes to maintain temporal context, effectively pruning intra-event redundancy. 3) Extensive experiments on multiple long video benchmarks show that GMM-EVA achieves competitive or superior performance while consuming approximately half the visual token budget, with robust generalization across diverse embedding backbones and downstream LVLMs.

2 Related Works↩︎

2.1 Keyframe Selection for Long Video Understanding↩︎

Keyframe selection aims to identify a condensed, task-relevant subset of frames or clips as input for downstream Large Vision-Language Models (LVLMs). Existing methodologies are generally categorized into training-based and training-free approaches.

Training-based approaches [5]–[9] typically train specialized modules for frame-wise scoring or index prediction. Methods such as VideoITG [7] and GenS [8] achieve high precision by employing LVLMs as selectors, yet suffer from significant inference latency and are constrained by the selector’s own context window. Alternatives like TSPO [6] and MSJoE [5] reduce computational cost with lightweight trainable selectors, but their reliance on large-scale annotated datasets remains a major scalability bottleneck.

Training-free methods [10]–[14], [22], [23] leverage off-the-shelf cross-modal embedding models, such as CLIP [15], to measure query-frame relevance without additional training. Beyond basic TopK selection [10], recent heuristics like AKS [12] and BOLT [10] refine keyframe composition but still treat sampled frames as atomic and equal units. F2C [22] extends selected anchors to surrounding clips, and Q-Frame [23] applies global resolution adjustments, yet neither captures higher-level temporal semantics. In contrast, GMM-EVA explicitly models event-level structure via generative modeling, transforming noisy frame-wise scores into a structured event representation and enabling adaptive, differentiated allocation within each event.

2.2 Gaussian Temporal Modeling in Video Understanding↩︎

Gaussian distributions are widely adopted as temporal priors in video understanding, owing to their inherent smoothness and ability to model continuous temporal spans, particularly in temporal grounding tasks [20], [24], [25]. For instance, GTAN [25] and CPL [20] utilize learnable Gaussian masks as temporal weights to aggregate visual features. To capture the multi-peak nature of video content [26], recent methods [19], [27] extend single Gaussian masks to Gaussian Mixture Models (GMMs), enabling the representation of multiple query-relevant events. Departing from these learnable approaches, GMM-EVA estimates GMM parameters directly from frame-wise relevance scores via the EM algorithm [21], enabling structured event discovery without supervised training or gradient-based optimization.

3 Methodology↩︎

Given a video with \(T\) candidate frames \(\mathcal{F} = \{f_t\}_{t=1}^T\) and a textual query \(Q\), GMM-EVA selects a compact subset of keyframes as the visual input for a downstream LVLM. As illustrated in Fig. 2 and Algorithm 3, the pipeline consists of three stages. First, a cross-modal embedding model computes frame-wise query-relevance scores as observations of temporal importance (§3.1). Then, a Gaussian Mixture Model is fitted to these observations via the EM algorithm, decomposing the score distribution into latent semantic events (§3.2). Finally, an event-aware allocation strategy assigns each keyframe to its most probable event and differentiates between a high-resolution primary keyframe and lower-resolution secondary keyframes within each event, yielding two subsets \(\mathcal{F}_{prim}\) and \(\mathcal{F}_{sec}\) (§3.3).

3.1 Query-Relevance Modeling↩︎

We first quantify the semantic relevance of each candidate frame \(f_t\) to the textual query \(Q\). Specifically, GMM-EVA employs CLIP [15], a lightweight pretrained cross-modal embedding model, whose visual and text encoders are denoted by \(\text{CLIP}_V(\cdot)\) and \(\text{CLIP}_T(\cdot)\), respectively. The query-relevance score \(s_t\) is computed as the cosine similarity between the frame and query embeddings: \[s_t = \text{sim}\bigl(\text{CLIP}_V(f_t),\;\text{CLIP}_T(Q)\bigr), \label{eq:1}\tag{1}\] where \(\text{sim}(\cdot,\cdot)\) denotes the cosine similarity. To interpret the scores set \(S = \{s_t\}_{t=1}^T\) as a valid probability distribution for subsequent GMM fitting, we apply a linear normalization: \[s_t' = \frac{s_t-\min(S)}{\sum_{i=1}^T (s_i-\min(S))}. \label{eq:2}\tag{2}\] The resulting set \(S' = \{s_t'\}_{t=1}^T\) serves as a discrete, empirical observation of how temporal importance is distributed across the video, providing the evidence required for latent event modeling in the next stage.

3.2 Latent Event Discovery via GMM↩︎

To move beyond isolated frame-level analysis and capture the coherent temporal structure of videos, GMM-EVA employs a Gaussian Mixture Model (GMM) to parametrically model the query-conditioned importance distribution. Specifically, the temporal importance density \(p(\tau)\) is formulated as a mixture of \(C\) Gaussian components, each representing a latent semantic event: \[p(\tau) = \sum_{j=1}^{C} \pi_j \mathcal{N}(\tau | \mu_j, \sigma_j^2), ~s.t. \sum_{j=1}^{C} \pi_j=1, \label{eq:3}\tag{3}\] where \(\tau \in [1,T]\) denotes the continuous temporal coordinate and \(\mathcal{N}\) denotes the Gaussian probability density function. Since candidate frames are uniformly sampled at a fixed rate, the temporal coordinate of frame \(f_t\) is identified with its index, i.e., \(\tau = t\). For each component, the mixing coefficient \(\pi_j\), mean \(\mu_j\), and standard deviation \(\sigma_j\) encode the event’s cumulative significance, temporal center, and duration, respectively.

The parameters \(\Theta = \{\pi_j, \mu_j, \sigma_j\}_{j=1}^{C}\) of the GMM are estimated by applying the Expectation-Maximization (EM) algorithm to the discrete observations \(S'\). GMM-EVA first adopts a uniform initialization strategy with a temporal stride \(\Delta t_c\). The initial number of components is set to \(C = \lfloor \frac{T}{\Delta t_c} \rfloor\), with the \(j\)-th component initialized as: \[\pi_j=\frac{1}{C},\quad\mu_j = \frac{jT}{C+1},\quad\sigma_j=\frac{T}{2C},\quad\text{for}\quad j=1,2,\dots,C. \label{eq:4}\tag{4}\] The EM algorithm then alternates between the E-step and M-step until convergence. The E-step computes the responsibility \(\gamma_{t,j}\), i.e., the posterior probability that frame \(f_t\) belongs to the \(j\)-th latent event: \[\gamma_{t,j} = \frac{\pi_j \mathcal{N}(t | \mu_j, \sigma_j^2)}{\sum_{i=1}^{C} \pi_i \mathcal{N}(t | \mu_i, \sigma_i^2)}. \label{eq:5}\tag{5}\] The M-step updates the parameters to maximize the weighted log-likelihood. Unlike standard EM, GMM-EVA incorporates \(s_t'\) as an importance weight for each frame. Let \(N_j = \sum_{t=1}^T s_t' \gamma_{t,j}\) denote the effective count of the \(j\)-th component; the parameters are updated as follows: \[\begin{align} \pi_j = \frac{N_j}{\sum_{k=1}^{C} N_k} = N_j,\quad \mu_j &= \frac{1}{N_j} \sum_{t=1}^T s_t' \gamma_{t,j} t,\quad \sigma_j = \sqrt{\frac{1}{N_j} \sum_{t=1}^T s_t' \gamma_{t,j} (t - \mu_j)^2}\\ \text{for}\quad t&=1,2,...,T,~j=1,2,...,C, \end{align} \label{eq:6}\tag{6}\] Furthermore, to refine the event structure, we incorporate a component merging scheme [25]. Two components \(\{\pi_{(1)}, \mu_{(1)}, \sigma_{(1)}\}\) and \(\{\pi_{(2)}, \mu_{(2)}, \sigma_{(2)}\}\) are merged if the Intersection over Union (IoU) of their 1-\(\sigma\) intervals exceeds a threshold \(\epsilon_m\). The merged parameters are derived as: \[\begin{align} \pi_{new} &= \pi_{(1)} + \pi_{(2)},\quad \mu_{new} = \frac{\mu_{(1)}\pi_{(1)} + \mu_{(2)}\pi_{(2)}}{\pi_{new}},\\ \sigma_{new} &= \sqrt{\frac{\pi_{(1)}\sigma_{(1)}^2 + \pi_{(2)}\sigma_{(2)}^2}{\pi_{(1)} + \pi_{(2)}} + \frac{\pi_{(1)}\pi_{(2)}(\mu_{(1)} - \mu_{(2)})^2}{(\pi_{(1)} + \pi_{(2)})^2}}. \end{align} \label{eq:7}\tag{7}\]

As outlined in Alg. 3, the EM iteration continues until the increment of the weighted log-likelihood \(\mathcal{L}(\Theta) = \sum_{t=1}^T s_t' \ln \left( \sum_{j=1}^{C} \pi_j \mathcal{N}(t | \mu_j, \sigma_j^2) \right)\) falls below a threshold \(\epsilon_L\), or the iteration count reaches a maximum limit \(I_{max}\). The converged parameter set \(\Theta^* = \{\pi_j^*, \mu_j^*, \sigma_j^*\}_{j=1}^{C^*}\) constitutes a structured representation of the video’s event-level temporal layout, where each Gaussian component probabilistically characterizes a latent semantic event, forming the foundation for the subsequent visual budget allocation.

3.3 Event-Aware Visual Allocation↩︎

With the latent event structure established via \(\Theta^*\), GMM-EVA implements a differentiated strategy to allocate visual budgets by categorizing keyframes into primary and secondary roles based on their event membership. We first sort the candidate frames in descending order of their relevance scores \(S\) and select the top-\(K\) allocation targets. For each top-\(K\) frame \(f_{(k)}\), we compute its posterior probability of belonging to the \(j\)-th latent event using the converged parameters \(\Theta^*\) via Eq. 5 , denoted as \(\gamma^*_{(k),j}\). GMM-EVA performs a hard assignment by associating the frame with its most probable event \(j^*\), formulated as: \[j^* = \arg\max_{j \in {1, \dots, C^*}} \gamma^*_{(k),j} \label{eq:8}\tag{8}\] The union of events assigned to the top-\(K\) frames constitutes the active event set \(\mathcal{E}_{Top\text{-}K}\).

To mitigate intra-event redundancy, we differentiate frames within each active event \(j \in \mathcal{E}_{Top\text{-}K}\): the highest-scoring frame is designated as the primary keyframe and collected into \(\mathcal{F}_{prim}\), serving as the semantic anchor of that event, while all remaining frames form the secondary set \(\mathcal{F}_{sec}\), providing auxiliary temporal context at a lower visual budget. Since the top-\(K\) frames are traversed in descending relevance order, this categorization is realized by checking whether each frame’s assigned event has been seen before (Algorithm 3).

When fed into downstream LVLMs, primary keyframes in \(\mathcal{F}_{prim}\) are allocated a higher visual budget \(B_p\) (e.g., high resolution), while secondary keyframes in \(\mathcal{F}_{sec}\) receive a lower budget \(B_s\) (e.g., low resolution). This differentiated approach concentrates computational resources on unique semantic anchors while maintaining temporal continuity through low-cost secondary frames, effectively optimizing the token budget without compromising understanding performance. Notably, GMM-EVA can be flexibly scaled by incorporating events beyond \(\mathcal{E}_{Top\text{-}K}\), as further analyzed in Sec. 4.3.3.

4 Experiments↩︎

4.1 Experimental Setups↩︎

4.1.1 Implementation Details↩︎

In our experiments, videos are sampled at 1 FPS to obtain candidate frames. We employ CLIP [15] as the default cross-modal embedding model, with additional experiments conducted using SigLIP [28], LanguageBind-Video [29], and Qwen3-VL-Embedding-8B [30]. For GMM parameters, the temporal stride \(\Delta t_c\) is set to 30, the merging threshold \(\epsilon_m\) to 0.7, the log-likelihood convergence threshold \(\epsilon_L\) to \(10^{-5}\), and the maximum iteration count \(I_{max}\) to 1000. The top-\(K{=}32\) target frames by relevance score are selected for event-aware allocation, with \(B_p{=}256\) tokens per primary keyframe and \(B_s{=}64\) tokens per secondary keyframe. We adopt Qwen2.5-VL-7B as the default downstream LVLM, with generalization experiments further conducted using Qwen2-VL-7B [31], Qwen3-VL-8B [1], and InternVL-2.5-8B [2]. Unless otherwise specified, all experiments use CLIP and Qwen2.5-VL-7B.

4.1.2 Evaluation↩︎

We evaluate GMM-EVA on three representative long-form video understanding benchmarks: LongVideoBench [32], LVBench [33], and Video-MME [34]. For LongVideoBench, we additionally construct a multi split comprising question types that involve multiple events (E3E, O3O, SSS, SOS, and SAA, per the official taxonomy) to specifically assess temporal structure modeling. For VideoMME, we additionally report results on the long split (average duration 2,466.7s). We use multiple-choice accuracy as the primary metric. Computational efficiency is measured by the relative token budget ratio \(\mathcal{R}_B\), defined as the percentage of consumed tokens relative to a baseline that processes all \(K{=}32\) frames at \(B_p{=}256\) tokens each.

4.2 Main Results↩︎

We compare GMM-EVA with several representative keyframe selection methods in Tab.1. All competing methods consume the full 100% budget (32 keyframes \(\times\) 256 tokens each) for downstream LVLM visual input.

With the CLIP backbone, GMM-EVA uses only 44.9%/52.7%/44.8% of the base budget on LongVideoBench/LVBench/VideoMME, yet achieves competitive performance. Compared with baseline methods, GMM-EVA with CLIP outperforms uniform sampling by a large margin across all benchmarks, and matches the Top-K baseline in accuracy while consuming approximately half the token budget. When compared with training-free methods, although GMM-EVA with CLIP does not surpass all competitors on every benchmark under its reduced budget, it achieves the best score on the multi split of LongVideoBench, underscoring the advantage of explicit event modeling for questions involving multiple events. For training-based methods, their specialized selection modules are trained to precisely measure frame-query relevance, thus outperforming GMM-EVA when paired with the lightweight CLIP encoder. Nevertheless, when equipped with the more powerful Qwen3-VL-Embedding-8B, GMM-EVA surpasses VideoITG/MSJoE by +2.7/+2.3 on average. These results validate the effectiveness and efficiency of GMM-EVA for long video understanding.

4.3 Analysis and Ablation Studies↩︎

4.3.1 Generalization across Embedding Backbones↩︎

As shown in Tab.¿tbl:tab:2?, GMM-EVA yields consistent gains of +3.6/+3.2/+5.3 over uniform sampling on LongVideoBench with SigLIP/LanguageBind-Video/Qwen3-VL-Embedding-8B, while matching or exceeding the full-budget Top-K baseline (\(\mathcal{R}_B{=}100\%\)) using roughly half the tokens and surpassing Top-K at comparable budgets (\(\mathcal{R}_B{=}50\%\)). Among the evaluated backbones, Qwen3-VL-Embedding-8B yields the best results, as its stronger cross-modal alignment provides higher-quality importance observations for GMM estimation. These results demonstrate that GMM-EVA generalizes seamlessly across diverse relevance sources, and its performance ceiling can be further elevated by integrating more advanced embedding models.

4.3.2 Compatibility with Downstream LVLMs↩︎

As shown in Tab.[tbl:tab:3], GMM-EVA improves over uniform sampling by +1.7/+2.7/+4.6 on LongVideoBench when paired with Qwen2-VL-7B/Qwen3-VL-8B/InternVL-2.5-8B. Across three downstream LVLMs, GMM-EVA with only 44.9% of the token budget consistently outperforms Top-K at 50% budget and achieves comparable accuracy to Top-K at 100%. Notably, on Qwen2-VL-7B, GMM-EVA also matches two training-based methods, GenS and TSPO, without any training and with substantially fewer tokens. The consistent efficacy across different model families (Qwen-VL and InternVL) and versions highlights the strong compatibility of our method.

4.3.3 Scaling Analysis↩︎

We analyze the scalability of GMM-EVA under varying token budgets. As discussed in Sec.3.3, GMM-EVA can be scaled by incorporating external latent events beyond the targeted \(K\) keyframes. We empirically allocate 1 primary and 2 secondary keyframes per additional event. We compare this with an internal variant that upgrades existing secondary keyframes to primary within the \(K\) targeted frames, as well as Uniform and Top-K baselines with varying frame counts.

As shown in Fig.4, the External strategy provides superior expansion capability over the Internal approach, and GMM-EVA consistently outperforms both baselines across all budget levels, demonstrating that our method enhances understanding performance even at equivalent budget scales. Furthermore, when externally scaled with 28 additional events, GMM-EVA achieves a 4.8% improvement on the long split, substantially exceeding the 2.0% gain on the full VideoMME set, suggesting that event-aware modeling is particularly advantageous for long-form videos.

4.3.4 Primary Keyframe Allocation↩︎

We compare the event-aware allocation of GMM-EVA against two alternatives for designating primary keyframes: 1) Top, which directly assigns the highest-scoring frames to \(\mathcal{F}_{prim}\); 2) K-means, which clusters keyframe timestamps and selects the highest-scoring frame per cluster. For fair comparison, \(|\mathcal{F}_{prim}|\) is set to the number of events \(C^*\) adaptively determined by GMM-EVA for each video, providing the two baselines with an oracle event count.

As shown in Tab.¿tbl:tab:4?, the event-aware strategy achieves the best performance across all benchmarks, validating the effectiveness of GMM-based adaptive event modeling. The Top method tends to select temporally adjacent frames, lacking diversity; K-means attempts temporal partitioning but cannot distinguish events with non-uniform densities. Moreover, despite receiving the oracle event count from GMM-EVA, neither baseline can match our method, as they lack the ability to adaptively discover the optimal event structure and must rely on external guidance for this critical hyperparameter.

4.3.5 Impact of Secondary Keyframe Budget↩︎

We vary the per-frame token budget \(B_s\) of secondary keyframes to assess their contribution. As shown in Tab.[tbl:tab:5], \(B_s{=}64\) yields the best accuracy; lower values incur notable information loss, while higher values bring only marginal gains at substantially greater cost. Removing secondary keyframes entirely (\(B_s{=}0\)) causes a 1.7-point drop compared to \(B_s{=}64\), confirming the importance of the temporal context they provide. Notably, even without secondary frames, GMM-EVA surpasses the Top-K baseline at comparable budgets by 1.8, further validating the superiority of our event-aware selection in distilling higher-quality semantic keyframes.

4.4 Qualitative Analysis↩︎

Fig.5 presents a qualitative comparison. The dashed GMM density closely fits the empirical score curve, confirming that the model faithfully captures the underlying event structure. GMM-EVA correctly identifies the key events relevant to the question (e.g., “visit sculpture”, “enter museum”) and allocates representative keyframes accordingly, producing a correct answer. In contrast, uniform sampling misses critical temporal cues and only captures “painting”-related content, leading to an incorrect answer.

5 Conclusion↩︎

In this paper, we revisited keyframe selection for long video understanding through the lens of event-level semantics, arguing that recognizing latent events offers a principled path toward more efficient visual budget utilization. Guided by this insight, GMM-EVA recasts noisy relevance scores into a structured Gaussian mixture, upon which an event-aware allocation strategy distinguishes primary semantic anchors from secondary contextual frames. Across three long-video benchmarks, GMM-EVA achieved competitive or superior accuracy with roughly half the conventional token budget, generalizing consistently across diverse embedding backbones and downstream LVLMs. The gain became more pronounced on the long-duration split of VideoMME, suggesting event-level structuring as a promising direction for further scaling LVLMs to longer videos.

Acknowledgements↩︎

This work was supported by Beijing Natural Science Foundation (L243015), Beijing Major Science and Technology Project under Contract no. Z251100008425008.

References↩︎