1 Introduction↩︎

Large Language Models (LLMs) [1]–[3] and Multimodal LLMs (MLLMs) [4]–[6] have recently demonstrated remarkable capabilities in unified vision-language understanding, enabling complex reasoning over images and videos [5]–[7]. By leveraging large-scale pre-training, these models have significantly advanced in a wide range of video understanding, e.g., video question answering [8], retrieval [9], and grounding [10].

However, despite their strong representational power, applying LLMs or MLLMs to long-form video remains a challenge. Long videos are inherently temporally redundant, and naively encoding dense frame sequences leads to substantial computational inefficiency. More importantly, when video frames are converted into visual tokens, long videos can easily exceed the token budget limitations of current MLLMs, resulting in truncated input or degraded reasoning performance. Under such constraints, selecting a small yet informative subset of frames is often more effective than processing dense but redundant frame sequences.

Existing approaches typically address this issue through uniform sampling [11], attention- or relevance-based selection [12], [13]. However, uniform sampling often overlooks semantically critical moments, as illustrated in Fig.1. Attention-based mechanisms offer fine-grained modeling, but they remain computationally expensive for long videos. And relying solely on query-frame similarity may overlook video content diversity, leading to redundant frame selection or insufficient information.

Therefore, effective long video understanding with MLLMs requires query- and content-aware keyframe selection. Intuitively, different temporal segments of a video contribute unequally to a given query, and selected frames should be both semantically relevant and content-diverse within each segment.

Motivated by this insight, we propose a Query- and Content-Aware keyframe selection framework, which is denoted as QCA. It first estimates the information contribution of each temporal segment by jointly modeling the semantic relevance and the content deviation. Then dynamically allocate keyframe budgets across segments, followed by an intra-segment keyframe selection strategy that balances semantic alignment and content diversity. Moreover, QCA is training-free and can be seamlessly integrated into existing Video-LLMs.

Extensive experiments on multiple long video understanding benchmarks show that our proposed QCA achieves the SOTA performance, e.g., \(66.9\%\) on LongVideoBench [14] and \(70.1\%\) on Video-MME [8]. Besides, it achieves average performance gains of \(4.00\%\) on LLaVA-Video [11] and \(4.78\%\) on Qwen3-VL [5], validating its generalization ability across different cutting-edge Video-LLMs.

Our main contributions are as follows:

• We formulate query-conditioned keyframe selection for long video understanding as a joint relevance–diversity modeling and frame allocation problem under a limited frame budget.

• We propose QCA, a query- and content-aware keyframe selection framework that integrates segment-level contribution estimation, adaptive budget allocation, and anchor-centric greedy frame selection.

• Extensive experiments demonstrate consistent improvements across benchmarks, Video-LLMs, and VL embeddings, demonstrating its superiority and generalization ability.

2 Related Work↩︎

2.1 MLLMs for Video Understanding↩︎

Multimodal Large Language Models (MLLMs) have recently achieved significant progress in video understanding by extending LLMs with visual perception modules. Early works such as Flamingo [15] demonstrated the effectiveness of aligning pretrained vision encoders with frozen language models for multimodal reasoning. Subsequent models, including LLaVA [4] and its variants, further improved vision–language alignment through instruction tuning, achieving strong performance across a wide range of multimodal tasks.

Building on these advances, recent studies have adapted MLLMs to video understanding by representing videos as sequences of visual tokens extracted from sampled frames. Representative approaches, such as Video-LLaMA [16] and VideoChat [17], allow MLLMs to process temporal visual information together with textual queries, enabling applications including video question answering, retrieval, and captioning.

However, most existing Video-LLMs rely on uniform frame sampling to compress videos into a limited set of frames. While simple and efficient, this strategy often struggles with long-form videos due to severe temporal redundancy and the limited context length of language models. Consequently, naively increasing the number of input frames leads to a higher computational cost or truncated visual context. These limitations motivate the need for more efficient video representations, such as query-aware keyframe selection.

2.2 Keyframe Selection for Video-LLMs↩︎

To address the scalability challenges of video inputs, extensive research has focused on designing efficient video representations for Video-LLMs. A common strategy is uniform sampling [7], [11], which reduces the number of frames while maintaining coarse temporal coverage. However, such methods are query-agnostic and often select redundant or irrelevant frames for specific queries.

In addition, some works [18], [19] have explored relevance-based frame filtering, where frames are ranked according to their semantic similarity to a given query and only the top-ranked frames are retained. While these approaches improve query awareness, they typically overlook the intrinsic structure and diversity of video content, leading to suboptimal coverage of important temporal segments. Clustering- and diversity-based methods [13], [20], [21] have also been investigated to select representative frames, but they are usually query-independent and may fail to capture query-specific evidence.

More recently, several studies have proposed segment-based video representations to better handle long videos under limited frame or token budgets. These methods aim to summarize or compress video content before feeding it into language models, for example, through temporal segmentation [22], [23], adaptive keyframe selection [24]–[26], or token pruning [27]–[29].

In contrast to prior works, our approach jointly models query relevance and content structure to select a compact yet informative set of keyframes. By dynamically allocating a limited frame budget across temporal segments and performing semantic-anchored and diversity-aware selection within each segment, our method provides an efficient video representation that is particularly well-suited for long video understanding operating under frame budget constraints.

3 Method↩︎

Rather than introducing new frame-scoring primitives, our goal is to provide a structured framework that integrates query relevance, content representativeness, and budget allocation for efficient keyframe selection in long videos. Fig.2 illustrates the overview of our proposed QCA, which first adaptively assigns the keyframe budget to different temporal segments by modeling semantic matching and content deviation. QCA then iteratively selects keyframes from the candidate set, satisfying both semantic alignment with the query and semantic diversity between selected keyframes.

3.1 Problem Formulation↩︎

Given a video uniformly sampled at 1fps, the resulting frame sequences are denoted as \[\label{eq:video95set} \mathcal{X}=\{x_1, x_2, ..., x_N\},\tag{1}\] where \(N\) is the total number of frames. Given a query \(q\), a natural language question, our goal is to select a compact keyframe subset \[\label{eq:keyframe95subset} \mathcal{K}\subset\mathcal{X}, |\mathcal{K}|=N',N'\ll N\tag{2}\] such that \(\mathcal{K}\) preserves the most query-relevant and informative visual content for the downstream video understanding task: \[\label{eq:task} answer = MLLMs(\mathcal{K},q)\tag{3}\]

3.2 Temporal Segmentation↩︎

To capture coarse temporal structure, we divide the video \(\mathcal{X}\) into \(S\) uniformly spaced segments: \[\label{eq:segment} \mathcal{X}=\bigcup_{s=1}^{S}\mathcal{X}_{s}, \mathcal{X}_{s}\cap\mathcal{X}_{s'}=\emptyset.\tag{4}\] where \(S\) is set to \(12\) by default. Intuitively, a larger \(S\) provides finer temporal granularity but reduces the number of frames allocated per segment, while a smaller \(S\) may overlook short but critical events.

3.3 Inter-Segment Keyframe Allocation↩︎

Semantic Matching. We first measure the average semantic matching degree \(M_s\in [0,1]\) between frames in \(\mathcal{X}_s\) and the given query: \[\label{eq:itm95matching} M_s=\frac{1}{|\mathcal{X}_s|}\sum_{x_i\in\mathcal{X}_s}ITM(x_i,q),\tag{5}\] where \(ITM(\cdot,\cdot)\) denotes Image-Text Matching function, e.g., BLIP-2 [30]. This term refers to the relevant video segment to the query at the semantic level.

Content Deviation. We define the segment content deviation as \[\label{eq:deviation} D_s=\|\mu_{\mathcal{X}_s}-\mu_{\mathcal{X}}\|_2^2 + Tr(\Sigma_{\mathcal{X}_s}),\tag{6}\] where \(\mu_{\mathcal{X}_s}\) and \(\Sigma_{\mathcal{X}_s}\) denote the mean feature and the covariance matrix of \(\mathcal{X}_s\), respectively. The first term measures how much a segment differs from the whole video, encouraging the selection of visually distinctive segments that are more likely to contain informative events. The second term, trace of the covariance matrix, captures intra-segment variation, where larger values indicate rich visual diversity. In practice, both terms are normalized to mitigate scale mismatch.

Keyframe Allocation. The final information contribution score of segment \(\mathcal{X}_s\) is defined as a weighted sum: \[\label{eq:contribution} c_s=\alpha\cdot M_s + \beta\cdot D_s,\tag{7}\] where \(\alpha\) and \(\beta\) are hyperparameters. And we respectively apply a softmax normalization to \(M_s\) and \(D_s\) across segments to ensure comparable scales before fusion. We compute the contribution weights via \[\label{eq:contribution95weight} w_s = \frac{{c_s}^\tau}{\sum_{j=1}^S {c_j}^\tau},\tag{8}\] where \(\tau\) controls the smoothness of allocation. Then the number of keyframes allocated to segment \(\mathcal{X}_s\) is \[\label{eq:quota} q_s = \lfloor w_s\cdot N'\rfloor, \sum_{i=1}^{S} s_i = N^{'}\tag{9}\]

Since the floor operation may yield fewer frames than the target budget, the remaining frames are assigned to the highest-scoring segments. Specifically, if a deficit of \(n\) Specifically, if a deficit of \(n\) highest-scoring segments, ensuring that the final frame count matches the target budget.

3.4 Intra-Segment Keyframe Selection↩︎

With each segment \(\mathcal{X}_s\), we anchor the most query-relevant frame and iteratively incorporate additional frames to maximize diversity while maintaining high semantic relevance.

Semantic Anchor. We first select the frame with the highest relevance to the query that is added to the keyframe set \(\mathcal{K}_s\) \[\label{eq:anchor} \mathcal{K}_s = \{\arg\max_{x_i\in\mathcal{X}_s}ITM(x_i,q) \}.\tag{10}\]

Semantic Constraint. Let \(R^*\) denote the matching score of the anchor frame. We construct a candidate set \[\label{eq:candidate} \mathcal{C}_s = \{x_j\in\mathcal{X}_s|ITM(x_j,q)\geq\gamma\cdot R^*\},\tag{11}\] in which \(\gamma\) controls the size of the candidate pool by filtering frames whose relevance is below a fraction of the anchor score. Smaller \(\gamma\) increases diversity but introduces noise, while larger \(\gamma\) improves relevance but reduces candidate diversity. We set \(\gamma=0.7\) by default.

Content Diversity. We iteratively expand \(\mathcal{K}_s\) by selecting a frame that maximizes its distance to the current keyframe set: \[\label{eq:expand} \mathcal{K}_s = \mathcal{K}_s \cup \{\arg\max_{x_i\in\mathcal{C}_s\setminus\mathcal{K}_s}\sum_{x_j\in\mathcal{K}_s}\phi(x_i,x_j) \},\tag{12}\] where \(\phi\) is a distance metric, e.g., Euclidean distance. The process repeats until \(|\mathcal{K}_s|=q_s\). Notably, \(\gamma\) can be reduced to broaden the range of the candidate set if \(|\mathcal{K}_s|+|\mathcal{C}_s|<q_s\) initially. The final keyframe set is obtained by aggregating selections from all segments: \[\label{eq:keyframe} \mathcal{K}=\bigcup_{s=1}^S\mathcal{K}_s,~|\mathcal{K}|=N'\tag{13}\]

The greedy strategy incrementally selects frames that maximize marginal information gain, allowing efficient approximation of diverse and representative keyframe sets.

4 Experiments↩︎

4.1 Evaluation Benchmarks↩︎

To evaluate the effectiveness of our proposed method, we conduct experiments on four widely used benchmarks for long video understanding.

LongVideoBench [14] highlights the referred reasoning questions that are posed on varying-length videos up to an hour long on diverse themes. These questions are dependent on long frame input and cannot be well-addressed by a single frame or a few sparse frames.

Video-MME [8] (w/o subs) spans 6 primary visual domains with 30 subfields to ensure broad scenario generalizability and encompasses both short-, medium-, and long-term videos, ranging from 11 seconds to 1 hour.

MLVU [31] is constructed from a wide variety of long videos, with lengths ranging from 3 minutes to 2 hours, and includes nine distinct evaluation tasks.

LVBench [32] comprises publicly sourced videos, including TV series, sports broadcasts, and everyday surveillance footage, and encompasses a diverse set of tasks aimed at long video comprehension and information extraction.

8pt

8pt

4.2 Implementation Details↩︎

We investigate LLaVA-Video [11], InternVL-3.5 [6] and Qwen3-VL [5] as our baselines. These MLLMs typically receive \(64\) uniformly sampled video frames, which may result in the loss of key information. For a fair comparison, we also set the number of retrained frames \(N'=64\) in QCA. Following previous research [24], we first sample the frame sequence from the raw video at \(1\) FPS before performing keyframe selection to reduce computational costs.

For the Image-Text Matching (ITM) function in Eq.@eq:eq:itm95matching and the Content Deviation term in Eq.@eq:eq:deviation , we both use BLIP-2 [30] as the default embedding model, in which the text and visual encoders receive the query and frame to measure their matching degree.

Also, we set the weighting coefficients for semantic relevance and content deviation to \(\alpha=\beta=0.5\), and use a unified softmax temperature \(\tau=0.5\) for all softmax operations. All experiments are conducted on 8\(\times\)A800 80G GPUs.

4.3 Quantitative Results↩︎

Comparison to existing keyframe selection methods. Across four long video understanding benchmarks, Tab.¿tbl:tab:main? presents a comprehensive comparison between QCA and several frame selection strategies, including uniform sampling, Top-\(k\) selection and recent keyframe selection methods, where Top-\(k\) selects keyframes with the highest Top-\(k\) image-text matching score.

Without additional training, QCA consistently achieves the best performance across representative Video-LLMs, including LLaVA-Video-7B, InternVL-3.5-8B, and Qwen3-VL-8B, demonstrating strong generalization and robustness. In particular, with Qwen3-VL-8B under a \(64\)-frame budget, QCA achieves \(66.9\%\) (\(3.8\%\uparrow\)) on LongVideoBench, \(69.5\%\) (\(1.9\%\uparrow\)) on Video-MME, \(75.7\%\) (\(4.7\%\uparrow\)) on MLVU, and \(51.8\%\) (\(8.0\%\uparrow\)) on LVBench. Although the absolute gains appear moderate, they are obtained under strict frame budgets and without additional training, making them particularly valuable for practical long video understanding scenarios.

Compared with both uniform sampling and Top-\(k\) selection, QCA consistently delivers superior results across diverse tasks requiring long-range reasoning, semantic localization, and holistic video understanding. Specifically, on LongVideoBench, which emphasizes semantically critical moments, the proposed QCA significantly outperforms uniform sampling, highlighting the limitation of uniform sampling in capturing key events. Meanwhile, on Video-MME, which requires more comprehensive scene understanding, QCA also surpasses Top-\(k\) selection. This suggests that selecting frames solely based on high ITM scores may overly focus on locally salient moments while overlooking broader contextual information.

We further compare QCA with recent keyframe selection methods, e.g., AKS [24] and Q-Frame [33]. Our approach almost outperforms these methods across all four benchmarks. Specifically, QCA outperforms AKS by \(2.1\%\) on LVBench and outperforms Q-Frame by \(2.2\%\) on Video-MME using InternVL-3.5-8B. Compared with FRAG [35] and E-VRAG [36], which rely on larger models for frame selection, QCA achieves comparable or superior performance. For instance, QCA gains a \(2.4\%\) improvement on Video-MME over FRAG and a \(3.9\%\) improvement on MLVU over E-VRAG using LLaVA-Video-7B. More surprisingly, although OneClip-RAG [34] benefits from additional training, QCA still achieves superior performance without any fine-tuning, e.g., gains a \(2.9\%\) improvement on MLVU using LLaVA-Video-7B. Importantly, the performance gains of QCA remain consistent across different Video-LLMs, indicating that the proposed method is model-agnostic and can be readily integrated into existing architectures.

Comparison to state-of-art Video-LLMs. Tab. ¿tbl:tab:sota? compares QCA with state-of-the-art Video-LLMs using Qwen3-VL-30B-A3B-Instruct and Qwen3-VL-8B as backbones. Although the Qwen3-VL series already achieve strong performance with uniform sampling of 64 frames, incorporating QCA further improves results across all benchmarks.

For example, with Qwen3-VL-30B-A3B-Instruct, QCA outperforms GPT-4o by \(12.5\%\) on MLVU and surpasses Gemini-1.5-Pro by \(19.5\%\) on LVBench, demonstrating that more informative frame selection can substantially enhance long video reasoning even for strong proprietary models.

Overall, these results indicate that QCA offers a simple yet effective approach for improving long video understanding by selecting a compact, informative, and diverse set of keyframes, without modifying model architectures or requiring additional training.

4.4 Qualitative Results↩︎

Fig.3 and Fig.4 present representative video question answering examples comparing our QCA with uniform sampling. Unlike uniform sampling, which selects a fixed set of frames regardless of the query, QCA dynamically adapts keyframe selection to the question while preserving sufficient global scene coverage.

Beyond identifying frames that are strongly aligned with the query, QCA also retains context frames that provide complementary visual information, even when their direct relevance to the query is weaker. This helps maintain holistic scene understanding under a strict frame budget and prevents errors caused by missing contextual cues. As illustrated, QCA captures key evidence such as the passer-by and salient objects while still preserving the overall scene context needed for disambiguation.

Fig.5 further visualizes the keyframe selection of QCA. The top-right plot shows the frame-query matching score, while the bottom-right plot illustrates the content deviation at the feature level. Based on the average matching score and feature deviation within each segment, QCA adaptively allocates different frame budgets across segments. The resulting keyframes jointly satisfy semantic relevance, content diversity, and temporal coverage.

4.5 Ablation Study↩︎

Matching and Deviation. In Tab.1, we first analyze the impact of semantic matching and content deviation by removing them from the full setting. Removing either component consistently degrades performance across all benchmarks, indicating that both query relevance and content representativeness are important for effective keyframe selection.

Specifically, removing the matching term (or both terms) reduces performance from \(66.9\%\) to \(65.8\%\) on LongVideoBench and from \(70.1\%\) to \(68.3\%\) on Video-MME, highlighting the dominant role of semantic relevance in query-driven video understanding. Interestingly, the \(2^{nd}\) row shows that using only \(M_s\) still achieves \(70.4\%\) on Video-MME, further confirming the importance of query-aware frame selection.

Fig.6 (middle) shows the sensitivity study in \(\alpha\), \(\beta\) and \(\gamma\), where \(\beta=1-\alpha\). We can observe that balanced relevance-diversity weighting generally provides the best trade-off. Regarding \(\gamma\), a smaller \(\gamma\) could introduce a noisy candidate set, while a larger one leads to insufficient diversity.

Anchor, Candidate, and Diversity. We further examine the anchor-centric selection strategy and the construction of the candidate set \(\mathcal{K}_s\), as shown in the bottom of Tab.1. When replacing the semantic anchor with a random frame, the candidate set is formed from the remaining frames in the segment. Performance degradation indicates that initializing the selection with the semantic anchor provides a strong semantic reference for subsequent selection.

Moreover, removing the candidate set and selecting frames from the entire segment leads to performance degradation, particularly on MLVU and LVBench. This result suggests that restricting the search space to high-relevance candidates helps suppress noisy frames and stabilize the diversity modeling process.

To further assess the role of diversity modeling, we replace the diversity-driven selection with a simple top-matching strategy. The performance drop across all benchmarks demonstrates that explicit diversity modeling is necessary to capture temporally dispersed yet semantically relevant events in long videos. In comparison, uniform selection causes substantial performance degradation, confirming that the improvements mainly stem from informed frame selection.

In summary, the components of QCA play complementary roles, e.g., semantic matching ensures query relevance, deviation modeling captures representative content, and diversity modeling improves temporal coverage.

Temporal Segmentation. The number of temporal segments \(S\) directly affects the effectiveness of the segment-level scoring. As shown in Fig. 6 (right), the performance first improves and then decreases as \(S\) increases. On LongVideoBench, the accuracy rises from \(66.0\%\) at \(S{=}8\) to a peak of \(66.9\%\) at \(S{=}16\), and then drops when \(S\) becomes larger. A similar trend is observed on MLVU.

When \(S\) is too small, each segment covers a long temporal span and may contain multiple events, causing overly averaged statistics and inaccurate frame allocation. Conversely, when \(S\) is too large, each segment contains fewer frames, making the statistics less stable. Therefore, a moderate segmentation granularity provides a better trade-off.

In addition, we explore a simple dynamic strategy in which \(S\) increases linearly with video length. However, this naive strategy did not consistently outperform using a fixed \(S\). Simply increasing the number of segments does not necessarily align with the video’s underlying semantic structure. This indicates that QCA is relatively robust to a moderate fixed \(S\).

Preprocessing Overhead. We also analyze the preprocessing overhead of QCA. As shown in Fig.6 (left), the keyframe selection process introduces only a marginal computational cost. For instance, as the number of input frames increases from 128 to 1024, the ITM matching time grows from 0.595s to 1.165s, while the selection process itself takes only about 0.005s. This overhead is substantially lower than the cost of processing dense visual tokens with the Video-LLM, demonstrating the efficiency of our approach.

3pt

VL Embeddings. Tab. 2 evaluates QCA with different Vision-Language embeddings, e.g., CLIP, BLIP, and LongCLIP, under a fixed frame budget of \(64\). QCA consistently improves performance across different VL embedding models and benchmarks, indicating that the gains mainly stem from the proposed selection strategy rather than the choice of pre-trained encoder. These results highlight the model-agnostic design of QCA and its ability to generalize across different VL embedding models.

6pt

Keyframe Budget. We further analyze the effect of different frame budgets to assess the frame efficiency of QCA. As shown in Tab.3, our method consistently outperforms uniform sampling across all benchmarks even with only 16 or 32 frames, demonstrating that QCA can significantly reduce the number of frames or tokens required by MLLMs while maintaining strong performance.

As the frame budget increases from 16 to 128, QCA shows steady performance improvements on LongVideoBench, MLVU, and LVBench. This trend indicates that QCA can effectively utilize additional frames without introducing severe temporal redundancy. Unlike uniform sampling, which often captures redundant frames, QCA prioritizes informative and diverse keyframes, leading to more efficient use of the available frame budget.

Furthermore, we compare our method with a recent token pruning approach, ForestPrune [29], under an identical effective visual token budget. As shown in Tab.4, QCA consistently outperforms ForestPrune on VideoMME and MLVU. This is likely because QCA proactively selects semantically informative frames before encoding, leading to higher information density within the fixed budget, whereas token pruning operates post-encoding and may discard tokens from less-relevant frames.

Overall, these results demonstrate a favorable trade-off between frame budget and performance, making QCA well-suited for long video understanding under strict computational constraints.

5 Conclusion↩︎

In this work, we study the problem of efficient long video understanding with MLLMs under a strict frame budget. We identify temporal redundancy as a key factor in existing Video-LLMs and propose a query- and content-aware framework that dynamically allocates frame budgets across video segments before selecting keyframes. By jointly modeling semantic relevance, content deviation, and diversity among selected frames, our method selects a compact yet informative set of keyframes for subsequent reasoning. Extensive experiments demonstrate that our approach consistently outperforms uniform sampling and recent frame selection baselines across different backbones. In particular, our method achieves strong performance even with substantially fewer frames, highlighting its superior frame efficiency and scalability. Notably, we adopt uniform temporal segmentation for simplicity and efficiency. Incorporating content-aware segmentation, e.g., shot boundary detection, is a promising direction for future work.

Acknowledgments↩︎

This work was supported by the New Generation Artificial Intelligence-National Science and Technology Major Project (No. 2025ZD0122701), the National Natural Science Foundation of China under Grant (No.U25B6003 and No.62425101), and the Shenzhen Science and Technology Program under Grant (No.KQTD2024 0729102051063).

References↩︎