1 Introduction↩︎

Multimodal foundation models for video understanding have achieved rapid progress in recent years, driven by architectural advances [8]–[12], large-scale pretraining [13]–[16], and sophisticated post-training strategies [3], [4], [17]–[19]. However, current models still face significant limitations when transitioning from surface-level video perception to video reasoning tasks that demand domain knowledge and multi-step inference [20]–[23]. A key bottleneck lies in the nature of the training corpora used to develop these models.

Existing large-scale video datasets are predominantly constructed for perceptual objectives such as action recognition, event localization, and short-range temporal understanding [3], [4], [17], [24], [25]. Their content is heavily skewed toward everyday activities, with limited coverage of specialized domains and little support for knowledge- and reasoning-intensive video understanding. Consequently, models trained on current corpora often struggle with tasks requiring multi-hop inference, scientifically grounded explanations, or interpretation of events governed by non-observable principles, limiting their reliability in real-world applications that demand accurate, domain-aware reasoning.

To bridge this gap, we open-source VideoKR, the first large-scale training corpus targeted for knowledge- and reasoning-intensive video understanding. We collect 145K CC-licensed videos across 82 professional subjects using a knowledge-driven collection protocol that targets real-world manifestations of domain knowledge. To transform these raw videos into effective video reasoning training data, we design a skill-oriented QA generation framework that decomposes knowledge- and reasoning-intensive video understanding into three complementary capabilities: basic video reasoning, knowledge-enhanced video perception, and knowledge-intensive video reasoning. For each video, the framework generates challenging QA examples tailored to each skill category, each paired with a high-quality CoT rationale. We apply rigorous quality control with human-expert involvement, yielding a high-quality supervised fine-tuning corpus, VideoKR-SFT-201K, and a reinforcement learning corpus, VideoKR-RL-114K. In addition, through a manual audit of existing knowledge-intensive video reasoning benchmarks, we find that many examples are solvable with little video understanding. To address these issues, we construct a new evaluation benchmark, VideoKR-Eval.

We adopt a standard SFT\(\rightarrow\)GRPO pipeline to isolate data design as the primary bottleneck and attribute performance gains more cleanly to VideoKR. We further establish a standardized evaluation framework to enable fair, reproducible model comparisons. Experiments show that even without sophisticated post-training algorithmic design, base models (i.e., Qwen2.5-VL-7B-Instruct and Qwen3-VL-8B-Instruct) post-trained on VideoKR already outperform prior post-training approaches. To inform how VideoKR can advance future work in video reasoning, we conduct comprehensive ablations that disentangle its key contributors, including the effect of CoT supervision, the impact of skill-based data composition, and controlled SFT and RL studies that compare VideoKR against prior post-training corpora.

We summarize our main contributions as follows:

• We open-source VideoKR, the first large-scale training corpus designed for knowledge- and reasoning-intensive video understanding. We apply rigorous quality control to ensure consistently high-quality training data (§3).

• We construct VideoKR-Eval, a new evaluation benchmark that mitigates single-frame answerability in prior benchmarks through multi-model single-frame probing and expert re-annotation of filtered videos (§4).

• We establish a standardized evaluation framework to ensure fair and reproducible model comparisons (§5.2).

• Models post-trained on VideoKR achieve the best knowledge-intensive performance among similar-sized models, while remaining competitive on general video benchmarks (§6).

• We conduct comprehensive ablations to isolate the contributions of VideoKR, including the effectiveness of CoT supervision, the impact of skill-based data composition, comparisons against prior post-training corpora, yielding actionable insights for future work (§6.4 to §6.5).

2 Related Work↩︎

2.0.0.1 Video Understanding Datasets.

Recent video understanding benchmarks have expanded their scope to evaluate a broader range of multimodal and reasoning capabilities [20], [22], [26]–[31]. General-purpose benchmarks such as Video-MME [28], MVBench [32], VSI-Bench [33], and VideoVista [29] assess perceptual skills, spatiotemporal comprehension, and cross-modal reasoning, providing a solid foundation for evaluating video understanding. Building on this trend, a growing set of knowledge-, science-, and reasoning-intensive evaluation benchmarks focuses on deeper, domain-aware reasoning that goes beyond surface-level video perception [30], [31]. For instance, MMVU [20] requires models to reason over specialized-domain videos and apply relevant domain knowledge; VideoMMMU [21] and Video-MMLU [23] target expert-level understanding of subject-specific lecture videos; and SciVideoBench [22] evaluates advanced reasoning over scientific videos.

2.0.0.2 Post-training for Video Understanding.

Current reasoning models are typically trained through a two-stage post-training pipeline that combines SFT and RL [34]–[37]. To enhance video reasoning capabilities, the SFT stage is usually initialized on video reasoning datasets that include explicit chain-of-thought annotations, temporal cues, and spatial grounding signals, helping the model form more structured and interpretable reasoning patterns [6], [38]–[41]. In the RL phase, recent work has concentrated on adapting reinforcement learning with verifiable rewards (RLVR) to video reasoning, exploring complex reward engineering that emphasizes spatial understanding [42], [43], temporal dynamics [3], [24], [44], [45], or integrated spatiotemporal relationships [17], [46]. Despite these advances, most post-training approaches still build on repurposed video understanding datasets that target basic perception: as shown in 1, existing open-source corpora largely rely on short videos from datasets released years ago, and synthesis-based efforts [1], [2] typically depend on a single model, which can introduce systematic biases. We address this need by open-sourcing VideoKR: every video is newly collected, depicts expert-domain scenarios, and is released under CC licenses, and we adopt a human-in-the-loop, skill-oriented example generation framework to ensure the difficulty, diversity, and reliability of the data. Our experiments show that, under a standard SFT\(\rightarrow\)GRPO pipeline, models post-trained on VideoKR already outperform prior post-training approaches, suggesting that better data remains crucial.

3 VideoKR Training Corpus Construction↩︎

This section describes the VideoKR data construction process, with an overview of the pipeline shown in 2. Because our goal is large-scale corpus synthesis, exhaustive manual construction is infeasible. However, model based generation can introduce systematic artifacts. We therefore adopt a quality-controlled, semi-automated pipeline: whenever a step involves model outputs, it is audited and validated by human experts (detailed in Section 3.4). We engage 34 domain experts, each with graduate-level background in the relevant discipline (see Appendix 8.2 for annotator biographies), to enforce quality criteria throughout the process.

3.1 Domain Knowledge Bank Construction↩︎

To achieve comprehensive coverage of domain-related videos, we begin by constructing a Domain Knowledge Bank, where each entry represents a knowledge point consisting of a term and its corresponding definition. The authors manually reviewed undergraduate curricula from top universities worldwide and identified 82 representative subjects distributed across four major disciplines: Natural Sciences, Healthcare, Humanities and Social Sciences, and Engineering. The complete subject list is provided in Appendix 8.1. To ensure systematic and fine-grained representation of domain knowledge within each discipline, we adopt a hierarchical knowledge organization framework with four layers: Subject \(\rightarrow\) Course \(\rightarrow\) Lecture \(\rightarrow\) Knowledge Point. Specifically, for each subject, we ask expert annotators to provide a list of 4 to 8 core undergraduate courses consistent with standard academic programs. For each selected course, annotators compile a structured syllabus based on well-established curricula from top universities, outlining major lecture topics and learning objectives. Then for each lecture, we prompt LLMs (see Section 3.4 for our expert-involved procedure to select and validate the models used here and in subsequent steps) to identify several key knowledge points, each paired with a term and a paragraph-length definition. We collect a total of 63,745 knowledge points.

3.2 Knowledge-Driven Video Collection↩︎

Building upon the constructed domain knowledge bank, we then collect a large-scale video corpus, outlined below.

3.2.0.1 Knowledge-based Video Scenario Generation.

In practice, directly using knowledge point terms (e.g.,”Newton’s Second Law”) as search queries often yields lecture recordings or purely instructional materials. While informative, such videos often lack the diversity and situational richness found in real-world contexts where the knowledge is implicitly applied. To obtain more authentic and engaging content, we first ask LLMs to generate 1–3 short scenarios that describe realistic situations involving each knowledge point. For example, instead of searching “Newton’s Second Law”, a generated scenario might be “a rocket launching into the sky”, which inherently reflects Newton’s Second Law. These scenarios are then transformed into semantically relevant search keywords, helping retrieve videos that embody the knowledge rather than merely explain it.

3.2.0.2 Scenario-Guided Video Search and Filtering.

For each generated search keyword, we employ the YouTube Data API¹ to retrieve metadata (e.g., titles, descriptions, and durations) for the top-\(10\) candidate videos. We restrict the search to videos released under CC licenses to ensure legal reusability, a critical aspect that has been ambiguous in prior training corpora. For each candidate video, we instruct models to evaluate its relevance to the expected knowledge point and scenario based on the textual metadata. Videos exceeding 30 minutes are excluded, as long-context video understanding falls beyond the scope of this work. For the remaining candidates, we download the videos and prompt MLLMs to perform a secondary relevance assessment using visual content to confirm alignment with the intended knowledge context. To remove potentially harmful or sensitive content, we randomly sample four frames from each video and run Azure AI’s image moderation APIs² to filter out unsafe videos. We collect 146,567 CC-licensed videos.

3.3 Skill-Oriented Example Generation↩︎

For each video, we generate multiple QA examples. Following recent video post-training work [3], [17], we adopt multi-choice and open-ended QA formats, as they offer verifiable supervision suitable for RLVR. To enable scalable and high-quality data creation, we design a skill-based example generation pipeline:

3.3.0.1 Core Skill Categorization.

As illustrated in 1, three complementary dimensions, i.e.,perception, knowledge, and reasoning, are essential for knowledge- and reasoning-intensive video understanding. Accordingly, we define three core skills that guide the example generation process: (1) Basic Video Reasoning (VidR), which involves direct comprehension of events observable from the visual sequence, such as tracking actions, spatial relations, or temporal order, without relying on external domain knowledge. (2) Knowledge-enhanced Video Perception (KnowVid), where visual perception is enriched by explicit domain knowledge. The model must align observed visual cues with relevant concepts across both spatial and temporal dimensions, for example, recognizing laboratory apparatus such as a “burette” or “condenser” and understanding their roles in a sequence of chemical procedures. (3) Knowledge-Intensive Video Reasoning (KnowVidR), which focuses on integrating visual understanding with domain knowledge to perform sophisticated, multi-hop inference, e.g., estimating the quantity of chemical product formed from observed reactant amounts, or inferring a patient’s likely diagnosis by interpreting symptoms and medical procedures depicted in a clinical video.

3.3.0.2 Seed Examples Curation By Human Experts.

To ensure the quality and domain accuracy of the generated examples, we engage expert annotators to curate a seed set of examples for every core skill defined above. For each core skill, the annotators select representative knowledge points and their corresponding collected videos, then construct question–answer pairs accompanied by detailed, step-by-step reasoning processes that clearly articulate how visual evidence, domain knowledge, and logical inference jointly lead to the final answer. Each annotated example then undergoes a manual review by the authors to verify the accuracy of the QA content and reasoning process. In total, 150 examples are created for each skill within every discipline, resulting in 1,800 high-quality, expert-curated seed examples. To improve reliability, each example is independently reviewed by a second annotator; 74 examples are revised in this stage.

3.3.0.3 Example Generation.

Building on the expert-curated seed set, we use frontier MLLMs to scale up example creation in a controlled, skill-aware manner. For each video, the model generates two examples per skill, producing six examples in total through six independent generation rounds (one per example). During each round, the model is provided with (1) video frames uniformly sampled at 0.2 fps with timestamps, (2) three randomly sampled human-curated examples from the same discipline and skill category, and (3) the knowledge point and associated subjects when targeting the KnowVid or KnowVidR skills. The model is instructed to emulate the seed examples in QA formulation and reasoning process generation, while maintaining fidelity to the video’s unique visual content and knowledge context.

3.3.0.4 Example Validation and Filtering.

To mitigate generation errors and reasoning bias, we adopt three complementary strategies: (1) Self-Consistency Verification: The model is re-prompted with the generated question and corresponding video frames to produce a detailed, step-by-step answer. An example is retained only if the re-derived answer matches the original, and the reasoning process from this verification step is used as the final reasoning trace. (2) Video Dependency Filtering: To ensure that each example in VideoKR genuinely requires visual understanding rather than relying on textual cues or shortcut reasoning, we instruct InternVL3.5-38B and Qwen3-VL-32B-Instruct to answer the question using only the text and four randomly sampled video frames. If both models successfully predict the correct answer under this limited setting, the example is removed from the dataset. Notably, this filter is stricter than what existing evaluation benchmarks use, which commonly rely on text-only [28], [47] or single-frame [48], [49] settings. (3) CoT Rationale Validation: To mitigate systematic generator bias in the reasoning traces, we use an independent strong MLLM as a verifier: given the question, reasoning trace, and videos, it checks that each key step is supported by observable evidence or standard domain knowledge, and that the reasoning decisively distinguishes the chosen answer from plausible alternatives. We discard examples with critical unsupported steps.

3.4 VideoKR Data Quality Control↩︎

Beyond the automated example validation and filtering described above, we further strengthen VideoKR through pipeline-level quality control and contamination mitigation.

3.4.0.1 Human-Validated Model Selection for Each Pipeline Step.

As summarized in 1, prior work on video reasoning corpus construction typically relies on a single model throughout the pipeline. This design can introduce model specific artifacts. To improve synthesis diversity and avoid overcommitting to any single model’s biases, we use a pool of seven frontier models (i.e., GPT-5.2, GPT-5-mini, Claude-4.5-Sonnet, Gemini-3-Flash, DeepSeek-V3.2, Qwen3-VL-235B-A22B, and GLM-4.6V). However, our pipeline is difficulty stratified: lightweight steps such as metadata relevance screening can be handled reliably by multiple models, whereas demanding stages such as QA example generation and verification are reliable only for a subset of models. We therefore introduce a human validated model selection protocol to determine stage eligibility: For each candidate model and each pipeline step, we sample 100 instances from that step’s real inputs and ask domain expert annotators to assess the model outputs and label errors. A model is eligible for a step only if its error rate falls below a predefined threshold. We provide details of this process and the human-validated models for each step in Appendix 8.4. During large-scale synthesis, for each instance at a given step, we randomly select one qualified model from the pool.

3.4.0.2 Data Contamination Mitigation.

To prevent evaluation leakage, we traverse all video benchmarks supported by LMMs-Eval [50] and apply a two-stage decontamination protocol over the videos in VideoKR: (1) YouTube-ID Filtering: When a benchmark (e.g., MMVU) provides YouTube video IDs, we directly filter out any training video whose YouTube ID matches an evaluation video ID, resulting in 131 videos removed. (2) Near-Duplicate Video Filtering: We also perform duplicate detection using frame level perceptual hashing and windowed sequence matching, resulting in 877 videos removed. We detail the process in Appendix 8.5.

3.4.0.3 Manual Quality Assessment.

To further assess the end-to-end quality of the finalized corpus and quantify residual noise, we randomly sample 800 examples from VideoKR-SFT-201K and ask ten expert annotators, who previously participated in seed-example curation, to evaluate them end to end. Of the sampled items, 52 questions are flagged as potentially non visual solvable. For reasoning traces, annotators identify 32 errors. Among these, 17 cases change the final answer, while 15 cases preserve the correct answer but rely on unsupported domain claims or fail to ground key steps in the relevant video evidence. These error rates are comparable to the error levels observed during human expert seed example curation (§3.3) and are therefore acceptable.

3.5 VideoKR-SFT-201K& VideoKR-RL-114K↩︎

We then randomly partition the generated 315,537 examples into two subsets while preserving video-level grouping, resulting in VideoKR-SFT-201K for supervised fine-tuning and VideoKR-RL-114K for RLVR training. For VideoKR-SFT-201K, each example retains its validated CoT rationale as the supervision target, whereas VideoKR-RL-114K keeps only the question and verifiable answer, since RLVR optimizes against the verifiable answer while the policy model generates its own reasoning during training. 2 presents the key data statistics. Randomly-sampled examples from VideoKR are shown in Appendix 8.3.

4 VideoKR-Eval Evaluation Benchmark↩︎

We next discuss the motivation and build of VideoKR-Eval.

4.1 Limitations of Existing Benchmarks↩︎

2pt

We observe that existing knowledge-intensive video reasoning benchmarks (e.g., VideoMMMU, MMVU, and SciVideoBench) contain a substantial fraction of examples that can be answered without continuous video understanding. We quantify this issue through single-frame probing: each model is given only the question, answer options, and one randomly sampled frame from the video, and each example is evaluated in three independent trials. As shown in 2, frontier models achieve surprisingly high single-frame answerability rates on existing benchmarks (e.g.,\(>\)​35% on MMVU and VideoMMMU).

4.2 VideoKR-Eval Benchmark Construction↩︎

We construct VideoKR-Eval from VideoMMMU, MMVU, and SciVideoBench by retaining 1,254 original examples that require continuous video understanding under multi-model single-frame probing, and augmenting them with 746 expert-reannotated examples from filtered videos.

4.2.0.1 Multi-Model Single-Frame Filtering.

For each example in VideoMMMU, MMVU, and SciVideoBench, we run single-frame probing with three frontier models: Qwen3-VL-235B-A22B, Claude-4.5-Sonnet, and GPT-5.2. Each model receives only the question, answer options, and one randomly sampled frame from the video, and is evaluated with three independent trials. For each model, an example is considered single-frame-solvable if the model answers it correctly in all three trials; otherwise, it is treated as requiring continuous video understanding for that model. We retain only the intersection of examples judged as requiring continuous video understanding by all three models, yielding 1,254 original examples.

4.2.0.2 Expert Re-annotation of Filtered Videos.

For examples outside this intersection, we discard the original QA pairs and ask domain experts to re-annotate new QA examples using the corresponding videos. Annotators are required to write questions grounded in clearly observable video evidence, requiring relevant domain knowledge, and paired with uniquely determined ground truth answers. This process yields 746 expert-reannotated examples. Together with the 1,254 retained original examples, VideoKR-Eval contains 2,000 examples. Detailed statistics for VideoKR-Eval are provided in Appendix 9.1.

5 Experiment Setup↩︎

In this section, we discuss the experiment setup for post-training on VideoKR and the subsequent model evaluation.

3.2pt

5.1 Post-Training on VideoKR↩︎

Recent post-training work for video reasoning emphasizes sophisticated RL variants and reward engineering. In contrast, we aim to isolate a different, and arguably more fundamental, bottleneck: whether data design is the primary limiting factor for knowledge and reasoning intensive video understanding. Accordingly, we deliberately adopt a standard, widely used SFT\(\rightarrow\)GRPO pipeline as a controlled scaffold, ensuring that algorithmic complexity does not become a confounder and that observed gains can be attributed more cleanly to the training data.

Specifically, we use Qwen2.5-VL-7B-Instruct and Qwen3-VL-8B-Instruct as base models to assess whether VideoKR yields consistent gains under various architectural designs and pretraining priors. For SFT, we fine-tune each base model on VideoKR-SFT-201K for one epoch. Starting from the resulting SFT checkpoint, we then run GRPO on VideoKR-RL-114K for one epoch. For Qwen3-VL-8B-Instruct, we also conduct Zero-RL training by directly running GRPO on VideoKR-RL-114K for one epoch. We set batch size as 32 for both SFT and GRPO. For the GRPO accuracy reward, we follow prior video reasoning work [3], [4] and use ROUGE for open-ended QAs and Exact Match for multiple-choice QAs. The maximum video token number is 4,096, and the maximum number of frames is 128. The training hyperparameters and details are provided in Appendix 10.1.

5.2 Evaluation Setup↩︎

We next describe our evaluation benchmarks and the standardized protocol for fair, reproducible model comparisons.

5.2.0.1 Evaluation Benchmarks.

We evaluate models on seven benchmarks grouped into two categories: (1) General Video Reasoning, including Video-MME [28], MVBench [32] and LongVideoBench [51], which measure broad video understanding; and (2) Knowledge-intensive Video Reasoning, including VideoMMMU [21], MMVU [20], SciVideoBench [22], and VideoKR-Eval, which focus on domain-specific, expert-level reasoning.

5.2.0.2 Reproducibility Challenges in Prior Post-Training Work.

We observe substantial cross-paper inconsistencies in reported results, particularly for base models used as post-training starting points. Based on careful follow-up experiments, we attribute these discrepancies primarily to prompt misalignment: base models are sometimes evaluated under prompt conditions that are misaligned with their intended inference mode. For example, Qwen2.5-VL-Instruct is not a “reasoning” model, yet some papers evaluate it with elaborate self-reflection and forced reasoning-trace instructions designed for post-trained ‘reasoning’’ variants.

5.2.0.3 Standardizing Evaluation for Fair Model Comparisons.

To ensure fair and reproducible comparisons, for each model, we use the official prompt released by the original paper whenever available; otherwise, we adopt the default prompt templates from LMMs-Eval [50]. For input frames, we follow the model-recommended inference configuration whenever it is specified by the model release. We run each model three times with independent sampling and report the mean. All evaluations are performed using the LMMs-Eval framework [50]. Full evaluation details (e.g., model prompts and inference parameters) are provided in Appendix 10.2.

6 Experiment Results and Analysis↩︎

We next discuss our main findings and ablation analysis.

6.1 Main Results↩︎

Table 3 presents the main experimental results. Post-training on VideoKR consistently improves Qwen2.5-VL-7B-Instruct across all evaluated benchmarks, and yields clear gains for Qwen3-VL-8B-Instruct especially on knowledge-intensive video reasoning benchmarks. The gains are most pronounced on knowledge-intensive tasks: SFT followed by RL on VideoKR raises the knowledge-intensive average of Qwen2.5-VL-7B from 41.9 to 46.6 (+4.7) and of Qwen3-VL-8B from 48.5 to 51.5 (+3.0), with the largest per-dataset improvements on MMVU and VideoKR-Eval(e.g.,+4.8 and +8.5 points for Qwen2.5-VL-7B). Notably, the post-trained Qwen3-VL-8B attains the best knowledge-intensive average among 7/8B-scale models (51.5, versus 50.0 for the strongest competing model, Qwen3-VL-8B-Thinking). These results underscore the value of our training data, which tightly integrates domain knowledge, visual grounding, and structured reasoning. Starting from an SFT-initialized model, subsequent RL training consistently delivers higher performance than the SFT-only baseline, highlighting that combining SFT with RL is important for fully leveraging the strengths of VideoKR data. Moreover, the RL-only variant also generally outperforms the SFT-only model, indicating that RL alone can induce stronger generalizable reasoning abilities.

6.2 Case Study↩︎

To better understand how post-training on VideoKR changes model behavior, we randomly sample 100 examples from VideoKR-Eval and compare outputs from different models. Our case study shows that the post-trained Qwen3-VL-8B model can integrate visual evidence with relevant domain knowledge to perform complex reasoning. It also exhibits “aha-moment” reasoning patterns indicative of deeper understanding. Examples are shown in Appendix 11.3.

6.3 Analysis of Inference-Time Frame Scaling↩︎

To analyze how input frame count affects performance, we evaluate our post-trained model, which was trained with 128 frames, under varying numbers of frames at inference. Specifically, we test 16, 32, 64, and 128 frames while keeping all other inference settings fixed. As shown in 3, increasing the number of input frames consistently improves performance for both the base model and our post-trained model. For example, on general video reasoning benchmarks, Qwen2.5-VL-7B (SFT+RL) improves from 60.1% at 16 frames to 65.5% at 128 frames. On knowledge-intensive video reasoning benchmarks, it further improves from 44.2% to 46.6%. These results suggest that our model benefits from richer visual and temporal evidence at inference time, and that the gains from VideoKR remain consistent across different frame budgets.

In the following subsections, we conduct controlled SFT and RL ablations on VideoKR to provide insights for future work. Unless otherwise specified, all experiments use Qwen2.5-VL-7B-Instruct as the base model and are trained and evaluated with 128 input frames.

6.4 Ablations on VideoKR-SFT-201K↩︎

Using VideoKR-SFT-201K, we ablate two design choices, the skill composition of the training examples and the use of CoT supervision, to quantify their individual contributions.

6.4.0.1 Skill-Oriented Data Composition.

To assess the contribution of each skill component, we fine-tune models on cumulative subsets of our skill-oriented data. Specifically, we construct three 80K-example variants from VideoKR-SFT-201K: (1) VidR only; (2) a balanced mixture of VidR and KnowVid(1:1); and (3) a randomly sampled subset from the full VideoKR-SFT-201K(i.e.,VidR+KnowVid+KnowVidR, we reuse examples in the previous ablation). We fine-tune Qwen2.5-VL-7B-Instruct on each variant for one epoch with a batch size of 16. We observe that incorporating all three skill components yields the best knowledge-intensive performance: training on VidR alone obtains 41.4% on knowledge-intensive benchmarks. Adding Knowledge-Enhanced Perception (VidR+KnowVid) gives 41.3%, while incorporating Knowledge-Intensive Reasoning (VidR+KnowVid+KnowVidR) further improves performance to 42.4%. The same trend holds on VideoKR-Eval, where accuracy rises monotonically from 35.3 (VidR) to 35.9 (VidR+KnowVid) to 36.8 (VidR+KnowVid+KnowVidR). These results indicate that combining domain knowledge and complex reasoning supervision is crucial.

4pt

6.4.0.2 CoT vs. Direct Output.

To validate the necessity of explicit CoT rationales in SFT, we randomly sample 80K examples from VideoKR-SFT-201K and create two variants: one with CoT rationales and one without. We fine-tune Qwen2.5-VL-7B-Instruct on each variant for one epoch with a batch size of 16. As illustrated in 4, the CoT-trained model improves over the Direct Output baseline on knowledge-intensive reasoning, lifting the average from 39.4% to 42.4% (a 3.0-point gain), underscoring the importance of high-quality CoT supervision for advanced knowledge-intensive video reasoning.

6.5 VideoKR vs Prior Post-Training Corpus↩︎

We next conduct a comprehensive comparison of VideoKR with prior open-source post-training corpora, under both SFT and zero-RL settings. For SFT, we randomly sample 80K examples from each SFT corpus and fine-tune Qwen2.5-VL-7B-Instruct on each variant for one epoch with a batch size of 16. For zero-RL, we randomly sample 50K QA examples from each RL corpus and train Qwen2.5-VL-7B-Instruct on each variant for one epoch with a batch size of 16; we exclude non-QA tasks since they require task-specific reward functions that cannot be unified under our standard GRPO training pipeline.

6.5.0.1 Main Findings.

As illustrated in 4, under SFT, the model trained on the VideoKR-SFT subset reaches a 42.4 knowledge-intensive average and is the only corpus to surpass the base model (41.9), whereas prior corpora such as Video-R1 and VideoRFT lower it to 36.2 and 38.4, respectively. Under zero-RL, VideoKR with VideoKR-RL-114K achieves the strongest gain (43.0, +1.1 over the base model, ahead of the next-best VideoAuto-R1 at 42.7), indicating that high-quality data is key to maximizing post-training benefits for advanced video reasoning.

6.5.0.2 Training-Data Difficulty Analysis.

To diagnose why prior corpora provide limited improvements relative to VideoKR, we analyze training-data difficulty with respect to the base models. Concretely, we randomly sample 3,000 video QA examples from each corpus and measure the zero-shot accuracy of Qwen2.5-VL-7B and Qwen3-VL-8B in the 128-frame setting. As shown in 5, all evaluated models attain high accuracy on prior corpora (e.g., Qwen3-VL-8B scores between 49.1% and 57.1%), suggesting these datasets are effectively saturated for current frontier base models and thus offer weak learning signals. In contrast, accuracy on VideoKR remains lower (42.3% for the same model), indicating a more challenging distribution that better supports continued capability gains during post-training.

7 Conclusion↩︎

This work offers a corpus-centric perspective on post-training foundation models for advanced video reasoning. Instead of viewing visual perception, domain knowledge, and advanced reasoning as loosely linked elements, we show that integrating structured domain concepts with visually grounded examples yields stronger reasoning performance, without relying on sophisticated RL reward engineering. Extensive experiments and analyses confirm that post-training on the VideoKR dataset produces strong improvements, especially on knowledge-intensive video reasoning.

Impact Statement↩︎

All videos in VideoKR are licensed under CC license, which enables reusers to distribute, remix, adapt, and build upon the material in any medium or format, so long as attribution is given to the creator. By restricting our dataset to CC-licensed videos, we ensure legal reusability and clear provenance, a critical aspect that has been ambiguous in prior training corpora. For human expert annotation and validation, we compensated annotators at an average rate of $13 USD per hour, which exceeds the prevailing hourly rates for comparable local work. All expert annotators provided informed consent to participate and explicitly authorized the public release and redistribution of their annotations as part of the resulting dataset and accompanying materials. The VideoKR and VideoKR-Eval data construction incurs approximately 70.4K US dollars in model inference costs.

Acknowledgments↩︎

We thank Dr. Yu Rong for valuable suggestions on the post-training experiment design and paper writeup improvement.

8 VideoKR Data Construction↩︎

8.1 Domain Knowledge Bank Construction↩︎

6pt

Based on a manual review of undergraduate curricula from leading universities worldwide, we identified 82 representative subjects spanning four major disciplines. Table 6 organizes these subjects across Natural Sciences, Healthcare, Humanities and Social Sciences, and Engineering, forming the top-level index of our four-layer knowledge base of subject, course, lecture, and knowledge point, and enabling broad cross-domain coverage and balanced sampling.

8.2 Annotator Information↩︎

3.5pt

8.3 VideoKR Data Example↩︎

8.4 Human-Validated Model Selection Protocol↩︎

To ensure that the VideoKR corpus is constructed with the highest possible quality while maximizing data diversity, we implemented a rigorous, human-in-the-loop qualification protocol for all foundation models used in our pipeline. Instead of relying on a single model (which risks imprinting specific model biases), we maintain a dynamic pool of eligible models for each synthesis stage. This section details the qualification methodology and the resulting model assignments.

We evaluate seven frontier models for potential inclusion in each pipeline step: GPT-5.2, GPT-5-mini, Claude-4.5-Sonnet, Gemini-3-Flash, DeepSeek-V3.2, Qwen3-VL-235B-A22B, and GLM-4.6V. For every pipeline stage defined in §3.1–§3.4, we conducted a controlled pilot study by sampling 100 representative input instances. Domain experts evaluated the model outputs against strict criteria, distinguishing between hard compliance failures (e.g., JSON format violations) and soft content failures (e.g., hallucinations or weak reasoning). A model was deemed eligible for a specific stage only if its total error rate \(\leq\) 3%. 8 shows the human-validated models for each pipeline step.

8.5 Data Contamination Mitigation↩︎

For Near-Duplicate Video Filtering, we uniformly sample both benchmark and VideoKR videos at \(1\) fps using ffmpeg and compute \(64\)-bit perceptual hashes per frame. We partition each training video into overlapping \(20\)-second windows with a \(1\)-second stride and build an index over window-level hash sequences to enable scalable retrieval. For each benchmark video, we retrieve the top-\(10\) candidate training windows and verify them by aligned-frame Hamming distance; we flag an overlap when the best \(20\)-second window has at least \(70\%\) of frames with distance \(\le 30\), and remove any matched training video.

9 VideoKR-Eval Benchmark↩︎

9.1 Detailed Statistics of VideoKR-Eval↩︎

As shown in 9, we construct VideoKR-Eval from three source benchmarks: VideoMMMU, MMVU, and SciVideoBench. We first perform multi-model single-frame probing and retain only original examples that are judged to require continuous video understanding by all three probing models. This yields 1,254 retained original examples. For the remaining filtered videos, domain experts re-annotate new visually grounded QA examples, contributing 746 additional examples. The final benchmark comprises 2,000 high-quality examples designed to require genuine video-level understanding and knowledge-intensive reasoning.

10 Experiment Setup↩︎

10.1 Post-Training Details↩︎

10.1.0.1 GRPO Reward Design.

We employ GRPO [52] as our reinforcement learning algorithm. Following the standard RLVR-style reward formulation, the total reward is defined as \(R = 0.1 \cdot R_f + 0.9 \cdot R_a\), where \(R_f\) and \(R_a\) denote the format and accuracy rewards, respectively. Specifically, \(R_f\) is set to \(1.0\) if the model output strictly satisfies the required format: <think>…</think><answer>…</answer>. For the accuracy reward \(R_a\), we adopt the ROUGE metric for open-ended QA, while employing Exact Match (EM) for multiple-choice tasks.

10.1.0.2 Training Details.

We train all models on up to 8 NVIDIA A800 GPUs (80 GB). For SFT, we use a learning rate of \(1\times10^{-5}\), while for RL we use a learning rate of \(5\times10^{-6}\). Both stages are optimized with AdamW, and the maximum response length is set to 2,048 tokens. For GRPO rollout generation, we set the rollout size \(G\) to 8 and use a temperature of 1.0 to encourage exploration. The KL penalty coefficient \(\beta\) is set to 0.01. Supervised fine-tuning is implemented with LLaMA-Factory [53], while reinforcement learning is implemented with verl [54].

10.2 Evaluation Setup↩︎

To ensure fair and reproducible comparisons, we standardize the inference configuration for all evaluations by setting the temperature to \(0.1\). The maximum response token is set to 8,192 tokens.

11 Experiment Results and Analysis↩︎

11.1 Performance with Different Frames↩︎

3.5pt

11.2 Ablation Studies↩︎

3.5pt

11.3 Case Study↩︎

References↩︎