1 Introduction↩︎

Large multimodal models (LMMs) have advanced video understanding toward omni-modal reasoning by jointly processing visual, audio, and textual signals [1]–[5]. Among related tasks, omni-modal video captioning remains fundamental: an ideal caption must recognize visual events, speech, sound effects, and music, and describe how they co-evolve over time [6], [7].

Despite recent progress, current models suffer from an illusion of integration (Figure 1), with two bottlenecks. First, modality isolation: models process audio and visual streams as weakly coupled channels. They may mention a visual action and a co-occurring sound but fail to express their temporal relations (e.g., “while”, “as”), yielding information-rich yet relation-poor captions. Second, speech-centric bias: acoustic representations are dominated by automatic speech recognition (ASR), leaving non-speech sounds (e.g., collisions, ambient effects, music) under-specified.

Current benchmarks also struggle to evaluate these issues [8]–[10]. Existing protocols often reward independent recognition: models can score highly by outputting visual and audio descriptions without demonstrating cross-modal synergy. Acoustic evaluation also mainly targets ASR correctness, largely ignoring background sound effects and music.

To address these gaps, we present AVSCap, a framework for explicit cross-modal event binding. First, we construct AVSCap-130K, a tri-modal corpus generated via a decoupled-then-fused pipeline that anchors unimodal evidence before composing audio-visual captions. Second, we train AVSCap-7B (based on Qwen2.5-Omni) by SFT and Group Relative Policy Optimization (GRPO) [11]. Guided by hybrid rewards, GRPO optimizes acoustic completeness and event binding. Finally, we introduce AVSCapBench, a human-curated benchmark decomposing captions into visual, audio, and synergistic events to evaluate audio sub-types and cross-modal synergy.

Our main contributions are: AVSCap-130K. A tri-modal corpus of 130K orchestrated captions, providing explicit supervision for isolated perception and cross-modal grounding. AVSCapBench. A human-annotated benchmark (1,226 videos) featuring a fine-grained, event-based matching protocol. It explicitly evaluates visual, audio, and synergistic events, preventing models from achieving high scores via modality isolation. AVSCap-7B. A 7B captioner integrating SFT and GRPO to explicitly optimize acoustic completeness (including sound effects and music) and event binding.

2 Related Work↩︎

Audio-Visual Captioning. The emergence of omni-modal models [1], [12], [13] has shifted video understanding from vision-centric perception to joint audio-visual modeling [7]. Representative works include UGC-VideoCaptioner [10] and video-SALMONN-2 [14] for multimodal integration, AVoCaDO [7] for audiovisual temporal coherence, Omni-Captioner [15] and ASID-Captioner [6] for detailed perception, TimeChat-Captioner [16] for structured multi-scene scripting, and OmniScript [17] for hierarchical script generation. While these methods improve audio-visual captioning, they do not center data construction, training, and evaluation around explicit event-level audio-visual binding. In contrast, AVSCap optimizes cross-modal synergy through a decoupled-then-fused data engine that anchors unimodal evidence before composing audio-visual captions.

RL for Video Captioning. RL [18]–[21] has become an important paradigm for aligning multimodal models with explicit objectives. CapRL [22] introduces verifiable rewards for caption generation, and VideoCap-R1 [23] uses GRPO to elicit structured thinking before captioning. AVoCaDO extends GRPO with content coverage and length regularization rewards, while TimeChat-Captioner trains its policy with task-specific rewards to align dense captions with fine-grained temporal boundaries. We adopt a regex-guided GRPO strategy with hybrid rewards to jointly optimize acoustic completeness and cross-modal event binding.

3 AVSCap↩︎

3.1 Data Construction Pipeline↩︎

3.1.1 Task Definition↩︎

Existing video captioning tasks mainly emphasize isolated visual or audio accuracy. We instead define a high-quality omni-modal caption by three criteria: (1) Acoustic Completeness, covering speech (Speech), sound effects (SFX), and music; (2) Visual Completeness, describing environments, characters, actions, object interactions, camera motion, and OCR; and (3) Audio-Visual Synergy, binding audio and visual events through coherent cross-modal relations, e.g., linking “a hammer falls” with “a striking sound (SFX)”.

3.1.2 Orchestrated Data Engine↩︎

To train AVSCap-7B, we construct AVSCap-130K, a 40K-video corpus with temporally grounded omni-modal captions. Videos are collected from AVoCaDO-107K [7], ASID-1M [6], FineVideo [24], TimeChatCap-40K [16], and Movie101 [25], and filtered to clips shorter than 2.5 minutes. As shown in Figure 2, our decoupled-then-fused data engine has three stages: unimodal anchoring, cross-modal orchestration, and automated verification. Prompts used in this section are provided in Appendix 9.2.

3.1.2.1 Step 1: Decoupled Unimodal Anchoring

To reduce cross-modal interference during initial perception, we adopt a decoupled unimodal anchoring strategy using Gemini-3-Flash [26]. For the visual stream, the parser extracts scene-level attributes, including environments, characters, actions, object interactions, camera motion, and OCR text. When an audio cue is detected, the parser inserts an empty placeholder to indicate that the current visual segment has corresponding audio information, without describing the acoustic content itself. This preserves the visual-only nature of the caption while retaining temporal anchors for later cross-modal fusion.

For the audio stream, the parser separately extracts three acoustic categories: human speech (Speech), sound effects (SFX), and background music (Music). To standardize acoustic descriptions, SFX captions follow the annotation style of AudioCaps [27], while music captions follow the descriptive format of MusicCaps [28].

3.1.2.2 Step 2: Cross-modal Fusion

Given the unimodal anchors, we use Gemini-3-Flash to synthesize a coherent omni-modal caption. The fusion module aligns each visual placeholder with its corresponding audio description and inserts the audio content at the matched temporal position. To make cross-modal relations explicit, the fused caption uses temporal conjunctions such as “As”, “While”, “Simultaneously”, and “Accompanied by”. The original audio tags (Speech), (SFX), and (Music) are retained for later verification.

3.1.2.3 Step 3: Automated Verification

To reduce hallucinations, duplication, and information loss during fusion, we apply a three-stage verification pipeline combining deterministic tag checks with local semantic validation.

Audio Tag Preservation. Since audio captions are chronologically ordered, we assign each audio event a unique tag ID before fusion, such as (Speech-1), (SFX-1), and (Music-1). After fusion, we verify that every source tag ID appears exactly once and that the retained tag count matches the number of source audio events for each type. Samples with missing, duplicated, or mismatched tags are discarded, as they indicate omitted, repeated, or misplaced audio events.

Tag-Event Consistency Check. Each retained audio tag must be attached to a local caption sentence and correspond to an integrated audio event. We use Qwen3-32B [29] to verify whether the fused audio event remains semantically consistent with its source audio caption. Samples with unmatched tags or altered audio meanings are discarded to filter hallucinated, duplicated, or temporally misplaced audio insertions.

Local Synergy Verification. We further use Qwen3-32B to check whether each tag-anchored sentence contains both the audio event and its corresponding visual context, expressed through temporal or associative cues such as “while”, “as”, or “accompanied by”. Captions failing this check are removed, since they may preserve both modalities without forming valid cross-modal event binding.

3.2 Training AVSCap-7B↩︎

AVSCap-7B is built on Qwen2.5-Omni-7B and trained in two stages: SFT establishes baseline captioning, and GRPO improves acoustic completeness and audio-visual event binding. Additional details are provided in Appendix 11.

3.2.1 Stage 1: Supervised Fine-Tuning (SFT)↩︎

We fine-tune Qwen2.5-Omni-7B with a multi-grained SFT dataset. Each video in AVSCap-130K provides three annotations: a visual caption, an audio caption, and a synergistic omni-modal caption. These prompt-response pairs enable both unimodal perception and cross-modal orchestration.

To improve non-speech audio understanding, we construct 10,000 audio-centric captions from AudioCaps [27] and MusicCaps [28]. Their raw audio tracks are processed with the same audio-stream pipeline in Section 3.1.2.1 to generate detailed (SFX) and (Music) captions. All video and audio captions are then jointly shuffled for unified SFT training. Ablations are provided in Appendix 6.1.

3.2.2 Stage 2: GRPO Optimization↩︎

Although SFT provides strong captioning ability, the model still produces occasional repetitive outputs and weak audio-visual event binding. We further optimize AVSCap-7B with Group Relative Policy Optimization (GRPO) on 2,000 additional videos that do not overlap with the SFT training set. Their reference captions are constructed with the same data pipeline as AVSCap-130K. For each training instance, the policy samples a group of candidate captions, and optimization is guided by three rewards for length control, speech preservation, and audio-visual consistency.

Reward 1: Length Regularization (\(R_{\mathrm{len}}\)). To prevent short outputs and repetition collapse, we assign full reward to captions within a valid length range and zero otherwise: \[R_{\mathrm{len}} = \mathbb{I}\left(\tau_{\min} \leq L_{\mathrm{gen}} \leq \tau_{\max}\right),\] where \(L_{\mathrm{gen}}\) is the generated caption length. We set \(\tau_{\min}=200\) and \(\tau_{\max}=2048\) based on caption-length statistics and context-budget analysis, with details provided in Appendix 7.

Reward 2: Regex-Anchored Speech Recall (\(R_{\mathrm{sp}}\)). To encourage faithful preservation of speech content without expensive judge calls, we extract all (Speech) segments via regex matching and compare them with the reference speech segments. Before matching, punctuation is removed from both sides. For each aligned speech pair, we compute an LCS-based recall score: \[R_{\mathrm{sp}} = \frac{1}{M}\sum_{i=1}^{M} \frac{ \mathrm{LCS}\left(s_i^{\mathrm{gen}}, s_i^{\mathrm{ref}}\right) }{ \left|s_i^{\mathrm{ref}}\right| },\] where \(M\) is the number of reference speech segments, \(s_i^{\mathrm{gen}}\) and \(s_i^{\mathrm{ref}}\) denote the generated and reference speech segments after tag-based alignment, and \(\mathrm{LCS}( , )\) denotes the length of their longest common subsequence. This reward penalizes omitted, reordered, or substantially rewritten speech while remaining fully rule-based.

Reward 3: Cross-modal Synergy Recall (\(R_{\mathrm{syn}}\)). To optimize the correctness and consistency of audio-visual event binding, we decompose the reference caption into synergy events and use GPT-5 [30] to judge whether each event is covered by the generated caption. The reward is computed as event-level recall: \[R_{\mathrm{syn}} = \frac{ \left|\mathcal{E}_{\mathrm{covered}}\right| }{ \left|\mathcal{E}_{\mathrm{syn}}\right| },\] where \(\mathcal{E}_{\mathrm{syn}}\) is the set of reference synergy events and \(\mathcal{E}_{\mathrm{covered}}\) is the subset covered by the generated caption. This reward encourages preserving both modalities and their temporal correspondence.

Total Reward. The final reward is the sum of the three components: \[R_{\mathrm{total}} = R_{\mathrm{len}} + R_{\mathrm{sp}} + R_{\mathrm{syn}}.\]

3.3 The AVSCapBench↩︎

To rigorously evaluate the capabilities of omni-modal models in video understanding and audio-visual synergy, we introduce the AVSCapBench.

3.3.1 Benchmark Construction↩︎

AVSCapBench consists of 1,226 manually annotated video clips collected from YouTube, TikTok, and Video-MME [8], lasting 30 to 120 seconds. It covers diverse domains, including movies, vlogs, gaming, sports, and news. A complete annotation case is provided in Appendix 8. Table 1 and Figure 3 compare AVSCapBench with existing benchmarks, and Figure 4 summarizes its statistics. We construct AVSCapBench through a three-stage human-in-the-loop pipeline.

Automated Segmentation. We first use Gemini-3-Flash [26] to identify segmentation timestamps that preserve both visual coherence and audio continuity. The original videos are then divided into shorter clips for annotation.

Segment-level Human Annotation. Human annotators independently caption each clip, covering visual content like scenes, characters, actions, objects, and OCR text alongside speech, sound effects, and music. Annotators are instructed to place audio descriptions near their corresponding visual events and to ensure that each caption reflects audio-visual consistency throughout the segment.

Merging and Cross-Verification. Segment-level captions are sequentially merged into dense full-video captions. A second group of annotators then cross-checks them to identify omissions, hallucinations, and incorrect audio-visual alignments. To stress-test fine-grained acoustic coverage, we retain only samples that contain all three audio categories—Speech, SFX, and Music; samples that do not satisfy this requirement are discarded.

3.3.2 Evaluation Protocol↩︎

Inspired by DREAM-1K [31], we adopt a fine-grained event-based matching protocol, as illustrated in Figure 1. We utilize GPT-5 [30] to decompose each ground-truth caption directly into a structured set of atomic events \(\mathcal{E}\) categorized across three distinct modality types. Visual events \(\mathcal{E}_{visual}\) describe objective visual content, including entities, actions, interactions, and scene states. Audio events \(\mathcal{E}_{audio}\) cover auditory information, including speech, sound effects, and background music, organized in chronological order. Synergy events \(\mathcal{E}_{synergy}\) bind a visual event with its corresponding audio cue when the two are temporally aligned, thereby capturing cross-modal audio-visual relations. We empirically demonstrate the limitations of traditional n-gram overlap metrics and justify our transition to event-based recall in Appendix 6.3.

During evaluation, we use Gemini-3.1-Pro [32] as the judge model. For each event \(e_i \in \mathcal{E}\), the judge determines whether the model-generated caption \(\hat{C}\) semantically covers the event. We then compute recall independently for each event type: \[\text{Recall}_{\text{type}} = \frac{1}{|\mathcal{E}_{\text{type}}|} \sum_{e_i \in \mathcal{E}_{\text{type}}} h(e_i, \hat{C}),\] where \(h(e_i,\hat{C})=1\) if the judge determines that \(\hat{C}\) covers event \(e_i\), and \(h(e_i,\hat{C})=0\) otherwise. The event type is defined as \(\text{type} \in \{\text{visual}, \text{audio}, \text{synergy}\}\). Since the numbers of events differ across modality types, the overall benchmark score is computed as the event-count-weighted average of the three recall metrics. The complete set of system instructions and evaluation prompts is detailed in Appendix 9.

4 Experiments↩︎

4.1 Main Results↩︎

We evaluate 13 leading omni-modal models, including Gemini-3-Pro [33], Gemini-3-Flash [26], Qwen3-Omni [34], Qwen2.5-Omni [1], ARC-Hunyuan-Video [35], HumanOmniV2 [36], MiniCPM-o [37], video-SALMONN-2 [14], ASID-Captioner [6], AVoCaDO [7], and UGC-VideoCaptioner [10]. We provide the detailed evaluation settings and an extended set of evaluation results in Appendix 10.1 and 10.2, respectively. The main results in Table 2 yield several key observations: (1) Audio-visual synergy remains the weakest capability. Synergy Recall is consistently the lowest dimension across all models. Open-source models often exhibit near-zero synergy despite strong unimodal perception, indicating a severe lack of temporal alignment. (2) Specialized models outperform general omni-modal models. Task-specific captioners (e.g., AVoCaDO) surpass general omni-modal models (e.g., Qwen2.5-Omni), underscoring the value of task-oriented training. (3) Audio understanding is highly inconsistent across open-source systems, with music and SFX remaining particularly challenging. (4) AVSCap-7B achieves competitive performance, outperforming open-source baselines and approaching commercial systems, validating the effectiveness of our curated data and GRPO-based optimization.

4.2 Cross-Benchmark Evaluation↩︎

To assess generalization, we evaluate AVSCap-7B on UGC-VideoCap [10], Daily-Omni [38], and Omni-Cloze [15]. Detailed descriptions of these external benchmarks can be found in Appendix 12. As shown in Table 3, AVSCap-7B demonstrates robust transferability, outperforming all open-source baselines and approaching proprietary models on both UGC-VideoCap and Daily-Omni. Furthermore, Table 4 confirms its superiority on Omni-Cloze, particularly in the Audio-Visual subset, validating that our synergy-focused training successfully transfers to dense cross-modal reasoning tasks.

4.3 Ablation Studies↩︎

Efficacy of GRPO vs. SFT Data Scaling. To examine whether the gains come from reinforcement learning or simply from more supervised data, we compare GRPO with SFT data scaling in Figure 5. Starting from the full AVSCap-130K training set, we construct a half-scale SFT variant by randomly sampling 50% of each data type, resulting in a 65K-example subset. Scaling SFT from 65K to 130K yields only marginal improvements on AVSCapBench and UGC-VideoCap. In contrast, applying GRPO on top of the 130K SFT model with only 2K additional optimization videos brings substantially larger gains, indicating that direct policy optimization is more effective than pure SFT data scaling for improving audio-visual synergy.

Effect of Video Duration. We evaluate three open-source models across four duration intervals (30–50s, 50–70s, 70–90s, and over 90s) in Figure 6. All evaluated models show a consistent downward performance trend as video length increases. This universal decline highlights a key limitation of current open-source omni-modal architectures: they struggle with long-context processing, making it difficult to maintain robust and aligned audio-visual perception across extended temporal horizons.

Effect of Frame Sampling Rate. We analyze the impact of visual sampling rates (0.5, 1, 2, and 4 FPS) in Figure 7. Open-source models show a non-monotonic trend, peaking at 2 FPS but degrading at 4 FPS, which suggests that moderate frame rates optimize perception while overly dense sampling introduces context redundancy and distribution shifts. In contrast, Gemini-3-Flash continuously improves up to 4 FPS, demonstrating stronger long-context robustness and redundancy filtering.

Modality Shielding and Leakage. Beyond overall captioning quality, an omni-modal model should also support controllable modality-specific generation: when asked to describe only the visual or audio stream, it should avoid leaking information from the suppressed modality. This ability is important for controllable captioning and for verifying whether the model truly separates unimodal evidence before cross-modal fusion. To evaluate this, we sample 100 videos from AVSCapBench that contain no subtitles or on-screen dialogue cues, preventing models from inferring speech content from visible text. For each video, we prompt each model to generate either a visual-only or an audio-only caption. We then use Gemini-3.1-Pro to check whether the generated caption contains information from the suppressed modality. A sample is counted as leakage if cross-modal content is detected, and the Modality Leakage Rate is computed as the percentage of leaked samples over the 100 videos. As shown in Table 5, general and task-specific baselines frequently leak suppressed modality information, whereas AVSCap-7B achieves substantially lower leakage rates, demonstrating stronger modality isolation and instruction-controllable generation. We further analyze prompt sensitivity in Appendix 6.2.

Judge–Human Agreement. To validate automated evaluation reliability, we assess the alignment between human annotations and three LLM judges (Gemini-3.1-Pro, DeepSeek-V4-Pro [39], and Qwen3.5-27B [40]) across visual, audio, and synergy dimensions on 200 benchmark samples (Table 6). All LLM judges demonstrate strong agreement with human ratings, with Gemini-3.1-Pro achieving the highest consistency; we thus adopt it as our primary automated evaluator. Importantly, we observe that while these judges exhibit different levels of agreement, they yield identical partial orderings for the evaluated models, ensuring the robustness of our ranking results (refer to Appendix 10.3 for detailed statistics and agreement calculations).

Error Analysis. To further analyze model failures, we randomly sample 200 cases and inspect the unmatched events across multiple models. We categorize errors according to the three event types used in our evaluation. Visual errors are divided into missing visual information and incorrect visual descriptions. Audio errors are divided into incorrect acoustic descriptions and partial audio omissions. Synergy errors are grouped into three types: missing audio-visual relations, incorrect cross-modal binding, and complete event omission. Overall, weaker open-source models often miss audio and synergy events entirely, while stronger models tend to fail through fine-grained incorrect descriptions or imperfect binding. Detailed statistics and examples are provided in Appendix 13.

5 Conclusion↩︎

We present AVSCap, a unified framework for fine-grained audio-visual synergy that addresses modality isolation and speech-centric bias in omni-modal video captioning. We construct AVSCap-130K, a tri-modal training corpus enforcing isolated unimodal perception before cross-modal grounding, and train AVSCap-7B with a two-stage SFT-GRPO paradigm to optimize event binding. We further introduce AVSCapBench, a human-curated benchmark with a fine-grained, event-based matching protocol for evaluating visual, audio, and synergy dimensions. Experiments show that AVSCap-7B substantially outperforms open-source baselines and approaches commercial-grade performance.

Limitations↩︎

While AVSCap demonstrates promising performance, several limitations remain. First, model capability declines as video duration increases, reflecting the general difficulty open-source architectures face in modeling long-term spatiotemporal audio-visual dependencies. Second, our dataset and benchmark are currently restricted to English; extending the framework to multilingual settings, non-English dialogue, and broader cultural contexts remains an important direction for future research.

Ethical Considerations↩︎

The video clips comprising AVSCapBench are gathered from publicly accessible online platforms. In order to respect intellectual property and align with standard copyright regulations, the benchmark will be released under a restrictive license that limits its application exclusively to academic research.

Impact Statement↩︎

This work advances omni-modal video understanding by improving fine-grained audio-visual alignment and temporal reasoning. The AVSCap framework holds potential to benefit various applications, including accessible video description, multimedia retrieval, and multimodal human–computer interaction. By promoting explicit cross-modal event binding, it provides a valuable pathway toward building more coherent and acoustically complete video understanding systems.

6 Additional Experiments↩︎

6.1 Effect of Audio-Centric Augmentation↩︎

As shown in our main experiments, while omni-modal foundation models exhibit robust Automatic Speech Recognition (ASR) capabilities, they frequently struggle with non-speech auditory events. To quantify this phenomenon and validate our solution, we conduct an ablation study analyzing the impact of our audio-centric augmentation during the Supervised Fine-Tuning (SFT) stage.

We first train a baseline model exclusively on the 40K video-text pairs. As detailed in the top row of Table 7, this model achieves a strong Speech Recall of 70.08%, confirming that the base model already possesses solid speech perception from its pre-training. However, it exhibits evident weaknesses in recognizing specific environmental sounds (SFX: 26.35%) and musical properties (Music: 35.98%).

To bridge this gap, we introduce the auxiliary set of 10K audio-only captions (sourced from AudioCaps and MusicCaps) into the training mixture. By integrating this targeted data, the Music and SFX Recall scores surge by +3.52% and +3.45% respectively, lifting the overall Audio score by 2.15%. Crucially, this unimodal auditory enhancement produces a positive ripple effect on cross-modal understanding, pushing the Synergy score up by 1.16%. This confirms that fine-grained acoustic features are essential prerequisites for accurate audio-visual event binding.

This ablation exposes a broader challenge in the current omni-modal industry: while models are heavily optimized for human dialogue (ASR), their comprehension of the wider acoustic world remains shallow. Our findings suggest that systematically scaling high-quality, non-speech auditory data is a critical future direction for advancing holistic video intelligence.

6.2 Prompt Sensitivity in Modality Shielding↩︎

As discussed in Section 4.3, modality shielding evaluates whether a model can selectively describe one modality while suppressing the other. We find that this diagnostic is highly sensitive to prompt wording. Under the default concise prompt (e.g., “Please describe the audio part...”), strong models may interpret the instruction broadly and inadvertently include information from the suppressed modality.

To investigate the boundaries of this behavior, we evaluate an explicit suppression prompt by adding negative constraints (e.g., “...and do not mention visual information”). As shown in Table 8, introducing explicit negative constraints reveals a stark contrast between model architectures. General-purpose instruction-tuned models (e.g., Gemini-3.1-Pro and Qwen2.5-Omni) exhibit dramatic reductions in both audio and visual leakage, demonstrating strong steerability and adherence to negative constraints. Conversely, specialized SFT-based captioners (e.g., ASID-Captioner-7B and AVoCaDO-7B) remain largely unaffected by the instruction change, suggesting that their multimodal descriptive behaviors are rigidly dictated by their task-specific training distribution rather than the prompt itself.

This disparity leads to a notable reversal in relative performance ordering. While the specialized ASID-Captioner-7B outperforms the Qwen2.5-Omni baseline under the default prompt, it is vastly surpassed by Qwen2.5-Omni when explicit suppression is applied. This shift highlights that modality isolation in task-specific captioners is often a rigid artifact of their SFT data distribution rather than a robust, instruction-controllable capability. Consequently, the modality shielding experiment should be interpreted as a prompt-dependent diagnostic of negative-constraint adherence rather than a prompt-invariant absolute benchmark.

6.3 Evaluation with Traditional Metrics↩︎

To empirically demonstrate the limitations of standard text-overlap metrics, we compare the generated captions from Gemini-3.1-Pro and Qwen2.5-Omni-3B against human ground truth using standard overlap metrics (BLEU, CIDEr, and METEOR) in Table 9.

Although Gemini-3.1-Pro outperforms Qwen2.5-Omni-3B, both models score extremely low and show narrow, uninformative performance gaps. These traditional n-gram metrics provide only coarse signals and are fundamentally incapable of capturing fine-grained errors in complex auditory components (e.g., background music, sound effects) or temporal alignment. These results further justify our transition to the proposed fine-grained, event-based matching protocol.

7 Details of GRPO Length Regularization↩︎

We use a simple length reward to prevent two common failure modes during GRPO: degenerate short captions and repetition collapse. The lower bound \(\tau_{\min}=200\) is set as a conservative minimum for valid omni-modal captions. Captions shorter than this threshold usually correspond to extreme under-generation, where visual details, non-speech audio, or audio-visual relations are likely to be omitted.

The upper bound \(\tau_{\max}=2048\) is chosen as a safe generation budget under our video processing setting. Our training videos are sampled at 1 FPS and are typically within about 90 seconds, while the base model has a 32K-token context window. Since the multimodal input already consumes a substantial portion of the context, \(\tau_{\max}=2048\) leaves sufficient room for dense caption generation without approaching the context limit. It also discourages overly long, repetitive outputs.

Overall, the length reward serves only as a coarse validity constraint. Captions within \([\tau_{\min}, \tau_{\max}]\) receive full length reward, while speech preservation and audio-visual event binding are optimized by the other reward components.

8 Example of AVSCapBench↩︎

In this section, we provide a human-annotated example from AVSCapBench. To clearly demonstrate the granularity of our evaluation protocol, we present a complete paragraph of the synthesized omni-modal caption alongside its correspondingly decomposed atomic events (Visual, Audio, and Synergistic). This fine-grained structure allows the judge model to accurately assess cross-modal alignment without being biased by holistic text matching.

[1] Omni-modal Caption

A whiteboard animation begins with upbeat acoustic guitar background music playing throughout the clip (Music) as a hand holding a grey marker sketches on a white background, accompanied by the sharp, squeaking friction of the marker against the board (SFX). The hand first draws a thick book being held open by two hands, and on the cover, the artist sketches a stylized icon of a building with columns and a triangular roof, resembling a courthouse or government institution, as a male narrator speaks in an educational tone, “Employment regulations derive from laws passed by Congress, state legislatures, and local governing bodies, as well as executive orders.” (Speech). As the scene progresses, the hand moves to a blank space and sketches a cluster of simple stick figures representing a group of people, creating rapid scratching sounds (SFX), then draws a single stick figure facing the crowd to illustrate a workplace dynamic while the male narrator states educationally, “These regulations commonly focus on fair treatment of people in the workplace.” (Speech).

[2] Decomposed Atomic Events

Visual Events (\(\mathcal{E}_{visual}\)):

• A hand holding a grey marker sketches on a white background.

• The hand draws a thick book held by two hands with a stylized icon of a building on the cover.

• The hand sketches a cluster of simple stick figures and a single stick figure facing the crowd.

Audio Events (\(\mathcal{E}_{audio}\)):

• Speech: “Employment regulations derive from laws passed by Congress...”

• Speech: “These regulations commonly focus on fair treatment of people...”

• SFX: The sharp, squeaking friction of a marker against a whiteboard.

• Music: Upbeat acoustic guitar background music plays.

Synergistic Events (\(\mathcal{E}_{synergy}\)):

• Upbeat acoustic guitar music plays throughout the animation, establishing an energetic atmosphere.

• As the hand physically sketches on the whiteboard, it is precisely accompanied by the squeaking friction sounds of the marker.

• As a male voice explains that regulations derive from laws, a hand draws a book featuring a courthouse icon.

9 Prompt↩︎

In this appendix, we present the complete set of system instructions and evaluation prompts utilized throughout our framework. To guarantee reproducibility, we detail the exact templates used for dataset construction, and benchmark evaluation.

9.1 Data Construction Prompts↩︎

[
  {
    "segment_id": 1,
    "start": "00:00",
    "end": "00:05",
    "reason": "Introduction speech by Speed"
  },
  {
    "segment_id": 2,
    "start": "00:06",
    "end": "00:12",
    "reason": "Scene cut to the truck accelerating and dragging the pool"
  },
  ...
]

9.2 AVSCap-130K Prompts↩︎

9.3 Benchmark Evaluation Prompts↩︎

{
  "visual_events": [],
  "pure_audio_events": [],
  "synergistic_events": [] 
}

{
  "visual_hits": [1, 0, ...],
  "audio_hits": [1, 1, ...],
  "synergy_hits": [0, 0, ...]
}

9.4 Modality Shielding Prompts↩︎

{
  "is_compliant": true/false,
  "leaked_content": ["List specific dialogue quotes or audio phrases found"]
}

{
  "is_compliant": true/false,
  "leaked_content": ["List specific visual details or specific names found"]
}

10 Evaluation Details↩︎

10.1 Evaluation Settings↩︎

We report the inference settings of locally deployed models in Table 10. The evaluated models include open-source models, our trained model, and closed-source models accessed through APIs. For open-source models, we use the released weights and code in accordance with their licenses and terms of use, and only for research-oriented benchmark evaluation. AVSCap-7B is our trained model and follows the same local evaluation protocol. Evaluation artifacts, including prompts, outputs, and logs, are used for research, analysis, and reproducibility, and their use and distribution remain compatible with the access conditions of the corresponding third-party resources. For closed-source models, we follow the official recommended API usage and the corresponding provider terms.

10.2 Complete Main Results on AVSCapBench↩︎

To provide a comprehensive performance overview, we present the complete evaluation results of all baseline models on the AVSCapBench benchmark in Table 11. This includes the additional baseline models that were omitted from the main text (Table 2) due to space constraints.

10.3 Agreement and Judge Consistency Analysis↩︎

To validate the reliability of our automated evaluation pipeline, we assess the decision-level alignment between human annotators and the three LLM judges (Gemini-3.1-Pro, DeepSeek-V4-Pro, and Qwen3.5-27B) on a validation subset of 200 videos.

For each atomic event \(e\) in the evaluated event set \(\mathcal{E}\), the human annotator and the LLM judge provide binary decisions representing whether the generated caption successfully recalls the event: \[y_{\text{human}}(e) \in \{0, 1\}, \quad y_{\text{judge}}(e) \in \{0, 1\}\] where \(1\) denotes a successful recall (Hit) and \(0\) denotes a failure (Miss). The decision-level Percentage Agreement (\(A_{\text{type}}\)) for a specific event type (Visual, Audio, or Synergy) is defined as the accuracy of the LLM judge’s binary decisions compared to the human ground truth:

\[A_{\text{type}} = \frac{1}{|\mathcal{E}_{\text{type}}|} \sum_{e \in \mathcal{E}_{\text{type}}} \mathbb{I}\Big(y_{\text{human}}(e) = y_{\text{judge}}(e)\Big)\]

where \(\mathcal{E}_{\text{type}}\) is the subset of atomic events belonging to the corresponding type, and \(\mathbb{I}(\cdot)\) is the indicator function that outputs \(1\) if the condition is met and \(0\) otherwise.

In addition to calculating the absolute human-judge agreement rates (as reported in Table 6), we evaluate the ranking consistency of the three LLM judges using three representative open-source models with close overall scores on the leaderboard (ARC-Hunyuan-Video-7B, HumanOmniV2-7B, and MiniCPM-o-2.6-8B).

As shown in Table 12, although the recall scores evaluated on this 200-video subset exhibit minor variations compared to the full leaderboard (Table 2) due to sample size constraints, all three LLM judges consistently preserve the identical partial ordering in the Overall score: ARC-Hunyuan-Video-7B \(>\) HumanOmniV2-7B \(>\) MiniCPM-o-2.6-8B. This solidifies the reliability of our automated evaluation pipeline for ranking comparative models.

11 Training Details↩︎

In the SFT stage, the model is trained for 2 epochs with a batch size of 128 and a learning rate of \(2 \times 10^{-5}\). During the GRPO stage, training is performed for 1 epoch with a batch size of 64 and a learning rate of \(1 \times 10^{-5}\). For each query, we sample 8 responses using a temperature of 1.0.

During both training and evaluation, video inputs are sampled at 1 fps, and the resolution of each frame is limited to a maximum of \(512 \times 28 \times 28\) pixels. Due to the base model’s context window limitation of 32K tokens, the total video pixels are restricted to \(25600 \times 28 \times 28\). All training is conducted on 16 NVIDIA H200 GPUs.

12 Details of Benchmarks↩︎

In this section, we will provide a detailed description of the benchmark we evaluated.

• UGC-VideoCap consists of 1,000 short TikTok videos, each under 60 seconds in duration and containing at least one meaningful audio segment lasting no less than 5 seconds. Each video’s caption is evaluated by a judge model that assigns scores on a 1-to-5 scale across three dimensions: visual, audio, and details. These dimension scores are then normalized and aggregated to produce a final caption quality score.

• Daily-Omni is an audio-visual question answering benchmark comprising 684 videos depicting diverse everyday life scenarios, sourced from multiple platforms. These videos are densely multimodal, offering rich visual and auditory cues. The benchmark includes 1,197 multiple-choice question-answer pairs, distributed across six core tasks. In our experimental setting, we assess the quality of generated captions by feeding them into a judge model and measuring their capacity to support accurate question answering.

• Omni-Cloze is a cloze-style benchmark comprising 2,340 video clips with 70,200 human-verified masked blanks across audio, visual, and audio-visual scenarios. Instead of resource-intensive multi-turn QA, it evaluates models via a single-pass protocol where an LLM completes masked spans in generated captions. Each blank includes a “Not Given” option to distinguish omission from hallucination, enabling a precise error breakdown across visual, auditory, and synergistic dimensions.

13 Error Analysis↩︎

To better understand the failure modes behind event-based recall, we conduct a manual analysis on 200 randomly sampled cases. For each model output, we collect events that are not matched by the evaluator and assign each unmatched event to one of the predefined error categories. For visual events, we distinguish between missing visual information and incorrect visual descriptions. For audio events, we distinguish between incorrect acoustic descriptions and partial audio omissions. For synergy events, we classify errors into missing audio-visual relations, incorrect cross-modal binding, and complete event omission. The ratios are normalized within each event type.

Figure 8 shows that weaker open-source models suffer mainly from omission-based failures. For example, Qwen2.5-Omni-7B and MiniCPM have a large proportion of audio omissions and complete synergy omissions, suggesting that they often fail to describe non-speech audio cues or to connect them with visual events. In contrast, stronger models such as Gemini-3-Flash and AVSCap-7B produce fewer complete omissions; their remaining errors are more concentrated in incorrect fine-grained descriptions and imperfect cross-modal binding. This pattern suggests that our training strategy improves the coverage of non-speech audio and audio-visual relations, shifting the dominant failure mode from missing events to more subtle grounding errors.

Example 1

Prompt:
Please describe all the information in the video without sparing every detail in it. As you describe, you should also describe as much of the information in the audio as possible, and pay attention to the synchronization between the audio and video descriptions.

Ground Truth A close-up shot reveals a man with shoulder-length brown hair, wearing a red plaid shirt and blue jeans, standing on the landing skid of a dark blue helicopter. He holds onto the frame of the aircraft as intense sunlight creates a lens flare across the screen, illuminating the word "BUNDESPOLIZEI" printed in white capital letters on the side. The perspective then cuts to a high-angle aerial shot looking down at a circular, elevated helipad located next to a river, where the helicopter is positioned in the center of the pad with its rotors spinning rapidly, creating a loud, rhythmic chopping and whirring of helicopter blades (SFX), the man still visible on the exterior skid. The scene shifts to a close-up exterior shot of a blue helicopter cockpit, accompanied by the loud, rhythmic whirring of helicopter rotor blades (SFX), where a bright sun flare initially obscures the view through the glass. As the lighting shifts, a man with shoulder-length dark hair, wearing a maroon long-sleeved shirt, is revealed inside the pilot’s seat, looking out with a focused, intense expression. He raises his left hand and firmly grabs the upper frame of the door, producing a mechanical clunking sound (SFX), his fingers curling around the metal structure as he exerts force to pull or brace the mechanism while the camera shakes slightly, accompanied by a tense, low-frequency orchestral action score. Elsewhere, a static shot shows a closed, dark grey metal door set within a rough, concrete wall, with a vertical pipe visible on the left side. Suddenly, the door is kicked open violently from the inside with a loud, sharp metallic crash and impact noise (SFX), swinging outward to the right as a man dressed in a dark suit and white shirt bursts through. He steps forward aggressively with his left leg, making a heavy footstep on concrete (SFX), raising a handgun in his right hand and aiming it directly ahead as he emerges into the corridor. The narrative returns to the action with a tight close-up of a muscular arm, clad in a red sleeve, firmly gripping a metal bar inside what appears to be a moving vehicle or structure. The scene abruptly cuts to Steve Rogers, wearing a grey t-shirt, sprinting with intense focus across an industrial rooftop.

Caption by Qwen2.5-Omni-7B The video opens with a man climbing into a helicopter marked "BUNDES POLIZ" on its side. The helicopter is parked on a helipad near a body of water. The man is wearing a brown jacket and seems to be preparing for a flight. The camera then cuts to a close-up of a man in a car, looking distressed and talking on the phone. The scene shifts to a man in a gray shirt and black pants running towards the helicopter. He grabs onto the helicopter’s rotor and is lifted into the air. The helicopter spins rapidly, and the man struggles to hold on. The camera shows him from different angles as he hangs on, with the cityscape visible in the background. The helicopter eventually crashes, and the man falls to the ground. The video ends with a screen showing two characters in a fight scene, with the words "WATCH MORE" and "SUBSCRIBE" displayed.

Error Analysisanalysisbg The baseline model suffers from severe acoustic omission, completely overlooking critical non-speech auditory cues such as the roaring helicopter rotor blades, mechanical clunking (SFX), and the dramatic background orchestral score (Music), while only capturing basic visual events.

Caption by AVSCap-7B(ours) A tense, rhythmic orchestral score with driving strings and percussion begins to play (Music) as a blue helicopter bearing the white text "BUNDESPOLIZEI" sits on a circular helipad beside a river and modern architecture. A man with shoulder-length dark hair and a maroon long-sleeved shirt approaches the aircraft, opens the door, and climbs into the pilot’s seat. The high-pitched whine of the helicopter engine starting up fills the air (SFX). Inside the cockpit, the man reaches up to flip several switches on the overhead control panel, producing sharp clicking sounds (SFX), before looking out the window with a stern expression. Suddenly, a muscular man with short blonde hair, wearing a tight grey t-shirt and dark pants, bursts through a metal door and sprints across the rooftop towards the helipad, his footsteps thudding rapidly (SFX). As the helicopter lifts off the ground, its rotors chopping the air with a heavy, rhythmic thrum (SFX), the blonde man leaps from the edge of the platform and grabs onto the aircraft’s black landing skid with both hands. Hanging suspended above the city skyline and the river below, the blonde man’s arms strain under his body weight as the helicopter continues to ascend and bank sharply. Inside the cockpit, the dark-haired pilot grips the control stick and pushes it forward (SFX), causing the helicopter to tilt downward. Reacting quickly, the blonde man swings his body toward the helipad structure, releasing one hand from the skid to grab a yellow metal railing on the edge of the platform. Now stretched horizontally between the ascending helicopter and the stationary railing, his muscles bulge visibly as he attempts to anchor the aircraft. The pilot peers down through the window, his face tight with exertion, while the helicopter engine roars loudly, struggling against the man’s immense grip (SFX). The orchestral music swells in intensity, matching the physical struggle (Music). A top-down view shows the helicopter hovering precariously close to the helipad, unable to break free from the man’s hold. With a massive effort, the blonde man pulls his arm inward, dragging the helicopter back toward the platform. The helicopter loses its battle against the man’s strength and crashes violently onto the helipad. The spinning rotor blades strike the concrete, shattering instantly and sending a massive cloud of dust, sparks, and metal debris flying across the platform with a deafening crunch and clatter (SFX). The music cuts out abruptly upon impact (Music). The blonde man is thrown to the ground, rolling through the thick grey smoke to avoid the wreckage. He pushes himself up onto his hands and knees, his face covered in dust, and stares intently toward the crashed helicopter as the debris settles. The scene then cuts to a static end screen featuring a digital painting of Iron Man and Captain America locked in combat, with bright sparks erupting where Iron Man’s repulsor beam meets Captain America’s shield. White text reading "SUBSCRIBE" appears on the left side of the screen, while "_WATCH MORE" is displayed on the right, both positioned above empty, semi-transparent rectangular placeholders. A new, dramatic, sweeping orchestral track begins to play over the static image (Music).

Example 2

Prompt:
Please describe all the information in the video without sparing every detail in it. As you describe, you should also describe as much of the information in the audio as possible, and pay attention to the synchronization between the audio and video descriptions.

Ground Truth On a stage with a black curtain background and a wooden floor, a young dancer is in the midst of a handstand while upbeat pop music plays, specifically the song ‘Call Me Maybe’, with the lyrics ‘I threw a wish in the well, don’t ask me I’ll never tell’ clearly audible. Text overlays appear in the lower left corner reading ‘MACKENZIE’S SOLO’ in white and ‘DANCE STYLE: ACRO’ in orange. The dancer is wearing a vibrant two-piece costume featuring neon pink, green, and yellow ruffles, topped with a matching party hat. She lowers her legs to land on her feet, immediately standing upright and raising her arms in a ‘V’ shape while smiling at the audience. She then performs a front walkover, her legs splitting vertically in the air as she rotates over her hands. The view shifts to a medium close-up of a woman in the audience holding a blue pen, gazing intently toward the stage. The scene cuts back to the stage where the backdrop features a large banner reading "TALENT COMPETITION" in white capital letters against a red and blue background decorated with stars. Continuing the routine, the dancer arches her back in a bridge position as a female singer sings melodically, "Yeah..." (Speech). She executes an extreme flexibility move, bending her spine backward until her head rests against her feet (Speech: "is a..."). Sitting center stage, she pulls her knees to her chest, then rolls backward onto her shoulders, extending her legs straight up in a shoulder stand. From a kneeling position, she suddenly springs upward as upbeat pop dance music begins with the lyrics ‘Let’s go...’ (Music), jumping high into the air. The camera cuts to the audience, showing a blonde woman, Melissa, wearing a red top and black blazer, smiling broadly and clapping. The view shifts to an interview where Melissa says in an explanatory tone, "Mackenzie is seven and before she" (Speech). A lower-third graphic identifies her as "MELISSA" with the subtitle "DANCE MOM". She says earnestly, "I want to really" (Speech). The scene abruptly cuts back to the stage performance where the young dancer executes a dynamic front aerial flip in mid-air as an upbeat reality TV instrumental plays in the background (Music). Back in the dimly lit auditorium, the blonde woman in the black jacket is captured in a close-up, beaming with a wide, toothy smile, with the sound of crowd cheering and applause mixed with the music (SFX). She finishes a cartwheel, landing on the dark stage floor with a thud (SFX). She raises her right arm and arches backward, executing a back walkover, landing on her feet with her right arm extended high in the air to audience applause and cheering (SFX). The scene cuts to two women in the front row with loud, distinct laughter from the audience members (SFX). Returning to the stage in front of the large red banner, the dancer performs a chin stand variation, balancing on her chest and hands with her legs split high in the air. On the glossy black floor, she transitions into a seated position, leaning back on her hands with her knees bent and feet flat on the floor, striking a final pose while smiling brightly toward the audience. A female audience member exclaims excitedly, "Oh my god!" (Speech), accompanied by loud applause and cheering from a crowd (SFX). The video concludes with a static promotional end card for the television show ‘Dance Moms’, with the logo and tagline ‘THE SHOW MUST GO ON...’ and navigation text overlays reading ‘WATCH MORE VIDEOS’, ‘SUBSCRIBE’, and ‘ABOUT’.

Caption by Qwen2.5-Omni-7B The video opens with a young girl, Mackenzie, sitting on a stage. She is wearing a colorful outfit and a party hat. The text "MACKENZIE’S SOLO DANCE STYLE: ACRO" appears on the screen. Mackenzie then begins her dance routine, performing various acrobatic moves. She does a handstand, a cartwheel, and a backbend. The audience, including a woman in the audience, claps and cheers for her. The backdrop behind her reads "NATIONAL TALENT COMPETITION." The music plays in the background, adding to the excitement of the performance. The video ends with a shot of the audience clapping and a woman in a black dress standing on a stage with the words "WATCH MORE VIDEOS" and "THE SHOW MUST GO ON... DANCE MOMS" displayed on the screen.

Error Analysisanalysisbg The baseline model exhibits incomplete transcription, completely omitting the spoken dialogue and on-screen interview speech (e.g., the mother’s remarks and audience exclamations), thereby failing to capture crucial semantic knowledge from the audio modality.

Caption by AVSCap-7B(ours) The scene opens on a dimly lit stage where a young girl stands in profile, wearing a vibrant two-piece dance outfit with pink, green, and yellow ruffles, topped with a yellow party hat featuring a pink pom-pom. On-screen text reads "MACKENZIE’S SOLO DANCE STYLE: ACRO". An upbeat pop song with a driving beat and female vocals singing, "Watch me do my dance, set the rules down, need to call the boys, welcome around," begins to play (Music). Mackenzie raises her arms and executes a handstand, followed by a front walkover. The camera briefly cuts to a woman in the audience with dark hair, wearing a black top and holding a blue pen, watching intently. Returning to the stage, the backdrop is now visible, displaying "NATIONAL TALENT COMPETITION" in large letters with star graphics. Mackenzie performs a backbend, lowering herself to the floor, then rolls onto her stomach, sits up, and completes a backward roll before running forward with a smile. The music continues with lyrics, "wanna have a good time, is it such a crime, over all the drama, let’s go have some fun" (Music). The view shifts to the audience, showing a woman in a red top smiling broadly next to a young girl. The scene then transitions to an interview setting with the woman, now wearing a beige jacket and a statement necklace, positioned in front of a rack of colorful costumes. On-screen text identifies her as "MELISSA DANCE MOM". She explains, "Mackenzie is seven, and before she just did it for fun, and now I think she wants to really be good." (Speech). The video cuts back to the stage as Mackenzie performs a leaping jump, landing gracefully. The camera briefly shows Melissa in the audience again, laughing joyfully. On stage, Mackenzie continues her routine, executing a one-handed cartwheel as the music’s chorus repeats, "party starts right now" (Music). The camera cuts to two women in the audience, one in a white jacket and the other in a sparkly black top, both laughing and enjoying the performance. Back on stage, Mackenzie performs a chin stand, her legs arched over her head, before rolling onto her stomach and sliding into a final seated pose with one hand resting on the floor. As she strikes her final pose, loud applause and cheering erupt from the audience (SFX). The camera shows Melissa clapping enthusiastically and exclaiming, "Oh, she did so good!" (Speech), while the woman in the sparkly black top also claps with a wide smile. The video concludes with a static end screen featuring a blue curtain backdrop. A woman dressed in black stands in the center, posing with one hand on her hip and one foot resting on scattered red rose petals. Text on the screen reads "WATCH MORE VIDEOS" on the left, and "THE SHOW MUST GO ON... DANCE MOMS" on the right, with "SUBSCRIBE" and "ABOUT" positioned below.

Example 3

Prompt:
Please describe all the information in the video without sparing every detail in it. As you describe, you should also describe as much of the information in the audio as possible, and pay attention to the synchronization between the audio and video descriptions.

Ground Truth A high-angle view shows a short track speed skating starting line on an indoor ice rink where five skaters are lined up horizontally, wearing distinct aerodynamic suits and helmets. From left to right, they are dressed in black with white stripes, white with red and blue accents, blue with black, white with red, and red with black. Initially, the athletes stand upright and relaxed, looking down at the ice markings as the official says authoritatively, "Ready." (Speech). Upon hearing the command, they remain still for a split second. Suddenly, a starting pistol fires with a loud gunshot (SFX), and all five skaters simultaneously drop their bodies low, bending their knees deeply and leaning their torsos forward into a crouched, ready position, poised to explode off the starting line. The five skaters launch from their crouched starting positions as the commentator notes in a professional and observant tone, "And they’re away cleanly for" (Speech). They drive their legs outward and swing their arms vigorously to gain momentum with the rhythmic scraping and whooshing of skates on ice (SFX). As the pack accelerates, a digital timer graphic in the bottom right corner counts up, displaying ‘2.2’. The camera pans smoothly to the left, tracking a tight pack of four skaters as they round a curve while the commentator states in an informative and excited tone, "500 meters four and a half laps around the track." (Speech). Leading the pack is a skater wearing a red helmet and a black suit with red shoulders representing China, leaning deeply with his left hand grazing the ice surface for balance. As they straighten out onto the track, a digital graphic overlay displays "4 LAPS TO GO" as the commentator excitedly says, "And they will be flying. Four laps to go now look at that speed 43.3 kilometers." (Speech). The skaters execute powerful crossover strides, their bodies low and aerodynamic, creating rhythmic, sharp scraping sounds of blades cutting into the ice (SFX). The commentator descriptively notes, "Wu Dajing of China has the lead right behind him..." (Speech) against the backdrop of purple rink barriers and a blurred crowd of spectators, accompanied by the continuous ambient roar from the stadium crowd (SFX). A leaderboard graphic displays the current race order: ‘1 CHN WU D’, ‘2 LAT R. ZVEJNIEKS’, and ‘3 JPN K. WATANABE’, alongside a status indicator reading ‘3 Laps to Go’ and a speed of ‘50.9 KM/H’. As the skaters round a curve, a racer in the back of the pack loses his footing and crashes, sliding helplessly across the rink into the purple padded barriers as the commentator states professionally, "And we have one racer down and out, the Kazakhstan is out." (Speech). The remaining three skaters continue their momentum down the straightaway to the continuous crowd cheering (SFX). A graphic overlay displays "2 Laps to Go." The commentator excitedly announces, "But Wu Dajing has opened up a sizable lead over Roberts Zvejnieks." (Speech). The commentator says in an anticipatory tone, "Korean racer Hwang Dae-heon trying to catch him." (Speech). Approaching the finish line, the leader extends his right skate forward in a lunge amidst crowd cheering (SFX), while the trailing skater simultaneously lunges to close the gap. They cross the line nearly together as the commentator declaratively states, "Photo finish." (Speech). A digital timer reads 39.81 as the race concludes. A results graphic lists Wu Dajing first, Hwang Daeheon second, and R. Zvejnieks third. The commentator emphatically declares, "But Wu Dajing, the definite winner of that heat" (Speech). The video concludes with a static promotional graphic for the Winter Olympics, displaying ‘THE WINTER OLYMPICS’ with the dates ‘FEB 8-25’ and a ‘SUBSCRIBE NOW’ call-to-action.

Caption by Qwen2.5-Omni-7B The video opens with a group of speed skaters on an ice rink, all wearing helmets and racing suits. They are lined up, ready to start a race. The skaters begin to move, gliding swiftly across the ice. The camera captures their speed and agility as they navigate the track. The crowd in the stands is visible, cheering and watching intently. The on-screen text shows the names and rankings of the skaters, with "WU D." in the lead. The speed of the skaters is displayed, reaching up to 50.9 km/h. As the race progresses, the skaters continue to push themselves, with some making sharp turns and others maintaining a steady pace. The final lap approaches, and the skaters are neck and neck, with "WU D." still in the lead. The video ends with a close-up of the skaters crossing the finish line, and the on-screen text confirms "WU D." as the winner. The video then transitions to a screen promoting "The Winter Olympics" with the dates "Feb 8-25" and a call to action to "Subscribe Now."

Error Analysisanalysisbg The baseline model fails to capture temporal alignment, listing isolated visual actions (such as the skaters starting and crossing the finish line) without syncing them with the auditory triggers, such as the starting pistol gunshot (SFX) or the commentator’s live voiceover (Speech).

Caption by AVSCap-7B(ours) The video begins on an indoor ice rink with five speed skaters lined up at the starting line in a crouched position. From left to right, the skaters wear black, white, blue, white, and red suits. A voice commands, "Ready" (Speech), followed immediately by the sharp crack of a starting pistol (SFX). The skaters launch forward, their blades scraping the ice (SFX). An announcer states, "And they’re away cleanly for this heat number two of the men’s 500 meters..." (Speech) while a timer graphic in the bottom right corner begins counting up. As the skaters navigate the first curve, the skater in red takes the lead, with the announcer continuing, "And they will be flying. Four laps to go now. Look at that speed, 43.3 kilometers" (Speech) as a ranking overlay appears on the bottom left. The skaters execute powerful crossover strides, creating a rhythmic scraping sound of blades cutting into the ice (SFX). The commentator excitedly notes, "Wu Dajing of China has the lead right behind him..." (Speech) amidst the continuous ambient roar of the cheering crowd (SFX). As the skaters round a curve, a racer in the back of the pack loses his footing and crashes, sliding helplessly into the padded barriers (SFX). The commentator notes, "And we have one racer down and out, the Kazakhstan is out." (Speech). The remaining three skaters maintain high speed down the straightaway to the crowd’s loud cheering (SFX). Approaching the final lap, the leader maintains a narrow lead over his opponent. He extends his right skate forward in a lunge to cross the finish line (SFX) as the commentator declares, "Photo finish." (Speech). A final results graphic lists Wu Dajing first, while the commentator states, "But Wu Dajing, the definite winner of that heat" (Speech). The video concludes with a static Winter Olympics promotional screen (Music).

14 Details of Human Annotation for AVSCapBench↩︎

In this section, we provide detailed information regarding the human annotation process for AVSCapBench, directly addressing the ARR Responsible NLP Research requirements (D1–D4).

14.1 Annotator Instructions and Workflow (D1)↩︎

To ensure high-quality annotations, annotators were provided with the required pre-segmented video clips alongside a comprehensive annotation guideline. The workflow was straightforward: annotators watched the provided clips and wrote omni-modal captions strictly following our detailed standards. They were not responsible for any upstream video segmentation or downstream caption merging.

The core instructions required annotators to accurately describe visual actions, comprehensively record auditory events (including speech, sound effects, and music), and explicitly use temporal conjunctions (e.g., “while”, “accompanied by”) to bind corresponding cross-modal events. After drafting a caption, annotators were required to cross-reference a predefined verification checklist to ensure the completeness of these three dimensions. Before entering the formal annotation phase, all participants underwent a qualification test where they annotated several sample videos and received feedback to ensure they fully grasped the required standards. Since the task only involves describing everyday videos, there were no risks of exposure to harmful materials, and a standard disclaimer was provided prior to the task.

14.2 Recruitment and Payment (D2)↩︎

We recruited 5 human annotators for this project. All annotators are graduate-level students with high English proficiency, ensuring the high linguistic quality of the captions.

Annotators were compensated based on the time spent on the task. On average, annotating a single video clip took approximately 20 minutes. The payment was set at approximately $12 USD per hour (which equates to roughly $4 USD per clip). This compensation rate is well above the local statutory minimum wage and is considered highly competitive and adequate for data annotation tasks in the annotators’ demographic region.

14.3 Data Consent (D3)↩︎

There are two aspects of data consent in our study. First, regarding the source videos: all video clips used to construct AVSCapBench were sourced from publicly available platforms (e.g., YouTube, TikTok) and existing open-source datasets. They are used strictly under fair use for academic research. We do not distribute the original video files; instead, we release the public URLs and timestamps. Second, regarding the annotators: before starting the task, all annotators were informed about the purpose of the research. They explicitly consented that their generated text annotations would be open-sourced and freely available to the academic community.

14.4 Ethics Review Board Approval (D4)↩︎

The data collection and annotation protocol was reviewed in accordance with our institution’s ethical guidelines. Because the annotation task strictly involves describing publicly available, non-harmful video content and does not involve collecting any personally identifiable information (PII), psychological profiling, or exposure to offensive materials, the protocol was determined to be exempt from formal Ethics Review Board (IRB) approval.

References↩︎