1 Introduction↩︎

Amidst the rapid ascendance of Multimodal Large Language Models (MLLMs) [1]–[9], video captioning plays a pivotal role in advancing the frontiers of video understanding and generation[10]–[22]. High-quality video captions serve not only as a foundational cornerstone for cross-modal alignment during pre-training but also as a critical mechanism for injecting rich semantic knowledge into downstream tasks [23]–[30]. Recent empirical studies suggest that training with detailed and factually grounded captions yields consistent performance gains across a broad spectrum of applications, including video question answering[31], action recognition[32], and generative video synthesis[24], [33]. Therefore, enhancing the capabilities of video captioning models has emerged as a fundamental pathway toward building more robust video understanding and generation systems.

Despite the remarkable progress of MLLMs, early research in video captioning has remained predominantly vision-centric[11], [12], [34], often overlooking the profound semantic cues embedded within auditory signals. In practice, auditory elements such as dialogues, voiceovers, and ambient soundscapes are indispensable for achieving a holistic and embodied understanding of video content. The emergence of omni-modal MLLMs, such as Qwen3-Omni[35] and Gemini 3 Pro [36], has signaled a paradigm shift toward joint audio-visual reasoning. Concurrently, specialized audiovisual captioners, including AVoCaDO[37], video-SALMONN 2[38], and UGC-VideoCaptioner[39], have begun to explore the complementary nature of auditory and visual streams to produce more comprehensive narratives.

Nevertheless, two critical challenges in joint audio-visual video captioning remain insufficiently addressed. The first is Audio-Visual Cross-Modal Binding, which refers to the precise association between specific auditory events and their corresponding visual entities. For instance, models frequently struggle to correctly attribute a segment of speech to its specific speaker in crowded scenes or link a transient impact sound to its physical origin, leading to attribute misattribution. The second challenge is Audio-Visual Temporal Coherence, which concerns the chronological ordering and causal dependencies across audio-visual modalities. Existing models often fail to capture the fine-grained temporal dynamics between audio signals and visual actions, such as the "sound-before-sight" anticipation or "action-followed-by-sound" feedback, thereby struggling to model the evolving logic of multimodal event streams. As illustrated in Fig. 1, Qwen3-Omni[35] exhibits a pronounced modality isolation in its descriptions; it tends to narrate visual content and auditory elements in a decoupled, sequential manner (i.e., describing all visual scenes before addressing any audio), which entirely neglects the temporal coherence between the two streams. Furthermore, it suffers from significant errors in cross-modal binding. Similarly, both Gemini-3-Pro[36] and AVoCaDO [37] demonstrate localized failures in binding and temporal synchronization within their generated captions. These deficiencies represent a significant barrier to achieving a truly unified perception of synchronized audio-visual content.

To bridge these gaps, we present TCA-Captioner along with a diagnostic benchmark, TCA-Bench, specifically engineered to enhance Temporal and Cross-modal Alignment in audiovisual video captioning. To facilitate high-fidelity model training, we first design a rigorous data synthesis pipeline termed Observer-Checker-Corrector (OCC), powered by the state-of-the-art Gemini 3 Pro. Unlike conventional single-pass annotation, the OCC pipeline adopts an iterative refinement strategy. In this framework, the Observer first generates initial descriptions, which are then rigorously vetted by the Checker to identify unimodal factual hallucinations and cross-modal alignment inconsistencies. By ensuring the narrative is faithful to both individual streams and their joint dynamics, this verification step allows the Corrector to precisely rectify misalignments, thereby producing meticulously grounded captions. Furthermore, to specifically address the intricacies of temporal and cross-modal dependencies, we curate a high-density human interaction Dataset. This dataset is characterized by dense human speech interleaved with complex physical actions, providing a rich corpus of synchronized audiovisual cues that are often absent in existing datasets. Leveraging this high-fidelity data, we optimize TCA-Captioner through a robust post-training regime centered on Supervised Fine-Tuning (SFT). Extensive experiments demonstrate that our TCA-Captioner achieves superior performance in modeling sophisticated cross-modal interactions, effectively setting a new standard for temporally-coherent audiovisual narratives.

To facilitate a more granular assessment of audiovisual video captioning, we introduce TCA-Bench, a diagnostic benchmark that advocates a paradigm shift from static detail recall to the evaluation of dynamic audio-visual entanglement. Unlike existing benchmarks that rely on holistic scoring or "bag-of-events" matching, which frequently neglect the structural coherence of narratives, TCA-Bench specifically addresses the prevalent failure mode of modality detachment. We provide structured annotations, including Audio-Visual Binding Lists and Temporal Relational Lists, to formalize the fine-grained dependencies between acoustic events and their visual origins. Furthermore, we propose a Decoupled Evaluation Protocol that disentangles foundational uni-modal perception from higher-order relational reasoning. This protocol employs an LLM-based judge to perform targeted, hallucination-resistant verification of cross-modal source localization and chronological ordering. By isolating these synchronization capabilities, TCA-Bench offers a highly interpretable and granular assessment, uncovering the primary bottlenecks of current models in achieving unified and temporally-coherent multimodal understanding. Our contributions can be summarized as follows:

• We introduce the Observer-Checker-Corrector (OCC) framework, which employs an iterative refinement strategy to disentangle multifaceted audiovisual caption generation, guaranteeing the production of meticulously grounded and high-fidelity video descriptions.

• We develop TCA-Captioner, specifically optimized for fine-grained cross-modal binding and temporal coherence, supported by a newly curated dataset of high-density human interactions.

• We propose TCA-Bench, a diagnostic benchmark utilizing a Decoupled Evaluation Protocol to quantify the model’s proficiency in audio-visual binding and temporal relational reasoning.

2 Related Works↩︎

2.1 Multimodal Large Language Models↩︎

The landscape of Multimodal Large Language Models (MLLMs) has undergone a rapid paradigm shift, expanding from basic image-text alignment to sophisticated, all-encompassing sensory perception [40]. Early endeavors, such as LLaVA[41] and BLIP-2 [42], focused on bridging the modality gap by aligning frozen visual encoders with LLMs via specialized adapters, a foundation later scaled by InternVL[43] and VILA[44]. This evolved into temporal modeling with Video-LLaVA[45], ShareGPT4Video[11], and LLaVA-Video[46], which utilized high-quality synthetic data to capture dynamic transitions in silent videos. Concurrently, dedicated models like SALMONN[47] and Qwen2-Audio[48] emerged to tackle non-visual acoustic reasoning and spatial audio understanding[4]. Building upon these modular successes, the field is converging toward Omni-modal MLLMs [35], [36], aiming for a holistic interpretation of real-world scenarios by processing interleaved text, vision, and audio tokens within a unified architecture. Industry leaders like GPT-4o[49] and the Gemini series (including Gemini 3 Pro)[36] have demonstrated the efficacy of native multimodal integration, exhibiting superior reasoning in long-context and real-time environments. In the open-source community, the Qwen-Omni series[35], [50] has mirrored this trend: while Qwen2.5-Omni[50] achieved seamless audio-visual streaming, the state-of-the-art Qwen3-Omni[35] further pushes these boundaries by introducing a "multimodal thinking mode" and significantly reducing latency for extended inputs. Our work is positioned at this frontier, aiming to systematically evaluate and enhance MLLM capabilities in these complex, omnimodal scenarios.

2.2 MLLM for Video Captioning↩︎

The integration of Large Language Models (LLMs) has fundamentally reshaped video captioning, evolving from template-based outputs to context-aware narratives. Early VideoLLMs[51]–[74] typically coupled pre-trained visual encoders with language backbones to capture dynamic transitions. To enhance granularity, recent models like OwlCap[75] and the Tarsier series[76] leveraged large-scale SFT datasets to balance high-level motion semantics with fine-grained static details. However, these vision-centric models remain "deaf" to the audio modality, limiting their interpretation of real-world videos where acoustic cues are indispensable. To overcome this, the field has pivoted toward omnimodal architectures [35], [77]–[83]. Proprietary models like Gemini 3 Pro[36] and open-source breakthroughs such as Qwen3-Omni[35]] treat vision and audio as synchronized streams within a unified transformer space. Despite their general reasoning prowess, they are not explicitly optimized for the dense, temporally-aligned captioning essential for high-fidelity video understanding. While concurrent efforts like Video-SALMONN-2[38] and UGC-VideoCaptioner[39] employ intensive DPO or focus on specific content types, achieving precise temporal alignment between transient acoustic events and visual frames remains an open challenge. In this work, we aim to enhance temporal and cross-modal alignment for joint audiovisual video captioning, ensuring that visual dynamics and acoustic semantics are harmoniously integrated into high-quality descriptions.

3 TCA-Captioner↩︎

3.1 Challenge for Audiovisual Video Captioning↩︎

Traditional video captioning models primarily follow a vision-centric paradigm, treat-ing audio as a redundant or auxiliary signal[11], [12], [84]–[88]. However, real-world audiovisual content is defined by high-density interactions where semantics are deeply entangled across modalities. Despite recent progress in joint audiovisual captioning, two primary bottlenecks remain largely overlooked: (1) Audio-Visual Cross-Modal Binding, the challenge of accurately associating a transient acoustic event (e.g., a specific utterance or a physical impact) with its corresponding visual source in a complex scene; and (2) Audio-Visual Temporal Coherence, the requirement to preserve the precise chronological and causal flow between auditory and visual streams. The neglect of these factors leads to "modality detachment," where captions may contain correct individual elements but fail to capture their synchronized relational logic.

3.2 Observer-Checker-Corrector Framework↩︎

To address the aforementioned challenges, we propose the Observer-Checker-Corrector (OCC) framework, an iterative data synthesis pipeline designed to generate high-fidelity, meticulously grounded audiovisual narratives. Unlike previous single-pass annotation, OCC employs a "divide-and-conquer" strategy through three collaborative modules, as shown in Fig. 2.

The Observer serves as the primary perception engine, functioning as an all-knowing sensor that performs a global analysis of the audiovisual stream. Its objective is to produce a raw, high-density description.The Observer is constrained by a strict mandate: (1) Audio Content and Details: Identify all sound events (speech, foley, ambient noise, background music) and their physical characteristics (loudness, frequency modulation, and timbre). (2) Visual Content and Details: Describe the scene, entities (appearance, actions), objects, lighting, and color palettes. (3) Motion and Cinematography: Catalog all subject movements (e.g., trajectories, gestures) and cinematographic techniques, including camera motion (pan, tilt, zoom) and shot transitions (cuts, fades). (4) Aesthetic Style and Atmosphere: Characterize the overall stylistic tone (e.g., cinematic, documentary, handheld) and the auditory atmosphere (e.g., tense, reverberant, industrial). (5) Precise Temporal Anchoring: All events must be localized with sub-second timestamps (e.g., [01.2s - 01.5s]) to ensure strict alignment with the raw signal. (6) Explicit Multimodal Binding: Avoiding vague pronouns in favor of entity-specific attribution (e.g., grounding a specific segment of speech to its corresponding speaker among multiple candidates in a scene). (7) High-Density Description: Prioritize a comprehensive inventory of atomic facts over abstract summaries. We employ Gemini-3-Pro, a formidable MLLM, as the Observer to navigate the inherent intricacies of global audiovisual analysis. To guarantee factual integrity, we introduce a Tripartite Forensic Checker consisting of three independent, specialized sub-systems. This decoupled verification ensures that unimodal errors do not propagate into cross-modal misalignments.

• Audio Forensic Checker: This module performs an independent audit of the audio track to verify auditory hallucinations and identify omissions in granular sound details. Focusing on Verbal Fidelity and Linguistic Integrity, it strictly penalizes phonetic discrepancies and flags "phantom sounds" that lack an acoustic basis. Beyond content validation, the checker scrutinizes fine-grained attributes, such as timbre characteristics, ambient interference, and rhythmic prosody, while rigorously ensuring the temporal coherence of acoustic events to maintain precise onset and offset alignment. We employ Gemini-3-Pro as the audio checker to ensure the factual integrity and perceptual accuracy of the auditory information.

• Visual Forensic Checker: Acting on silent video tracks, this module is engineered to audit visual hallucinations and recover omitted fine-grained details. It prioritizes the validation of entity dynamics and spatio-temporal logic, meticulously scrutinizing micro-expressions and motion trajectories to ensure the narrative aligns with the physical causality of the visual scene. Furthermore, it supplements any overlooked granular nuances while rigorously enforcing the temporal coherence of visual events, ensuring that the chronological flow of actions remains logically sound and consistent. We observe that Gemini-3-Pro introduces a significant amount of hallucinations when processing silent video streams. Therefore, we employ the powerful silent video understanding model, Doubao-1.8, as the Visual Checker, which possesses more accurate and fine-grained visual understanding capabilities.

• Audio-Visual Clip Checker: This is the core module for evaluating audio-visual cross-modal binding and temporal coherence. It distinguishes between diegetic sounds (on-screen), off-screen sounds, and non-diegetic voice-overs. It specifically targets "lip-sync" discrepancies and ensures that the narrative preserves the chronological sequence of cross-modal interactions. To achieve this, the module employs a sliding-window strategy, partitioning the video into 2-second clips that are processed iteratively to generate detailed verification logs. Notably, each input segment is augmented with its sequence index and corresponding timestamps, facilitating superior calibration of temporal coherence. Given the intensive computational demands of this granular analysis, we utilize Gemini-3-Flash as the underlying engine for the Clip Checker, balancing high-fidelity reasoning with operational efficiency.

The Corrector acts as the final synthesis agent. It receives the initial description from the Observer and the comprehensive audit reports (violation lists) from the tripartite Checkers. Its task is to perform an optimal fusion: rectifying hallucinations identified by the unimodal checkers and resolving alignment conflicts flagged by the audiovisual checker. Through this iterative feedback loop, the Corrector produces a final caption that is both semantically rich and structurally aligned with the raw sensor data. We employ Gemini-3-Pro as the refinement corrector.

As detailed in Table 1, processing a 10-second video consumes approximately 22,000 tokens (both input and output) across the OCC pipeline. The A/V-Checker accounts for the majority of this cost due to its iterative processing of the entire audio-visual sequence.

3.3 High-Density Human Interaction Dataset↩︎

Existing video datasets[89], [90], Farré2024FineVideo? often feature sparse audiovisual events, rendering them insufficient for training models on complex alignment tasks. To address this gap, we curate the High-Density Human Interaction (HDI) Dataset, comprising 3,500 meticulously selected clips sourced from YouTube and cinematic content. The HDI dataset is specifically filtered for "hard cases," such as multi-speaker dialogues in noisy environments, rapid physical actions coupled with impact sounds, and subtle cross-modal causal chains. Each clip is processed through our full OCC pipeline, yielding high-fidelity annotations that provide the necessary supervision for the TCA-Captioner to transition from coarse scene-level understanding to fine-grained, synchronized audiovisual perception. Furthermore, to enhance the diversity of the training distribution, we incorporate an additional 5,000 samples from FineVideoFarré2024FineVideo? and 7,000 samples from TikTok-10M[89]. Given the extended duration and relatively straightforward content of these videos, we streamlined the annotation process by bypassing the Clip Checker module within the OCC framework. Leveraging this comprehensive multi-source corpus, we perform LoRA fine-tuning on the Qwen3-Omni[35] backbone to develop our final TCA-Captioner.

4 TCA-Bench↩︎

4.1 Motivation↩︎

While recent audiovisual captioning benchmarks have advanced the measurement of detailed perception, they fundamentally assess only the static presence of individual details, not the dynamic interplay between modalities. Video-SALMONN-2 [38] decomposes videos into flat atomic events to compute missing and hallucination rates; UGC-VideoCap [39] assigns a single holistic LLM score across entangled dimensions; Omni-Cloze [91] transforms evaluation into cloze-style multiple-choice questions to penalize hallucinations on isolated attributes. A common structural limitation underlies all three: they treat multimodal content as an unordered bag of independent facts, ignoring the relational structure that connects modalities. Consequently, none of these paradigms can systematically evaluate whether a model correctly binds a sound to its visual source or preserves the temporal ordering of cross-modal events, two capabilities we identify as the primary failure modes of state-of-the-art models. Table 2 contrasts these benchmarks with TCA-Bench. To bridge this gap, TCA-Bench introduces a Decoupled Evaluation Protocol, specifically designed to disentangle foundational unimodal perception from higher-order relational reasoning.

4.2 Benchmark Construction↩︎

4.2.0.1 Data curation.

TCA-Bench targets videos with dense audio-visual interplay, specifically multi-person dialogues coupled with complex physical actions (e.g., variety shows, group interactions)—sourced from YouTube. Such scenes produce frequent, entangled cross-modal events that stress-test audiovisual binding and temporal relational reasoning. We focus on short clips (5–30 seconds), as these densely interleaved segments already pose substantial challenges for current models. As illustrated in Fig. 3, the construction pipeline proceeds in two macro-stages: (I) automated clip curation via multi-modal temporal analysis, and (II) structured annotation on a purpose-built labeler platform.

In Stage I, two parallel analysis streams operate on each source video: a visual stream uses YOLOv8 for person detection to produce a per-frame person-count timeline, while an auditory stream applies Silero VAD to generate speech-activity intervals. The two timelines are fused under a joint co-occurrence constraint, retaining only intervals where multiple persons are visually present and speech is simultaneously active, followed by quality filtering on person-detection ratio and speech overlap. Human annotators then perform a final curation pass to ensure sufficient density and diversity of cross-modal interactions. In Stage II, each curated clip enters a purpose-built web-based labeler platform where all three annotation layers are produced through a unified LLM-draft-then-human-validation paradigm: a multimodal LLM first generates an initial dense caption, which human annotators correct and finalize; the verified caption then serves as context for the LLM to draft binding pairs and temporal triples, each again undergoing human validation.

4.2.0.2 Structured annotation.

Unlike prior benchmarks that rely solely on flat textual ground truths, TCA-Bench formalizes cross-modal dependencies with three complementary annotation layers:

1. Ground-truth captions that exhaustively describe visual details (subjects, actions, scenes) and auditory elements (speech, sounds).

2. Audio-Visual Binding Lists — structured JSON lists of [sound, source] pairs linking each acoustic event to categories: foreground character, off-screen character, or environment.

3. Temporal Relational Lists — triples of [first_event, second_event, relation] encoding relation types: A_then_V, V_then_A, or AV_simultaneous.

These three layers are explicitly aligned with the evaluation protocol: ground-truth captions support Base Perception scoring, while Binding Lists and Temporal Relational Lists serve as structured references for the cross-modal Binding Accuracy and Temporal \(F_1\) metrics, respectively. Fig. 4 illustrates the three annotation layers on a representative benchmark clip.

4.3 Decoupled Evaluation Protocol↩︎

Existing evaluation paradigms conflate uni-modal perception errors with cross-modal reasoning failures, making it impossible to diagnose where a model fails. TCA-Bench addresses this with a two-tier Decoupled Evaluation Protocol using an independent LLM judge.

4.3.0.1 Base Perception (Uni-modal).

An LLM judge compares the candidate against the GT caption on a 0–10 scale:

• Audio: Transcription Accuracy—verbatim fidelity of speech content including meaning-critical words and fillers; Tone & Emotion—accuracy of the described emotional delivery style and non-speech audio such as music mood and ambient sounds.

• Visual: Subject & Action—correctness of primary subjects’ appearance, quantity, spatial relationships, and sequential movements; Scene & Atmosphere—fidelity of the setting, background layout, spatiotemporal attributes, and visual mood; Cinematography—accuracy of shot size, camera angles, and camera movement patterns.

4.3.0.2 Audiovisual Integration (Cross-Modal).

This track uses structured GT annotations for targeted checklist evaluation:

• Binding Accuracy: For each GT binding pair, the judge performs a two-stage verification: first confirming that the acoustic event is present in the candidate caption, then checking whether it is attributed to the correct source. A sound is scored as correct only when both stages pass. Crucially, this decoupled design ensures that transcription errors (e.g., mishearing speech content) do not penalize binding scores—such errors are isolated in the Base Perception tier. We report accuracy separately for Character (foreground + off-screen) and Environment sounds.

• Temporal \(F_1\): For each triple in the GT temporal list, the judge assigns a three-way verdict: correct (both events present with a consistent temporal link), incorrect (both events present but the temporal relationship is clearly reversed or contradicted), or skipped (one or both events absent). This three-way protocol decouples temporal accuracy from event coverage: omitting an event yields a skipped verdict rather than a false temporal error. We compute:

\[\text{Prec} = \frac{|\text{correct}|}{|\text{correct}| + |\text{incorrect}|}, \quad \text{Cov} = \frac{|\text{correct}| + |\text{incorrect}|}{|\text{GT}|}\]

\[F_1 = \frac{2 \cdot \text{Prec} \cdot \text{Cov}}{\text{Prec} + \text{Cov}}\]

We report \(F_1\) separately for Sequential and Simultaneous relations.

5 Experiment↩︎

5.1 Caption Evaluation on Public Benchmarks↩︎

We first evaluate the audiovisual captioning proficiency of TCA-Captioner across two public benchmarks: the Video-SALMONN-2 [38] and the UGC-VideoCap [39] benchmark. To ensure a fair and consistent comparison, we strictly adhere to the evaluation protocol established by AVoCaDO[37], employing GPT-4.1 as the judge model for Video-SALMONN-2 and GPT-4o for UGC-VideoCap. Our model is benchmarked against a comprehensive suite of representative baselines, including prominent open-source models, such as HumanOmniV2[94], ARC-Hunyuan-Video[95], MiniCPM-o-2.6[96], Qwen2.5-Omni[50], Qwen3-Omni[35], video-SALMONN-2[38], UGC-Captioner[39], and the latest AVoCaDO[37], as well as the powerful closed-source Gemini series [36]. As illustrated in Table 3, TCA-Captioner achieves state-of-the-art (SOTA) performance among all open-source models and demonstrates formidable competitiveness even when positioned against top-tier closed-source systems.

5.2 Caption Evaluation on Our TCA-Bench↩︎

We conduct extensive experiments on our proposed TCA-Bench to evaluate the multi-modal proficiency of TCA-Captioner against a broad spectrum of models, including the closed-source Gemini series (v2.5 and v3.0) and various leading open-source MLLMs. The performance is measured across four primary dimensions: Audio (Transcription, Tone \(\&\) Emotion), Visual (Subject \(\&\) Action, Scene \(\&\) Atmosphere, Cinematography), Audiovisual (AV) Binding (Character, Environment), and AV Temporal (Sequential, Simultaneous). The experimental results in Table 4 reveal several key insights:

: A significant performance disparity exists between existing open-source models and the closed-source Gemini series. This gap is particularly pronounced in the AV Binding and AV Temporal tracks, where open-source models struggle to accurately associate acoustic events with their corresponding visual sources or maintain the logical chronological order of cross-modal interactions.

: Gemini-3.0-Pro demonstrates substantial improvements in Audio (70.2 avg.), AV Binding (76.8 total) and AV Temporal (80.0 total) compared to its predecessor, Gemini-2.5-Pro (65.3, 74.3 and 78.0, respectively). However, a performance degradation is observed in the Visual dimension, where its average score drops from 66.2 to 60.9, suggesting a trade-off in its visual perception tuning.

: Our model consistently outperforms all existing open-source models across both the average and total metrics within the Audio, Visual, AV Binding, and AV Temporal dimensions. Notably, in the AV Binding (73.2) and AV Temporal (76.9) categories, TCA-Captioner exhibits a profound understanding of cross-modal interactions, nearly doubling the performance of the baseline Qwen3-Omni.

: Compared to the Gemini family, TCA-Captioner surpasses Gemini-3.0-Flash in every evaluated dimension. Furthermore, our model achieves a superior Video average score (63.6) compared to Gemini-3.0-Pro (60.9). This advantage is primarily attributed to our specialized Visual Checker module, which leverages the high-fidelity perception of Doubao-1.8 to rectify hallucinations and supplement granular visual details that generic models often overlook.

5.3 Exploring Another Base Models and DPO↩︎

As shown in Table 5, we conduct additional experiments based on Qwen2.5-Omni, which similarly achieves competitive performance. Furthermore, we construct 2,000 DPO preference pairs based on our OCC pipeline. As demonstrated in Table 5, applying DPO to our SFT captioner significantly enhances performance across all dimensions. This validates the high quality of the OCC refinement data and proves its strong potential for automated preference alignment.

5.4 Analysis for Our OCC Framework↩︎

Since the generation of LLM drafts in our benchmark construction utilized the OCC framework, a direct comparison within the full TCA-Bench would introduce inherent bias. To conduct a more rigorous and equitable ablation study of the OCC framework’s contributions, we constructed an additional mini-bench comprising 20 samples. These samples utilized Gemini-2.5-Pro for initial draft generation and underwent stringent human auditing and manual refinement to ensure the same structural integrity as the main TCA-Bench. As illustrated in Figure 5, we compared the OCC framework against Gemini-3.0-Pro. The results demonstrate significant improvements across all evaluated dimensions. The visual total score increased from 77.7 to 84.4, underscoring the critical role of the Visual Checker in refining visual details. The audio total score rose from 78.5 to 92.0, highlighting the substantial impact of the Audio Checker in rectifying auditory content. While AV Temporal alignment saw a robust 9.9 improvement, the gain in AV Binding was more modest at 5.5. This discrepancy suggests that for complex multi-person dialogue cases, severe hallucinations, particularly regarding the attribution of ambient sounds, persist even when the video is segmented into finer clips. Moreover, Table 6 presents an ablation study where models are finetuned on data generated by specific OCC components. The results confirm that the Audio and Visual Checkers significantly boost their respective scores, while the A/V-Clip-Checker markedly enhances AV Binding and Temporal capabilities.

6 Ethical Statement↩︎

Our dataset and benchmark comprise publicly available YouTube videos (e.g., variety shows and films) paired with precise timestamps and fine-grained annotations. The suite will be released under a CC BY-NC-SA 4.0 license, strictly restricted to non-commercial research and explicitly prohibiting biometric or surveillance applications. Furthermore, we remain committed to privacy rights; any featured individual may request content removal, which we will fulfill by promptly excluding the relevant segments from future releases.

7 Conclusion↩︎

In this work, we present TCA-Captioner, a specialized framework designed to bridge the gap in audiovisual video captioning. By integrating the iterative OCC framework and training on a curated high-density interaction dataset, our approach effectively alleviates modality detachment and temporal incoherence. Furthermore, the introduction of TCA-Bench enables a granular diagnostic evaluation of cross-modal interaction. Experimental results confirm that TCA-Captioner significantly advances the state-of-the-art in generating precise, temporally-aligned, and contextually rich audiovisual narratives. Acknowledgments. This work was supported by Gusu Innovation and Entrepreneur Leading Talents: No. ZXL2024362, National Natural Science Foundation of China : No. 62406135, Natural Science Foundation of Jiangsu Province: BK20241198, and Meituan.

References↩︎