1 Introduction↩︎

In recent years, the rapid evolution of multimodal large language models (MLLMs) has significantly advanced long-form video understanding. However, deciphering long-form TV dramas remains a formidable challenge, particularly regarding storyline that hinge on the complex interactions and evolving relationships of a large cast. A foundational requirement for this task is speaker recognition (SR), the subtask of attributing each utterance to its respective character. While a vast corpus of research exists for speaker diarization and verification, existing literature rarely addresses the unique complexities of TV dramas. In such settings, an intelligent agent must synthesize disparate multimodal cues, including visual frames, acoustic signals, and linguistic context, to achieve accurate attribution.

To address this gap, we introduce DramaSR-532K, a large-scale benchmark curated from 13 long-form TV series, some of which span multiple seasons. Within DramaSR-532K, SR is formulated as an open-set classification problem: characters are defined either by proper names for primary roles (e.g., Alice, Bob) or by descriptive attributes for secondary roles (e.g., Man in Black, Female Doctor). Our annotation pipeline consists of two stages. First, we extracted hard-rendered subtitles to define utterance boundaries and computed corresponding acoustic biometrics (voiceprints). Leveraging face recognition results, we employed data clustering and affinity propagation to generate initial pseudo-labels. Second, we conducted a rigorous human-in-the-loop revision process to refine these labels and resolve ambiguous cases. The resulting dataset comprises over 532K utterances across more than 900 primary and more than 6.6K secondary roles, establishing DramaSR-532K as the most comprehensive benchmark to date for speaker recognition in movies or TV dramas (see Table 1).

To provide a robust solution for this benchmark, we propose DramaSR-LRM, a Large Reasoning Model (LRM) architected to utilize three specialized tools: (a) \(\mathtt{voice\_sim}\): an acoustic utility that computes cosine similarity between the target voiceprints and known character embeddings; (b) \(\mathtt{video\_cap}\): a video captioning module that extracts visual context of the utterance; (c) \(\mathtt{char\_relation}\): a relational database that assists parsing social hierarchies and appellations (e.g., ‘Mom’ and ‘Darling’) within the dialogue. To train DramaSR-LRM, we leveraged a single drama comprising 50K utterances, and utilized Gemini-3-Pro [5] to generate high-quality supervised fine-tuning (SFT) trajectories. We fine-tuned a Qwen3-8B backbone [6] using a pipeline of SFT followed by reinforcement learning (RL) to optimize reasoning consistency.

Expect for 2 dramas leveraged for training process, evaluation across the remaining 11 dramas demonstrates that DramaSR-LRM elevates the accuracy of the label propagation baseline from 85.49% to 87.79% (a 2.30% gain). These gains are particularly pronounced in hard scenarios, in particular, short (a 3.33% gain) and very short (a 9.20% gain) utterances, as well as dramas with a relatively low baseline (e.g., Lost: a 5.16% gain, and Qin Empire 2: a 4.06% gain). Our work underscores the critical role of speaker recognition in long-form video understanding and provides a scalable framework for future research, which can potentially address more challenges in our benchmark, e.g., open-world speaker description and end-to-end speaker recognition.

2 Related Work↩︎

Speaker Recognition aims to identify a speaker’s identity through vocal utterances [7], [8], serving as a fundamental component in diverse audio-visual understanding pipelines. Our study is highly related to three established research domains.
(a) Speaker Diarization (SD) involves partitioning an audio or video stream into homogeneous segments attributed to specific individuals (i.e., ‘who spoke when’) [9]. While traditional SD focused primarily on audio-only inputs [1], the field has transitioned toward multimodal frameworks [3], [4] and joint diarization-recognition models in the era of deep learning [10]–[12]. However, significant gaps remain. First, existing SD benchmarks typically involve a limited number of speakers (e.g., 10–25 in AVA-AVD [3]), whereas a single TV drama may feature over 100 named characters alongside numerous transient roles. Second, since our work leverages timestamped dialogue transcriptions, the challenge shifts from temporal segmentation to character attribution in complex narrative contexts.
(b) Speaker Verification (SV) requires determining whether a given utterance matches a target identity [13], [14]. Prominent benchmarks like VoxCeleb [15], [16] and CN-Celeb [17] have scaled this task using millions of web-sourced utterances, while others like 3D-Speaker [18] explored cross-device and cross-dialect robustness. While acoustic features derived from state-of-the-art SV models [19], [20] provide a strong foundation for our work, TV dramas introduce unique difficulties: multiple characters often interact within the same acoustic environment, leading to overlapping speech and varying recording conditions that go beyond standard verification settings.
(c) Active Speaker Detection (ASD), in the visual domain, focuses on identifying which visible characters are currently speaking by correlating facial movements with audio signals. Established datasets such as Columbia [21] and AVA-ActiveSpeaker [22] have driven the development of unified audio-visual architectures [23], [24] and temporal modeling techniques [25], [26]. While ASD identifies the presence of a speaker in a frame, our task, in the context of long-form dramas, extends it by maintaining character identity across multiple scenes and seasons, even when the speaker is off-screen.
In summary, these speaker recognition paradigms are mutually reinforcing [27] and provide critical metadata for downstream applications, including dense video captioning [28], speech-driven video generation [29], and semantic video editing [30].

Long-form video understanding has gained significant traction within the research community. While several benchmarks for hour-long video QA have recently emerged (e.g., EgoSchema [31], LongVideoBench [32], and Video-MME [33]), efforts toward long-form TV dramas remain relatively scarce. In narrative-driven media, speaker recognition is a pivotal task, as establishing ‘who speaks what’ is essential for deciphering intricate storylines and character dynamics. Our study includes preliminary evaluations demonstrating that robust speaker attribution is a critical prerequisite for holistic drama understanding.

Large language models (LLMs) [5], [34], [35] have demonstrated versatile capabilities across a wide range of AI applications. Two critical evolutions of this technology are particularly relevant to our work: multimodal LLMs (MLLMs) [36]–[40], which integrate visual perception with linguistic reasoning, and large reasoning models (LRMs) [6], [41], [42], which utilize extended inference-time processing to solve complex problems. In our framework, an MLLM is employed for fine-grained video clip description, while an LRM is specifically trained to synthesize disparate multimodal cues for speaker recognition. It is related to [43], which applied an LLM for speaker diarization post-processing.

The optimization of our LRM relies on reinforcement learning (RL). We build upon classical algorithms like PPO [44] and more recent advancements such as GRPO [45], which enhances policy optimization efficiency. While recent literature has explored RL-based policy refinement for both LLMs [46]–[48] and MLLMs [49], [50], its application to narrative speaker identification remains underexplored. Notably, while SpeakerLM [12] and D-ORCA [51] utilize MLLMs for speaker identification, our approach is distinct in its use of a reasoning-centric model to explicitly aggregate multimodal evidence in complex scenarios.

3 The DramaSR-532K Benchmark↩︎

A conceptual illustration of the benchmark, including our data preparation and annotation pipeline, is shown in Figure 1. Formally, processing a long-form TV drama (which typically encompasses dozens of hours of video) requires a multi-stage pipeline: temporal speech segmentation, character discovery (indexing the cast), and ultimately, character attribution. To isolate the challenges of high-level reasoning, we adopt a simplified setting by assuming that: (1) utterances are pre-segmented and synchronized with their corresponding dialogue transcriptions, and (2) a comprehensive candidate character list is provided a priori. Thus, the primary objective is to accurately attribute each utterance to its respective character from the candidate set.

Despite these boundary conditions, the task remains inherently complex. As depicted in Figure 1, several ‘corner cases’ frequently arise: (a) short utterances provide insufficient signal for stable voiceprint extraction, (b) environmental complexity: in multi-character scenarios, background noise and overlapping speech introduce significant signal interference, and (c) visual absence: the speaking character may be off-screen or occluded, raising difficulties for visual-based identification. These challenges necessitate the development of a sophisticated reasoning model (see Section 4) capable of synthesizing disparate multimodal cues, including character detection (primarily via facial recognition), the alignment of acoustic biometrics (voiceprints), and the logical parsing of dialogue context, to achieve high-fidelity recognition.

3.1 Data Preparation↩︎

Our benchmark is established upon 13 popular long-form TV dramas (3 English and 10 Chinese productions), encompassing a total duration of 525 hours. We denote each raw video data as \(\mathsf{V}\) and the total temporal duration as \(T\). The data preparation pipeline consists of two primary phases: transcript extraction and character library construction.

Transcript extraction. We first deploy an OCR model (i.e., PaddleOCR [52], [53]) to extract hard-rendered subtitle text directly from the video stream. To ensure linguistic and temporal precision, we deduplicate the results and utilize an MLLM (i.e., Qwen2.5VL [40]) to refine the transcriptions by analyzing corresponding keyframes. For approximately 1% of the data identified as anomalous, where the ratio of temporal duration to text length deviates significantly from normal rates, we conduct manual verification and correction. This process yielded a final corpus denoted as \(\mathcal{S}=\{(\mathbf{t}_n,a_n,b_n)\}_{n=1}^N\), where each \(\mathbf{t}_n\) denotes the transcription text of utterance \(\mathbf{u}_n\), and \(a_n<b_n\in(0,T)\) represent the start and end timestamps, respectively.

Character library construction. To define the candidate speaker set, we first extract actor-role mappings from the series’ credits using OCR (supplemented by manual verification) and cross-referenced with public web metadata. A character (facial) library is initialized with professional portraits of the identified cast members, and iteratively expanded by scanning the entire drama at 1 FPS to collect diverse, in-situ facial samples. Then, we perform frame-wise facial detection and recognition across the dataset. Consequently, we obtain collection of \(M\) visual instances \(\mathcal{F}=\{(\mathrm{id}_m,c_m)\}_{m=1}^M\), where \(\mathrm{id}_m\in\{1,2,\ldots,P,\ldots,P+U\}\) identifies the character ID from a pool of \(P\) identified individuals (where \(\mathrm{id}\leqslant P\)) and \(U\) ancillary (unrecognized or background) individuals, and \(c_m\) denotes the timestamp of the facial instance.

Task formulation. Formally, the objective of speaker recognition is to produce \(\mathcal{R}=\{r_n\}_{n=1}^N\), where \(r_n\) is the character assignment for utterance \(\mathbf{u}_n\), denoted as \(r_n\in\{0,1,\ldots,P\}\), where \(r_n>0\) corresponds to a specific character ID from the library, and \(r_n=0\) serves as the null class, indicating that the speaker is an ancillary character. For further technical details of data preparation, please refer to Appendix 7.1.

3.2 Speaker Label Annotation↩︎

To establish a high-fidelity ground truth, we conducted a human annotation procedure. To reduce the annotation overhead, we employed a semi-automated pipeline: an initial pass was performed using the label propagation baseline (see Section 4.2), which achieved an estimated accuracy of 90%. Human annotators were then tasked with auditing and refining these pseudo-labels. We developed an graphical user interface that synchronizes a video player with real-time dialogue contexts, acoustic biometric statistics, and candidate character profiles to maximize labeling efficiency.

Beyond standard character attribution, we established protocols for three special taxonomies. (a) When a speaker is not present in the existing character library, labelers infer the character’s name from narrative context or assign a descriptive identifier (e.g., Man in Black, Alice’s Father). (b) In instances where the speaker’s identity remains ambiguous despite multimodal evidence, a special \(\mathtt{[UNKNOWN]}\) tag is assigned to maintain dataset integrity. (c) For dialogue lines spoken by multiple characters (e.g., choral speech or rapid-fire interruptions), annotators partition the transcription into segments and assign the appropriate speaker(s) to each specific sub-unit. On average, an experienced annotator requires \(1.5\) to \(2.5\) hours to process one hour of video content. Comprehensive details regarding the quality control measures are provided in Appendix 7.3.

We evaluated the inter-annotator agreement in about 10% of data. On average, two annotators agreed on about 99.6% of subtitle lines. By estimation, the final label noise is about 0.5%, far higher than all the algorithms (10+%).

3.3 Evaluation Protocol↩︎

The efficacy of speaker recognition models is assessed by comparing predicted labels against the manually verified ground truth. For each utterance, we evaluate the prediction accuracy based on the following taxonomic logic:
(a) Single-speaker utterances. If the ground truth identifies a specific primary character, the prediction is marked correct only if it matches that character’s ID. For utterances attributed to ancillary roles in the ground truth, the prediction is considered correct if it maps to the ‘null’ class.
(b) Multi-speaker utterances. A prediction is deemed correct if it identifies any of the contributing speakers within the ground-truth set or correctly assigns the ‘null’ label.
(c) Unknown ground truth. Any prediction is considered valid to ensure these ambiguous samples do not unfairly penalize model performance.
We will show that the evaluation protocol does not change the conclusion (that our method is better than the baseline) in Section 5.2.

We report utterance-wise prediction accuracy for each drama. To provide a granular understanding of model robustness, we also evaluate performance across several challenging subsets:
(a) Temporal duration. Categorizing utterances by length, as brief speech fragments often lack stable acoustic biometrics for recognition.
(b) Character density. Categorizing utterances by the number of candidate characters to assess the model’s ability to disambiguate in crowded environments.
(c) Visual occlusion. Isolating utterances where the speaker is not visible in context, forcing the model to rely exclusively on acoustic and linguistic reasoning.

Benchmark extensibility. Beyond the core task described above, the rich annotations in DramaSR-532K support more sophisticated research frontiers. These include open-world speaker identification, where models must generate linguistic descriptions for ancillary characters, and end-to-end speaker recognition, which requires simultaneous speech segmentation, character discovery, and attribution from raw video. These extensions ensure the long-term utility of the benchmark for the evolving field of multimodal AI.

4 Methodology↩︎

4.1 The LRM-based Pipeline↩︎

The pipeline of our method is detailed in Algorithm 2. It begins with an initialization phase utilizing a label propagation algorithm, followed by an iterative refinement phase driven by a large reasoning model (LRM). During each iteration, the LRM dynamically invokes a suite of multimodal tools, including voiceprint similarity, video captioning, and character relationship, to adjudicate ambiguous cases and resolve conflicting evidence. These tool outputs are updated periodically to incorporate the refined labels from the preceding round, enabling the model to converge on a high-fidelity narrative understanding. In the following subsections, we first detail the mechanics of the label propagation baseline and the technical implementation of our multimodal toolset. Subsequently, we describe the training paradigm for the LRM, including the integration of supervised fine-tuning and reinforcement learning.

4.2 Label Propagation↩︎

Feature extraction. To initialize speaker attribution, we first extract acoustic embeddings for each dialogue line. Using the start and end timestamps, we segment the raw audio into discrete clips. Each clip is padded with a \(100\mathrm{ms}\) buffer at both boundaries; to prevent temporal overlap with adjacent segments, the buffer is truncated at the midpoint between neighboring utterances where necessary. We utilize ERes2Net [54], a pre-trained speaker verification model¹, to extract a 192D feature vector \(\mathbf{v}_n\) for the \(n\)-th utterance. These vectors are \(\ell_2\)-normalized to facilitate cosine similarity measurements within the range of \([-1,1]\).

The neighborhood assumption. Our initialization relies on a spatiotemporal neighborhood assumption: the visual presence of a character suggests a possibility of that character speaking within a temporal window of \(\tau=30\mathrm{s}\)². Formally, for each character \(p\in\{1,2,\ldots,P\}\), we define the facial instance set \(\mathcal{F}_p=\{(\mathrm{id}_m,c_m)\in\mathcal{F}\mid\mathrm{id}_m=p\}\). We then identify the candidate utterance set \(\mathcal{S}_p\) for character \(p\) as \(\mathcal{S}_p=\{(\mathbf{v}_n,a_n,b_n)\in\mathcal{S}\mid\exists m,(\mathrm{id}_m,c_m)\in\mathcal{F}_p\wedge a_n-\tau\leqslant c_m\leqslant b_n+\tau\}\).

Seed cluster generation. Within each set \(\mathcal{S}_p\), dense clusters in the acoustic embedding space typically correspond to the target character. However, to mitigate false attributions, particularly for secondary characters who frequently co-occur with primary characters, we implement a greedy step-wise clustering procedure. In each step, we identify the maximum cluster across all \(\{\mathcal{S}_p\}\) and assign it to the respective character. Subsequent clusters exhibiting excessive similarity to previously labeled sets are discarded to maintain purity. This process continues until each known character is associated with at least one seed voiceprint set³. The \(p\)-th character’s seed set is denoted as \(\mathcal{V}_p\).

Affinity propagation. We subsequently apply the neighborhood assumption in reverse: for any given utterance, the candidate speaker pool is restricted to characters whose faces appear within the \(\pm\tau\) window⁴. The affinity between an utterance \(\mathbf{v}_n\) and character \(p\), denoted \(k_{n,p}\), is calculated as the mean cosine similarity of the top-\(N_p'\) most similar samples in \(\mathcal{V}_p\), where \(N_p'=|\mathcal{V}_p|^{0.4}\) is an empirically derived scaling factor. We employ an iterative refinement strategy with a monotonically decreasing similarity threshold \(\theta\); utterances exceeding \(\theta\) are attributed to the matching character and integrated into the respective seed set. The process terminates when \(\theta\) reaches a lower bound (e.g., \(0.4\)), at which point any unassigned utterances are labeled as \(\mathtt{[UNKNOWN]}\).

For further technical details of the label propagation algorithm, please refer to Appendix 7.3.

4.3 Multimodal Toolset↩︎

Voiceprint similarity. We compute a similarity matrix \(\mathbf{K}\in\mathbb{R}^{N\times P}\), where each entry \(k_{n,p}\) represents the similarity between the \(n\)-th utterance and the \(p\)-th character’s voiceprint set. This value is computed using the mean top-\(L\) cosine similarity metric established in our label propagation framework (see Section 4.2).

Hierarchical video captioning. We employ a hierarchical captioning strategy. The video is first partitioned into shots using the PySceneDetect library and subsequently aggregated into clips of \(10\)–\(15\) seconds. Given the scale of our data (e.g., a \(40\)-hour drama yields over \(10^4\) clips), we process these units in two stages. (a) Base-level captioning: we extract keyframes from each clip and utilize the Qwen3-VL-32B model [56] to generate dense visual descriptions. (b) High-level summarization: We group base clips into semantically cohesive segments (averaging 10 clips per segment) and then utilize the Qwen3-32B model [6] to synthesize the clip-level descriptions into a high-level segment summary. Each utterance is thus associated with a dual-context representation: a (detailed) local clip caption and a (brief) global segment summary. Further details are provided in Appendix 7.4.

Character relationship. To leverage social context and appellations found in dialogue, we maintain a dynamic relational ontology \(\mathcal{X}\). By processing the dialogue transcript alongside current speaker attributions, we utilize Qwen3-32B to extract a set of character triplets \((p_1,p_2,\text{relation})\). Here, \(\text{relation}\) is an open-domain descriptor (e.g., father, colleague, friend) defining the relationship of character \(p_2\) to \(p_1\). Further details are provided in Appendix 7.4.

4.4 Training the Large Reasoning Model↩︎

The primary objective of the large reasoning model (LRM) is to iteratively refine the initial speaker attributions generated by the label propagation stage. The model is provided with a contextual window of \(20\)–\(30\) dialogue lines and is tasked with synthesizing multimodal evidence to produce a final, high-fidelity attribution. A typical chain-of-thought (CoT) trajectory is illustrated in Figure 3. We adopt a two-stage training paradigm consisting of supervised fine-tuning (SFT) followed by reinforcement learning (RL).

Data curation. To establish the model’s fundamental tool-use and reasoning capabilities, we curate an SFT dataset using Gemini-3-Pro [5] as a teacher model. For a given training drama, the teacher model is supplied with the dialogue transcripts and the outputs of the multimodal toolset (\(\mathtt{voice\_sim}\), \(\mathtt{video\_cap}\) [both brief and detailed], and \(\mathtt{char\_relation}\)). Gemini-3-Pro is prompted to generate rationales that justify correct attributions by cross-referencing these cues (e.g., ‘The voiceprint similarity for Alice is high (0.75)’, and ‘the caption identifies a female speaker in a lab coat, consistent with Alice’s occupation’). To ensure the veracity of the CoT trajectories, we employ a feedback-driven distillation process: if the teacher model produces an incorrect prediction, it is re-prompted with the ground-truth label and instructed to synthesize a refined rationale that logically aligns with the correct answer. Both prompts are detailed in Appendix 7.5.

Since the majority of utterances are easy cases (i.e., one character possesses a dominant acoustic similarity), simple classification would not necessitate complex reasoning. To prevent the training set from being overwhelmed by trivial samples, we select a strategic subset (approximately 20% of total utterances) that involves (a) failure recovery: all utterances incorrectly attributed by the label propagation baseline, (b) ambiguity resolution: borderline samples where the margin between the top-1 and top-2 acoustic similarity scores is less than \(0.03\)⁵, and (c) balanced control: a randomly sampled set of clearly correct utterances to balance the dataset.

SFT training. We fine-tune a Qwen3-8B model [6] on all the curated trajectories, training it to autonomously invoke tools and generate structured reasoning. Another set of samples (with ground-truth labels, but not necessarily with curated trajectories) are reserved for the reinforcement learning phase.

RL post-training. To optimize decision-making under multimodal uncertainty, we employ Group Relative Policy Optimization (GRPO) [45]. GRPO enhances training efficiency by normalizing rewards across a sampled group of outputs, bypassing the need for a separate value model. Our reward function \(R\) is defined by two primary signals, (a) accuracy reward (\(r_\mathrm{acc}\)): a binary reward of \(+1\) for matching the ground-truth character ID \(p_n\), and \(0\) otherwise, and (b) format reward (\(r_\mathrm{fmt}\)): a penalty-based reward to ensure strict adherence to the structured template.

Iterative inference. At test time, the LRM operates on the initial labels established during propagation. We employ an iterative refinement loop: if the LRM modifies the attribution for a significant portion of the window, the updated labels are fed back into the context for a subsequent pass. Within this loop, the dynamic tools (\(\mathtt{voice\_sim}\) and \(\mathtt{char\_relation}\)) are re-computed based on the updated pseudo-labels, while the static, computationally intensive \(\mathtt{video\_cap}\) data remains constant. This allows the model to progressively resolve complex multi-character dependencies across the drama.

5 Experiments↩︎

5.1 Experimental Setup↩︎

Datasets and setting. We partition our benchmark to facilitate a rigorous cross-drama evaluation. SFT is performed on approximately 10K CoT trajectories from A Lifelong Journey, while RL utilizes 10K labeled utterances from Empresses in the Palace. The model’s generalization capabilities are evaluated on the remaining 11 held-out dramas, totaling 428K utterances. These test dramas represent a diverse spectrum of genres (see Table 4 in Appendix 7.1), production styles, and acoustic environments. To initialize the label propagation baseline, we provide a sparse seed set containing ground-truth labels for approximately 1% of the utterances for each known character (with a minimum of 1 utterance per character).

Implementation details. Our DramaSR-LRM is built upon the Qwen3-8B backbone [6]. The model undergoes 3 epochs of SFT followed by 2 epochs of RL. For the RL phase, we employ GRPO [45] with a group size of \(G=8\) and a KL-divergence penalty coefficient of \(\beta=0.0001\). The training was conducted on an 8-node NVIDIA H800 GPU server, requiring approximately 40 hours for completion.

Confidence sampling. While our training focuses on ‘hard’ cases, where top-1 and top-2 acoustic similarities are close, the majority of the test set consists of ‘easy’ samples. To mitigate the risk of LLM hallucinations on these easy cases, we only invoke the LRM for utterances where the margin between the top-1 and top-2 acoustic scores is less than a threshold \(\rho\). This selective reasoning not only improves overall accuracy, but also significantly reduces the computational burden by bypassing LLM on about 80% of test data (for \(\rho=0.10\)).

Computational overhead. Inference was benchmarked on a single NVIDIA H800 GPU. Acoustic feature extraction is highly efficient, requiring less than \(0.01\)s per utterance. The label propagation baseline, executed on a CPU, takes \(2\)–\(3\) minutes for a drama of 50K utterances. For the LRM, we utilize the vLLM framework to support \(256\) concurrent threads. This setup achieves an amortized inference time of about \(0.33\) GPU-seconds per utterance; in other words, an 8-GPU server can finish 50K subtitles within \(40\) minutes.

5.2 Main Results↩︎

Table 2 summarizes the speaker recognition accuracy across the benchmark test suite. DramaSR-LRM achieves a substantial performance breakthrough over the competitive label propagation baseline, increasing the mean accuracy from 85.49% to 87.79% (an absolute gain of +2.30%, representing a 16% reduction in the relative error rate). We trained five DramaSR-LRM models individually, and the standard deviation of mean accuracy is 0.39%, much smaller than the 2.30% gain. These results underscore that while acoustic features are foundational, the LRM’s capacity to synthesize visual and relational evidence is crucial for resolving identity ambiguity in complex narrative contexts.

Disclaimer: the impact of the evaluation protocol. As said in Section 3.3, we applied special treatment for the multi-speaker lines and \(\mathtt{[UNKNOWN]}\) tags. While this protocol is not final, it does not impact the advantage of DramaSR-LRM over the LP baseline. Specifically: (a) The [UNKNOWN] tag appears rarely, occupying <100 out of 532K (<0.02%) utterances. Excluding these from evaluation reduces both label propagation (LP) and LRM accuracy by only 0.002%. (b) Multi-speaker lines occupy  2K out of 532K (<0.4%) utterances. Since current methods (LP and LRM) treat each line as an inseparable unit, requiring an exact match causes all methods to fail, reducing overall accuracy by 0.34% and 0.35%, respectively. In any choice of the protocol, DramaSR-LRM outperforms LP by 2.3%. We agree that distinguishing multiple speakers is important. In the future, we will train the LRM to detect multi-speaker lines and use an audio separation tool for voiceprint extraction. We can easily build training data by merging adjacent single-speaker lines.

Comparison to pyannote. Furthermore, we benchmark our approach against pyannote [57], a state-of-the-art speaker diarization framework. We map diarization clusters to character IDs using the Hungarian matching algorithm; however, the results indicate that standard diarization algorithms struggle with the long-range temporal dependencies and high character density characteristic of TV dramas. This confirms that the proposed pipeline is perfectly suited for this domain.

Short utterances. The right-hand columns of Table 2 highlight a critical failure mode of acoustic-only methods: baseline performance degrades sharply as utterance duration drops below 1.0 or 0.5 seconds. In these ‘acoustic sparsity’ scenarios, voiceprints are often unstable or non-discriminative. In contrast, DramaSR-LRM maintains robust accuracy, partly via invoking the \(\mathtt{video\_cap}\) and \(\mathtt{char\_relation}\) tools and effectively complements the weakened acoustic signal (see ‘tool ablation’ in Section 5.3).

Environmental complexity. We group the utterances by the number of candidates (i.e., the number of characters in the \(\tau=30s\) neighborhood). In the easy (1–2 candidates, occupying 40.6% of utterances), medium (3–4, 40.8%), and hard (5+, 18.6%) subsets, DramaSR-LRM improves the recognition accuracy of the LP baseline from 84.91%, 86.12%, 85.46% to 87.98%, 87.94%, 86.07%, respectively, showing consistent gains.

Visual absence. There are about 9.6K (1.8%) off-screen utterances (i.e., GT speaker’s face not appearing in the \(\tau=30s\) neighborhood). To correctly attribute these utterances, the model must assign an \(\mathtt{[OTHERS]}\) label and then extend the candidate list with 3–5 characters sharing similar voiceprints and reasons again. DramaSR-LRM succeeds in 52.4% of such cases, much higher than the LP baseline that succeeds in merely 13.4% of cases.

Cross-genre and cross-lingual generalization. Although the SFT and RL phases utilize data from only two slow-paced Chinese dramas, DramaSR-LRM demonstrates remarkable zero-shot transferability. Significant gains are observed in fast-paced war dramas (e.g., Qin Empire 2, +4.06%) and complex crime procedurals (e.g., The Knockout, +2.50%), where character interactions are significantly more volatile. Most notably, the model transfers effectively to English-language dramas, delivering significant accuracy improvements (e.g., +2.30% on Friends and +5.14% on Lost) without specific linguistic adaptation. This suggests that the underlying reasoning patterns for character attribution are language-agnostic.

5.3 Diagnostic Experiments↩︎

Tool ablation. To study the importance of each tool, we ablate the use of tools. We inherit the model trained with the complete tool set, but in the inference stage, when a specific tool is called, the system returns a message saying ‘no information can be given’. Results are shown in Table 3. We can draw the following conclusions from the results: (i) \(\mathtt{voice_sim}\) is the most useful tool, without which the accuracy drops a lot (below the label propagation baseline, which completely relies on this tool), (ii) \(\mathtt{video_cap}\) also helps, in particular when captions include speaking actions, and (iii) \(\mathtt{char_relation}\) brings a small gain, but it offers useful clues for hard cases, mostly very short relation-calls (e.g., "Dad" or "Mom"). Additionally, although \(\mathtt{video_cap}\) and \(\mathtt{char_relation}\) contribute less to the overall accuracy, they become more important in shorter utterances, where audio features are less stable.

Impact of confidence sampling. Figure 4 illustrates the relationship between the sampling threshold \(\rho\) and recognition accuracy (full numerical results in Tables 5 and 6). Our results indicate that the optimal \(\rho\) is inversely correlated with the fidelity of the initial label propagation. For dramas with high initial accuracy (e.g., Stand by Me, baseline 93.77%), a conservative threshold (\(\rho=0.04\)) is preferable, invoking the LRM only for highly ambiguous cases. Conversely, for dramas where acoustic cues are less reliable (e.g., Lost, baseline 73.89%), a larger threshold (\(\rho=0.20\)) allows the LRM to intervene more frequently, correcting systematic errors in the propagation. This trend is also observed in subsets with ‘acoustic sparsity’ (short and very short utterances), which consistently demand higher \(\rho\) values to achieve performance gains. To provide a generalized solution across the entire benchmark, we fix \(\rho=0.1\) for all main experiments.⁶

Iterative reasoning. Since DramaSR-LRM utilizes context speaker labels as part of its reasoning input, it is inherently compatible with an iterative refinement loop as detailed in Algorithm 2. Evaluate on the drama The Long Night, the first pass improves the label propagation baseline from 88.18% to 90.15% (+1.97%), while a second pass further elevates performance to 90.40% (+0.25%). This demonstrates that the LRM can ‘self-evolve’ by re-evaluating its own previous outputs. However, to maintain a favorable trade-off between accuracy and computational latency, we report results using only a single iteration in the main experiments.

Robustness to contextual variation. DramaSR-LRM exhibits robustness to the length of the dialogue context window. We tested the model (trained on 30-line contexts) across varying window sizes, and found that the overall accuracy fluctuations do not exceed 0.2%. Furthermore, since the dialogue text represents a relatively small fraction of the total token count compared to the dense multimodal tool logs and long CoT trajectories, adjusting the window size does not impose a prohibitive computational overhead on inference time. Consequently, we employ a standardized 30-line context for all main experiments.

5.4 Downstream Utility: Video Understanding↩︎

To evaluate the cascading benefits of accurate speaker recognition on holistic video understanding, we perform downstream assessments on video captioning and video question-answering. We provide a state-of-the-art MLLM (Qwen3-VL-32B [56]) with video clips paired with two transcription variants, produced label propagation and DramaSR-LRM. An example is illustrated in Figure 13, where the integration of DramaSR-LRM’s labels improves the factual grounding of generated descriptions. This is particularly evident in resolving complex narrative queries, such as ‘who claimed what’, where the model must distinguish between subtle character motivations and conflicting dialogue. These results confirm that speaker attribution is not merely an auxiliary task, but a foundational prerequisite for deep, character-centric narrative comprehension.

To quantitatively study the question-answering (QA) accuracy, we collected 20K+ QA pairs (for plot understanding) on these 13 dramas. In the subset of 18,399 QAs (covering 11 dramas in the SR test set), we used Qwen3-VL-32B for multimodal QA, and different speaker recognition results (i.e., no labels, label propagation, DramaSR-LRM, and ground-truth) are used as auxiliary clues. The QA accuracy under these four settings are 21.6%, 70.3%, 72.0%, and 80.8%, respectively. These results not only show the importance of speaker recognition in video QA, but also demonstrate the downstream gain of our method, as well as the room for improvement (compared to GT).

6 Conclusion↩︎

In this paper, we have addressed the formidable challenge of speaker recognition within the context of long-form TV dramas. We introduced DramaSR-532K, a comprehensive benchmark comprising 532K annotated utterances across 13 distinct dramas, providing the research community with a high-fidelity resource. We proposed DramaSR-LRM, a large reasoning model that moves beyond isolated acoustic biometrics by autonomously synthesizing multimodal evidence through autonomous tool-use. Our empirical evaluations demonstrate that DramaSR-LRM significantly outperforms the label propagation baseline, particularly in the scenarios such as acoustic sparsity, secondary characters, high character density, and off-screen dialogue. Furthermore, we validated that improved speaker recognition directly translates to superior performance in downstream video understanding tasks. By releasing DramaSR-532K and our reasoning framework, we provide a foundation for future research into character-centric video analysis. In the future, we will investigate more challenging settings, including open-world speaker description and end-to-end speaker recognition, i.e., operating directly on raw audio signals rather than pre-segmented dialogue transcriptions.

Impact Statement↩︎

This work introduces DramaSR-532K, a benchmark for speaker recognition in long-form TV dramas, and DramaSR-LRM, a multimodal reasoning framework. While the primary focus of this research is to advance the state-of-the-art in narrative video understanding, we recognize several broader implications:

• Large reasoning models (LRMs). As this paper explores the use of LRMs for evidence synthesis, it contributes to the ongoing discussion regarding the reliability and transparency of AI reasoning. Our use of verifiable chain-of-thought trajectories and tool-use logs is a step toward making multimodal AI decisions more interpretable.

• Content accessibility. Precise speaker recognition is a foundational technology for generating high-quality automated closed-captioning and audio descriptions. By improving attribution in complex, multi-character scenarios, this work can contribute to more accessible media experiences for the hearing-impaired and visually-impaired communities.

• Ethical use of data. The DramaSR-532K benchmark is derived from publicly available television dramas. We have taken care to ensure that the data is used for non-commercial research purposes. However, we acknowledge that advancements in facial recognition and voiceprint analysis must be handled with caution regarding privacy. We advocate for the use of these technologies in accordance with established ethical guidelines and legal frameworks.

As a standard research contribution in video content analysis, this work does not present any immediate societal risks beyond those typically associated with the development of large-scale language, audio, and vision models.

7 Details↩︎

7.1 Data Preparation↩︎

Video Pre-processing. TV dramas used in this paper are displayed in Table 4. Each drama in our dataset comprises multiple episodes. To ensure narrative continuity, we first excise the opening and closing sequences (titles and credits) of each episode. The remaining footage is concatenated into a single, continuous video stream for each drama. This concatenation provides extended temporal context for dialogue occurring at episode boundaries, which significantly assists the DramaSR-LRM in maintaining character consistency and inferring speakers through broader narrative flow.

Transcription extraction. We extract dialogue text from hard-rendered subtitles using an OCR and refinement pipeline.

• Initial subtitle extraction. For each drama, we define a spatial bounding box encompassing the subtitle region to minimize background noise. We then utilize PaddleOCR-v4 [52], [53] for frame-by-frame recognition, yielding a raw text sequence \(\{\mathbf{t}_n\}\).

• Grouping texts into utterances. To aggregate consecutive frames into discrete dialogue lines, we compute the normalized Levenshtein similarity (a.k.a., \(1\) minus edit distance) between adjacent frames: \[\mathrm{sim}(\mathbf{t}_n,\mathbf{t}_{n+1})=1-\frac{\mathrm{Levenshtein}(\mathbf{t}_n,\mathbf{t}_{n+1})}{\max(|\mathbf{t}_n|,|\mathbf{t}_{n+1}|)}. \label{eq:subtitle95sim}\tag{1}\] Frames with a similarity score exceeding \(0.8\) are grouped into a single utterance.

• MLLM refinement. For each identified segment, the median frame is processed by Qwen3-VL-32B [56]. This step corrects OCR artifacts, resolves text segmentation for overlapping subtitles, and enforces linguistic formatting consistency.

• Quality control. We perform manual verification on approximately \(1\%\) of the corpus, specifically targeting anomalies where the text length is disproportionate to the temporal duration or where contextual logic appears fractured.

Character library construction. To achieve high-fidelity facial reference data, we implement a two-stage construction and enrichment pipeline that integrates external and in-situ visual information:

• Initial face library. Cast metadata and reference portraits are harvested from public databases (e.g., IMDB and TMDB for English dramas, Douban and BaiduBaike for Chinese dramas). We perform manual cross-verification to establish a definitive mapping between actors and their respective in-drama roles.

• In-situ library enrichment. To account for variations in lighting, aging, and makeup within the drama, we scan the entire video at 1 FPS. Using the initial library as a reference, we extract all matching facial instances. To manage the scale and quality of this data, we apply a sequential filtering pipeline:

  • Multi-face exclusion. Samples containing multiple faces are discarded to avoid identity ambiguity.

  • Deduplication. We remove redundant samples using a cosine similarity threshold of \(0.8\).

  • Quality filtering. Low-resolution or heavily occluded faces are removed based on a quality threshold of \(0.45\).

  • Quality control. From the remaining high-quality pool, we randomly sample \(15\) representative images per character to form the final reference library.

• Frame-wise scan. Finally, we utilize the InsightFace library [58] to perform frame-by-frame detection and recognition across the full duration of each drama, establishing the foundational visual evidence for speaker attribution.

7.2 Speaker Label Annotation↩︎

Human annotators are provided with three primary information streams via a specialized graphical user interface (GUI): (1) video streams with overlaid facial detection bounding boxes, (2) initial speaker hypotheses generated by the label propagation algorithm, and (3) acoustic biometric statistics (e.g., voiceprint similarity scores). Annotators are instructed to watch the videos sequentially and perform corrective edits on the initial labels as necessary.

To ensure high-fidelity labeling, particularly for ancillary roles that lack pre-existing voiceprints, we implement a stringent quality control protocol. Following the initial annotation, an automated script flags high-entropy labels for second-pass human verification. A label is marked for review if it meets any of the following criteria: (a) a primary character assigned to an utterance with low acoustic similarity to their reference set, (b) a primary character speaking while off-screen, (c) an newly introduced ancillary character, or (d) an \(\mathtt{[UNKNOWN]}\) attribution.

On average, the primary annotation phase requires \(1.5\)–\(2.5\) hours per video hour, while the targeted quality-control validation requires an additional \(10\)–\(15\) minutes per video hour. This tiered approach minimizes errors in the ground truth, particularly for the difficult edge cases analyzed in Section 5.

7.3 Label Propagation↩︎

Acoustic feature extraction. We utilize the ffmpeg library to extract raw audio from the video containers at a sampling rate of \(16\mathrm{kHz}\). For each dialogue segment \(n\) in the set \(\mathcal{S}=\{(\mathbf{t}_n,a_n,b_n)\}_{n=1}^N\), we isolate the corresponding audio clip using the pydub library. Voiceprint embeddings are generated via the ERes2Net model [54] within the 3D-Speaker toolkit [59]. To ensure a stable similarity metric, all extracted feature vectors are \(\ell_2\)-normalized, mapping them to a hyperspherical space where cosine similarity is equivalent to the dot product.

Seed cluster generation. Given the candidate utterance sets \(\{\mathcal{S}_p\}\) associated with character \(p\) via the neighborhood assumption, we initialize a high-confidence clustering procedure controlled by a similarity threshold \(\theta_\mathrm{seed}\) (initially \(0.85\), a high value to guarantee that two utterances with a higher similarity belong to the same character) and a minimum cluster size \(\eta_\mathrm{seed}\). Then, we employ a two-phase greedy search strategy:

• Phase 1: Global high-confidence search. For all active characters, we construct an acoustic graph where an edge exists between two nodes (utterances) if their similarity is greater than \(\theta_\mathrm{seed}\). We identify the maximum connected component (MCC) for each character using a floodfill algorithm. The MCCs are ranked by cardinality. If the largest MCC meets the \(\eta_\mathrm{seed}\) requirement and maintains sufficient separation from existing seeds (similarity difference is greater than \(\Delta\theta_\mathrm{seed}\)), it is accepted. If no MCC is accepted, we iteratively decay \(\theta_\mathrm{seed}\) (step size \(0.01\)) until a lower bound of \(0.70\) is reached.

• Phase 2: Targeted character search. For any character \(p\) lacking a seed after Phase 1, we perform a local MCC search within \(\mathcal{S}_p\). We apply the same validation criteria against existing clusters to prevent identity collision. The threshold \(\theta_\mathrm{seed}\) is gradually relaxed until every active character is associated with one seed voiceprint set.

This process includes an optional human-in-the-loop verification to ensure the purity of the seed sets. For a typical drama involving approximately \(50\) characters, this supervised seed generation requires roughly \(2\)–\(3\) hours.

Iterative affinity propagation. Once seeds are established, we expand the attributions to the entire corpus through an iterative propagation strategy. Starting from a high threshold \(\theta_\mathrm{prop}\), we repeat a dual-pass search:

• Identity expansion. Each unlabeled utterance is compared against all existing character clusters. If the maximum similarity exceeds \(\theta_\mathrm{prop}\), the utterance is assigned to that character and integrated into the reference set, dynamically updating the cluster centroid.

• Latent identity discovery. Within a sliding window of \(20\) dialogue lines, we search for MCCs using a more stringent threshold (\(\theta_\mathrm{prop}+\Delta\theta_\mathrm{prop}\)). MCCs exceeding \(\eta_\mathrm{prop}\) in size are initialized as new ancillary-character clusters⁷. This allows the model to track consistent voices that were not captured by the initial visual-anchor sets.

We decrement \(\theta_\mathrm{prop}\) by \(0.02\) per iteration. The process terminates at \(\theta_\mathrm{prop}=0.45\). Any utterances remaining unassigned at this stage are designated with the \(\mathtt{[UNKNOWN]}\) tag, pending final adjudication by the LRM.

7.4 Multimodal Toolset↩︎

Clip segmentation via adaptive merging. We utilize the PySceneDetect library to perform initial content-aware shot segmentation. Using an adaptive thresholding algorithm that monitors visual intensity and color histogram shifts, we partition the video into discrete shots with an average duration of \(2\)–\(3\) seconds.

To aggregate these shots into semantically cohesive clips, we employ a multi-view representation strategy. For each shot, we extract the first, middle, and last frames and compute their embeddings using the CLIP ViT-L model [60]. The concatenation of these three vectors forms a robust representation of the shot’s visual content. We then calculate pairwise cosine similarities between adjacent shots. To account for stylistic variations across different dramas, we apply a dynamic quantile thresholding strategy: the merging threshold is defined as the top \(1/5\) quantile of all similarity scores within a specific drama. Adjacent shots exceeding this threshold are merged until a target duration of \(8\)–\(15\) seconds is reached. This process ensures that clips are long enough to provide narrative context but short enough to be processed efficiently by the multimodal model.

Base clip captioning. Let the generated clip sequence be \(\mathsf{V}=(\mathsf{C}_1,\ldots,\mathsf{C}_L)\). For each clip \(\mathsf{C}_l\), we sample frames at 1 FPS, prioritizing frames with high facial density. We represent each clip as a sequence of these sampled frames and utilize Qwen3-VL-32B [56] for caption generation. To enhance the grounding of these captions, we implement two visual-linguistic augmentations: (a) Textual anchoring: We integrate the dialogue transcript and current speaker pseudo-labels directly into the model’s context, and (b) Visual overlay: we overlay red bounding boxes on detected faces, labeled with the corresponding character ID. This results in detailed descriptions averaging \(300\) words per clip, ensuring that the visual narrative is tightly coupled with character identities. Figure 6 shows the prompts for video clip captioning.

Hierarchical Segment Summarization. To manage the computational overhead of processing millions of words per drama, we aggregate clips into consecutive sections \(\mathsf{S}_m\) via semantic partitioning. We extract text embeddings for each clip caption using the BAAI’s bge-large-zh-v1.5 model [61], and compute the cosine similarity between temporal neighbors \((\mathsf{C}_l,\mathsf{C}_{l+1})\). Clips are merged into a section if their caption similarity exceeds \(0.8\), subject to a constraint on the section size (\(8\pm4\) clips).

For each resulting section \(\mathsf{S}_{l'}\), Qwen3-32B [6] is tasked with synthesizing the constituent clip captions into a single, high-level segment summary \(\mathbf{T}_{l'}\). This summary filters redundant visual details and focuses on core plot developments and character interactions. Figure 7 shows the prompts for video segment summarization.

Character relationship extraction. To capture the evolving social dynamics inherent in drama series, we implement a temporal relational ontology. We process the transcripts and speaker labels episode-by-episode, prompting Qwen3-32B to extract character triplets \((p_1,p_2,\mathrm{relation})\). These episode-wise relational lists are then consolidated into a global mapping where each relationship is timestamped (e.g., Alice is Bob’s colleague in Episodes 1–10; Alice is Bob’s rival in Episodes 11–20). This temporal awareness allows the LRM to resolve identity ambiguities that arise as characters’ social roles shift over the course of the storyline. Figure 9 shows the prompts for character relationship extraction.

7.5 Data Curation↩︎

We utilize Gemini-3-Pro [5] as the teacher model to curate our SFT trajectories. The system prompts governing the model’s tool-use behavior are detailed in Figures 10 and 11. During the generation phase, the model is initialized with the user prompt described in Figure 12, which incorporates the initial speaker labels. To ensure high-quality reasoning, we implement a two-pass fallback mechanism: the ‘cheat message’ is withheld during the initial inference. If the model fails to produce a valid or consistent result in the first pass, the cheat message is provided for a secondary re-evaluation. Empirical observation suggests that this iterative refinement allows the model to successfully converge on the correct speaker attribution in the vast majority of cases.

8 Additional Results↩︎

8.1 Full Numerical Results↩︎

Table 5 shows drama-wise speaker recognition accuracy with respect to different confidence sampling thresholds. Table 6 shows speaker recognition accuracy in the subsets of different durations with respect to different confidence sampling thresholds.

8.2 Downstream Video Understanding↩︎

Figure 13 shows an example how speaker recognition impacts video captioning and question-answering. Upon the DramaSR-532K benchmark, we will establish a test set comprising thousands of QA pairs to quantitatively evaluate the benefit of speaker reocgnition to video understanding.

References↩︎