1 Introduction↩︎

Broadcast television is monitored through minute-level audience measurement systems that report metrics such as Average Minute Rating (AMR), capturing how many viewers are watching at each moment and their demographic composition [1], [2]. These indicators quantify engagement but do not explain it: connecting minute-level audience variation to the semantic structure of the underlying content is still largely manual and does not scale [3], [4].

Multimodal large language models (MLLMs) can jointly reason over visual frames, speech transcripts, and structured metadata, and are increasingly used for automated video annotation [5], [6]. In principle they could label broadcast programs at minute resolution and align those labels with audience data. In practice, deployment introduces constraints that rarely surface in lab settings: content is temporally structured and heterogeneous, ASR introduces noise that propagates downstream [7], large-scale annotation must remain computationally sustainable, and it is not obvious that richer inputs (e.g., full video) justify their latency and token cost [8]–[10].

This paper reports a real-world case study conducted with a national broadcast media group. We build a multimodal semantic annotation system for professional TV audience analysis and validate it on more than 3,000 minutes of broadcast content integrated with disaggregated audience measurement data. Two questions frame the work from a deployment standpoint:

• RQ1. Which pipeline configuration is production-viable for broadcast annotation when task quality is weighed jointly with computational cost (tokens, latency)?

• RQ2. Can the resulting annotations be operationally integrated with minute-level audience data—and with content audits such as on-screen guest composition—to enable demographic engagement analysis at broadcast scale?

We proceed in two stages. First, a diagnostic benchmark of 100 annotated clips is used to compare frame-based and video-based configurations across nine MLLMs and four tasks, explicitly accounting for token consumption, in order to select a single production configuration (Section 4). Second, the selected pipeline is deployed on 14 full episodes aired in 2025 (3,104 minutes); minute-level annotations are aligned with normalized AMR disaggregated by age cohort (Section 5). Our central practical result is that, for the language-oriented tasks (topic and entity recognition), speech- and text-derived context (ASR plus metadata) is a more robust performance driver than temporal visual continuity [10], [11], with direct consequences for how such systems should be built. We close with the lessons learned (Section 6).

2 Related Work↩︎

Broadcast programming differs from short-form web video: stable production formats, recurring guest ecosystems, and strong coupling between spoken discourse and visual context [12]. Prior work mostly optimizes individual components—speaker recognition from face and audio embeddings [13], scene-level environment labels, and ASR-driven topic analysis [14]—rather than end-to-end pipelines under heterogeneous tasks and constrained budgets. Vision–language models offer a unified framework for visual and textual reasoning [6], [15]–[17], but show persistent limits in fine-grained spatiotemporal reasoning: extending the context window is insufficient without careful input structuring, and frame sampling strongly shapes outcomes [8]–[10]. Methodologically, single automatic metrics can mislead system-level comparison and are better treated as diagnostics [18], performance is highly sensitive to prompt and input formulation [19], and domain-specific benchmarks must be purpose-built [20].

A third strand links audiovisual content to audience behavior: [12] show that large-scale visual analysis of TV news reveals systematic editorial exposure patterns, and [21] link automated subtitle analysis to segment-level audience dynamics. Both remain unimodal and depend on existing subtitles or curated metadata; fully automated multimodal pipelines that annotate raw broadcast video and integrate it with minute-level, disaggregated audience data remain underexplored—the gap this work addresses.

3 Methodology↩︎

Pipeline. The production system (Figure 1) shares a single architecture across all configurations. An audio–textual branch transcribes speech with Faster-Whisper, optionally adds speaker diarization via pyannote, and optionally appends episode-level metadata (program title, broadcast date, genre, and expected guests). A visual branch is instantiated either as sampled static frames or as the native video clip. The branches feed an MLLM that returns, per minute, a structured JSON record over four tasks. Activating subsets of the optional components yields a grid of input configurations, ranging from visual-only to visual\(+\)ASR\(+\)diarization\(+\)metadata, that lets us isolate the contribution of each textual source. The exact configuration set differs slightly between the two visual branches and is enumerated in Appendix 10: notably, the video branch includes a visual\(+\)metadata configuration without ASR—the one ultimately selected for deployment (Section 5).

Tasks and taxonomy. Each minute is annotated along four dimensions, inspired by the IAB Content Taxonomy [22] and adapted to Italian prime-time television. Topic classification assigns one label from a 28-label taxonomy (14 macro-areas). Environment classification assigns one of 22 setting labels. Visual named-entity recognition lists individuals visually present (empty if none). Sensitive-content detection flags material across six categories drawn from content-moderation and broadcast regulatory frameworks. The taxonomy was refined iteratively with broadcast domain experts; the full label set is in Appendix 7.

Deployment setting. The selected configuration (Gemini-3-Pro with video and episode-level metadata) is deployed on 14 full episodes of a single fixed-slot weekly program, annotating 3,104 minutes at one-minute granularity (advertising excluded). Audience metrics are supplied by an Italian media company in aggregated, normalized form; no personally identifiable information is accessed.

4 Choosing the Production Configuration↩︎

Diagnostic benchmark. To select a configuration under realistic constraints, we built a diagnostic test set of 100 one-minute clips from Italian broadcast programs aired in 2025, stratified across four editorial macro-categories—current-affairs talk shows, investigative, cultural, and lifestyle programs—with 25 clips sampled per category (one per episode) to balance editorial conditions by design; program identities are anonymized. Clips are extracted without enforcing scene boundaries and may span multiple shots, speaker turns, and topic transitions, reflecting operational conditions rather than curated single-topic segments. Two annotators with TV-content experience labeled all clips; inter-annotator agreement (Cohen’s \(\kappa\)) was 0.70 (topic), 0.67 (environment), and 0.67 (entities); sensitive content reached \(\kappa{=}1.0\), interpreted cautiously given only 12 positive clips. Disagreements were resolved by consensus. We evaluate nine MLLMs spanning proprietary and open-weight families—Gemini-3-Pro, GPT-5-mini, Llama-4-Maverick, Qwen2.5-VL-72B, Qwen3-VL (235B/30B/8B), Gemma-3-27B, and Ministral-14B—in the frame setting, and the six models supporting native video input (Gemini-3-Pro, Qwen3-Omni, Qwen3-VL-235B, Qwen3-VL-30B, Molmo-2-8B, and Seed-1.6) for the cross-pipeline comparison, in a zero-shot setting with a single prompt template that includes the full taxonomy (Appendix 8). Predictions outside the label set are scored as incorrect. Input token counts from model APIs are reported as an explicit proxy for cost.

Component choices. Preliminary studies fixed the visual and ASR components. Accuracy plateaus beyond 12 frames across all models, and shot-based sampling offers no advantage on this material; uniform sampling at 0.2 fps (12 frames) cuts token use by \({\sim}40\%\) relative to an 18-frame budget at comparable accuracy. Across the ASR systems compared, Faster-Whisper Medium offers competitive downstream quality without the latency overhead of larger variants, making it suitable for cost-efficient large-scale inference. This fixes the ASR component for the benchmark configurations that include it; the configuration ultimately deployed omits ASR altogether (Section 5).

What drives quality. Task quality is shaped by the interaction of input modality and model capacity, not by raw visual richness (Table 2). The frame pipeline’s only/asr/asr_meta configurations let us isolate each textual source within a fixed model: only\(\rightarrow\)asr isolates the ASR increment, asr\(\rightarrow\)asr_meta the metadata increment. For topic, the isolated ASR increment is large and consistent (\(+0.14\) to \(+0.21\) accuracy for Gemini-3-Pro, Llama-4-Maverick, and the Qwen models), while the subsequent metadata increment is near zero: speech-derived text is the driver and diarization adds no systematic benefit. For visual NER—the most demanding task—the pattern reverses: the ASR increment is negligible whereas the isolated metadata increment dominates (precision \(0.08\rightarrow0.52\) for Qwen3-VL-235B and Qwen3-VL-30B, \(0.42\rightarrow0.60\) for Gemini-3-Pro), though enrichment is not uniformly safe—diarization noise degrades identity resolution for several models, and smaller models hallucinate identities without textual grounding. Environment is primarily perceptual; whether native video helps is model-dependent (Table 1), with gains concentrated in motion-dependent or spatially ambiguous scenes. Sensitive-content detection is dominated by false-positive control: transcripts can contextualize ambiguous scenes but also trigger spurious activations via lexical shortcuts, especially in smaller models; with only 12 positive clips, its metrics are diagnostic and do not drive configuration selection.

Video is not universally better (controlled comparison). Critically, native video input does not systematically outperform sampled frames. To isolate the visual representation from confounds of model capacity and textual input, we compare the two pipelines within each of the three architectures common to both (Gemini-3-Pro, Qwen3-VL-235B, Qwen3-VL-30B) at matched input compositions (Table 1). Holding model and textual inputs fixed, the video\(-\)frame accuracy gap is small and, in most cells, not statistically distinguishable from zero: of the eighteen matched comparisons, twelve have a 95% bootstrap confidence interval (paired over clips) that spans zero. For topic, video helps only when no transcript is present (Gemini-3-Pro \(+0.12\), CI \([{+}0.05,{+}0.20]\); Qwen3-VL-30B \(+0.14\), CI \([{+}0.05,{+}0.24]\)) and the gain becomes indistinguishable from zero once ASR is added. For environment, the significant effects are model-specific and of opposite sign: video helps Qwen3-VL-30B substantially (\(+0.36\), CI \([{+}0.23,{+}0.48]\)) but hurts the larger Gemini-3-Pro (\(-0.10\), CI \([{-}0.19,{-}0.01]\)). The residual token gap between the two encodings is intrinsic to the representation and is reported, not equalized (Appendix 10). This within-model evidence—bounded to three architectures and a 100-clip set—substantiates the headline result under controlled conditions: the video benefit does not track model size and is confined to a few model–task cells, while degradation under extended multimodal context appears among the smaller video-only models in the full results (Appendix 10). Combined with the token-cost accounting, this argues against adopting video ingestion as a default. We select Gemini-3-Pro with video plus metadata for deployment: it is not Pareto-dominant (accepting a small environment loss, 0.73 vs. for frames\(+\)ASR\(+\)metadata) but gives the best topic accuracy (0.83) at roughly half the token cost. It omits ASR—for the highest-capacity model, metadata already supplies enough textual context (topic 0.83 vs. with ASR)—which in a corporate setting also drops a transcription stage and its hardware, lowering latency and improving portability.

6pt

Error patterns. Manual inspection reveals failure modes the aggregate metrics obscure. Topic errors are systematic collapses between adjacent labels (international \(\rightarrow\) domestic politics), driven by shared lexical fields in the transcript; environment errors cluster within perceptually similar settings (home apartment vs.corporate office), while distinctive studio layouts stay stable. Sensitive-content errors stem from threshold instability—under weak visual evidence, lexical triggers induce spurious activations, turning text into a liability—and entity-recognition failures concentrate in smaller models and in clips lacking textual anchors, where models hallucinate identities. Model capacity moderates these effects but does not remove the underlying bottlenecks.

4pt

5 Deployment and Audience Analytics↩︎

Production cost and throughput. Table 3 summarizes deployment economics. The selected configuration consumes \({\sim}6.4\)K input tokens per annotated minute and emits only four structured fields, so annotating the full corpus costs \({\approx}\$40\) at list pricing¹ (\({\approx}\$20\) via the Batch API)—under a cent and a half per minute. Median latency (\({\sim}31\) s/minute, real-time factor \({\approx}0.5\)) means a single request stream keeps pace with broadcast; annotation is offline and trivially parallelizable. Notably, native video\(+\)metadata is also more token-economical than the frame configuration (\({\sim}6.4\)K vs \({\sim}14.6\)K tokens/minute): the model encodes native video at a low fixed token rate per sampled second, whereas twelve full-resolution stills each consume a large token block. The production choice is therefore simultaneously the higher-quality and the cheaper one.

6pt

Human-cost comparison. The media partner’s manual workflow for the same deliverable is estimated at roughly two analyst-hours per episode (one to annotate content, one to ingest audience data and produce the analysis): \({\approx}28\) hours per 14-episode cycle, or \({\approx}\,\)€840 at an illustrative €30/h, against \({\approx}\$40\) of compute plus the residual human time to confirm sensitive-content flags—an order-of-magnitude gap that widens with scale, since manual effort grows linearly with airtime while per-minute annotation cost is fixed. The manual figure reflects a coarse operational pass, not the research-grade annotation used for the benchmark, and only bounds the saving.

Content characterization. On the 3,104 deployed minutes, editorial structure is highly concentrated: domestic politics (40.8%) and international politics (21.7%) account for over 60% of airtime, followed by economy and finance (11.7%), humor and satire (9.2%), and crime and justice (6.0%); remaining topics form a long tail. The NER module identifies 143 unique guests (hosts excluded) over 3,717 minute-level occurrences. Participation is structurally asymmetric: male guests account for 81.1% of occurrences (89 individuals, 33.9 appearances each on average) versus 700 occurrences across 54 female guests (13.0 each), and 80.2% of minutes feature exclusively male guests—a pattern replicated across episodes. Since visual NER operates at F1\(\,{\approx}\,\)​0.60 at deployment, these absolute participation figures should be read as approximate estimates rather than exact counts; the asymmetry itself, however, is large and directionally robust to this error rate, illustrating the kind of large-scale content audit the pipeline makes feasible.

Baseline audience composition. Figure 2 reports normalized AMR per episode by age cohort (Young 15–34, Adults 35–54, Seniors 55+). The senior segment is the largest and most stable component; younger viewers show markedly higher inter-episode variability, indicating that episode-level fluctuations are driven mainly by changes in younger participation rather than the senior base.

Topic-level sensitivity and divergence. Audience values are normalized intra-episode via per-cohort \(z\)-scores (Appendix 9), so positive values indicate above-average engagement controlling for episode popularity and seasonality. Among the ten topics with the largest inter-cohort \(z\)-score gap, art and literature is positive for Young viewers yet negative for Adults and Seniors—the clearest case of cohort-selective engagement. A gradient pattern characterizes music, health and wellness, and sports–football: negative across cohorts, with the decline steepening with age. Family and relationships and environment and climate are associated with disengagement across all groups, while food and cooking and sports–other act as shared attractors that differ in amplitude rather than direction.

Interpretation. High-frequency structural topics sustain a stable engagement backbone, while low-frequency episodic segments generate localized deviations that differentially affect age cohorts—a shift from static aggregate measurement toward dynamic audience-sensitivity analysis of direct value for editorial planning. We present this integration as an illustrative downstream use enabled by the annotations rather than a standalone analytical contribution; all reported patterns are observational and no causal claims are made.

6 Lessons Learned and Conclusion↩︎

Speech and text beat video for the language-oriented tasks. Video pipelines do not systematically outperform frame configurations: under a controlled within-model comparison, most matched gaps are statistical ties and the few significant effects are model-specific and of opposite sign, not a function of model size (degradation under extended multimodal context shows up mainly in the smaller video-only models). Video ingestion should be a conditional design choice, validated per task and per model before its cost is accepted.

Metadata is the cheapest high-impact lever. Isolating each textual increment, episode-level metadata was the dominant driver for entity resolution—outweighing transcription quality there at negligible token cost—whereas speech transcripts, not metadata, drove topic. In a deployment, cheap structured side information is often more valuable than a heavier perceptual modality for the tasks it targets.

Taxonomy design is a first-order engineering decision. Recurring errors—semantic collapse between adjacent topics, perceptual confusion between similar environments—were attributable to label structure, not model capacity: overlapping or underspecified categories introduce systematic ambiguity that no extra modality resolves. The taxonomy should be a maintained engineering artifact, not a fixed schema.

Automated annotation enables scale but needs error-aware use. Annotating at inference speed makes longitudinal audits and cross-program comparisons feasible, but systematic label biases (environment misclassification, topic conflation) propagate downstream. We add two safeguards: the lowest-precision task (sensitive-content detection) is human-confirmed before any downstream use rather than driving autonomous decisions, and a rolling sample of deployed minutes is periodically re-annotated to monitor drift, seeded by the diagnostic benchmark. Studies built on automated annotations should carry reliability proxies, and findings from borderline categories should be read as indicative.

Taken together, these lessons show that deployed multimodal annotation should match each task to the cheapest reliable signal rather than defaulting to the richest input. In our broadcast setting, speech supported topic interpretation, metadata stabilized entity resolution, and vision remained necessary for perceptual labels. Under these safeguards, such annotations can support scalable audience analysis and content audits without treating automated labels as ground truth.

Limitations↩︎

The diagnostic benchmark (100 clips) limits statistical robustness and generalizability and should be read as diagnostic rather than as a large-scale corpus. The cross-pipeline comparison is controlled within the three architectures common to both pipelines and at matched textual inputs, but video and frame encodings cannot be equalized in token count; the residual token gap is an intrinsic property of the representation rather than a nuisance variable we control away. Bootstrap confidence intervals are reported for the controlled cross-pipeline comparison; the remaining per-model tables report point estimates only. Although the clips are stratified by editorial macro-category, they remain internally heterogeneous (multiple shots, speakers, and topic transitions per minute), so per-signal effects are reported descriptively and may partly reflect clip composition. The taxonomies are tailored to a specific broadcast format and may require adaptation elsewhere. The evaluation is monolingual (Italian) and centered on a single program type, leaving multilingual content and other genres untested. All experiments use zero-shot prompting of general-purpose models, without task-specific fine-tuning. The audience analyses are purely observational: intra-episode per-cohort \(z\)-score normalization removes episode-level popularity and seasonality, but the analysis is not benchmarked against simpler scheduling baselines, reported associations between content and engagement do not establish causal relationships, and they may be confounded by concurrent programming. Finally, downstream analyses inherit the annotation error profile of the pipeline, which is only partially characterized at this scale.

Ethical Considerations↩︎

Audience data were provided by the media partner in aggregated and normalized form, disaggregated only by coarse age cohorts; no personally identifiable information was accessed or stored. The system performs sensitive-content detection and visual entity recognition over public broadcast material; we report these capabilities for analytical and editorial-planning purposes and caution against punitive or surveillance uses. Automated annotation can encode and amplify bias: the observed gender asymmetry in guest participation is a property of the broadcast content, but pipeline errors could distort such measurements, so demographic findings should be validated before informing decisions. Program and company identities are anonymized, and proprietary data are not released. We follow the ACL Code of Ethics.

7 Complete Annotation Taxonomy↩︎

Table 4 lists the full label sets for topic, environment, and sensitive-content annotation.

8 Prompt Strategies↩︎

Both pipelines share the output schema and hard constraints: select exactly one label for topic and environment, at most one sensitive-content flag, include a guest name only if the person is visually recognized or explicitly named, and return JSON only. Configurations differ in which sources are provided and in their relative authority. When metadata is added, it is explicitly subordinated to visual evidence (used only to confirm or correct visually recognized identities, never to assume presence). When ASR is added, a three-level hierarchy is established: video is primary for visual elements, ASR for spoken content, and metadata acts as a correction layer for proper nouns and context.

9 Audience Normalization↩︎

Topic-level audience sensitivity (Section 5) is computed on per-cohort, intra-episode \(z\)-scores. Let \(c\) index the age cohort, \(e\) the episode, and \(t\) the topic. Let \(x_{c,e,t}\) be the mean normalized AMR of cohort \(c\) over the minutes labeled with topic \(t\) in episode \(e\), and let \(\mu_{c,e}\) and \(\sigma_{c,e}\) be the mean and standard deviation of cohort \(c\)’s per-minute normalized AMR within episode \(e\). The \(z\)-score is \[z_{c,e,t} \;=\; \frac{x_{c,e,t}-\mu_{c,e}}{\sigma_{c,e}} .\] Normalizing within each episode and cohort removes differences in baseline popularity and seasonality, so positive (negative) values denote above-average (below-average) engagement of a cohort with a topic relative to its own episode-level mean. The inter-cohort gap discussed in Section 5 is the spread of \(z_{c,e,t}\) across cohorts, averaged over episodes.

10 Full Per-Model Results↩︎

Complete per-model, per-configuration results for all four tasks. We report Accuracy, Precision, Recall, and F1, together with input tokens (Tok) and average API latency in milliseconds (Lat). The video pipeline covers the six models supporting native video; the frame pipeline covers all nine models. Tables 1–4 report the video pipeline and Tables 5–8 the frame pipeline.

References↩︎