1 Introduction↩︎

Music is a cornerstone of compelling video content, shaping emotional tone, narrative pacing, and audience engagement across domains from cinematic productions to short-form social media. Yet selecting appropriate background music is far from trivial: it requires not only an understanding of high-level semantic compatibility, such as mood and genre, but also a fine-grained understanding of the temporal correspondence between evolving visual dynamics and musical elements. Identifying music that satisfies both criteria is time-consuming and often yields suboptimal results.

Automatic video-to-music (V2M) recommendation has therefore attracted growing interest as a cross-modal music retrieval task [1]–[4], aiming to learn joint embedding spaces between visual content and music [5]–[8]. Despite steady progress, current frameworks leave substantial room for improvement across two major dimensions. First, while general-purpose multimodal models [9]–[11] have demonstrated the power of joint audio-visual-text representations, established V2M methods still rely almost exclusively on raw visual features. Even though some previous works [12] have utilized both visual and textual modalities for V2M recommendation, existing architectures have yet to fully exploit the multimodal richness of both video and music signals.

Second, all existing V2M methods reduce retrieval to a single global embedding similarity score, compressing entire video and music streams into static vectors. While global embeddings are effective for capturing coarse semantic compatibility, such as overall mood or genre, this single-vector bottleneck fundamentally cannot represent how localized musical events align with specific video moments. The temporal dimension is collapsed by design, making it impossible to distinguish music that is globally compatible from music that is temporally aligned with specific video moments.

To address both limitations, we propose VTMR, a two-stage framework capable of capturing both unified multimodal semantics and temporal dynamics (see Figure 1). Stage 1 (Semantic Retrieval) addresses the first limitation by projecting comprehensive multimodal video signals, music signals, and LLM-generated text descriptions into a tightly aligned shared embedding space, enabling efficient large-scale retrieval of the top-\(N\) candidates. Stage 2 (Temporal Reranking) resolves the second limitation by deploying a fine-grained cross-encoder. It directly attends over the dense, unpooled temporal sequences of both the video and the retrieved music candidates, capturing the intricate cross-modal dynamics that global embeddings inevitably lose.

2 Methods↩︎

Let \(\mathcal{M} = \{m_1, \dots, m_n\}\) be a music corpus of \(n\) tracks. Given a query video \(v\), the goal is to retrieve and rank tracks from \(\mathcal{M}\) by relevance. We represent \(v\) as a multimodal tuple \((x^v, x^a, x^t)\), where \(x^v\) denotes raw RGB frames, \(x^a\) the non-musical audio (speech, sound effects), and \(x^t\) an LLM-generated scene description. Symmetrically, each music track \(m \in \mathcal{M}\) is defined as a multimodal composition \((x^m, x^{mt})\), where \(x^m\) represents the acoustic music audio stream and \(x^{mt}\) denotes the LLM-generated music caption. Additionally, \(x^{meta}\) denotes raw video metadata (e.g., title, channel, category information).

2.1 Multimodal Media Encoding↩︎

We use (PE_AV-base) [13] as a frozen backbone to encode all modalities. PE_AV-base is trained on \(\mathcal{O}(100\text{M})\) synthetic audiovisual pairs via multiple cross-modal contrastive objectives.

2.1.0.1 Video Encoding.

A video \(v = (x^v, x^a)\) contains two complementary streams. The visual stream \(x^v\) is encoded by the PE-L frame encoder into a dense temporal sequence \(\mathbf{e}^v = {PE_{frame}}(x^v) \in \mathbb{R}^{L_v \times C}\). The non-musical audio stream \(x^a\) (speech, sound effects) is tokenized by DAC-VAE at 25 Hz and contextualized by Transformer Layers with RoPE to yield \(\mathbf{e}^a = {PE_{audio}}(x^a) \in \mathbb{R}^{L_a \times C}\). The two streams are temporally aligned and fused by a shallow Transformer into a joint audiovisual [CLS] token \(\mathbf{h}^{av} = {PE_{fuse}}(\mathbf{e}^v,\, \mathbf{e}^a)_{\texttt{[CLS]}} \in \mathbb{R}^{C}\), which compactly represents what is seen and heard in the video.

2.1.0.2 Music Encoding.

A music track \(m = (x^m)\) is encoded by the same PE_AV audio encoder into a temporal sequence \(\mathbf{e}^m = {PE_{audio}}(x^m) \in \mathbb{R}^{L_m \times C}\). A global music embedding \(\mathbf{h}^m \in \mathbb{R}^C\) is obtained by taking the [CLS] token.

2.1.0.3 Text Encoding.

The video scene description \(x^t\), music caption \(x^{mt}\), and video metadata \(x^{meta}\) are encoded by the PE_AV text encoder (ModernBERT), yielding global representations \(\mathbf{h}^{t} = {PE_{text}}(x^t) \in \mathbb{R}^{C}\), \(\mathbf{h}^{mt} = {PE_{text}}(x^{mt}) \in \mathbb{R}^{C}\), and \(\mathbf{h}^{meta} = {PE_{text}}(x^{meta}) \in \mathbb{R}^{C}\), respectively. The global representations \((\mathbf{h})\) feed the Stage 1 semantic retrieval, while the temporal sequences \((\mathbf{e})\) feed the Stage 2 temporal reranker.

2.2 Semantic Retrieval↩︎

2.2.0.1 Retrieval Modules.

We map the global representations extracted from the pre-trained encoders (Section 2.1) into a shared \(d\)-dimensional embedding space (\(d{=}1024\)) to facilitate efficient large-scale retrieval. Formally, we define the projection networks \(f_v\), \(f_m\), and \(f_t\) for the video, music, and text modalities, respectively, where each head consists of a LayerNorm followed by a bias-free 3-layer MLP with intermediate GELU activations. The resulting joint embeddings are formalized as the set \(\mathcal{Z} = \{\mathbf{z}^{v}, \mathbf{z}^{m}, \mathbf{z}^{vt}, \mathbf{z}^{mt}, \mathbf{z}^{meta}\}\), where the isolated projected vectors are mapped via \(\mathbf{z}^{v} = f_v(\mathbf{h}^{av})\), \(\mathbf{z}^{m} = f_m(\mathbf{h}^{m})\), \(\mathbf{z}^{vt} = f_t(\mathbf{h}^{t})\), \(\mathbf{z}^{mt} = f_t(\mathbf{h}^{mt})\), and \(\mathbf{z}^{meta} = f_t(\mathbf{h}^{meta})\).

2.2.0.2 Modality Fusion and Dropout.

To ensure the model remains robust to missing modalities at inference time (e.g., querying with videos that lack text descriptions), we employ a dynamic modality fusion and dropout strategy during training [14], [15]. For a given batch, with a probability of \(0.5\), we fuse the parallel representations within \(\mathcal{Z}\) by mean-pooling the projected media and text embeddings, dynamically updating the active video and music keys via \(\mathbf{z}^{v} \leftarrow \frac{1}{2}(\mathbf{z}^{v} + \mathbf{z}^{vt})\) and \(\mathbf{z}^{m} \leftarrow \frac{1}{2}(\mathbf{z}^{m} + \mathbf{z}^{mt})\). Otherwise, we drop the text streams entirely and rely solely on the isolated audiovisual (\(\mathbf{z}^{v}\)) or acoustic (\(\mathbf{z}^{m}\)) vectors.

2.2.0.3 Loss Function.

For training, the semantic retrieval module is optimized using a multi-pair contrastive framework based on the SigLIP loss [16], where pairwise sigmoid log-likelihoods are computed across all \(\binom{5}{2} = 10\) unique modality combinations derived from the projected embedding set \(\mathcal{Z}\).

2.3 Temporal Reranking↩︎

2.3.0.1 Architecture.

Given the top-\(N\) candidates retrieved in Stage 1, the temporal reranker scores each pair \((v, m)\) using a cross-encoder architecture [17], [18]. We extract the audio-visual frame sequence \(\mathbf{x}^{{av}}\) and the music acoustic feature sequence \(\mathbf{x}^{{m}}\) using PE_AV-base [13]. To accommodate variable-length inputs, both sequences are resampled along the temporal dimension to a fixed length (\(T_{\text{target}} = 64\)), using window averaging for downsampling and linear interpolation for upsampling. The resampled sequences are then linearly projected into a shared \(d_j\)-dimensional space, yielding aligned representations \(\mathbf{e}^{{av}}, \mathbf{e}^{{m}} \in \mathbb{R}^{T_{\text{target}} \times d_j}\). To distinguish the two modalities while allowing their temporal representations to interact, we add learnable modality-type embeddings \(\mathbf{t}^{{av}}, \mathbf{t}^{{m}} \in \mathbb{R}^{d_j}\) and construct the concatenated embedding sequence \(\mathbf{e}^{v+m} = [\,\texttt{[CLS]};\; \mathbf{e}^{{av}} + \mathbf{t}^{{av}};\; \texttt{[SEP]};\; \mathbf{e}^{{m}} + \mathbf{t}^{{m}}\,]\). A 4-layer Transformer encoder with 8 attention heads then applies self-attention over the entire concatenated sequence. Finally, the contextualized [CLS] representation is passed through a two-layer GELU MLP head to produce a scalar classifier logit \(s(v,m)\).

2.3.0.2 Loss Functions.

During training, the reranker is optimized using a hybrid objective. For each query video \(v\), its paired music track \(m^+\) serves as the positive, while a negative track \(m^-\) is sampled from the other music tracks in the batch based on their semantic retrieval scores (Section 2.2). We compute the corresponding logits as \(s^+=s(v,m^+)\) and \(s^-=s(v,m^-)\). To encourage both pointwise discrimination and relative ordering, we combine binary cross-entropy (BCE) with a margin-based ranking loss: \(\mathcal{L}_{\text{rerank}}=-\log\sigma(s^+)-\log(1-\sigma(s^-))+\max(0,\gamma-(s^+-s^-))\), where \(\sigma(\cdot)\) denotes the sigmoid function and \(\gamma=0.2\) is the ranking margin.

3 Dataset↩︎

To build a robust framework for multimodal video-to-music recommendation, we construct our dataset through a three-stage pipeline: collecting videos from diverse source datasets, filtering them to retain salient and high-quality background music, and generating pseudo annotations from visual and separated audio streams (illustrated in Figure 2).

3.0.0.1 Source Datasets.

To build a diverse media dataset, we establish three collection criteria. First, the videos should preserve their original mixed audio, containing real-world combinations of speech, sound effects, and background music. Second, background music should be present for at least 50% of each video’s duration, ensuring a sufficiently strong video–music association. Third, the dataset should cover diverse content sources and production contexts, ranging from professionally produced broadcast programs to user-generated YouTube videos. Our data are drawn from three distinct sources: (1) MMtrail-2M [19], a large-scale public dataset of YouTube videos; (2) MovieLens-Content [20], a dataset of 16K movie trailers produced by professional filmmakers; and (3) an internal dataset comprising broadcast content.

3.0.0.2 Dataset Filtering.

To ensure the fidelity and semantic relevance of the target background music, we apply a rigorous filtering process. First, we filter out non-music video clips based on music logits derived from a pre-trained music detection model [21], discarding samples with a music logit score of 0.7 or lower. Furthermore, we exclude video sequences exceeding 2 minutes to maintain dense temporal alignment and optimize computational efficiency. Finally, to guarantee music quality, we utilize an aesthetic prediction model [22], removing music tracks with a content enjoyment score below 5.5.

3.0.0.3 Pseudo Annotation.

To enable scalable content descriptions, we leverage pseudo annotations [23]–[25]. Since the original audio contains overlapping speech, sound effects, and background music, we employ an audio source separation model [26] to disentangle the mixed audio into three independent streams: speech, sound effects, and music. Following this audio separation, we annotate semantic descriptions by employing a suite of multimodal Large Language Models. Specifically, we extract visual captions using  [27], transcribe spoken dialogue using  [28], capture ambient sound contexts via  [29], and describe musical attributes using  [30]. Finally, for audio-visual description, we aggregate vision, speech, and sound-effect descriptions with an powered by  [31].

4 Experiments↩︎

4.0.0.1 Quantitative Evaluation.

To evaluate our video-to-music recommendation system, we assess cross-modal retrieval performance on the human-verified VidMuse benchmark [33]. We re-crawled the dataset to incorporate all multimodal signals extracted via our preprocessing pipeline. At inference, the video query vector is obtained by averaging the audiovisual and text embeddings; the music vector averages its audio and text embeddings. We use re-ranker with-top 40 retrieval sample. We report Recall@10 (R@10) and Median Rank (MedR) as primary metrics, benchmarking against AudioCLIP [9], Wav2CLIP [32], ImageBind [10], LanguageBind [11], and PE_AV-base [13].

Table 1 summarizes the quantitative evaluation results. Our proposed multimodal fusion retrieval module, VTMR (w/o Reranker), consistently outperforms all existing baseline models. Notably, it yields a substantial margin of improvement over the strongest baseline, ImageBind, increasing the R@10 from 14.2 to 15.9 and reducing the Median Rank from 75 to 58. Furthermore, integrating the Stage 2 temporal reranker—VTMR (w/ Reranker)—delivers a powerful performance boost, achieving the highest R@10 of 18.3 and reducing the MedR further to 46. This remarkable improvement underscores the critical importance of modeling fine-grained, unpooled temporal sequences via cross-encoder to align cross-modal dynamics effectively.

4.0.0.2 Qualitative Evaluation.

To evaluate the subjective quality of the video-to-music system, we conducted a human evaluation using an A-vs-B preference test. We selected 20 diverse video clips as evaluation stimuli: 10 professional broadcast clips featuring original scores curated by music directors, and 10 background-music-free YouTube clips. To ensure high-quality retrieval candidates, our model retrieves from an external music database of 500K professionally produced tracks, providing broad coverage across genres, moods, and instrumentation. Our proposed framework was benchmarked against three distinct configurations: (1) Human-Reference (upperbound), (2) Adobe Firefly (commercial baseline), and (3) VidMuse [33] (generative model baseline).

Human annotators evaluated the paired samples based on two primary perceptual dimensions: Overall Preference and Music Quality. A total of 30 expert evaluators—comprising academic music researchers and audio industry professionals—participated in the user study, yielding a robust dataset of 1,200 pairwise preference annotations.

As shown in Figure 3, the win+tie rates reveal two complementary trends. In terms of Overall Preference, our model is competitive with both the commercial baseline Adobe Firefly (77%) and the generative baseline VidMuse (73%), while falling short of professional human reference curation (46%), which remains a strong upper bound. In terms of Music Quality, our retrieval-based model substantially outperforms the generative model VidMuse (96% win+tie vs.% loss), demonstrating that retrieving from a curated music corpus yields consistently higher intrinsic audio quality than end-to-end generation. Taken together, these results suggest that our framework reaches the practical quality range of commercial tools while avoiding music quality degradation.

5 Conclusion↩︎

We presented VTMR, a two-stage framework for video-to-music recommendation that combines multimodal semantic correspondence with fine-grained temporal matching. Its key contribution is a retrieve-and-rerank architecture that first identifies semantically compatible music using visual, audio, and textual cues and then captures temporal correspondence between video and music. Future work will explore end-to-end training of the retrieve-and-rerank pipeline.

References↩︎