1 Introduction↩︎

Multimodal music systems increasingly rely on heterogeneous cues such as text, video, and audio context. Text prompts provide high-level semantic intent, such as the desired instrument or acoustic scene, while visual inputs provide evidence about visible instruments, performers, and scene context. In practical settings, however, these cues may be available under different operating conditions. A user may want to synthesize music from text or video when no reference audio exists, or recover a specified source from an observed mixture when mixture audio is available. These two settings are related but distinct: conditional music generation produces audio without a reference mixture [1]–[5], whereas target-source extraction recovers a specified source from an observed mixture using text, visual, or other target-identifying conditions [6]–[12]. We address these two settings within a shared conditional latent model: conditional music generation and mixture-grounded target-source extraction.

Existing methods are typically designed for one side of this problem. Music demixing and audio–visual separation models estimate target sources from mixtures and remain strong task-specific baselines [13]–[24]. Promptable extraction systems further allow the target source to be specified through language, visual prompts, or other flexible cues [6]–[12]. These methods, however, are grounded in an observed mixture and do not address mixture-free music generation. Conversely, conditional generation models synthesize music from text, video, or other semantic conditions, but generally do not recover a specified source from an existing mixture [1]–[5]. This creates a gap between generation systems that operate without mixture audio and extraction systems that require it.

We introduce MAGE, a modality-agnostic framework that supports conditional music generation and mixture-grounded target-source extraction within a shared continuous latent formulation. MAGE represents audio using a learned continuous codec and uses a Controlled Multimodal FluxFormer to model a conditional flow from noise to the target audio latent. When no mixture is provided, the model generates music from text, visual input, or both. When a mixture is available, its encoded latent is supplied as an acoustic condition, while text or visual input specifies the source to be recovered. Both operating modes use the same model parameters and flow-based inference procedure. A central challenge is that text, visual input, and mixture audio provide different types of information. Text provides global semantic guidance, visual frames provide scene-level evidence, and mixture audio provides acoustic context for extraction. Directly fusing these signals can obscure their roles and does not align frame-level visual evidence with the audio latent sequence. MAGE addresses this with Audio–Visual Nexus Alignment (AVNA), which maps visual frame features to the audio latent resolution before conditioning. The aligned visual representation is then used by cross-gated modulation to regulate intermediate audio features, while text is handled through a separate semantic-conditioning pathway. This design allows visual information to condition the audio-token stream without treating text as temporally aligned evidence.

MAGE is trained with dynamic modality masking, exposing the same model to text-only, visual-only, joint text–visual, mixture-conditioned, and unconditional configurations. We use the term modality-agnostic to describe the shared conditioning interface through which the model handles the supported text, visual, mixture-conditioned, and unconditional configurations. We evaluate MAGE on MUSIC using separate protocols for mixture-free generation and mixture-grounded target-source extraction [13]. For generation, we evaluate audio quality and conditioning alignment. For extraction, we evaluate reconstruction quality and interference suppression against specialized and promptable baselines. Dedicated separation systems are treated as valid task-specific baselines, while MAGE evaluates how a shared conditional model performs across both mixture-free generation and mixture-grounded extraction.

Our contributions are summarized as follows:

• We propose MAGE, a modality-agnostic framework that treats conditional music generation and mixture-grounded target-source extraction as two operating modes of a shared continuous latent model.

• We introduce the Controlled Multimodal FluxFormer, a flow-matching Transformer that predicts target audio latents under supported combinations of text, visual, and mixture conditions using the same model parameters and inference procedure.

• We propose Audio–Visual Nexus Alignment and cross-gated visual modulation to map frame-level visual features onto the audio latent sequence, and train the model with dynamic modality masking to support text-only, visual-only, joint text–visual, mixture-conditioned, and unconditional configurations.

• We conduct separate evaluations for mixture-free generation and mixture-grounded target-source extraction on MUSIC, compare extraction against specialized and promptable baselines, and analyze the effects of alignment, gated visual conditioning, and modality masking through ablations.

2 Related Work↩︎

Music Demixing and Target-Source Extraction. Music source separation aims to recover target sources from an observed mixture. Dedicated demixing systems such as Demucs [20], Hybrid Demucs [21], [22], and related architectures [23] achieve strong reconstruction performance on benchmarks such as MUSDB18 [24]. Audio–visual methods further use visual evidence to identify the source of interest, including Sound-of-Pixels [13], Co-Separation [14], Music Gesture [15], CCoL [16], audio–visual correspondence models [17], [18], iQuery [19], and DAVIS [10]. Promptable extraction methods such as Universal Sound Separation [25], AudioSep [6], CLIPSep [7], FlowSep [8], SoloAudio [9], SAM Audio [11], and PromptSep [12] broaden the target specification interface through language, visual, or other prompts. These methods remain grounded in an observed mixture, whereas MAGE supports both mixture-grounded extraction and mixture-free generation within the same conditional latent model.

Conditional and Multimodal Music Generation. Conditional music-generation models synthesize audio from semantic or perceptual cues without requiring a reference mixture. MusicGen [1], AudioLDM 2 [2], and MusicFlow [3] generate music from language descriptions using autoregressive, diffusion, or flow-based modeling. Video-to-music systems such as GVMGen [5] use visual features to improve video–music correspondence, while AudioX [4] supports broader multimodal audio generation through masked multimodal training. These systems focus on mixture-free generation, whereas MAGE supports both mixture-free generation and mixture-grounded target-source extraction within a shared conditional latent model and evaluates them under separate task-specific protocols.

Multimodal Conditioning and Audio–Visual Alignment. Multimodal audio models combine signals with different roles: text provides semantic guidance, visual frames provide scene-level evidence, and mixture audio provides acoustic context. Prior systems such as CLIPSep [7], AudioSep [6], DAVIS [10], GVMGen [5], and AudioX [4] incorporate these signals through feature fusion, attention, adaptive modulation, or masked multimodal training. Audio–visual methods such as Co-Separation [14], Music Gesture [15], CCoL [16], iQuery [19], and GVMGen [5] further address cross-modal mismatch using object localization, motion cues, cyclic consistency, query-based interaction, or hierarchical temporal modeling. MAGE differs by mapping frame-level visual features to the audio latent resolution with Audio–Visual Nexus Alignment, applying cross-gated visual modulation, and using modality masking so the same model operates under text-only, visual-only, joint text–visual, mixture-conditioned, and unconditional configurations.

3 Proposed Method: MAGE↩︎

Problem Formulation. We study two related conditional audio-generation tasks: conditional music generation without a reference mixture and mixture-grounded target-source extraction. Let \(x^\star \in \mathbb{R}^{T}\) denote the target waveform and \(\hat{x} \in \mathbb{R}^{T}\) denote the model output. The available inputs may include a mixture waveform \(x_m\), a visual input \(v\) given as a video clip or sampled frames, and a text prompt \(c\) describing either the intended musical content or the target source. In the generation setting, no mixture is provided, and the model samples \(\hat{x} \sim p_{\theta}(x \mid \mathcal{C}_{\mathrm{gen}})\), where \(\mathcal{C}_{\mathrm{gen}} \in \{\{c\}, \{v\}, \{c,v\}\}\). In the extraction setting, the observed mixture is provided together with at least one target-identifying condition, and the model samples \(\hat{x} \sim p_{\theta}(x \mid x_m, \mathcal{C}_{\mathrm{ext}})\), where \(\mathcal{C}_{\mathrm{ext}} \in \{\{c\}, \{v\}, \{c,v\}\}\). The mixture supplies the acoustic context, while the text or visual condition specifies the source to be recovered; we therefore exclude mixture-only extraction, since the mixture alone does not define a unique target. We refer to the mixture-conditioned task as target-source extraction. Our goal is to support both operating modes with a single model, a shared continuous latent representation, and a common inference procedure without task-specific output heads.

Method Overview. MAGE represents audio in a continuous latent space and models both conditional music generation and mixture-grounded target-source extraction with a shared flow-based backbone, as shown in Fig. 1. An audio waveform is first mapped to latent tokens using MixWavCodec, whose encoder and decoder are trained to reconstruct audio from the latent representation. The target latent is then generated by the Controlled Multimodal FluxFormer, a Transformer-based flow model that learns a conditional trajectory from noise to the target audio latent. When no mixture is provided, the model generates audio from text, visual input, or both. When a mixture is available, its encoded latent is supplied as an additional acoustic condition, while text or visual input specifies the target source to be recovered. To incorporate visual information, MAGE uses Audio–Visual Nexus Alignment (AVNA) to map frame-level visual features onto the audio latent sequence. The aligned visual representation is then used by a cross-gated modulation mechanism to condition intermediate audio features, while text is introduced through a separate semantic-conditioning pathway. During training, dynamic modality masking exposes the same model to text-only, visual-only, joint text–visual, mixture-conditioned, and unconditional configurations. This allows a single model to be evaluated across the supported generation and extraction settings without task-specific output heads.

3.1 Continuous Audio Representation↩︎

MAGE operates in a continuous audio-latent space defined by a learned waveform codec, which we refer to as MixWavCodec. The codec reduces the temporal length of the audio sequence before flow modeling and decodes generated latents back to waveform space. Given an audio waveform \(x\in\mathbb{R}^{T}\), the encoder maps it to latent tokens \(z=\mathcal{E}_{\phi}(x)\in\mathbb{R}^{N\times D}\), and the decoder reconstructs the waveform as \(\tilde{x}=\mathcal{D}_{\phi}(z)\), where \(N\) is the number of latent tokens and \(D\) is the latent dimension. We use a continuous bottleneck rather than discrete audio tokens so that the flow model can predict target audio latents directly in a differentiable latent space. The codec is trained as an audio reconstruction model using both isolated sources and randomly mixed audio segments. The mixture examples are used as data augmentation to expose the codec to overlapping musical content encountered in the extraction setting, while the main modeling contribution lies in the shared conditional latent flow built on top of this representation. For two sampled audio segments \(x^{(1)}\) and \(x^{(2)}\), we form a training mixture \(x_m=x^{(1)}+\lambda x^{(2)}\), where \(\lambda\sim\mathcal{U}[\lambda_{\min},\lambda_{\max}]\).

To make the decoder less sensitive to small deviations in latent space, we apply latent perturbation during codec training. Specifically, we perturb the encoded latent as \(\tilde{z}=z+\sigma\epsilon\), where \(\epsilon\sim\mathcal{N}(0,I)\), and reconstruct the waveform from \(\tilde{z}\). This augmentation exposes the decoder to latent states near the encoded data manifold, which may occur when the flow model generates latents at inference time. The codec is optimized with reconstruction and adversarial objectives: \[\min_{\mathcal{E}_{\phi},\mathcal{D}_{\phi}} \mathcal{L}_{\mathrm{rec}} + \alpha \mathcal{L}_{\mathrm{adv}} + \beta \mathcal{L}_{\mathrm{fm}}, \qquad \min_{\mathcal{D}_{\mathrm{adv}}} \mathcal{L}_{\mathcal{D}_{\mathrm{adv}}}. \label{eq:codec95objective}\tag{1}\] Here, \(\mathcal{L}_{\mathrm{rec}}\) combines waveform and spectral reconstruction terms, \(\mathcal{L}_{\mathrm{adv}}\) is the adversarial loss, and \(\mathcal{L}_{\mathrm{fm}}\) is the feature-matching loss. The discriminator is used only for codec training. In the full MAGE pipeline, the trained codec provides the latent representation for both mixture-free generation and mixture-grounded target-source extraction. The codec and latent perturbation are evaluated through reconstruction metrics and downstream ablations to quantify their effect on the shared latent modeling pipeline.

3.2 Conditional Flow Matching↩︎

MAGE models target audio generation as conditional flow matching in the continuous latent space of the pretrained codec. Let \(z_1=\mathcal{E}_{\phi}(x^\star)\in\mathbb{R}^{N\times D}\) denote the target audio latent, and let \(z_0\sim\mathcal{N}(0,I)\) be a Gaussian noise latent of the same shape. For \(t\sim\mathcal{U}(0,1)\), we construct the interpolation \(z_t=(1-t)z_0+t z_1\) with target velocity \(u_t=z_1-z_0\). The Controlled Multimodal FluxFormer \(F_{\theta}\) predicts the conditional velocity \(\hat{u}_t=F_{\theta}(z_t,\widetilde{\mathcal{C}},t)\), where \(\widetilde{\mathcal{C}}\) is the effective condition set after modality masking. The model is trained with the velocity-matching objective \[\mathcal{L}_{\mathrm{FM}} = \mathbb{E}_{z_0,z_1,t,\widetilde{\mathcal{C}}} \left[ \left\| F_{\theta}(z_t,\widetilde{\mathcal{C}},t)-u_t \right\|_{1} \right]. \label{eq:flow95matching95loss}\tag{2}\]

In the generation setting, \(\widetilde{\mathcal{C}}\) contains text, visual input, or both, with the mixture branch set to its null condition. In the extraction setting, \(\widetilde{\mathcal{C}}\) additionally contains the mixture latent \(z_m=\mathcal{E}_{\phi}(x_m)\) and at least one target-identifying condition. At inference, we sample \(z_0\sim\mathcal{N}(0,I)\), integrate the learned vector field from \(t=0\) to \(t=1\), obtain the predicted target latent \(\hat{z}_1\), and reconstruct the waveform as \(\hat{x}=\mathcal{D}_{\phi}(\hat{z}_1)\). The same FluxFormer parameters and inference procedure are used for both mixture-free generation and mixture-grounded target-source extraction; only the selected condition set changes.

Multimodal Condition Encoders. MAGE conditions the latent flow on text, visual input, and, when available, mixture audio. A text encoder \(\phi_{\mathrm{text}}\) maps the prompt \(c\) to a semantic embedding \[p = \phi_{\mathrm{text}}(c), \label{eq:text95encoder}\tag{3}\] which provides global guidance about the intended musical content or the target source. A visual encoder \(\phi_{\mathrm{vis}}\) extracts frame-level features from a visual input \(v=\{I_k\}_{k=1}^{K}\): \[s_k = \phi_{\mathrm{vis}}(I_k), \qquad k=1,\ldots,K . \label{eq:visual95encoder}\tag{4}\] For mixture-grounded target-source extraction, the mixture waveform is encoded by the audio codec as \[z_m = \mathcal{E}_{\phi}(x_m). \label{eq:mixture95encoder}\tag{5}\] The mixture latent provides acoustic context, while the text or visual condition identifies the target source to be recovered. When a modality is unavailable, it is replaced by a null condition and handled through the modality-masking strategy described in Sec. 3.3.

Audio–Visual Nexus Alignment. The audio latent sequence contains \(N\) tokens, whereas the visual encoder produces \(K\) frame-level features. Directly fusing these sequences is not well defined because they have different temporal resolutions. To address this, Audio–Visual Nexus Alignment (AVNA) maps the frame-level visual features to the audio latent resolution before visual conditioning is applied.

Let \(\tau_i^a\) and \(\tau_k^v\) denote normalized positions of audio latent token \(i\) and visual frame \(k\): \[\tau_i^a = \frac{i-1}{N-1}, \qquad \tau_k^v = \frac{k-1}{K-1}.\] For each audio latent index \(i\), AVNA assigns the nearest visual feature in normalized time: \[\pi(i) = \arg\min_k \left|\tau_i^a-\tau_k^v\right|, \qquad \bar{s}_i = s_{\pi(i)} . \label{eq:avna}\tag{6}\] This produces an aligned visual sequence \(\bar{s}\in\mathbb{R}^{N\times d_v}\) with the same length as the audio latent sequence. AVNA provides feature-resolution alignment between sampled visual frames and audio latent tokens, allowing visual cues to be applied consistently across the latent trajectory. Since the visual encoder operates on sampled frames, the aligned features primarily capture instrument and scene-level evidence rather than explicit onset-level motion cues.

Cross-Gated Visual Modulation. After AVNA, the aligned visual features are used to modulate intermediate audio representations in the FluxFormer. Let \(h_i^{(\ell)}\) denote the audio feature at layer \(\ell\) and latent index \(i\). We compute a visual gate \[g_i^{(\ell)} = \sigma \left( W_s^{(\ell)}\bar{s}_i + b_s^{(\ell)} \right), \label{eq:visual95gate}\tag{7}\] where \(\sigma(\cdot)\) is a sigmoid activation and \(W_s^{(\ell)}\) is a learnable projection. The gated audio feature is then \[\tilde{h}_i^{(\ell)} = h_i^{(\ell)} \odot g_i^{(\ell)} . \label{eq:cgm}\tag{8}\]

This modulation lets visual information guide the audio-token representation without directly injecting visual features into the audio stream. Text is handled through a separate semantic-conditioning pathway, reflecting its role as global guidance rather than frame-level evidence. Together, AVNA and cross-gated modulation provide a structured mechanism for incorporating visual cues into the latent flow. Their contribution is evaluated through ablations on generation quality, extraction quality, and interference suppression.

3.3 Dynamic Modality Masking and Inference↩︎

Dynamic Modality Masking. MAGE is trained with dynamic modality masking so the same conditional flow model can support multiple valid conditioning settings. Let the available condition set be \(\mathcal{C}=\{z_m,p,\bar{s}\}\), where \(z_m=\mathcal{E}_{\phi}(x_m)\) is the mixture latent, \(p=\phi_{\mathrm{text}}(c)\) is the text embedding, and \(\bar{s}\) is the AVNA-aligned visual sequence. During training, we sample a binary mask \(m=(m_m,m_t,m_v)\in\{0,1\}^{3}\) and construct the effective condition set \(\widetilde{\mathcal{C}}=\{\tilde{z}_m,\tilde{p},\tilde{s}\}\), where missing modalities are replaced by null conditions: \(\tilde{z}_m=m_m z_m+(1-m_m)z_{\varnothing}\), \(\tilde{p}=m_t p+(1-m_t)p_{\varnothing}\), and \(\tilde{s}=m_v\bar{s}+(1-m_v)s_{\varnothing}\).

We sample masks only from supported task configurations. For mixture-free generation, the valid masks are \(\mathcal{M}_{\mathrm{gen}}=\{(0,1,0),(0,0,1),(0,1,1)\}\), corresponding to text-only, visual-only, and joint text–visual conditioning. For mixture-grounded target-source extraction, the valid masks are \(\mathcal{M}_{\mathrm{ext}}=\{(1,1,0),(1,0,1),(1,1,1)\}\), where the mixture is paired with at least one target-identifying condition. We exclude the mixture-only mask \((1,0,0)\) because the mixture alone does not specify which source should be recovered. The null mask \((0,0,0)\) is also sampled during training for unconditional/CFG learning.

Given target latent \(z_1=\mathcal{E}_{\phi}(x^\star)\), noise \(z_0\sim\mathcal{N}(0,I)\), and \(t\sim\mathcal{U}(0,1)\), we form \(z_t=(1-t)z_0+t z_1\) with target velocity \(u_t=z_1-z_0\). The FluxFormer predicts \(\hat{u}_t=F_{\theta}(z_t,\widetilde{\mathcal{C}},t)\) and is trained with the masked flow-matching loss \[\mathcal{L}_{\mathrm{FM}} = \mathbb{E}_{z_0,z_1,t,m} \left[ \left\| F_{\theta}(z_t,\widetilde{\mathcal{C}},t)-u_t \right\|_1 \right]. \label{eq:masked95fm95loss}\tag{9}\] This trains one conditional vector field across the supported generation and extraction settings without task-specific output heads.

Inference. At inference time, no random masking is applied; the condition set is chosen according to the requested operating mode. For mixture-free generation, the mixture branch is set to its null condition and the model is conditioned on text, visual input, or both. For mixture-grounded target-source extraction, the mixture latent \(z_m=\mathcal{E}_{\phi}(x_m)\) is provided together with at least one target-identifying condition. Starting from \(z_0\sim\mathcal{N}(0,I)\), we integrate the learned vector field from \(t=0\) to \(t=1\) to obtain \(\hat{z}_1\) and decode the waveform as \(\hat{x}=\mathcal{D}_{\phi}(\hat{z}_1)\). The same model parameters, ODE solver, and decoder are used for all supported configurations; only the selected condition set changes.

Classifier-Free Guidance. We optionally use classifier-free guidance (CFG) at inference. During training, the null-conditioning mask \((0,0,0)\) is sampled with non-zero probability, allowing the FluxFormer to learn both conditional and unconditional vector fields. Given a valid condition set \(\widetilde{\mathcal{C}}\), we compute \(\hat{u}_t^{\mathrm{cond}}=F_{\theta}(z_t,\widetilde{\mathcal{C}},t)\) and \(\hat{u}_t^{\mathrm{uncond}}=F_{\theta}(z_t,\mathcal{C}_{\varnothing},t)\), where \(\mathcal{C}_{\varnothing}\) denotes the null condition for all modalities. The guided velocity is \[\hat{u}_t^{\mathrm{cfg}} = \hat{u}_t^{\mathrm{uncond}} + \gamma \left( \hat{u}_t^{\mathrm{cond}} - \hat{u}_t^{\mathrm{uncond}} \right), \label{eq:cfg}\tag{10}\] where \(\gamma\) is the guidance scale. We keep \(\gamma\) fixed for reported comparisons unless otherwise stated.

4 Experiments↩︎

Dataset. We evaluate MAGE on the MUSIC dataset [13], a widely used benchmark for audio–visual musical source separation. MUSIC contains in-the-wild videos of musical performances with instrument-level labels and corresponding audio tracks. We use the dataset for two evaluation settings. For mixture-grounded target-source extraction, the model receives an observed mixture together with text and/or visual conditions that identify the target source. For mixture-free generation, the model receives text, visual input, or both, but no mixture audio. Instrument labels are converted into simple text prompts, while sampled video frames provide the visual condition. We follow the standard train/validation/test protocol used by prior MUSIC evaluations and keep the generation and extraction test settings separate.

Evaluation Protocols and Metrics. We evaluate MAGE under two task-specific protocols. For mixture-grounded target-source extraction, we report standard source-separation metrics: SDR, SIR, and SAR. SDR measures overall reconstruction quality, SIR measures suppression of interfering sources, and SAR measures artifact-related distortion. These metrics are computed only when a ground-truth target source is available.

For mixture-free generation, separation metrics are not applicable because there is no observed mixture and no uniquely defined separated target. We therefore report generation-oriented metrics that measure audio realism and conditioning alignment. Specifically, we use CLAP-based audio–audio similarity to measure semantic similarity between generated and reference audio, and CLAPScore to evaluate alignment between generated audio and the text condition. Because text conditioning and automatic evaluation both use CLAP-based representations, these metrics may favor semantic alignment captured by CLAP. We therefore interpret them as automatic alignment indicators rather than substitutes for perceptual listening evaluation.

This separation of protocols is important because MAGE supports both operating modes, but the two modes measure different capabilities. Extraction evaluates recovery of a specified source from a mixture, while generation evaluates whether the model can synthesize plausible and condition-aligned music without using mixture audio.

Notation. Let \(x \in \mathbb{R}^{T}\) denote an audio waveform, \(x_m\) denote an observed mixture waveform when available, and \(x^\star\) denote the ground-truth target waveform. The codec encoder \(\mathcal{E}_{\phi}\) maps audio to a continuous latent sequence, and the decoder \(\mathcal{D}_{\phi}\) maps latents back to waveform space: \[z = \mathcal{E}_{\phi}(x) \in \mathbb{R}^{N \times D}, \qquad \hat{x} = \mathcal{D}_{\phi}(z).\] We denote the target latent as \(z_1=\mathcal{E}_{\phi}(x^\star)\) and, when a mixture is provided, the mixture latent as \(z_m=\mathcal{E}_{\phi}(x_m)\). A text prompt \(c\) is encoded as \(p=\phi_{\mathrm{text}}(c)\), and a visual input \(v=\{I_k\}_{k=1}^{K}\) is encoded into frame-level features \(s_k=\phi_{\mathrm{vis}}(I_k)\). AVNA maps these frame-level features to the audio latent resolution, producing aligned visual tokens \(\bar{s}\in\mathbb{R}^{N\times d_v}\).

To support multiple conditioning settings, we define a modality mask \(m=(m_m,m_t,m_v)\in\{0,1\}^{3}\) over mixture, text, and visual conditions. The effective condition set is \[\widetilde{\mathcal{C}} = \{\tilde{z}_m,\tilde{p},\tilde{s}\},\] where unavailable modalities are replaced by their corresponding null conditions. We use only valid task masks: text-only, visual-only, and text–visual masks for mixture-free generation; and mixture plus at least one target-identifying condition for target-source extraction. The mixture-only configuration is excluded because the mixture alone does not specify which source should be recovered.

Implementation Details. We implement MAGE in PyTorch and train it on MUSIC using 64,640-sample audio segments at 11.025 kHz. MixWavCodec maps each waveform to \(N=202\) continuous latent tokens with dimension \(D=128\). After codec training, its encoder and decoder are frozen while training the FluxFormer. For visual conditioning, we sample 47 frames at 8 fps and encode them with CLIP ViT-B/32; AVNA then maps the frame features to the audio latent resolution. Text prompts are encoded using a frozen CLAP text encoder, with MUSIC instrument labels converted into simple text prompts. The Controlled Multimodal FluxFormer is a flow-matching Transformer with hidden size 1024 operating on the 202-token audio latent sequence. The input latent \(z_t\) is projected to the model dimension and combined with learned positional embeddings. When available, the mixture latent \(z_m\) is encoded by the same codec and provided as an additional condition. Text, visual, and mixture inputs are incorporated through their respective conditioning pathways, and cross-gated modulation is applied only to AVNA-aligned visual features.

We train the FluxFormer with the masked flow-matching objective in Eq. 9 , using an \(\ell_1\) velocity loss in latent space. Task configurations are sampled as mixture-grounded extraction with probability 0.70, mixture-free generation with probability 0.20, and null-conditioning for unconditional/CFG training with probability 0.10. Extraction always includes the mixture plus at least one target-identifying condition; mixture-only extraction is not used. When visual conditioning is present, static-frame augmentation is applied with probability 0.25. We use AdamW with learning rate \(10^{-4}\), batch size 32, gradient clipping at 5.0, and train for 2000 epochs. During this stage, the FluxFormer and trainable visual-conditioning layers are optimized, while the audio codec and CLAP text encoder remain frozen. At inference, we sample Gaussian noise in latent space and integrate the learned vector field with an Euler solver; all reported comparisons use the same integration steps and CFG scale unless otherwise stated. Additional implementation details are provided in the Supplementary Material.

4.1 Comparison with Prior Methods↩︎

We evaluate MAGE under separate extraction and generation protocols. Extraction measures target-source recovery from a mixture, while generation measures synthesis quality and conditioning alignment without mixture audio.

Mixture-Grounded Target-Source Extraction. Table 1 compares MAGE with representative audio–visual source-separation and target-source extraction methods on MUSIC [13], including Sound-of-Pixels [13], Co-Separation [14], CCoL [16], iQuery [19], and DAVIS [10]. These methods are specialized for mixture-conditioned extraction and remain strong task-specific baselines. MAGE targets a broader operating regime, using one shared conditional model for both extraction and mixture-free generation. In the extraction setting, MAGE achieves competitive interference suppression, as reflected by SIR, while SDR and SAR reflect the reconstruction-quality advantage of specialized extraction systems.

Mixture-Free Conditional Generation. Table 2 evaluates MAGE in the zero-mixture setting, where the model receives text, visual input, or both, but no mixture audio. This setting is not applicable to conventional separation systems because they require an observed mixture as input. We therefore evaluate mixture-free generation using CLAP-based semantic similarity and conditioning-alignment metrics. The text-only, visual-only, and text–visual results are close, indicating that MUSIC often provides overlapping instrument-level cues across modalities. This suggests that zero-mixture generation on this benchmark is primarily driven by coarse instrument semantics, while joint text–visual conditioning becomes more informative in the mixture-grounded setting where the target source must be identified within an observed mixture. These results demonstrate the additional zero-mixture operating mode supported by the same model, while extraction performance is evaluated separately with SDR, SIR, and SAR.

5 Ablation Study↩︎

We analyze the effect of MAGE’s main design choices on MUSIC [13]. Unless otherwise stated, all ablated models use the same backbone, training schedule, conditioning encoders, and evaluation protocol. We report SDR, SIR, and SAR for mixture-grounded target-source extraction.

Effect of AVNA and Cross-Gated Modulation. Table 3 shows that removing either Audio–Visual Nexus Alignment (AVNA) or cross-gated modulation reduces extraction performance. The drop in SIR is especially large, indicating that these components improve interference suppression in the mixture-grounded extraction setting. AVNA aligns visual features to the audio latent sequence, while cross-gated modulation uses the aligned visual features to regulate intermediate audio representations.

Fusion Operator Comparison. Table 4 compares cross-gated modulation with alternative fusion operators under the same training setup. Additive fusion, self-attention, and cross-attention all yield lower SDR, SIR, and SAR than cross-gated modulation. This suggests that using aligned visual features as multiplicative gates is more effective than directly adding or attending to visual features in this setting.

6 Conclusion↩︎

We presented MAGE, a modality-agnostic framework for conditional music generation and mixture-grounded target-source extraction within a shared continuous latent formulation. MAGE combines a learned audio codec, a conditional flow-matching Transformer, Audio–Visual Nexus Alignment, cross-gated visual modulation, and dynamic modality masking to support multiple text, visual, and mixture-conditioning settings with the same model. Experiments on MUSIC show that MAGE operates in both mixture-free generation and mixture-grounded extraction regimes, while ablations indicate that visual alignment and gated modulation improve extraction performance, particularly interference suppression. The results show that MAGE provides a unified conditioning interface across generation and extraction, while specialized separators continue to define strong task-specific references for reconstruction quality.

7 Supplementary Material↩︎

7.1 Implementation Details↩︎

Dataset Construction and Training Protocol. We construct training examples from MUSIC videos with instrument-level labels. Each example uses a fixed 5.86-second audio–video segment. During training, temporal crops are sampled from the available clip duration; during evaluation, fixed deterministic crops are used. Video frames are sampled uniformly from the same temporal window as the audio crop to keep the visual and audio inputs paired at the segment level. For mixture-grounded target-source extraction, each example is constructed from two source clips. One clip is designated as the target source and the other as the non-target source. After audio loading and augmentation, the observed mixture is formed by combining the target and non-target waveforms. The target source, non-target source, and mixture are then encoded into the codec latent space. The model receives the mixture together with at least one target-identifying condition, while the supervision target is the isolated target-source waveform. Instrument labels are converted into simple text prompts using the corresponding source label. For mixture-free conditional generation, the mixture branch is replaced by its learned null condition, and the model is conditioned on text, visual input, or both. This setting uses the same target-source audio as supervision but does not provide the observed mixture as input. During audio loading, waveform amplitudes are randomly scaled by a factor sampled from \([0.5,1.5]\) and clipped to \([-1,1]\). Standard spatial augmentations are applied to sampled video frames during training, while evaluation uses fixed resizing and center cropping. All compared extraction models are evaluated on the same held-out mixtures and target sources.

Audio Codec. All audio is resampled to 11.025 kHz, converted to mono, and cropped or padded to 64,640 samples, corresponding to approximately 5.86 seconds. We first train MixWavCodec as a continuous waveform codec and then use its frozen encoder and decoder as the audio representation layer for MAGE. The codec provides a compact continuous latent space in which the conditional flow model operates. Mixture examples are used during codec training as reconstruction augmentation, exposing the codec to overlapping musical content, but the codec itself is not treated as a standalone source-separation or generation model. The codec is trained on both isolated sources and randomly constructed mixtures so that the decoder observes the types of isolated and overlapping audio content encountered by the downstream MAGE pipeline. The same frozen codec is used to encode target sources, non-target sources, and observed mixtures. During MAGE training, the codec parameters remain fixed; the trainable components are the FluxFormer and the visual-conditioning layers.

The encoder–decoder follows an Oobleck-style architecture with weight-normalized 1D convolutions, residual units, and Snake activations. The encoder uses base width 128, channel multipliers \([1,2,4,8]\), stride pattern \([2,4,4,10]\), and a continuous VAE bottleneck. The decoder mirrors this hierarchy with transposed convolutions and residual refinement. The total temporal downsampling ratio is 320, yielding approximately 35 latent tokens per second. For each 64,640-sample input segment, the codec produces a continuous latent sequence of length \(N=202\) with token dimension \(D=128\). During codec training, we optimize waveform reconstruction, spectral reconstruction, adversarial, and feature-matching losses. We also apply latent perturbation by adding small Gaussian noise to encoded latents before decoding. This exposes the decoder to small deviations around the encoded latent manifold, which can occur when the flow model generates latent trajectories at inference time. This two-stage design separates waveform reconstruction from conditional latent modeling: MixWavCodec provides the frozen audio latent representation, while MAGE learns a conditional latent vector field under text, visual, and mixture conditions.

Controlled Multimodal FluxFormer. As shown in Fig. 4, the generative backbone of MAGE is the Controlled Multimodal FluxFormer, a Transformer-based flow model operating on continuous audio latent tokens. It uses hidden size 1024 and an encoder–bottleneck–decoder layout with skip connections between corresponding blocks. The model predicts the conditional velocity field used for flow matching in the codec latent space. At each flow step, the current latent state \(z_t\) is projected from \(\mathbb{R}^{128}\) to the Transformer hidden space and combined with learned positional embeddings. When a mixture is available, the mixture waveform is encoded by the same frozen audio codec to obtain \(z_m\), which is projected separately and provided as an acoustic condition. The projected current latent and projected mixture latent are concatenated before being passed to the Transformer. When no mixture is provided, the mixture branch is replaced by a learned null condition. Therefore, mixture-free conditional generation and mixture-grounded target-source extraction use the same backbone, latent representation, and inference procedure; only the selected condition set changes.

Text and Visual Conditioning. Text prompts are encoded using a frozen CLAP text encoder (laion/clap-htsat-unfused) [28]. We use the pooled 768-dimensional text representation, project it to the Transformer hidden space, and insert it as a global semantic-conditioning token. This provides prompt-level semantic guidance without assuming temporal correspondence between text and audio tokens. Visual conditioning is extracted using CLIP ViT-B/32 [29]. For each 5.86-second clip, we uniformly sample 47 frames at 8 fps and project each frame embedding to the Transformer hidden space. The resulting frame-level visual sequence is passed to AVNA, which maps visual features to the audio latent resolution before cross-gated visual modulation. These features provide instrument- and scene-level evidence about the depicted source. Since the visual encoder operates on sampled frames rather than dense motion features, fine-grained sound-producing actions and precise audio–visual synchronization are not explicitly modeled.

Audio–Visual Nexus Alignment. As shown in Fig. 3, Audio–Visual Nexus Alignment (AVNA) maps frame-level visual features to the audio latent resolution before visual conditioning is applied. The visual encoder produces \(T_v\) sampled-frame features, while the audio codec produces \(N\) continuous audio latent tokens. Since these sequences have different resolutions, direct token-wise fusion is not well defined. AVNA provides a deterministic feature-resolution alignment by assigning each audio latent token to the nearest sampled visual feature in normalized time.

Let \(\tau_i^a=(i-1)/(N-1)\) denote the normalized position of audio latent token \(i\), and let \(\tau_k^v=(k-1)/(T_v-1)\) denote the normalized position of visual frame \(k\). AVNA assigns \[\pi(i) = \arg\min_k |\tau_i^a-\tau_k^v|, \qquad \bar{s}_i = s_{\pi(i)} ,\] producing an aligned visual sequence \(\bar{s}\in\mathbb{R}^{N\times d_v}\). This resampling step introduces no additional trainable alignment parameters and simply provides visual features at the same token resolution as the audio latent sequence. We therefore treat AVNA as feature-resolution alignment for conditioning.

Cross-Gated Visual Modulation. After AVNA maps visual features to the audio latent resolution, we apply Cross-Gated Modulation (CGM) to condition intermediate audio representations. Let \(\mathbf{h}_{\mathrm{audio}}\in\mathbb{R}^{N\times d}\) denote the audio token stream at a FluxFormer layer, and let \(\bar{\mathbf{s}}\in\mathbb{R}^{N\times d_v}\) denote the AVNA-aligned visual features. CGM computes a visual gate and applies it multiplicatively: \[\mathbf{g} = \sigma \left( W_s \bar{\mathbf{s}} + b_s \right), \qquad \mathbf{h}_{\mathrm{joint}} = \mathbf{h}_{\mathrm{audio}} \odot \mathbf{g}, \label{eq:cgm95impl}\tag{11}\] where \(W_s\) and \(b_s\) are learnable parameters and \(\sigma(\cdot)\) is the sigmoid function. This operation lets aligned visual features regulate the audio-token representation through a learned gate rather than through direct feature addition. Text conditioning is handled separately through a global semantic token, while visual conditioning is applied at the audio-token resolution through the aligned visual sequence. We compare CGM with additive fusion, gated residual fusion, self-attention, and cross-attention in Sec. 7.3.

Flow-Matching Objective. We train MAGE with continuous-time flow matching in the codec latent space. Let \(z_1=\mathcal{E}_{\phi}(x^\star)\) denote the target audio latent, and let \(z_0\sim\mathcal{N}(0,I)\) be a Gaussian noise latent with the same shape. For \(t\sim\mathcal{U}(0,1)\), we define the linear interpolation path \[z_t = (1-t)z_0 + tz_1 . \label{eq:fm95path}\tag{12}\] The corresponding target velocity is \[u_t = z_1 - z_0 . \label{eq:fm95target95velocity}\tag{13}\] Given the masked condition set \(\widetilde{\mathcal{C}}\), the FluxFormer predicts \[\hat{u}_t = F_{\theta}(z_t,\widetilde{\mathcal{C}},t). \label{eq:fm95prediction}\tag{14}\] We optimize the latent velocity-matching objective \[\mathcal{L}_{\mathrm{FM}} = \mathbb{E}_{z_0,z_1,t,\widetilde{\mathcal{C}}} \left[ \left\| \hat{u}_t-u_t \right\|_1 \right]. \label{eq:fm95loss95final}\tag{15}\] This objective trains the model to predict the conditional vector field from Gaussian noise to the target audio latent. The same objective is used for all supported conditioning configurations. In mixture-free conditional generation, \(\widetilde{\mathcal{C}}\) contains text, visual input, or both, with the mixture branch set to its learned null condition. In mixture-grounded target-source extraction, \(\widetilde{\mathcal{C}}\) additionally includes the mixture latent and at least one target-identifying condition. Thus, the same model learns a conditional latent vector field across both operating modes; only the selected condition set changes.

Dynamic Modality Masking. We train MAGE with dynamic modality masking so that one conditional backbone supports the valid generation and extraction interfaces. For each training sample, we select one of three regimes: mixture-grounded target-source extraction with probability 0.70, mixture-free conditional generation with probability 0.20, and null-conditioning for classifier-free guidance with probability 0.10. In the extraction regime, the mixture latent is always provided together with at least one target-identifying condition. The valid extraction masks are mixture+text, mixture+visual, and mixture+text+visual. In the generation regime, the mixture branch is replaced by its learned null condition, and the valid masks are text-only, visual-only, and text+visual. We exclude mixture-only extraction because the mixture alone does not define a unique target source. This masking strategy exposes the same conditional flow model to all supported conditioning configurations without using task-specific output heads.

Static-Frame Augmentation. When visual conditioning is present during training, we apply static-frame augmentation with probability 0.25 by repeating one randomly selected frame across the visual sequence. This exposes the visual-conditioning pathway to both sampled video-frame sequences and still-frame inputs. We use this only as a visual-conditioning augmentation; it is not intended to model fine-grained audio–visual synchronization.

Optimization and Data Construction. We train the FluxFormer and trainable visual-conditioning layers with AdamW using learning rate \(10^{-4}\), batch size 32, and gradient clipping at 5.0, while keeping the audio codec and CLAP text encoder frozen. Training runs for 2000 epochs. To increase variation in source pairings and temporal crops, the training set is repeated five times per epoch with independently sampled pairings and augmentations. Each extraction training example is constructed from two source clips. One clip is treated as the target source and the other as the non-target source. After audio loading and augmentation, the observed mixture is formed by combining the target and non-target waveforms. The target source, non-target source, and mixture are then encoded into the codec latent space. During audio loading, waveform amplitudes are randomly scaled by a factor sampled from \([0.5,1.5]\) and clipped to \([-1,1]\). Standard spatial augmentations are applied to sampled video frames during training, while evaluation uses fixed deterministic preprocessing.

Inference. At inference time, we solve the learned latent flow with an Euler integrator using the same number of solver steps and the same guidance scale for all reported settings. For mixture-free conditional generation, the trajectory starts from Gaussian noise and is conditioned on text, visual input, or both, with the mixture branch set to its learned null condition. For mixture-grounded target-source extraction, the mixture latent is provided together with at least one target-identifying condition. The predicted target latent sequence is decoded to waveform using the frozen MixWavCodec decoder. The same latent representation, conditioning interface, decoder, and solver are used in both operating modes; only the selected condition set changes.

Sampling-Step and Guidance Sensitivity. We evaluate sensitivity to the number of Euler solver steps and classifier-free guidance (CFG) scale for mixture-grounded target-source extraction. The trained model, conditioning setup, and evaluation procedure are kept fixed; only the solver-step count and CFG scale are varied. As shown in Table 5, the default setting of 2 solver steps and CFG scale 1.5 gives the best SDR and SIR, indicating the strongest overall reconstruction and interference suppression. Increasing the guidance scale to 2.0 improves SAR but reduces SDR and SIR, suggesting a trade-off between artifact reduction and source selectivity.

8pt

7.2 Evaluation Metrics↩︎

We evaluate MAGE under two separate protocols. In mixture-grounded target-source extraction, the model receives an observed mixture and a target-identifying condition, and the goal is to recover the specified source. In mixture-free conditional generation, the model receives text, visual input, or both, but no mixture audio. Since these settings measure different capabilities, we report source-separation metrics only for extraction and CLAP-based automatic alignment metrics for mixture-free generation.

Source-Separation Metrics. For mixture-grounded target-source extraction, we report SDR, SIR, and SAR using the standard BSS-Eval protocol. Let \(\hat{s}\) denote the predicted source and \(s\) denote the ground-truth target source. SDR measures overall reconstruction quality, SIR measures suppression of non-target sources, and SAR measures artifact-related distortion. Higher values indicate better extraction quality. These metrics are used only in the mixture-grounded setting, where a reference target source is available. Dedicated separation methods are treated as task-specific baselines for extraction. MAGE is evaluated as a shared conditional model that supports both mixture-grounded extraction and mixture-free conditional generation, rather than as a replacement for specialized source-separation systems.

Generation and Alignment Metrics. For mixture-free conditional generation, SDR, SIR, and SAR are not applicable because there is no observed mixture and no uniquely defined separated target. We therefore report CLAP-based automatic alignment metrics separately from the extraction metrics. CLAP text–audio similarity measures semantic alignment between the generated audio and the conditioning text prompt, while CLAP audio–audio similarity measures semantic similarity between generated audio and the reference target audio when a reference is available. Because CLAP is used both for text conditioning and automatic evaluation, these scores may favor semantic relationships captured by the CLAP embedding space. We therefore interpret CLAP-based metrics as automatic semantic-alignment indicators rather than substitutes for perceptual listening evaluation. These metrics are reported separately from SDR, SIR, and SAR to avoid conflating mixture-free generation quality with source-extraction quality.

CLAP Audio–Audio Similarity. We use CLAP audio–audio similarity as an automatic semantic-alignment measure between a predicted audio sample and a reference audio sample when such a reference is available. Let \(f_a(\cdot)\) denote the CLAP audio encoder. Given predicted audio \(\hat{s}\) and reference audio \(s\), we compute \[\mathrm{Sim}_{\mathrm{AA}} = \frac{ f_a(\hat{s})^\top f_a(s) }{ \|f_a(\hat{s})\|_2 \, \|f_a(s)\|_2 }.\] Higher values indicate stronger similarity in the CLAP audio embedding space. We use this metric to measure semantic correspondence to the reference target audio, while SDR, SIR, and SAR remain the signal-level metrics for mixture-grounded target-source extraction.

CLAP Text–Audio Similarity. We use CLAP text–audio similarity as an automatic measure of alignment between the predicted audio and the conditioning text prompt. Let \(f_t(\cdot)\) denote the CLAP text encoder and \(f_a(\cdot)\) denote the CLAP audio encoder. For prompt \(c\) and predicted audio \(\hat{s}\), we compute \[\mathrm{Sim}_{\mathrm{TA}} = \frac{ f_t(c)^\top f_a(\hat{s}) }{ \|f_t(c)\|_2 \, \|f_a(\hat{s})\|_2 }.\] Higher values indicate stronger text–audio alignment in the CLAP embedding space. Since CLAP is also used for text conditioning, we interpret CLAP-based scores as automatic semantic-alignment indicators rather than substitutes for perceptual listening evaluation. Together, the metrics are used under task-specific protocols. SDR, SIR, and SAR evaluate signal-level reconstruction, interference suppression, and artifact-related distortion for mixture-grounded target-source extraction. CLAP audio–audio similarity evaluates semantic similarity to the reference target audio when available, and CLAP text–audio similarity evaluates alignment with the conditioning text prompt. Since CLAP is also used for text conditioning, we interpret CLAP-based scores as automatic semantic-alignment indicators rather than substitutes for perceptual listening evaluation. We report these metrics separately to avoid conflating source-extraction quality with mixture-free generation quality.

7.3 Fusion Operator Ablation↩︎

We ablate how AVNA-aligned visual features are incorporated into the audio latent stream. All variants use the same backbone, training schedule, conditioning encoders, AVNA alignment, and evaluation protocol; they differ only in the visual fusion operator. We evaluate each variant on mixture-grounded target-source extraction and report SDR, SIR, and SAR in Table 6. We compare four alternatives against cross-gated modulation. Additive fusion directly adds projected visual features \(W_s\bar{s}\) to the audio representation \(h\). Gated residual fusion applies a learned gate to the projected visual features before adding them to the audio stream. Self-attention and cross-attention allow audio and visual tokens to interact through attention-based updates. These operators provide increasingly flexible feature interaction, but they either add visual information into the audio stream or update the audio representation through attention.

Cross-Gated Modulation instead uses the AVNA-aligned visual features to compute a multiplicative gate over the audio tokens: \[h_{\mathrm{out}} = h \odot \sigma(W_s\bar{s}+b_s).\] This design conditions the audio representation by modulating existing audio-token features rather than directly adding visual features. As shown in Table 6, cross-gated modulation obtains the best SDR, SIR, and SAR among the tested operators. The largest improvement is in SIR, suggesting that visual gating is particularly useful for suppressing non-target sources in the mixture-grounded extraction setting. This supports the use of aligned visual features as a conditioning gate rather than as directly fused auxiliary tokens.

8pt

7.4 Temporal vs. Static Visual Conditioning↩︎

We compare three visual-conditioning settings for mixture-grounded target-source extraction: static-image conditioning, shuffled-frame conditioning, and temporal-frame conditioning. In the static-image setting, one frame is sampled from the video segment and repeated across the visual sequence. In the shuffled-frame setting, the model receives the sampled frames but with their temporal order randomly permuted. In the temporal-frame setting, the model receives the uniformly sampled frame sequence in its original order. This comparison separates instrument- and scene-level evidence from temporal ordering effects. Table 7 shows that temporal-frame conditioning gives the best SDR and SIR. The improvement over static-image conditioning indicates that using multiple frames provides more useful target-identifying evidence than a single repeated frame. The improvement over shuffled-frame conditioning suggests that preserving the original frame order is beneficial for mixture-grounded extraction. The static-image variant obtains the highest SAR, suggesting fewer artifact-related distortions but weaker source selectivity. Overall, these results support the use of sampled temporal frame sequences, while also showing a trade-off between interference suppression and artifact control.

8pt

References↩︎