1 Introduction↩︎

Driven by systems like Sora-2 [1], [2], Veo-3 [3], [4], and Seedance-2 [5], [6], audio-visual generative models [7]–[11] are evolving from visual synthesizers into nascent “world simulators” [2]. While these models achieve single-modality fidelity, the synchronization of cross-modal events remains fragile (shown in upper row of Fig. 1). Such failures matter not only for perceptual realism, but also as a key observable proxy for whether a model has learned how actions and sounds relate in the real world.

However, measuring this new capability with current evaluation paradigms reveals a fundamental mismatch. Traditional metrics, such as onset-based synchronization accuracy [12], cross-modal contrastive similarity [13], and offset-detection scores [14], [15], are all grounded in a structural correctness assumption: they presume that audio and video content are drawn from reality, semantically matched, and require only simple temporal alignment. Nevertheless, modern generative outputs [1], [4], [6] often break this assumption entirely. Instead of clean temporal shifts [14], their failures frequently involve structural hallucinations (e.g., generating a sharp glass-shattering sound when a plastic cup falls onto a soft carpet), asymmetric relationships [10] (e.g., a visual action implying a sound that never occurs), and temporally diffuse events that defy simple onset detection [12] (e.g., the continuous pouring of liquid where the acoustic pitch fails to dynamically rise with the visual fluid level).

Because standard algorithmic matching cannot resolve these unpredictable structural errors, establishing a reliable ground truth for audio-visual synchronization fundamentally requires human cognition. The high perceptual fidelity of modern generative artifacts limits ordinary rule-based discrimination, which often fails to distinguish subtle structural violations from overall audio-visual quality. Consequently, expert human preference serves as the necessary oracle, shifting the evaluation paradigm from signal-level temporal offsets to genuine causal perception.

Translating this human intuition into a scalable automated metric, however, introduces a central methodological paradox. When evaluating complex generative errors, human annotators naturally rely on relative comparisons—judging which of two flawed audio tracks fits a video better is easier than directly scoring a single one in isolation. In contrast, a practical and deployable evaluation metric must be reference-free, assessing a single generated video-audio pair in the wild and outputting an absolute, continuous score. This disconnect between relative human perception and the need for an absolute measurement motivates our core research question: How can we distill the complex, comparative nature of human judgment regarding synchronization into a continuous, reference-free, and globally consistent automated metric?

In this paper, we systematically resolve this paradox through a three-stage methodological progression, as shown in the lower row of Fig. 2. First, to ground the subjective nature of human judgment, we construct the SynthSync dataset. Instead of using artificial time shifts [14], we curate authentic generative failures by sampling outputs from 10 state-of-the-art video-to-audio models (see Section 4 for complete list) given the same visual anchor. This allows us to 1) simulate the complex topology of synchronization errors in modern AV-Gen systems; and 2) collect fine-grained pairwise rankings (\(<\mathbf{v},\mathbf{a}_1>\) vs. \(<\mathbf{v},\mathbf{a}_2>\)) over expert human annotations.

Second, to enable reference-free scoring, we architecturally redesign an Omni-LLM. Specifically, we inherit the structural correctness knowledge embedded in Omni-LLM and adapt it for scoring tasks by replacing its discrete language head with a continuous projection head from the latent manifold. We initialize this scorer using a curriculum-based preference learning strategy, mapping relative human rankings into absolute scalar values via a Bradley-Terry-Luce formulation [16]. Finally, to enforce global consistency beyond local pairwise comparisons, we introduce a novel reinforcement learning framework: \(\mathbb{R}\)-GRPO (Real-Valued Group Relative Policy Optimization). By treating the estimator’s output as a stochastic Gaussian policy, we optimize the entire listwise distribution of scores against the ground truth topology, allowing the model to internalize the global causal structure of synchronization.

Empirically, extensive experiments demonstrate that our continuous estimator achieves state-of-the-art (SOTA) alignment with human preference in causal evaluation. Leveraging this validated metric, we establish a comprehensive benchmark, SyncBench, for contemporary AV-Gen models, providing the field with a standardized tool to assess the physical grounding of nascent world simulators.

Our contributions are fourfold:

• SynthSync: The first dataset of authentic generative synchronization failures, moving the field beyond artificial time-shift benchmarks.

• Continuous Omni-LLM Evaluator: A novel architecture that repurposes multimodal foundation models for continuous, reference-free causal consistency scoring.

• \(\mathbb{R}\)-GRPO Framework: A scalable reinforcement learning algorithm that aligns continuous latent policies with discrete human preferences, establishing a new standard for physically grounded generative evaluation.

• SOTA Alignment and SyncBench: We demonstrate SOTA performance in human preference alignment and establish a standardized benchmark for evaluating AV-Gen models.

2 Related Work↩︎

2.1 Audio-Video Joint Generation↩︎

Early audio-visual synthesis commonly adopted a cascaded design, in which video generation [17]–[29] and audio generation [30]–[35] were treated as separate stages. A silent video was first generated, and a downstream video-to-audio [36]–[46] (V2A) model was then used to synthesize the soundtrack.

More recent work moves toward native joint generation. A prominent design is the Dual-Branch Diffusion Transformer, adopted by the Seedance family [5], [6] and related systems [7]–[11], [47]–[55], where audio and video are modeled in parallel by modality-specific transformer branches coupled through cross-modal interaction modules. By contrast, closed-source systems such as Sora-2 and the Veo-3 series provide little public technical detail, leaving their underlying solutions unavailable for open analysis.

2.2 Audio-Video Synchronization Evaluation↩︎

Audio-visual synchronization has traditionally been measured by algorithmic metrics such as SyncNet [56], PEAVS [57], and AlignNet [15]. SyncNet evaluates alignment mainly through global temporal offsets between speech audio and lip motion. PEAVS improves perceptual modeling, but generative evaluation in practice is still often dominated by unimodal fidelity measures such as FAD and FID, which do not explicitly model cross-modal temporal dependence. AlignNet, meanwhile, is designed for human motion–audio alignment and is not readily transferable to general open-domain scenes.

These approaches become increasingly inadequate for modern generative outputs. They typically assume that audio and video are structurally correct up to a simple temporal shift, that synchronization is expressed by sparse transient events such as lip closures or percussive onsets, and that cross-modal similarity can be captured by symmetric matching. Such assumptions break down in realistic generative failures, where errors are often semantic, asymmetric, and temporally diffuse.

Recent methods increasingly rely on pretrained cross-modal encoders, e.g., ImageBind [13], to assess synchronization [10]. However, such models are still constrained by pretraining objectives that emphasize global and compressed cross-modal correspondence rather than fine-grained causal timing. Related methods such as AV-Align further formulate synchronization as an onset- or event-matching problem, which is less suitable for continuous phenomena such as flowing water, fire, or explosions, where synchronization is gradual rather than event-localized.

2.3 LLM-as-Judge↩︎

2.3.0.1 Evaluating Language Responses.

LLM-as-Judge effectiveness hinges on evaluator design rather than model scale. Prior work mitigates bias via structured prompting[58], dynamic multi-agent judges [59], [60], and contrastive scoring [61]; or enhances reasoning through pairwise comparison [62], continuous ranking [63], and RL-based training [64]. However, these overlook temporally grounded cross-modal generation, where errors are asymmetric, structurally unconstrained, and ill-suited to text-centric evaluation criteria.

2.3.0.2 Evaluating Multi-Modal Synthesis.

LMM/VLM-based evaluators primarily target image/video quality via discrete text-defined scoring[65], pairwise-to-score conversion [66], [67], and RL-based quality understanding [68]–[70], with extensions to document images [71], and diagnostic flaw maps for text-to-image generation [72]. Yet these address perceptual quality or aesthetics, not the directed causal-semantic consistency central to audio-visual synchronization—where visual events must yield physically coherent acoustic outcomes. This gap motivates our specialized dataset, supervision signal, and continuous RL-based judge formulation tailored to authentic V2A synchronization failures.

3 Method↩︎

Evaluating audio-visual synchronization in generative models demands perceiving complex, structurally unconstrained semantic errors. Operationalizing this requires resolving a central methodological paradox. On one hand, human annotation of generative errors is inherently relative—annotators require an anchor to make reliable psychophysical judgments. On the other hand, a practical evaluation metric must be absolute and reference-free—capable of observing a single generated video-audio pair in the wild and outputting a continuous scalar. Our methodology systematically dismantles this tension through a novel dataset curation strategy (SynthSync), a continuous Omni-LLM architecture, and a two-stage topological alignment paradigm, with Fig. 2 depicting the framework.

3.1 The Annotation Paradox and the SynthSync Dataset↩︎

Developing a robust evaluator requires a human-aligned supervisory signal. However, the community currently lacks a dataset capable of benchmarking the true nature of modern generative failures. Existing evaluation protocols rely almost exclusively on artificial linear offsets—mechanically shifting ground-truth audio forward or backward. This approach fundamentally misrepresents the generative landscape, heavily rewarding models that merely detect offset artifacts rather than those that understand physical causality.

To bridge this gap, we curate SynthSync, the first dataset dedicated to authentic generative synchronization failures. Given an underlying real-world visual stream \(\mathbf{v}\), we generate multiple distinct acoustic tracks by passing the video through \(K\) distinct, cutting-edge video-to-audio (V2A) foundation models, forming a candidate pool \(\mathcal{A}(\mathbf{v})=\{\mathbf{a}_1,...,\mathbf{a}_K\}\). This strategy offers two critical advantages over traditional dataset construction:

1. Simulating Authentic Structural Chaos: Unlike manual temporal shifting, multi-model generation natively surfaces the complex, non-linear failure modes endemic to modern AV-Gen systems. This includes semantic hallucinations (e.g., a dog barking with a bird’s chirp), temporal diffusion (continuous water flow without distinct onsets), and severe causal omissions.

2. Visual Anchoring for Psychophysical Consistency: Given a single heavily hallucinated audio-video pair, attempting to annotate an absolute synchronization “score” without references is an ill-posed psychophysical task, introducing extreme inter-annotator variance. By setting the underlying video \(\mathbf{v}\) as a fixed causal anchor, annotators are instead tasked with a comparative judgment: does candidate \(\mathbf{a}_i\) better match the video than candidate \(\mathbf{a}_j\)? This grounds the subjective nature of multimodal physics into strictly mathematically operable partial orderings (\(\mathbf{a}_i \succ_\mathbf{v} \mathbf{a}_j\)).

3.2 The Inference Requirement: Reference-Free Continuous Scoring↩︎

While SynthSync relies on relative rankings to guarantee human consistency, a viable Omni-LLM evaluator must be structurally reference-free at inference—capable of emitting a standalone metric for any \((\mathbf{v}, \mathbf{a})\) pair. Furthermore, the dominant “LLM-as-a-Judge” paradigm typically prompts models to emit discrete linguistic categories (e.g., “Good”, “Poor”). In the context of audio-visual physics, fine-grained temporal and semantic variations defy clean mapping into broad linguistic buckets; doing so inevitably introduces severe quantization artifacts and lexical biases.

Rather than forcibly categorizing human perception, we bypass token decoding altogether. Given a video \(\mathbf{v}\) and a generative candidate audio \(\mathbf{a}\), we extract a scalar quality representation directly from the underlying causal manifold of the model. We force it to output a persistent inference token [SCORE] into the multimodal context sequence. Letting \(\mathbf{h}_{\text{score}} \in \mathbb{R}^d\) denote the model’s ultimate hidden representation corresponding to this token, we append a specialized continuous projection head \(\text{MLP}_\phi\): \[s = f_\theta (\mathbf{v}, \mathbf{a}) = \text{MLP}_\phi \left(\mathbf{h}_{\text{score}}\right) \in \mathbb{R}\]

This parameterization cleanly strips away language-generation stochasticity, transmuting the Omni-LLM into a fully differentiable mapping \(f_\theta\) optimized exclusively for relative continuous scaling.

3.3 Stage I: Latent Preference Alignment via Curriculum↩︎

We are now faced with an optimization gap: our model \(f_\theta\) must output an absolute, reference-free scalar \(s\), but our ground truth consists exclusively of relative preference boundaries. Standard Mean Squared Error (MSE) regression introduces active harm by compressing highly contextual psychophysical phenomena into rigid numeric bounds.

We consequently abandon regression-based absolute distance, directly optimizing the global topological rank mapping via the Bradley-Terry-Luce (BTL) [16] formulation. For each paired ranking \(\mathbf{a}_i \succ_\mathbf{v} \mathbf{a}_j\), we minimize the localized loss over the logit decision boundary: \[\label{eq:BTL} \mathcal{L}_{\text{BTL}}(\theta) = -\mathbb{E}_{\mathbf{v}, \mathbf{a}_i \succ_\mathbf{v} \mathbf{a}_j} \left[ \log \text{Sig.} \big(f_\theta(\mathbf{v}, \mathbf{a}_i) - f_\theta(\mathbf{v}, \mathbf{a}_j)\big) \right],\tag{1}\] where Sig. represents the sigmoid activation. Although supervised only within each anchor, fitting all margins through a single shared \(f_\theta\) induces a globally consistent implicit value space—rather than a per-anchor calibration—making the absolute scalar \(s=f_\theta(\mathbf{v},\mathbf{a})\) directly usable as a reference-free metric across anchors at inference, as formalized in the reward-modeling analysis of DPO [73].

Curriculum Pacing: Uniformly sampling pairwise preferences risks gradient collapse due to the massive variance in generative errors—ranging from obvious semantic hallucinations to nuanced sub-frame asynchronies. To stabilize optimization, we structure the learning trajectory using a dynamically annealed curriculum. Given a set of pairwise annotations for a single video, we first establish a global ordinal ranking of the candidate audios by aggregating their empirical win-rates (the total number of head-to-head victories per candidate). Let \(\delta(i, j)\) denote the distance between two candidates in this derived global ranking. We initialize training by exclusively sampling pairs with large rank discrepancies—for instance, contrasting the top-ranked candidate (No. 1) with a severely degraded one (No. 9)—ensuring a strong, unambiguous initial gradient signal. As training progresses, we decay a threshold constraint \(\delta(i, j) \ge \gamma(t)\), safely uncoiling the margin to expose the model to finer, microscopic comparisons (e.g., No. 5 vs. No. 6) and steadily refining its perceptual boundaries.

3.4 Stage II: Causal Topologies via \(\mathbb{R}\)-GRPO↩︎

From Local Pairs to Global Lists: Pairwise learning of Eq. 1 suffers from metric myopia: establishing local margins (\(\mathbf{a}_i \succ_\mathbf{v} \mathbf{a}_j\)) does not guarantee a globally transitive scale. Consequently, the model may satisfy individual constraints while producing distorted, non-linear score distributions that fail on holistic ranking metrics. To enforce global topological consistency, we transcend pairwise proxies \(\{i,j|\forall i,j\le K,i\ne j\}\) and optimize the entire listwise distribution \(\{1,...,K\}\) directly using reinforcement learning (RL).

From Deterministic Regression to Stochastic Policy. However, applying standard RL presents a structural paradox: policy gradient methods fundamentally require stochastic exploration over a probability distribution to estimate advantages, whereas our evaluator \(f_\theta\) is a deterministic regressor outputting a fixed scalar. To bridge this gap, we propose \(\mathbb{R}\)-GRPO (Real-Valued Group Relative Policy Optimization). We reparameterize the deterministic scalar output not as a static value, but as the central manifold \(\mu_\theta\) of a Gaussian policy, injecting the necessary uncertainty to enable differentiable policy optimization.

Exploratory Listwise Rollouts: Rather than treating the final projection \(f_\theta(\mathbf{v}, \mathbf{a}_k)\) as an inflexible terminal state, we reconstruct it as the central manifold \(\mu_\theta\) of an unconstrained exploratory Gaussian policy space. During optimization rollouts for a single anchor video \(\mathbf{v}\), the current deterministic estimator issues base continuous values \(\{\hat{s}_1, \dots, \hat{s}_K\}\). To trigger dynamic evaluation environments, we sample \(N\) parallel execution paths by systematically perturbing the scores within a designated active-exploration variance \(\sigma^2\): \[\hat{s}_{k}^{(n)} = \hat{s}_{k} + \epsilon_{k}^{(n)}, \qquad \epsilon_{k}^{(n)} \sim \mathcal{N}(0, \sigma^2)\] This formulation projects the configuration group \(\mathcal{G}_i = \{\hat{\mathbf{s}}^{(1)}, \dots, \hat{\mathbf{s}}^{(N)}\} \subset \mathbb{R}^K\) directly into the continuous domain, natively unlocking deep state-space trajectories that bypass auto-regressive decoder constraints.

Composite Causal-Semantic Rewards: To guarantee the preservation of global causal topologies across the candidate space, we sample all of generated audios from the same video, and obtain their scores, which is used to against the ground-truth reference ranking of corresponding audio using a holistic Composite Ranking Reward \(r \in [0, 1]\). This formulation directly optimizes the structural consistency of the entire distribution list, which evaluates the sequence natively through three complementary topological lenses: 1) structural attenuation, 2) monotonic topological order, and 3) extreme target bounding. (See Appendix for detailed formulation.)

Group Bounded Policy Objectives: We adapt the GRPO framework [74] to continuous latent spaces, leveraging group-relative normalization to eliminate the need for a separate value critic. For a sampled ensemble of \(N\) trajectories, we compute variance-reduced advantages \(A^{(n)}\) relative to the local group mean \(\bar{r}\): \[A^{(n)} = \frac{r^{(n)} - \bar{r}}{\text{std}\big(\{r^{(n)}\}_{1}^{N}\big) + \epsilon}\] To ensure monotonic policy improvement, we constrain gradient updates using a proximal trust region. Since our policy is parameterized as a Gaussian \(\pi_\theta(\cdot|\mathbf{v},\mathbf{a}) = \mathcal{N}(\mu_\theta, \sigma^2)\), the importance sampling ratio \(\rho_{k}^{(n)}\) admits a closed-form analytical solution: \[\rho_{k}^{(n)} = \frac{\pi_\theta \left(\hat{s}_{k}^{(n)}\right)}{\pi_{\theta_{\text{old}}}\left(\hat{s}_{k}^{(n)}\right)} = \exp \!\left(-\frac{\left(\hat{s}_{k} - \hat{s}_{k}^{(n)}\right)^2}{2\sigma^2}\right)\]

The final objective \(\mathcal{L}_{\mathbb{R}\text{-GRPO}}\) integrates the clipped surrogate loss with a standard KL divergence penalty to strictly regularize the updated policy against the reference semantic backbone \(\pi_{\theta_{\text{ref}}}\): \[\begin{align} \mathcal{L}_{\mathbb{R}\mathtt{-GRPO}} = &-\mathbb{E}_{k,n}\Bigg[ \min \!\Big(\rho_{k}^{(n)}\, A^{(n)},\; \text{clip}\!\big(\rho_{k}^{(n)},\, 1-\varepsilon,\, 1+\varepsilon \big) A^{(n)}\Big) \Bigg] \\ &+ \lambda_{\text{KL}}\; \mathbb{E}_{k}\!\Bigg[\frac{\sigma^2 + \left(\hat{s}_{k} - \hat{s}_{k}^{\,\text{ref}}\right)^2}{2\sigma^2} - \frac{1}{2}\Bigg] \end{align}\]

This formulation allows the Omni-LLM to optimize macroscopic structural constraints directly in continuous space, remaining robust to the arbitrary modality perturbations inherent in generated streams.

4 Experiments↩︎

4.1 SynthSync Dataset↩︎

Dataset Curation. We construct the dataset by generating acoustic tracks for real-world visual streams using 10 state-of-the-art video-to-audio (V2A) foundation models. To mitigate data contamination, source videos are drawn exclusively from the test splits of VGGSound[75] and FoleyBench[76]. Rigorous human

Annotation Protocol. To establish a reliable ground truth for synchronization, we bypass absolute scoring and instead rely on relative comparative judgments. We recruited an expert annotation pool of 20 individuals, dividing them into three independent groups. Each group provided complete pairwise annotations for the entire dataset, judging which candidate audio track (\(a_{i}\) or \(a_{j}\)) better matched the fixed visual anchor. Following the collection of these pairwise preferences, we calculated the empirical win-rates (total head-to-head victories) for each audio candidate. By aggregating these win-rates across all three annotation groups (\(2,267\times {}_{10}C_{2}\times3\approx 306K\) annotations), we derived a robust, final global ordinal ranking for all generated audios associated with the same video. This strictly operable partial ordering forms the ground-truth topology for our continuous latent alignment.

4.2 Evaluation Setup↩︎

Benchmarks. We train and evaluate our continuous estimator exclusively on the proposed SynthSync dataset, assessing audio-visual temporal synchronization using evaluation metrics such as NDCG[77], Kendall’s \(\tau\)[78], and Pairwise Accuracy (Pair-ACC)[79] to measure both structural causal coherence and fine-grained alignment. Detailed descriptions of the dataset and curation pipeline are provided in Section 4.1.

Implementation Details. Built on Qwen2.5-Omni-3B, our model replaces the discrete language head with a continuous projection MLP initialized via pairwise SFT. Inputs are standardized to 5s duration and \(140\times140\) resolution at 12 FPS. We employ \(\mathbb{R}\)-GRPO with listwise batches (\(K=6\)), sampling 12 Gaussian variants per output (\(\sigma^2=2.5\)). Optimization uses AdamW (lr=\(1e-6\)) with PPO clipping (\(\epsilon=0.2\)) and KL penalty (\(\beta=0.001\)) over 100 epochs. See Appendix for further hardware and training specifications.

4.3 Main Results↩︎

As shown in Table 1, our proposed framework establishes a new state-of-the-art on the SynthSync benchmark, consistently outperforming off-the-shelf Omni-LLMs, traditional non-LLM metrics, and specifically trained LLM evaluators. \(\mathbb{R}\)-GRPO yields substantial gains: direct listwise optimization boosts Top-1 Acc by 5.6% and achieves 72.38% Pairwise Acc. While VQ-Insight slightly leads in Bottom-1 Acc, our full \(\mathbb{R}\)-GRPO dominates cumulative gain and rank correlation metrics, confirming its ability to model physical causality. RL trajectories are in Fig. 5.

4.4 Benchmarking AV-Gen with SyncBench↩︎

while accurately reflecting the synchronization gap in open-weight models like LTX-2. Conversely, traditional evaluation metrics (e.g., AV-Align, JavisScore, and DeSync) exhibit extreme variance and produce rankings that often contradict human perceptual consensus. These legacy metrics frequently penalize advanced models by over-indexing on rigid, low-level signal matching rather than authentic structural causality. This stark divergence confirms that our continuous, R-GRPO-aligned metric provides a highly reliable, human-aligned standard necessary for advancing AV-Gen assessment.

4.5 Reward Model of Test-Time Best-of-N Scaling for AV-Gen↩︎

To prevent reward hacking, we strictly analyze cross-metric generalization, excluding diagonal cases where the reward and evaluation metrics match. As shown in Table 11, traditional metrics yield poor cross-generalization, often underperforming random selection. Conversely, our metric (+ Ours) achieves the highest off-diagonal scores across three of four independent metrics. This confirms our metric captures universal causal-semantic synchronization, proving effective for AV-Gen supervision.

4.6 Ablation Studies↩︎

How should an Omni-LLM parameterize rating scores for continuous evaluation? We validate our continuous scoring design by ablating alternative formulations. We compare Scalar Ranking against: 1) Scalar Regression, regressing ranks via MSE; 2) Score as Text-CE, autoregressively generating score strings; and 3) Score as Text-KL, minimizing KL divergence over digit probabilities.

Text-based decoding and scalar regression yield poor correlation (Kendall \(\le\) 0.1628) and discrimination (Top-1 Acc \(<\) 28%). Conversely, Scalar Ranking preserves relative topological order, boosting Pairwise Accuracy to 71.07% and Top-1 Accuracy to 55.80%. This confirms continuous rank optimization avoids the quantization and framing biases of discrete vocabularies.

strategies against the PFT baseline. As shown in Table 17, optimizing solely with a Pair Reward yields consistent but marginal improvements across all metrics. Conversely, relying exclusively on a List Reward degrades rank correlation (Kendall’s \(\tau\) drops by 5.5%) and pairwise accuracy, indicating that macroscopic distributional optimization alone can destabilize fine-grained local ordering. However, integrating both constraints into a Dual Reward strategy achieves the optimal balance. This composite approach establishes the highest overall performance, yielding a 4.9% relative improvement in MRR and an 8.5% improvement in Kendall’s \(\tau\) over the baseline. This confirms that global structural consistency and local comparative accuracy are mutually reinforcing objectives during latent continuous policy optimization.

How does the exploration variance of Gaussian policy impact RL-post training?

into the rollout sampling, which destabilizes the proximal trust region and degrades the optimization gradient. We find that an intermediate variance of \(\sigma = 2.5\) provides the optimal equilibrium—affording the policy enough flexibility to explore complex global topologies while remaining stable enough to accurately exploit learned comparative margins. This configuration achieves the peak Pairwise Accuracy of 72.38% and an NDCG of 0.9435.

4.7 Qualitative Visualization↩︎

To complement the quantitative leaderboard, Fig. 20 shows four SyncBench examples comparing generated audio against the ground-truth pattern with the corresponding evaluator scores. Our metric consistently penalizes missing acoustic events, premature or delayed responses, timbre mismatch, and event-count inconsistency, more faithfully reflecting whether the audio preserves the causal-semantic structure implied by the visual content than low-level signal matching.

5 Conclusion↩︎

We identified a critical bottleneck in audio-visual generation: the lack of metrics capable of understanding intricate physical interactions between sight and sound, as traditional metrics fail on modern artifacts by assuming structural correctness and relying on simple temporal offsets. To overcome this, we present a continuous, reference-free framework for causal-semantic synchronization, built on SynthSync, a dataset of authentic generative failures ranked via human preference. By replacing the discrete language head of an Omni-LLM with a continuous latent projection and optimizing the listwise score distribution via Real-Valued Group Relative Policy Optimization (\(\mathbb{R}\)-GRPO ), we bridge the gap between relative human perception and the need for an absolute, deployable metric. Our estimator achieves state-of-the-art human preference alignment and also generalizes robustly across metrics when deployed as a test-time Best-of-N reward model. This dual role—as both a reliable evaluator and an actionable reward signal—advances audio-visual assessment from low-level signal correlation to physically grounded causality.

Acknowledgements↩︎

This work was supported by the Zhejiang Key Laboratory of Advanced Intelligent Warehousing and Logistics Equipment under Grant No. 2024E10007. This work was partially supported by RGC Collaborative Research Fund (No. C5055-24G), the Start-up Fund of The Hong Kong Polytechnic University (No. P0045999), the Seed Fund of the Research Institute for Smart Ageing (No. P0050946), and Tsinghua-PolyU Joint Research Initiative Fund (No. P0056509), and PolyU UGC funding (No. P0053716).

6 Appendix↩︎

6.1 Dataset Statistics and Split↩︎

Dataset Sources and Composition. Our SynthSync dataset is curated from two complementary video corpora: VGGSound[75] and FoleyBench[76]. After temporal standardization to 5-second clips, the final dataset contains 2,267 visual anchors in total. Among them, 1,016 samples originate from VGGSound, while the remaining 1,251 samples are drawn from FoleyBench. The source origins are systematically tracked throughout preprocessing and annotation to ensure balanced representation and transparent evaluation.

Data Split. To prevent content leakage and ensure a strict evaluation protocol, we partition the dataset at the video level rather than at the generated-audio level. Specifically, all candidate audio generations associated with the same visual anchor are assigned to the same split. The resulting partition contains 1,767 training samples and 500 test samples. This design prevents near-duplicate visual content from appearing across training and evaluation, ensuring that the reported performance reflects genuine generalization in causal-semantic synchronization assessment.

6.2 Best-of-N Selection and Cross-Metric Evaluation Protocol↩︎

Candidate Generation Protocol. For each prompt in SyncBench, we construct a fixed candidate pool under the same textual condition using the LTX-2 generator. Specifically, we first sample one batch of 6 videos using random seed 42, and then repeat the same generation procedure with random seed 1234 to obtain another batch of 6 videos. Therefore, each prompt is associated with 12 candidate videos in total. This two-seed design increases sample diversity while keeping the generation budget and prompt condition strictly controlled across all reward models.

Best-of-N Selection. We then evaluate every candidate video under all five scoring schemes, namely AV-Align, JavisScore, DeSync, RALI, and our proposed metric. For a given reward model, BoN selection is performed only within the 12 candidates corresponding to the same prompt: we identify the single candidate that achieves the highest score under that reward model and mark it as the selected sample for that prompt. Importantly, after this selection is made, we record not only the reward score used for selection itself, but also the scores assigned to the same selected sample by the other four evaluators. Repeating this procedure over all prompts yields one cross-metric performance profile for each reward model.

Cross-Metric Generalization. This protocol is deliberately designed to test generalization rather than self-consistency. A reward model is not judged solely by whether it can select samples that maximize its own score, since such behavior can arise from reward overfitting or metric-specific bias. Instead, a reward model is considered effective only if the samples it selects are also rated favorably by other independent evaluators. Consequently, the off-diagonal entries in Table 11 constitute the principal evidence for reward quality. Strong off-diagonal performance indicates that the reward captures transferable causal-semantic synchronization, whereas weak transfer suggests the reward is overly tied to local signal artifacts or narrow objective shortcuts.

6.3 Comprehensive Ablation Studies↩︎

Extended Ablations. We include the complete ablation results in the appendix to complement the compact presentation in the main paper. In addition to the core component analysis, the appendix consolidates the full reward-function comparison, Gaussian exploration variance study, and any supplementary ablations omitted from the main paper for space considerations. This expanded presentation clarifies the individual contribution of dataset curation, preference-aligned initialization, and reinforcement learning to the final synchronization estimator.

Complete Tables. Tables 2–5 expand the ablations in the main paper with the full evaluation suite. Table 2 shows that scalar ranking remains the strongest score parameterization across all ranking and decision metrics. Table 3 presents the cumulative contribution of SynthSync training, preference-aligned fine-tuning, and RL post-training, showing consistent gains in both global ranking quality and top/bottom decision accuracy. Table 4 further confirms that the dual-reward design provides the best overall trade-off, whereas a list-only reward weakens several metrics despite preserving reasonable cumulative gain. Finally, Table 5 indicates that an intermediate exploration variance of \(\sigma=2.5\) achieves the best overall balance, while a larger variance slightly improves Last-1 accuracy at the cost of weaker overall alignment.

6.4 Global Reward and Metric Formulations↩︎

Evaluation Metrics. For each visual anchor \(x\), let \(\mathcal{Y}_{x}=\{y_{1},\dots,y_{K}\}\) denote the candidate audio-video pairs, with predicted scores \(\hat{\mathbf{s}}=[\hat{s}_{1},\dots,\hat{s}_{K}]\) and ground-truth scores \(\mathbf{g}=[g_{1},\dots,g_{K}]\). The predicted ranking is obtained by sorting candidates in descending order of \(\hat{\mathbf{s}}\), while the ground-truth ranking is induced by descending order of \(\mathbf{g}\). All ranking metrics are computed from these ordered lists.

Pairwise Accuracy. Given a valid pair \((i,j)\) with \(g_i \neq g_j\), PairAcc evaluates whether the predicted ordering agrees with the ground-truth ordering: \[\mathrm{PairAcc} = \frac{1}{|\mathcal{P}|} \sum_{(i,j)\in\mathcal{P}} \mathbb{I}\!\left[ \left(\hat{s}_{i}>\hat{s}_{j}\right) \Leftrightarrow \left(g_{i}>g_{j}\right) \right],\] where \(\mathcal{P}\) is the set of annotated comparison pairs. Pairs with tied ground-truth scores are skipped.

NDCG. Let \(\pi\) be the predicted permutation and let the relevance at rank \(r\) be defined by the ground-truth score of the item placed at that position, namely \(\mathrm{rel}_{r}=g_{\pi(r)}\). The discounted cumulative gain is \[\mathrm{DCG}(\pi)=\sum_{r=1}^{K}\frac{\mathrm{rel}_{r}}{\log_{2}(r+1)},\] and the normalized metric is \[\mathrm{NDCG} = \frac{\mathrm{DCG}(\pi)}{\mathrm{IDCG}},\] where \(\mathrm{IDCG}\) is computed from the ideal descending order of \(\mathbf{g}\).

MRR, Top-1, and Last-1 Accuracy. Let \(y^{\star}=\arg\max_{i} g_i\) be the ground-truth best candidate, and let \(\mathrm{rank}_{\pi}(y^{\star})\) be its 1-based position in the predicted ranking. The mean reciprocal rank is \[\mathrm{MRR}=\frac{1}{\mathrm{rank}_{\pi}(y^{\star})}.\] Top-1 Accuracy measures whether the predicted best candidate matches the ground-truth best: \[\mathrm{Top1} = \mathbb{I}\!\left[\arg\max_{i}\hat{s}_{i}=\arg\max_{i}g_{i}\right].\] Similarly, the supplementary evaluation protocol also reports Last-1 Accuracy, defined as \[\mathrm{Last1} = \mathbb{I}\!\left[\arg\min_{i}\hat{s}_{i}=\arg\min_{i}g_{i}\right].\]

Kendall’s \(\tau\) and Spearman’s \(\rho\). Let \(r^{\mathrm{pred}}_i\) and \(r^{\mathrm{gt}}_i\) denote the predicted and ground-truth ranks of candidate \(i\). We compute rank correlation using \[\tau = \frac{C-D}{\binom{K}{2}},\] where \(C\) and \(D\) are the numbers of concordant and discordant pairs, respectively, and \[\rho = \mathrm{corr}\!\left(\mathbf{r}^{\mathrm{pred}}, \mathbf{r}^{\mathrm{gt}}\right).\] In the standalone evaluation framework, these correlations are computed over the predicted and ground-truth rank indices. In the training objective, the same two statistics are computed directly from the predicted score vector and the ground-truth score vector; both serve to measure global rank consistency.

Pairwise Concordance. During RL training, the reward module uses a dense pairwise concordance score over all valid pairs within the candidate set: \[\mathrm{Conc}(\hat{\mathbf{s}},\mathbf{g}) = \frac{1}{|\mathcal{Q}|} \sum_{1\le i<j\le K} \mathbb{I}[g_i \neq g_j] \cdot \mathbb{I}\!\left[ \left(\hat{s}_{i}>\hat{s}_{j}\right) \Leftrightarrow \left(g_{i}>g_{j}\right) \right],\] where \(\mathcal{Q}=\{(i,j)\mid 1\le i<j\le K,\;g_i\neq g_j\}\). This quantity is the listwise analogue of PairAcc.

Global Ranking Reward. The final reward is defined as a weighted combination of several ranking metrics: \[\mathcal{R}_{\mathrm{rank}} = \lambda_{\mathrm{ndcg}}\mathrm{NDCG} +\lambda_{\tau}\widetilde{\tau} +\lambda_{\rho}\widetilde{\rho} +\lambda_{\mathrm{top1}}\mathrm{Top1} +\lambda_{\mathrm{pair}}\mathrm{Conc} +\lambda_{\mathrm{mrr}}\mathrm{MRR},\] where Kendall’s \(\tau\) and Spearman’s \(\rho\) are first normalized from \([-1,1]\) to \([0,1]\): \[\widetilde{\tau}=\frac{\tau+1}{2},\qquad \widetilde{\rho}=\frac{\rho+1}{2}.\] In our experiments, we use \[(\lambda_{\mathrm{ndcg}},\lambda_{\tau},\lambda_{\rho},\lambda_{\mathrm{top1}},\lambda_{\mathrm{pair}},\lambda_{\mathrm{mrr}}) =(0.25,\,0.20,\,0.20,\,0.15,\,0.10,\,0.10).\] This reward encourages the policy to align not only with local pairwise preferences, but also with the global topology of the full candidate list.

6.5 Training Details↩︎

Model Initialization. Our evaluator is built upon Qwen2.5-Omni-3B and replaces the original discrete language head with a one-layer continuous regression head. RL post-training is initialized from the pairwise preference-aligned checkpoint rather than from the untuned backbone, which substantially stabilizes early optimization and preserves the comparative structure learned during the cold-start stage. During optimization, the reference branch is kept frozen, while the trainable branch updates the regression head together with the selected transformer layers used for cross-modal reasoning.

Input Processing and Batch Construction. All training and evaluation samples are standardized to 5-second clips, resized to \(140\times 140\), and processed at 12 FPS. In the listwise RL setting, each training instance consists of one shared visual anchor together with \(K=6\) candidate methods. To preserve the full listwise structure within each sample and avoid excessive memory growth from parallel multimodal forward passes, we use a per-device batch size of 1 during RL post-training. Validation is conducted independently from training with a small batch size for stable evaluation throughput.

Optimization Setup. We optimize the model using AdamW with learning rate \(1\times 10^{-6}\) and weight decay \(5\times 10^{-2}\). The GRPO objective uses clipping radius \(\epsilon=0.2\), KL regularization weight \(\beta=10^{-3}\), rollout number \(N=12\), and Gaussian exploration variance centered at \(\sigma=2.5\). Training is performed for 100 epochs with gradient clipping at 1.0. Following the main experiments, validation is executed once per epoch, and the best checkpoint is selected according to validation pairwise accuracy.

Distributed Training. RL post-training is carried out with mixed-precision distributed optimization using six GPUs under a memory-efficient sharded training strategy. This configuration is necessary because each listwise sample requires multiple multimodal forward passes, one for each candidate method, and the rollout-based policy optimization further increases memory pressure. The chosen setup allows the full listwise objective to be optimized without reducing the candidate set size or truncating rollout evaluation.

Reproducibility. Unless otherwise stated, all experiments use a fixed random seed of 42. We additionally enable deterministic cuDNN behavior and disable benchmark auto-tuning to reduce nondeterminism across runs. Dataloaders are refreshed at the epoch level so that shuffled listwise groupings remain synchronized with the distributed training process. Taken together, these settings yield stable RL trajectories and reproducible validation performance across repeated runs.

6.6 Computational Efficiency↩︎

Inference Cost. A practical evaluation metric must be cheap enough for large-scale deployment, particularly when used as a reward model during generative model training. Although our evaluator is built upon a 3B Omni-LLM, it computes a synchronization score from a single forward pass over the multimodal context, reading the hidden state at the [SCORE] token without any autoregressive decoding. Table 6 reports the parameter count, peak memory, latency, and FLOPs of representative evaluators measured on a single RTX 4090 under vanilla HuggingFace inference. Our metric runs at \(0.23\) s per pair with \(10.0\)G memory, which is comparable to lightweight CNN-based baselines and roughly \(5\times\) faster and \(6\times\) cheaper in FLOPs than the 8.3B LLM-as-Judge (VQ-Insight). This confirms that the substantial gains in human-preference alignment do not come at a prohibitive computational cost.

8pt

6.7 RL Post-Training Pseudocode↩︎

Algorithmic Summary. Finally, we provide pseudocode for the RL stage of \(\mathbb{R}\)-GRPO, summarizing the rollout construction, Gaussian score sampling, reward computation, and policy update procedure. This presentation makes explicit how continuous scalar scoring is reparameterized as a stochastic policy and optimized under the group-relative objective.

References↩︎