1 Introduction↩︎

Multimodal large language models (MLLMs) such as GPT-4o, Gemini 1.5, and Qwen2.5-Omni are rapidly advancing in their ability to process and reason over complex multimodal inputs [1]–[3]. Yet, achieving consistent reliability and high-fidelity reasoning remains a significant challenge, since raw capability improvements do not automatically yield trustworthy outputs across modalities. This is frequently manifested in issues such as brittle visual grounding, temporal hallucinations in videos, or unfaithful audio interpretations [4]–[6]. A central component in addressing these gaps is the reward model (RM): it serves as a provider of critical discriminative reward signals, offering training-time supervision for post-training (e.g., preference optimization) and inference-time guidance for performance enhancement (e.g., reranking). From a principal–agent perspective, reward models act as incentive functions through which researchers encode task-specific requirements and nuanced qualitative preferences; consequently, the information density and structural clarity of these reward signals largely determine how effectively a model’s potential can be steered, audited, and scaled.

Despite their importance, current multimodal RMs remain limited in both signal granularity and modality coverage. Most existing models are still largely vision-centric and compress nuanced preferences into single scalar scores, resulting in sparse reward signals that offer little criterion-level transparency for debugging or controllable improvement [7]. While recent reasoning reward models begin to provide free-form textual critiques or chain-of-thought rationales [8]–[10], such reward signals are typically not grounded in an explicit, fixed rubric, making dimension-wise assessments harder to audit and less consistent across modalities. Moreover, high-quality preference supervision still heavily relies on expensive human annotation, which creates a bottleneck for rapidly adapting reward modeling to under-explored modalities such as audio [11], [12]. These gaps motivate a scalable and efficient reward modeling framework that: (i) can construct high-fidelity preference supervision in an automated manner, and (ii) provides rubric-grounded reward signals that are unified yet modality-aware.

To overcome these barriers, we introduce Omni-RRM, an open-source, rubric-grounded reward model. The term omni highlights its versatility across text, image, video, and audio. Distinct from opaque scalar RMs, Omni-RRM enforces the generation of structured, rubric-grounded justifications prior to the final verdict. This design introduces structural regularization that encourages the reward model to align its preferences with human-auditable criteria, shifting reward modeling from a ‘black-box’ scoring task toward a more auditable criterion-level judgment process.

To achieve this, we first propose an automated preference synthesis pipeline that generates high-fidelity supervision without human annotation. Using this pipeline, we construct Omni-Preference, a large-scale multimodal dataset where candidate pairs are generated via capability-contrast sampling and annotated with teacher-reconciled rationales. Omni-RRM is then trained through a progressive SFT-RL regimen: initial supervised fine-tuning internalizes the rubric schema, followed by Group Relative Policy Optimization (GRPO) to sharpen discrimination on nuanced, low-margin pairs. Empirically, Omni-RRM achieves state-of-the-art preference accuracy on video (80.2%) and audio benchmarks (66.8% on Audio-HH and 65.0% on TA2T), reaching 70.4% five-benchmark Overall accuracy with a +17.0% relative gain over its backbone. It also effectively guides Best-of-\(N\) selection and exhibits positive transfer to text-only benchmarks, proving its versatility as a unified reward source.

In summary, our contributions are three-fold:

• Automatic Rubric-Grounded Preference Synthesis. We propose a fully automated pipeline for preference synthesis that overcomes the scalability and consistency bottlenecks inherent in manual multimodal annotation. By leveraging teacher-reconciled consensus, this pipeline systematically generates Omni-Preference, a diverse omni-modal dataset enriched with high-fidelity, rubric-grounded rationales. This approach encourages cross-modal reward calibration and provides dense, discriminative supervision that facilitates more robust preference modeling across modalities.

• Omni-RRM. We develop Omni-RRM, an omni-modal reward model supporting text, image, video, and audio. By enforcing a shared rubric within a single omni-modal framework, Omni-RRM encourages cross-modal reward calibration. It is optimized via a progressive SFT + GRPO regimen, utilizing a rubric-grounded reward design to generate interpretable, rubric-grounded reward signals that are consistent and auditable across modalities.

• Extensive Validation. Omni-RRM-7B achieves state-of-the-art accuracy among open-source RMs, yielding a +17.0% relative gain over its backbone in five-benchmark Overall accuracy. It matches or exceeds top-tier proprietary models on video and audio benchmarks, achieving 80.2% on ShareGPT-Video, 66.8% on Audio-HH, and 65.0% on TA2T. Its efficacy is further validated through Best-of-\(N\) inference and robust generalization to text-only preference tasks.

2 Related Work↩︎

2.1 Multi-modal Large Language Models↩︎

Open-source multimodal large language models have advanced rapidly recently, evolving from early vision-language instruction tuning (e.g., LLaVA) [4], [13], [14] to stronger billion-parameter backbones (e.g., InternVL) [5], [15], and more native audio-visual processing in the Qwen family [3], [6], [16]. Meanwhile, frontier models have begun to handle long-form, multi-turn multimodal reasoning [17], [18], and image/video understanding [19]. These advances substantially broaden the scope of multimodal applications, but also increase the demand for reliable and scalable alignment mechanisms that can generalize across modalities.

2.2 Preference Alignment↩︎

Preference alignment is commonly achieved via Reinforcement Learning from Human Feedback (RLHF) [11], with complementary directions including AI feedback (RLAIF) [20] and direct policy optimization methods such as DPO and GRPO [21], [22]. These paradigms typically rely on a reward model to instantiate preferences as a learnable signal: It provides training-time supervision for post-training and can also serve as an inference-time reward score for reranking or Best-of-\(N\) selection. Extending preference optimization to multimodal settings is non-trivial due to heterogeneous inputs, modality-specific failure modes, and benchmark fragmentation [12]. Consequently, the reliability and coverage of the underlying RM often become the bottleneck in multimodal post-training pipelines, motivating reward supervision that is both scalable and interpretable.

2.3 Reward Modeling↩︎

Multimodal Reward Models. Multimodal reward models (RMs) have lagged behind the generators they aim to align, with two recurring limitations. First, modality coverage remains imbalanced and vision-centric: widely used preference resources and RM pipelines mainly target images and videos, supporting strong vision-oriented datasets, benchmarks, and reward models (e.g., ImageReward, Pick-a-Pic, and BaseReward) [7], [23], [24], while audio remains less explored and standardized [12]. Recent work such as Omni-Reward moves toward generalist omni-modal reward modeling by introducing multi-modality benchmarks and data, including audio, and training omni-modal RMs [10]. Second, interpretability remains limited: many RMs collapse judgments into scalar scores, while critic-style models provide free-form feedback that is hard to audit or control at the criterion level, as in LLaVA-Critic [8]. To improve transparency, recent work incorporates Chain-of-Thought (CoT) reward reasoning in multimodal and text settings, such as UnifiedReward-Think, VR-Thinker, and RM-R1 [9], [25], [26]. However, free-form traces do not enforce fixed criterion-wise accountability, and are often studied under restricted modality coverage or inconsistent evaluation interfaces.

Rubric-grounded Reward Modeling. Rubric-grounded reward modeling constrains evaluation with explicit multi-criterion rubrics and dimension-wise justifications, improving controllability and auditability beyond scalar scores or unstructured critiques. Prior work shows that criterion-level feedback can stabilize post-training and support targeted debugging in open-ended domains [27]. Recent studies further scale rubric supervision via automatic rubric construction and large rubric datasets [28]–[30], and analyze how rubric design affects reliability and robustness under misspecification or spurious cues [31], [32]. Rubric-aware optimization and calibration have also been explored in post-training pipelines [33]–[36]. However, most rubric-grounded methods remain text-centric. In contrast, Omni-RRM extends rubric-grounded reward modeling to multimodal alignment with a fixed five-criterion rubric, modality-aware instantiations, and a unified model that produces criterion-wise comparative judgments across text, image, video, and audio. Our novelty does not lie in omni-modality, reasoning-style outputs, or RL refinement alone, but in unifying them through a shared rubric-grounded reward formulation, automatic rubric-grounded preference synthesis, and rubric-aware GRPO optimization.

3 Method↩︎

3.0.0.1 Problem setup and structured outputs.

Given a multimodal context \(x\) and a response pair \((y_A, y_B)\), Omni-RRM formulates reward modeling as structured generation rather than direct scalar preference prediction. It generates a rubric-grounded record \(r = (\hat{s}_A, \hat{s}_B, \hat{\ell}, \hat{\mathbf{j}}_{1:5}, \hat{\ell}')\), where \(\hat{s}_A, \hat{s}_B \in [0, 10]\) are overall scores, \(\hat{\ell} \in \{A, B, E\}\) is the preference label (\(E\) denotes a tie), \(\hat{\mathbf{j}}_{1:5}\) denotes five criterion-level comparative justifications, and \(\hat{\ell}'\) is a redundant verdict copy for robust parsing. The five criteria—fluency/coherence, relevance, accuracy/completeness, reasoning quality, and safety/ethical alignment—provide a shared evaluation interface across text, image, video, and audio.

This structured format turns opaque scoring into a more auditable criterion-level judgment process. By requiring justifications before the final verdict, Omni-RRM encourages preferences grounded in human-inspectable evidence rather than holistic score patterns. The same schema is used in both SFT targets and GRPO reward computation, tying the rubric to optimization rather than only inference-time prompting.

3.0.0.2 Method overview.

Our method comprises Omni-Preference, an automatically constructed preference dataset with teacher-reconciled rubric supervision (Table 1), and progressive training of Omni-RRM via SFT\(\rightarrow\)GRPO to sharpen discrimination on low-margin pairs (Figure 1).

3.0.0.3 Rubrics design.

We adopt five broad rubrics to align with widely used axes in RLHF and reward-modeling practice and preference datasets.

• Fluency & Coherence: linguistic clarity and internal consistency of the response; consistent with reward-modeling datasets that explicitly annotate Coherence as a fine-grained attribute (e.g., HelpSteer2) [37].

• Relevance: adherence to the user request and provided context, reflecting the helpfulness / instruction-following emphasis in RLHF preference data and instruction-tuning with human feedback (e.g., HH-RLHF; InstructGPT) [38], [39].

• Accuracy & Completeness: factual correctness and sufficient coverage of key requested information, aligning with Correctness (and related truthfulness/correctness objectives) commonly tracked in RM datasets and instruction-following RLHF evaluations (e.g., HelpSteer2; InstructGPT) [37], [39].

• Reasoning Quality: soundness of inference and absence of contradictions, introduced to make judgments actionable beyond surface-level correctness and to complement consistency-focused analyses in preference modeling [38], [39].

• Safety & Ethical Alignment: avoidance of harmful or unsafe content, matching the “helpful vs.harmless” framing and explicit safety-alignment formulations in HH-RLHF and Safe RLHF, as well as safety/toxicity-focused RLHF evaluation goals [38]–[40].

3.0.0.4 Modality-aware instantiation.

Although the five rubrics are shared across modalities, the evidence used to instantiate each criterion is modality-specific. For image inputs, accuracy and reasoning emphasize visual grounding, object attributes, spatial relations, and OCR-sensitive details. For video inputs, the same criteria additionally consider temporal ordering, action progression, and event consistency. For audio inputs, they focus on acoustic event recognition, spoken content, and consistency between audio cues and textual responses. This design keeps the evaluation interface unified while allowing criterion-level evidence to reflect modality-specific failure modes.

3.1 Omni-Preference: A Rubric-Grounded Dataset for Omni-Modal Reward Modeling↩︎

We construct Omni-Preference, a fully automated dataset synthesized via our rubric-grounded preference synthesis pipeline. It is designed to provide high-fidelity reward signals across image, video, and audio. Unlike traditional preference sets, each sample in Omni-Preference is anchored by a multidimensional record \((x, y_A, y_B)\), annotated with teacher-reconciled preferences and structured rationales (Table 1).

Stage 1: Diverse Candidate Pair Generation. To populate the reward model’s training distribution, we generate candidate pairs via a capability-gap strategy. For each context \(x\) sampled from omni-modal datasets, we sample responses from a strong generator \(M_S\) and a weak generator \(M_W\): \[(y_A, y_B) = \big(M_S(x), M_W(x)\big).\] This ensures a wide coverage of quality margins. All definitive reward labels are assigned via the rubric-grounded consensus mechanism in Stage 2 to handle potential model reversals. We empirically compare against single-model rollouts in Supplementary Sec. B.3. This comparison shows that single-model rollouts are frequently judged as ties and yield very few large-margin pairs, making them inefficient for constructing high-confidence preference supervision under a fixed budget. In contrast, capability-gap pairing increases response diversity while leaving all final labels to the teacher-reconciled Stage-2 consensus.

Stage 2: Rubric-Grounded Annotation and Reconciliation. To ensure the precision of the reward signal, we employ two heterogeneous teachers (GPT-4o-mini and Gemini-2.0-Flash). Each teacher provides: (i) scalar scores, (ii) a categorical verdict \(\ell \in \{A, B, E\}\), and (iii) a rubric-grounded justification across five shared criteria.

We apply a strict consensus filter: a pair is retained only if both teachers reach an identical non-tie verdict that is numerically consistent with their score ranking and rubric-grounded justification. This dual-teacher reconciliation minimizes individual bias and reinforces the density of the rubric-grounded justification. We quantify sample difficulty using the reconciled score margin \(\Delta = |s_A - s_B|\), where \(\Delta < 2\) defines Hard pairs critical for refining the reward model’s discriminative boundary. Details of teacher prompting, reconciliation, and filtering rules are in Supplementary Sec. B.1.

Data scale and signal quality. We prioritize supervision reliability and cross-modality coverage over raw scale. This reliability-first design is particularly important for low-margin preference pairs, where noisy labels can easily blur the reward boundary and destabilize subsequent RL refinement. Detailed reliability analyses of synthetic preferences, including human validation and train–test contamination checks, are provided in Supplementary Sec. B.5.

3pt

ActivityNet, Charades, Ego4D, NextQA, and YouCook2.

3.2 Training Omni-RRM: A Progressive SFT–RL Approach↩︎

We train Omni-RRM with a progressive two-stage strategy: supervised fine-tuning (SFT) to learn rubric-grounded structured outputs, followed by reinforcement learning to refine discrimination on low-margin preference pairs.

The RL stage uses a bounded composite objective that combines: (i) preference correctness, (ii) the quality of the rubric-grounded justification, and (iii) a lightweight schema-validity guardrail to prevent format collapse during exploration. Such schema constraints are commonly used to stabilize structured critics under RL [37], [41].

3.2.0.1 Stage 1: Supervised Fine-Tuning (SFT).

SFT teaches the model to follow the output interface and generate coherent rubric-grounded justifications. Given input \((x,y_A,y_B)\) and a target structured record represented as a token sequence \(\mathbf{o}_{1:T}\), we optimize the standard negative log-likelihood loss: \[\mathcal{L}_{\text{SFT}}(\theta) = - \sum_{i=1}^{T} \log P_\theta(o_i \mid \mathbf{o}_{<i}, x, y_A, y_B).\] We use lightweight LoRA fine-tuning [42] for efficiency. By the end of SFT, the model reliably produces schema-compliant outputs with non-trivial rubric-grounded justifications, providing a stable initialization for RL.

3.2.0.2 Stage 2: Reinforcement Learning with GRPO.

SFT provides a schema-stable cold start so that RL updates can focus on improving low-margin preference discrimination rather than repairing output format [41], [43]. We then refine Omni-RRM with Group Relative Policy Optimization (GRPO), following the group-sampling and relative-advantage optimization recipe popularized in DeepSeek-R1-style training [22], [43]. For each input \((x,y_A,y_B)\), we sample a group of \(k\) structured outputs and assign each output a bounded composite reward: \[R = w_{\text{fmt}} R_{\text{fmt}} + w_{\text{pref}} R_{\text{pref}} + w_{\text{rub}} R_{\text{rub}}.\] \(R_{\text{fmt}}\) enforces strict JSON/schema validity and required fields (e.g., overall scores \(\hat{s}_A/\hat{s}_B\), the categorical verdict, and the rubric-grounded justification fields). \(R_{\text{pref}}\) rewards correct preference prediction and score–verdict consistency, i.e., the predicted verdict \(\hat{\ell}\in\{A,B,E\}\) should agree with the score ordering (e.g., \(\hat{\ell}=A\Rightarrow \hat{s}_A>\hat{s}_B\), \(\hat{\ell}=B\Rightarrow \hat{s}_B>\hat{s}_A\), and \(\hat{\ell}=E\Rightarrow \hat{s}_A=\hat{s}_B\)). \(R_{\text{rub}}\) encourages substantive, rubric-grounded justifications, serving as the structural regularization term that compels the reward model to derive its verdict through a disciplined, criterion-based analysis; it is computed using justification-consistency heuristics and consistency checks (details below). We optimize the policy using group-normalized advantages and a clipped policy-gradient objective with a KL penalty to the SFT reference policy. Full GRPO update equations, reward rules, and weights are provided in Supplementary Sec. A.1.

Overall, the progressive SFT–GRPO regimen stabilizes the rubric-grounded output interface and improves discrimination on low-margin preference pairs. A detailed compute budget for data synthesis, teacher annotation, training, and evaluation is provided in Supplementary Sec. A.2.

4 Experiments↩︎

4.1 Benchmarks and Metrics↩︎

We evaluate Omni-RRM on preference benchmarks spanning image, video, and audio. Our primary metric is Preference Accuracy (%), i.e., the fraction of test pairs where the predicted winner \(\hat{\ell}\in\{A,B\}\) matches the benchmark’s ground-truth preference under its official protocol.

Image. We report results on VL-RewardBench (VL-Reward) [44] using its recommended Reasoning split, and on MM-RewardBench [45] using its primary preference accuracy metric.

Video. We derive a pairwise preference test set from ShareGPT-Video DPO (ShareGPT-Video) [46] by retaining pairs with sufficiently large human score margins to obtain clear winners.

Audio. We use two audio-conditioned preference benchmarks. Audio-HH-RLHF (Audio-HH) is constructed by applying TTS to prompts from HH-RLHF [38], while retaining the original human-written chosen/rejected responses and preference labels. This yields a controlled audio-conditioned preference setting, though it does not cover real-world acoustic noise or speaker variability. We further evaluate on Omni-RewardBench-TA2T (TA2T) [10], a public audio-conditioned reward benchmark, to assess external audio generalization.

2.0pt

4.2 Baselines↩︎

We compare Omni-RRM against baselines spanning proprietary MLLMs, open-source MLLMs, and open-source multimodal reward models. Proprietary models (e.g., GPT-4o-mini [1] and Gemini-2.0-Flash [47]) serve as strong reference points. Open-source MLLMs are represented by the Qwen2.5 family [3]; we include the original Qwen2.5-Omni backbones as unaligned baselines to quantify gains from our post-training. We further compare against open-source reward models trained for multimodal preference judgment, including Omni-Reward [10], LLaVA-Critic-7B [8], R1-Reward-7B [48], and UnifiedReward-Think-7B [9]. To isolate the effect of rubric-structured supervision and rubric-aware RL, we include two matched variants under the same SFT+RL budget: Omni-RRM (w/o rationale), which removes five-criterion rubric rationales during SFT+RL, and Omni-RRM (w/o rubric RL), which ablates the rubric-related reward term during GRPO.

4.3 Experimental Results↩︎

Our main results in Table ¿tbl:tab:main95results? yield several primary insights. Additional analyses on Hard/Easy subsets and text-only transfer are provided in Supplementary Secs. C.1–C.2.

4.3.0.1 Omni-RRM achieves state-of-the-art omni-modal performance.

Omni-RRM-7B sets a new state-of-the-art among open-source reward models on video and audio benchmarks, achieving 80.2% on ShareGPT-Video, 66.8% on Audio-HH-RLHF (Audio-HH), and 65.0% on TA2T. Its five-benchmark Overall accuracy of 70.4% closely matches Gemini-2.5-Pro (70.6%), despite being an order of magnitude smaller. On image benchmarks, it also achieves the highest score on MM-RewardBench (72.9%), outperforming both specialized reward models and general-purpose proprietary MLLMs. Including TA2T in the main evaluation ensures that our audio conclusion is not tied to a single controlled benchmark. The rankings on Audio-HH and TA2T are highly correlated across common models, supporting Audio-HH as a meaningful controlled audio proxy; detailed with-/without-tie results and correlation analysis are provided in Supplementary Sec. C.6.

4.3.0.2 Rubric-grounded rationales are vital for high-fidelity reward modeling.

To isolate the impact of rubric-grounded supervision, we compare against Omni-RRM (w/o rationale), a variant trained without generating rationales (Table ¿tbl:tab:ablations?). The sharp decline in this variant indicates that rubric-grounded rationales are not merely auxiliary text, but provide useful structural regularization for preference discrimination. By requiring criterion-level evidence before the final verdict, the model becomes less reliant on holistic score patterns or shortcut multimodal cues. We further ablate rubric-aware reinforcement via Omni-RRM (w/o rubric RL), which removes the \(R_{\text{rub}}\) term during GRPO; this variant likewise underperforms the full model in the four-benchmark ablation suite (68.1% vs.% Overall for 7B). Together, these ablations show that rubric-grounded rationales and rubric-aware optimization jointly improve omni-modal reward modeling.

4.3.0.3 Progressive SFT+GRPO training drives substantial performance gains.

Our two-stage training regimen yields significant improvements over both the unaligned backbone and the SFT-only baseline. While SFT establishes a foundational capability for reward modeling, the subsequent GRPO refinement further sharpens the reward boundary. For example, Omni-RRM-7B improves from 62.6% (SFT) to 70.4% (SFT+RL) in five-benchmark Overall accuracy, with particularly strong gains on MM-RewardBench (61.0% \(\rightarrow\) 72.9%), ShareGPT-Video (70.5% \(\rightarrow\) 80.2%), and TA2T (58.5% \(\rightarrow\) 65.0%). Compared to the original Qwen2.5-Omni-7B backbone, our full model achieves a +17.0% relative gain in overall accuracy, validating that RL-based optimization effectively improves reward calibration. This improvement is especially pronounced on low-contrast examples: difficulty-stratified evaluation in Supplementary Sec. C.1 shows that Omni-RRM yields larger average gains on Hard pairs than Easy pairs (+12.0 pp vs. +6.6 pp), supporting our design focus on low-margin preference discrimination.

4.3.0.4 Specialized reward modeling outweighs raw model scale.

We find that reward modeling is a specialized capability that does not scale linearly with general pre-training: reward models explicitly optimized for preference supervision can significantly outperform much larger general-purpose MLLMs. For example, on MM-RewardBench, the proprietary Gemini-2.5-Pro (63.3%) lags substantially behind specialized open-source reward models, including our Omni-RRM-7B (72.9%), R1-Reward-7B (72.3%), and UnifiedReward-Think-7B (71.4%). This performance gap suggests that high-fidelity reward modeling depends critically on targeted supervision and rubric-grounded post-training objectives rather than sheer parameter count or pre-training scale alone.

2.4pt

4.4 Inference-Time Alignment via Best-of-\(N\)↩︎

To validate the practical utility of Omni-RRM as a plug-and-play alignment tool, we employ a Best-of-\(N\) (BoN) re-ranking protocol using a fixed base generator. Specifically, we use Qwen2.5-Omni-7B to sample \(N\) candidate responses per query and select the final output via two distinct strategies: (i) Self-consistency@N, which performs a majority vote over the multiple-choice options among the \(N\) candidates; and (ii) BoN@N, which selects the optimal candidate based on an external reward model. For the latter, we implement a single-elimination tournament: the reward model performs pairwise comparisons under a fixed bracket order, with the preferred response advancing until a single winner remains (\(N{-}1\) comparisons per query). This setup isolates the effect of inference-time alignment from any changes to the generator’s underlying parameters.

We evaluate BoN performance on three representative benchmarks: MMMU (Image) [49], Video-MME (Video) [50], and AVQA (Audio-only setting) [51]. We compare five open-source reward models: Qwen2.5-Omni-7B, Qwen2.5-VL-7B, LLaVA-Critic-7B, UnifiedReward-think-7B, and our Omni-RRM-7B. Results are reported only for modalities supported by each respective model, with \(N \in \{2,4,6,8,10\}\).

As illustrated in Figure 2, the choice of reward model significantly influences BoN effectiveness, with the performance gap over self-consistency typically widening as \(N\) increases. Across all modalities, Omni-RRM-7B consistently yields the most substantial improvements. At \(N{=}10\), Omni-RRM improves upon Self-consistency by +1.9 on MMMU (\(54.5 \rightarrow 56.4\)), +1.7 on Video-MME (\(52.4 \rightarrow 54.1\)), and +2.6 on AVQA (audio-only; \(77.7 \rightarrow 80.3\)).

Although Omni-RRM introduces extra RM decoding latency due to structured rationales, it delivers stronger BoN selection gains on video and audio tasks, suggesting that criterion-level reasoning is most beneficial when fine-grained discrimination among diverse candidates is required.

These results underscore the scalability of our rubric-grounded reward modeling in complex inference-time scenarios. As the candidate pool \(N\) expands, the base generator yields a more diverse set of potential responses, increasing the likelihood that the pool contains high-quality candidates that surpass standard greedy decoding. However, identifying these optimal responses among a larger set of plausible alternatives requires enhanced discrimination granularity. Omni-RRM thrives in this high-\(N\) regime by leveraging its structured rubric rationales to conduct exhaustive, multi-dimensional comparisons. This systematic evaluation allows the model to effectively capture subtle quality gains and consistently surface the best-performing responses from the expanded search space, leading to the superior performance observed across all three modalities. Detailed numerical results and a latency analysis are provided in Supplementary Secs. C.3–C.4.

4.5 Inter-Modality Dynamics in Omni-Modal Reward Modeling↩︎

A fundamental challenge in omni-modal alignment lies in understanding how preference knowledge is encoded and transferred across heterogeneous sensory inputs. Rather than viewing reward modeling as a collection of disjoint tasks, we treat it as a unified omni-modal reasoning problem anchored by a shared rubric. By systematically varying the supervision mixture of image, video, and audio data, we aim to map the omni-modal dynamics and quantify how preference signals in one domain facilitate or reinforce the reward boundaries in others. This investigation moves beyond simple coverage to explore whether a unified rubric interface can effectively harmonize diverse omni-modal inputs into a consistent, modality-agnostic preference space.

4.5.0.1 Experimental setup.

All ablations utilize the same 3B backbone and identical SFT+GRPO hyperparameters. We maintain a constant total training budget (in terms of updates and pairs) across all settings by subsampling, ensuring that performance variance is strictly driven by the composition of omni-modal supervision rather than raw data volume. We evaluate preference accuracy (%) across VL-RewardBench (Image), ShareGPT-Video (Video), and Audio-HH-RLHF (Audio).

4.5.0.2 Results and Analysis.

Figure 3 reveals a complex landscape of omni-modal synergy and reciprocal reinforcement:

(1) Global convergence through omni-modal joint training. The full-modality configuration (I+V+A) achieves a superior performance envelope that transcends any modality-specific or partially combined variant. This demonstrates that joint training across all three modalities does not lead to optimization conflicts; instead, the diversity of omni-modal inputs facilitates a more robust and generalized convergence on the underlying rubric criteria, outperforming models trained on narrower distributions.

(2) Reciprocal dependency and semantic anchoring. We observe a collateral performance decline whenever any single modality is excluded, suggesting that preference knowledge is not siloed within specific domains. For instance, removing video-level reasoning patterns (I+A) not only eliminates video capabilities but also compromises the model’s judgment on image and audio tasks. This confirms that our rubric-grounded rationales act as a shared semantic anchor: the "logic" learned in one modality serves as a reference for others, creating a mutually reinforcing omni-modal reward landscape.

(3) Asymmetric gains in low-resource omni-modal domains. A pivotal finding is the asymmetric performance boost in the audio domain. Omni-modal training (65.1%) significantly outperforms audio-only training (63.8%) under an identical budget. This highlights that the high-granularity preference structures inherited from vision and video datasets provide a structural scaffold for audio. By "borrowing" the well-defined boundaries from high-resource modalities, Omni-RRM effectively overcomes the inherent scarcity of audio preference data, validating the strategic efficiency of unified omni-modal reward modeling.

5 Conclusion↩︎

In this paper, we presented Omni-RRM, a rubric-grounded framework for omni-modal reward modeling. Our approach alleviates the scalability and consistency bottlenecks of manual labeling through automatic rubric-grounded preference synthesis, yielding the Omni-Preference dataset enriched with dense criterion-level supervision signals. By shifting from coarse, black-box scoring to structured five-criterion justifications, Omni-RRM introduces structural regularization that makes reward judgments more auditable and less dependent on opaque scalar patterns or shortcut cues across image, video, and audio. Extensive evaluations demonstrate that Omni-RRM achieves state-of-the-art accuracy among open-source reward models, reaching 70.4% five-benchmark Overall accuracy and closely matching strong proprietary MLLMs on challenging multimodal benchmarks. Our analysis of omni-modal dynamics further suggests that a unified rubric-grounded framework can induce cross-modal transfer, allowing preference signals from heterogeneous modalities to reinforce a more consistent omni-modal reward boundary. Furthermore, we validate its utility as a plug-and-play alignment tool via Best-of-\(N\) inference, demonstrating its practical effectiveness for inference-time response selection. By harmonizing omni-modal coverage with structured transparency, this work provides a robust foundation for building more trustworthy, scalable, and efficient human-AI alignment systems.

Acknowledgements↩︎

This work was supported in part by the National Natural Science Foundation of China under Grant Nos. 62576046, 62301066, 62406028, and 62476224; the Beijing Academy of Artificial Intelligence under Grant No. Z251100008125041; the Key Project of Philosophy and Social Sciences Research, Ministry of Education of China, “Research on Risk Governance and Capability Building of Generative Artificial Intelligence Systems,” under Grant No. 24JZD040; the Graduate Education and Teaching Reform Research Project of Beijing University of Posts and Telecommunications under Grant No. 2025YZ010; and the Beijing Natural Science Foundation under Grant No. L259043.

References↩︎