Unison: Benchmarking Unified Multimodal Models via Synergistic Understanding and Generation


Abstract

Unified multimodal models capable of both understanding and generation have achieved remarkable strides. However, despite their unified designs, existing evaluations typically assess understanding and generation capabilities in isolation, overlooking the synergy between comprehension and generation. To bridge this gap, we introduce Unison, a comprehensive benchmark comprising 2,169 high-quality unified task samples, designed to evaluate joint understanding and generation in unified multimodal models. Unison offers three key strengths: 1) Comprehensive Dimensions: Unison encompasses internal consistency, understanding-guided generation, generation-guided understanding, and mutual enhancement to enable holistic evaluation. 2) Diagnostic Evaluation: it provides both unified and decoupled tracks for understanding and generation, allowing fine-grained attribution of failure modes and quantitative analysis of the gains from unified modeling. 3) Human Alignment: we also introduce Unison-Judge, an evaluation model well aligned with human judgments to ensure reliable assessment. Based on systematic evaluations of state-of-the-art models on Unison, we uncover critical limitations in current unified multimodal systems and highlight promising directions for future research. Codes, Unison and Unison-Judge are publicly available at https://github.com/FudanCVL/Unison.

1 Introduction↩︎

Multimodal large language models (MLLMs) have advanced rapidly in recent years, demonstrating impressive comprehension capabilities in complex reasoning tasks. On the other hand, visual generative models, particularly diffusion-based, have achieved unprecedented fidelity and controllability in synthesizing images and videos. Motivated by their success, unified multimodal models (UMMs) have emerged as a rapidly advancing research direction, which aim to integrate complementary understanding (textual output) and generation (visual output) capabilities within a single framework. Current UMMs are increasingly moving beyond composition of separate understanding and generative models toward natively unified designs that support shared representations across both perception and synthesis.

Nowadays, UMMs have demonstrated strong performance in multimodal understanding, exhibiting notable complex reasoning capabilities on benchmarks such as MMBench [1] and MMMU [2]. They also achieve compelling results in compositional image generation, producing fine-grained, instruction-aligned outputs, as evaluated on GenEval [3], DPG-Bench [4] and OneIG-Bench [5]. However, existing evaluations remain confined to capability-specific tasks. There is a critical need for a benchmark to holistically evaluate UMMs on unified tasks requiring both understanding and generation capabilities.

To this end, we propose Unison, a comprehensive unified benchmark designed to holistically evaluate UMMs through synergistic understanding and generation. Our work is motivated by three foundational questions: 1) Do current UMMs truly integrate understanding and generation capabilities? Prior work [6] points out that most UMMs suffer from a performance trade-off between these capabilities due to conflicting visual tokenization strategies. UniGen [7] leverage understanding ability as a self-verifier to assess the quality of generated images by evaluating image-text semantic coherence, suggesting a critical insight: If a UMM correctly comprehends an input image, can it generate a new image consistent with its own semantic interpretation? We define this notion as internal consistency, as shown in [fig1] (a), to evaluate the alignment between understanding and generation. 2) For challenging tasks, what is the synergistic effect of UMMs in leveraging their own reasoning to guide generation, and vice versa? Bagel [8] claims that reasoning before visual generation may enhance the final output in multimodal tasks. By using this approach to filter training data, Bagel achieves superior performance. Motivated by this, we design understanding-guided generation and generation-guided understanding, as shown in [fig1] (b) and (c), to evaluate the synergistic effect in UMMs. 3) Can these dual capabilities synergistically interact to enhance the performance of generation or understanding tasks? X-Omini [9] shows that shared autoregressive modeling between image and text generation, augmented by reward models, reduces cumulative errors, demonstrating that understanding and generation can mutually reinforce each other. Thus, we propose mutual enhancement to assess UMMs’ self-refinement capability through iterative cycles of alternating multimodal understanding and image generation, as shown in [fig1] (d).

Through the Unison framework, we observe that while certain UMMs achieve state-of-the-art performance on specialized understanding or generation benchmarks, they often fall short in unified tasks requiring joint comprehension and synthesis. Subsequent experiments and analysis further reveal key limitations of current UMMs and point to promising directions for future research.

In summary, our main contributions are as follows:

  • In this work, we explore a promising yet underresearched space for evaluating unified multimodal models via synergistic understanding and generation. Based on this, we introduce Unsion, consisting of 2,169 human-validated unified task samples, which encompasses internal consistency, understanding-guided generation, generation-guided understanding, and mutual enhancement to enable unified evaluation.

  • Unison supports both joint and decoupled assessment of understanding and generation within the same framework, providing interpretable and fine-grained analysis.

  • Extensive experiments demonstrate that Unison-Judge exhibits strong alignment with human judgments. Additionally, we provide valuable insights into current UMMs’ limitations and future directions.

2 Related works↩︎

Unified Multimodal Models. Recent years have witnessed remarkable advancement in both multimodal large language models (MLLMs) and visual generative models, leading to increasing research efforts toward unifying these capabilities within a single architecture. This pursuit has given rise to unified multimodal models (UMMs) [8], [10][22], which generally fall into three categories: auto-regressive models, diffusion-based models and hybrid models. Auto-regressive models, such as SEED-X [15] and UniWorld [22], typically tokenize both visual and text inputs into discrete sequencial tokens using transformer-based architectures, enabling seamless multimodal reasoning and generation. On the other hand, diffusion-based unified models such as D-DiT [11], which introduces a dual-branch diffusion framework that jointly generates text and images, facilitating coherent and controllable synthesis of aligned image-text pairs starting from pure noise, enabling synchronized generation across visual and textual modality. In addition, hybrid approaches [8], [13], [21] integrate auto-regressive and diffusion-based components within a unified framework. These methods aim to preserve strong language reasoning while enabling high-fidelity visual generation, but they often increase training complexity and computational cost.

Multimodal Understanding Benchmarks have evolved from basic perception toward complex reasoning. Early datasets such as Flickr30k [23] and MS COCO Captions [24] focused on image–text alignment, while VQA-style benchmarks [25][27] introduced free-form question answering requiring scene-level comprehension. More recent works evaluate diverse multimodal capabilities: MMBench [1] emphasizes cross-lingual perception, MMMU [2] targets domain-specific reasoning, and SEED-Bench [28] assesses fine-grained skills with external knowledge integration.

Image Generation & Editing Benchmarks. Previous image generation relied on automatic metrics like Inception Score (IS) [29] for basic image-only quality assessment. Frechet Inception Distance (FID) [30] and CLIPScore [31] estimates image-text alignment in feature level. However, contemporary works [3], [4], [32] reveal significant gaps between these metrics and human judgment, particularly for complex scenes. To address this, benchmarks now emphasize compositional generation and semantic fidelity. GenEval [3] and T2I-CompBench [32] evaluate attributes like object color, counting, and spatial relationships. DPG-Bench [4] focuses on generation from dense prompts, and OneIG-Bench [5] supports multilingual and stylized evaluation. For image editing, where models modify images according to textual instructions, InstructPix2Pix [33] and MagicBrush [34] assess the correctness of local and global edits. AnyEdit [35] introduces a dedicated evaluation set covering 25 editing types for real-world applicability, while ImgEdit [36] provides a large-scale benchmark with 1.2 million high-quality edit pairs, emphasizing instruction adherence and visual realism.

Table 1: Benchmark Comparison. MU denotes multi-modal understanding, IG denotes image generation, IE denotes image editing, IC denotes internal consistency, UF denotes unified understanding and generation, Det denotes detection, Seg denotes segmentation, HE denotes hybrid editing, HT denotes hybrid type, RI denotes real image, SI denotes synthetic image.
Evaluation Localization Editing Scenario Multi- Judge Task
MU IG IE IC UF   Det Seg HE HT RI SI Turn Model Type
MMBench [1]    0
SEED-Bench [28]    0
MMMU [2]    0
GenEval [3]    0
DPG-Bench [4]    0
OneIG-Bench-EN [5]    0
AnyEdit [35]    0
ImageEdit [36]    6
Unison (ours)    11 14

3 Unison↩︎

In this work, we propose Unison, a comprehensive benchmark designed to holistically assess the unified capabilities of UMMs, focusing on the synergy between understanding and generation. To facilitate systematic evaluation, we decompose unified capability into four dimensions: internal consistency, understanding-guided generation, generation-guided understanding, and mutual enhancement.

3.1 Internal Consistency↩︎

Internal consistency, referring to the alignment of UMM’s semantic encoding of understanding and generation within its internal representations, is designed by leveraging fine-grained compositional understanding of images alongside text-to-image tasks. Both two tasks fundamentally require holistic comprehension of all input information. In fine-grained compositional understanding, the model must analyze visual elements such as objects, attributes and spatial relationships, typically achieved through tasks like Visual Question Answering (VQA). On the other hand, text-to-image generation requires instruction-following and compositional reasoning to accurately synthesize images. Without this foundational consistency, the model struggles to perform complex tasks effectively, as errors in basic alignment can propagate and hinder performance.

In particular, given an image-caption pair \((I, C)\), a set of questions \(Q = \{ q_i^{ic} \}_{i=1}^n\) and a set of attributes \(A = \{ a_i \}_{i=1}^n\), where attributes contain object, color, texture, count, light text rendering and spatial relation. The image-caption pair \((I, C)\) is manually aligned, where each attribute \(a_i\) explicitly represented in the corresponding question \(q_i\). For understanding, UMM perform VQA on image \(I\) to get answer \(s_i^{und} = \mathcal{M}(I, q_i^{ic})\). For generation, we first prompt UMM to perform text-to-image synthesis to obtain a generated image \(I_{t2i} = \mathcal{M}(C)\), then apply a dedicated judge model \(\mathcal{J}\) to answer the same question on \(I_{t2i}\), yielding \(s_i^{gen} = \mathcal{J}(I_{t2i}, q_i^{ic})\). Finally, we compare answers \(s_i^{und}\) and \(s_i^{gen}\) for each attribute-aligned question \(q_i\) calculating final internal consistency score \(\text{\textit{Score}}_\text{\textit{ic}}\): \[\label{eq1} \text{\textit{Score}}_\text{\textit{ic}} = \sum_{i=1}^{n} f(a_i, s_i^{und}, s_i^{gen}))/n ,\tag{1}\] where \(f(a_i,s_i^{und},s_i^{gen}) \in \{0,1\}\) returns 1 if \(s_i^{und}\) and \(s_i^{gen}\) are consistent with respect to attribute \(a_i\), and 0 otherwise.

Notably, extreme failure cases can lead to spurious consistency, where a UMM may misidentify a critical attribute (e.g., an image-text pair depicts five people, but the model perceives six and subsequently generates an image showing six people). Since consistency derived from a faulty premise is meaningless, our evaluation protocol assigns a score of zero to such cases. Our strict criterion ensures that high scores only represent faithful consistency, effectively penalizing hallucinations or inherent model biases.

3.2 Understanding-Guided Generation↩︎

In this part, we leverage the reasoning capability of UMM as prior to guide contextually accurate generation for synergy’s effect assessment. In short, we prompt the UMM to autonomously reason precise edit parts from complex instructions then executing generation. Based on comparison with direct image manipulation, understanding-guided generation provides a systematic analysis for causal impact of a model’s comprehension on its generative fidelity.

Specifically, given an image \(I\) and a complex editing instruction \(E\). The model first comprehends \(E\) and reason about \(I\) to output the target region \(\mathcal{R}\) via bounding box/segmentation mask. Then obtaining its own localization as conditional input, the model executes the editing operation to get final edited image \(I_{ie}\). We average the localization score and image editing score to get final score \(\text{\textit{Score}}_\text{\textit{ugg}}\): \[\label{eq2} \text{\textit{Score}}_\text{\textit{ugg}} = \left(IoU(\mathcal{R},\mathcal{R}_{gt}) + \mathcal{J}(I,I_{ie})\right)/~2 ,\tag{2}\] where \(\mathcal{R}_{gt}\) is the ground truth target region, \(IoU(,)\) returns Intersection over Union score, \(\mathcal{J}\) is a MLLM-based rating model to provide scores for image editing.

3.3 Generation-Guided Understanding↩︎

For this paradigm, we employ visual output as a proxy for complex reasoning, exploring the untapped potential of UMMs. We formalize this as generation-guided understanding, which requires a tight coupling between generative synthesis and the comprehension of implicit structural constraints. For example, given a cub net, determining what color of its top face after it is folded into a cube requires visual imagination, which challenges even human cognition. We reveal that UMMs can leverage their generative priors to simulate these transformations, providing an intuitive visual reference that facilitates the resolution of otherwise intractable reasoning problems.

We formalize this task as visual question answering, where an intermediate image generation step is employed to synthesize relevant visual contexts. Based on an image \(I\) (2D spatial, 3D spatial) or description \(D\) (complex relation), as illustrated in 1, the task is to select the ground-truth option \(O_{gt}\) from a set of four candidates in response to a question \(q^{ggu}\). To achieve this, UMM first performs an intermediate generative phase explicitly conditioned on the question’s context. Subsequently, the model reasons over the synthesized image to derive the final output. To evaluate this process, we define an understanding score \(S^{ggu}_u \in \{0,1\}\) based on the accuracy of the predicted option. The generation score \(S^{ggu}_g\) follows the principles of internal consistency. Finally, the holistic performance is measured by the average score \(\text{\textit{Score}}_\text{\textit{ggu}} = ( S^{ggu}_u + S^{ggu}_g )/2\).

3.4 Mutual Enhancement↩︎

In this part, we assess self-refinement ability on capability-specific tasks of UMMs by alternately motivating them alternately comprehend and regenerate their own outputs, or vice versa, referring to mutual enhancement. Specifically, the model first generates an initial output, justifies the coherence between the generated content and the input instruction, and then summarizes a textual operation to produce optimized outputs. This iterative loop tests whether UMM can identify and reinforce its own errors without external supervision.

For clarity, this process follows a four-step loop (up to \(R\) rounds): 1. Generation: The model produces an initial textual/visual output from the input instruction. 2. Evaluation: The model acts as an evaluator, judging if the current output satisfies the input instruction. 3. Refinement: If misalignments are found, the model needs to generate a specific refinement instruction to reproduce a new output. Then re-evaluates. 4. Termination: The loop stops when the model judges no further conflicts, or reaches the maximum round.

Self-Refining Image Editing extends standard editing by incorporating iterative feedback loops.

Formally, assuming there are at most \(R\) execution rounds. Given an initial instruction \(T_1 \in \{T_r\}_{r=1}^R\), the model generates an edited image \(I^{me\_ie}_1 \in \{I^{me\_ie}_r\}_{r=1}^R\). For subsequent rounds \(r \in \{2,\dots, R\}\), the model evaluates if \(I^{me\_ie}_{r-1}\) fully aligns with the initial instruction \(T_1\), If misaligned, it will be prompted to generate a refinement instruction \(T_r\) and a corresponding image \(I^{me\_ie}_{r}\), otherwise, the refinement loop terminates early. Moreover, we define self-refining gain \(\Delta S_{g}\) to quantify the cumulative improvement. This metric represents the absolute increase achieved by final refined image \(I_k\) compared to \(I_1\). We directly use editing score at \(k\) round as final score \(\text{\textit{Score}}_\text{\textit{me\_ie}}\).

Self-Refining Multimodal Understanding focuses on ability to diagnose and rectify cross-modal inconsistencies. Unlike self-refining image editing, this task begins with a pre-existing image-caption pair \((I_1, C_1)\) that contain misalignments. The model must critically recognize these discrepancies to drive the refinement process.

In each round \(r\), the model performs a “detect-and-correct” cycle, it first identifies semantic mismatches between the current image \(I^{me\_mu}_r\) and the original caption \(C_1\), then formulates a corrective instruction \(T_r\) to produce a better-aligned image \(I^{me\_mu}_{r+1}\). This loop terminates when the model perceives no further conflicts or reaches the maximum \(R\) rounds. The final performance \(\text{\textit{Score}}_\text{\textit{me\_mu}}\) is quantified by measuring the alignment between the model-identified mismatches and human-annotated ground truth. Similar to the image editing task, we define self-refining gain \(\Delta S_{u}\) as the absolute improvement from initial state to final round.

The overall mutual enhancement performance is defined as \(\text{\textit{Score}}_\text{\textit{me}} = (\text{\textit{Score}}_\text{\textit{me\_ie}} + \text{\textit{Score}}_\text{\textit{me\_mu}}) / 2\).

Figure 1: Data distribution of Unison. Unison comprises four core categories, each designed to evaluate a specific capability.

4 Dataset & Evaluation↩︎

Unison supports understanding, generation, and unified assessment. Data distribution is shown in 1. 4.1 details the data structure and compares Unison with existing benchmarks. 4.2 defines evaluation metrics. 4.3 introduces Unison-Judge, an evaluation model closely aligned with human preference for evaluation.

4.1 Benchmark Construction↩︎

To ensure a high-quality, rigorous evaluation framework, our benchmark is constructed through a collaborative pipeline involving state-of-the-art foundation models, followed by three weeks of verification and refinement by ten people.

Data Corpus and Image Generation. Our benchmark construction begins with a diverse data curation process tailored to multi-dimensional capabilities. For general vision-language tasks, we derive comprehensive captions from predefined attributes following GenEval [3] protocols. For complex scene understanding, we select a subset from RefCOCOg [37] characterized by multi-object clusters and significant occlusions. High-quality aesthetic samples are further sourced from LAION-Aesthetics [38] by filtering for scores exceeding 5.0. To generate the visual content, we employ Qwen-Image [39] for text-to-image synthesis, while spatial reasoning images are generated by mapping logical constraints of unfolded diagrams onto 3D structures using OpenCV. For image editing tasks, we define 11 compositional types by combining four meta-operations: Alter, Remove, Replace, and Add, ensuring each instruction contains at least two edits to maintain multi-step complexity.

Instruction Generation and Detection / Segmentation. To facilitate fine-grained interaction, we implement a multi-stage instruction generation pipeline. We leverage GPT-5.2 [40] and Gemini 3 Pro [41] to produce initial image captions and high-level semantic prompts, which are then refined through customized templates to ensure information integrity. For VQA task construction, we utilize the Davidsonian Scene Graph (DSG) [42] framework, prompting Qwen3-VL-235B-A22B-Instruct [43] to generate complex questions and candidate options. In tasks requiring precise localization, target objects are manually annotated with bounding boxes, which serve as prompts for SAM 3 [44] to produce segmentation masks. These masks undergo post-processing to ensure pixel-level accuracy for subsequent editing and understanding evaluations.

Automated Filtering and Human Annotation. To guarantee the highest fidelity and logical consistency, all generated data undergoes a rigorous dual-validation process. An initial automated filtering phase is applied to eliminate structural ambiguities and samples that fail to meet aesthetic or alignment thresholds. Subsequently, we conduct an extensive expert manual review. This phase includes: 1) inspecting image-text pairs to ensure all textual attributes are accurately reflected in the visual content; 2) correcting logical inconsistencies in the VQA reasoning chains; and 3) refining instruction prompts to eliminate any potential ambiguities. This refinement process ensures that the benchmark provides high-quality ground truth for evaluating both generative and discriminative capabilities of UMMs.

Table 2: Comparative results of human alignments between different models. P. denotes parameters.
P. Human Alignment (\(\rho\)) Average
IC UGG GGU ME
Qwen3-VL 8B 0.885 0.863 0.524 0.806 0.770
GPT-5.2 - 0.892 0.870 0.824 0.891 0.869
Gemini 3 Pro - 0.905 0.887 0.831 0.911 0.884
Unison-Judge 8B 0.896 0.893 0.863 0.895 0.887

4.2 Evaluation Metrics↩︎

We evaluate UMMs using a suite of task-specific metrics. For localization and segmentation, we utilize Intersection over Union (IoU) to quantify predicted regions. For UniWorld, since it generates masks directly on images, we compute pixel-wise alignment by overlaying these masks onto the original images. Image generation is benchmarked against human-annotated VQA data to ensure compositional semantic fidelity. For image editing, we follow ImgEdit [36] to employ qualitative ratings on a scale of 1 to 5, which are transformed into a normalized scale of \((0, 1)\), specifically evaluating attribute consistency of edited regions and preservation of unedited areas.

4.3 Unison-Judge↩︎

To ensure a more reasonable and fine-grained evaluation, we shift from traditional feature-level similarity metrics [29][31] to a customized MLLM-based assessment framework, which aligns more closely with human preference. We introduce Unison-Judge, an evaluation model derived by fine-tuning Qwen3-VL-8B-Instruct on a meticulously curated corpus. This corpus consists of 8,000 manually labeled samples in total, with 2,000 allocated to each dimension to ensure a uniform distribution. Furthermore, we conduct human alignment experiments on 231 samples uniformly sampled across all tasks, as shown in ¿tbl:table3?. Despite Gemini 3 Pro and GPT-5.2 show competitive performance on certain subsets, Unison-Judge achieves the superior average alignment score. This result underscores both the effectiveness and computational efficiency of our proposed framework.

5 Experiment↩︎

Experimental Setups. We evaluate UMMs across a comprehensive suite of both open-source and closed-source platforms. Open-source lineup encompasses Show-o/Show-o2 [13], [14], Janus-pro [10], D-DiT [11], ILLUME+ [18], OmniGen2 [45], TokenFlow [20], Bagel [8], SEED-X [15], and UniWorld [22], respectively. Closed-source models are Gemini 3 Pro [41] and GPT-5.2 [40]. To ensure a fair comparison, all models adhere to a standardized input/output format, with image generation and editing resolution fixed at \(512 \times 512\). In mutual enhancement experiments, we cap the process at 5 rounds. Notably, early termination is triggered if a model determines that no further errors remain.

5.1 Evaluation Results↩︎

As shown in ¿tbl:table3?, we evaluate various open-source and closed-source models. We averaged the scores from the four dimensions to obtain the final score.

In internal consistency evaluation, we find that understanding performance typically exceeds generation performance. Notably, a model’s high comprehension or generation capability does not guarantee good consistency. For example, D-DiT [11], despite trailing in an absolute score of 86.5 for understanding, maintains higher consistency which is reflected by the Uni. score compared to TokenFlow [20]. This difference may be attributed to TokenFlow’s separated architecture, which decouples understanding and generation, while D-DiT’s unified diffusion-based design enables better cross-capability alignment.

For the Und.-Guided Gen. task, while reasoning capability generally improves contextual understanding. ILLUME+-7B [18] shows an unsatisfactory generation score of only 10.4. Its low understanding score of 12.4 indicate that ineffective reasoning due to semantic misalignment limits its overall image editing performance.

In the Gen-Guided Und. task, the Und. scores remain low, revealing numerous inaccuracies. When the Uni. score falls below the average of the Und. and Gen. scores, it indicates that the generated image has a negative impact on the model’s comprehension. Conversely, a score above the average suggests a beneficial effect. Additionally, imperfectly generated images may paradoxically enhance the model’s understanding performance, by providing contrastive boundaries that reduce ambiguity, as shown in 2.

Finally, in the Mutual Enhancement task, models like Omnigen2 [45] show significant performance gains, with an Uni. score of 47.7. However, BAGEL [8] and UniWorld[22] obtain relatively low Uni. scores due to their lower Und. or Gen. scores. These results suggest that UMMs clearly benefit from multi-modal synergies, but strong capabilities in either understanding or generation does not necessarily guarantee superior performance in multi-turn collaborative settings.

Figure 2: Qualitative Visualizations. IC, UGG, GGU and ME denotes Internal Consistency, Understanding-Guided Generation, Generation-Guided Understanding and Mutual Enhancement, respectively. For GGU, visualizations only include the 3D spatial task.

6 Insights and Discussions↩︎

Importance of Internal Consistency. The results in Table ¿tbl:table3? highlight that internal consistency is a fundamental prerequisite for high-performing UMMs. We observe a strong correlation between a model’s overall score and its performance in the internal consistency dimension. For instance, closed-source models like Gemini 3 Pro [41] and GPT-5.2 [40] exhibit near perfect consistency score, which serves as the foundation for their superior reasoning capabilities. Among open-source models, BAGEL [8] and OmniGen2 [45] achieve the highest overall scores (\(53.2\) and \(51.3\) respectively) by maintaining robust consistency across both understanding and generation tasks. This emphasizes the importance of a UMM’s ability to maintain self-alignment, which plays a critical role in minimizing errors.

UMM’s understanding and generation capabilities can synergistic work. Contrary to the belief that understanding and generation are disparate tasks, our benchmark demonstrates a clear synergistic effect. In the Understanding-Guided Generation and Generation-Guided Understanding, models do not merely perform well in isolation, they leverage one capability to bootstrap the other. For example, BAGEL achieves a state-of-the-art open-source score of \(67.9\) in the UGG task. This indicates that the latent representations learned during pre-training provide rich structural priors that enhance generation, and vice versa.

UMMs can self-reinforce autonomously. A pivotal finding from the mutual enhancement category is that UMMs possess the latent ability for autonomous self-improvement. Even when the reference information provided to the model is sub-optimal or contains noise, as implied by the gap between guided and unified scores, models like Omnigen2 and UniWorld show significant gains in the Uni. score (\(47.7\) and \(31.3\)) of Mutual Enhancement. This suggests that these models can perform internal cross-verification, where the generation process acts as a check on understanding, and the understanding module filters generative outputs. This mechanism allows the model to extract value even from incorrect or noisy internal states, effectively achieving a degree of self-reinforcement without external supervision.

7 Limitations↩︎

Computational Cost of Evaluation. Assessing mutual enhancement involves iterative cycles of generation followed by comprehension, and vice versa. This recursive process significantly increases computational overhead, presenting scalability challenges when extending the benchmark to high-resolution imagery or multi-frame video sequences.

Metric Sensitivity. Although we introduce Unison-Judge, demonstrating competitive human alignment compared to Gemini 3 Pro, the models under assessment may still exhibit sensitivity to prompt engineering or generative stochasticity. This can introduce a degree of variance that might not fully reflect the model’s inherent reasoning limits.

8 Future Directions↩︎

Inernal Consistency Improvement. Future research should move beyond standard pre-training objectives by incorporating explicit consistency objective. By penalizing discrepancies between a model’s discriminative understanding and its generative output during training, we can foster more robust self-alignment and minimize the hallucinations and error accumulations inherent in current UMMs.

Self-Refinement Training via Agentic Planning. A promising direction lies in integrating UMM self-refinement with autonomous agentic planning strategies. By enabling models to autonomously design task-specific plans, they can treat understanding and generation as iterative feedback steps in a self-play loop. This allows the model to generate its own pseudo-supervision signals, facilitating continuous self-evolution and reinforcement learning

Additional Modality Support. While current open-source UMMs largely lack video synthesis capabilities, integrating the video modality is essential. Future work should bridge the gap between video understanding and generation, enabling models to simulate physical world dynamics and improve long-range causal reasoning.

9 Conclusion↩︎

We introduce Unison, a benchmark designed to evaluate the synergistic understanding and generation capabilities of unified multimodal models. Unlike prior benchmarks that assess these skills in isolation, Unison integrates four key dimensions: internal consistency, understanding-guided generation, generation-guided understanding, and mutual enhancement, and offers both unified and decoupled evaluation tracks for fine-grained analysis. To ensure reliable scoring, we train Unison-Judge, a human-aligned evaluator. Our evaluations reveal critical gaps in current models and demonstrate performance comparable to Gemini 3 Pro.

Impact Statement↩︎

Unison provides a comprehensive framework for unified multimodal research by systematically evaluating the essential synergy between understanding and generation. This work contributes to the reliability of AI systems by identifying internal semantic conflicts and hallucinations, while enabling efficient, human-aligned automated assessment through Unison-Judge. By revealing critical performance gaps in current models, Unison offers practical observations that can help researchers design and develop more consistent and robust unified multimodal architectures in the future.

Acknowledgment↩︎

This work was supported in part by the Science and Technology Commission of Shanghai Municipality under Grant No. 25511103600.

References↩︎

[1]
Y. Liu et al., “Mmbench: Is your multi-modal model an all-around player?” 2024.
[2]
X. Yue et al., “Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi,” 2024.
[3]
D. Ghosh, H. Hajishirzi, and booktitle=NIPS. Schmidt Ludwig, “Geneval: An object-focused framework for evaluating text-to-image alignment,” 2023.
[4]
X. Hu, R. Wang, Y. Fang, B. Fu, P. Cheng, and G. Yu, “Ella: Equip diffusion models with llm for enhanced semantic alignment,” arXiv preprint arXiv:2403.05135, 2024.
[5]
J. Chang et al., “OneIG-bench: Omni-dimensional nuanced evaluation for image generation,” 2025.
[6]
Y. Li et al., “MANZANO: A simple and scalable unified multimodal model with a hybrid vision tokenizer,” arXiv preprint arXiv:2509.16197, 2025.
[7]
R. Tian et al., “UniGen: Enhanced training & test-time strategies for unified multimodal understanding and generation,” 2025.
[8]
C. Deng et al., “Emerging properties in unified multimodal pretraining,” arXiv preprint arXiv:2505.14683, 2025.
[9]
Z. Geng et al., “X-omni: Reinforcement learning makes discrete autoregressive image generative models great again,” arXiv preprint arXiv:2507.22058, 2025.
[10]
X. Chen et al., “Janus-pro: Unified multimodal understanding and generation with data and model scaling,” arXiv preprint arXiv:2501.17811, 2025.
[11]
Z. Li et al., “Dual diffusion for unified image generation and understanding,” 2025.
[12]
Y. Wu et al., “Vila-u: A unified foundation model integrating visual understanding and generation,” 2025.
[13]
J. Xie et al., “Show-o: One single transformer to unify multimodal understanding and generation,” 2024.
[14]
J. Xie, Z. Yang, and booktitle=NIPS. Shou Mike Zheng, “Show-o2: Improved native unified multimodal models,” 2025.
[15]
Y. Ge et al., “SEED-x: Multimodal models with unified multi-granularity comprehension and generation,” arXiv preprint arXiv:2404.14396, 2024.
[16]
S. Xiao et al., “Omnigen: Unified image generation,” 2025.
[17]
J. Chen et al., “Blip3-o: A family of fully open unified multimodal models-architecture, training and dataset,” arXiv preprint arXiv:2505.09568, 2025.
[18]
R. Huang et al., “Illume+: Illuminating unified mllm with dual visual tokenization and diffusion refinement,” arXiv preprint arXiv:2504.01934, 2025.
[19]
J. Wu et al., “Liquid: Language models are scalable multi-modal generators,” IJCV, 2025.
[20]
L. Qu et al., “Tokenflow: Unified image tokenizer for multimodal understanding and generation,” 2025.
[21]
C. Zhou et al., “Transfusion: Predict the next token and diffuse images with one multi-modal model,” 2025.
[22]
B. Lin et al., “Uniworld: High-resolution semantic encoders for unified visual understanding and generation,” arXiv preprint arXiv:2506.03147, 2025.
[23]
B. A. Plummer, L. Wang, C. M. Cervantes, J. C. Caicedo, J. Hockenmaier, and booktitle=ICCV. Lazebnik Svetlana, “Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models,” 2015.
[24]
X. Chen et al., arXiv preprint arXiv:1504.00325, 2015.
[25]
S. Antol et al., “Vqa: Visual question answering,” 2015.
[26]
Y. Goyal, T. Khot, D. Summers-Stay, D. Batra, and booktitle=CVPR. Parikh Devi, “Making the v in vqa matter: Elevating the role of image understanding in visual question answering,” 2017.
[27]
A. Singh et al., “Towards vqa models that can read,” 2019.
[28]
B. Li, R. Wang, G. Wang, Y. Ge, Y. Ge, and booktitle=CVPR. Shan Ying, “Seed-bench: Benchmarking multimodal llms with generative comprehension,” 2024.
[29]
T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and booktitle=NIPS. Chen Xi, “Improved techniques for training gans,” 2016.
[30]
M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and booktitle=NIPS. Hochreiter Sepp, “Gans trained by a two time-scale update rule converge to a local nash equilibrium,” 2017.
[31]
J. Hessel, A. Holtzman, M. Forbes, R. Le Bras, and booktitle=EMNLP. Choi Yejin, “Clipscore: A reference-free evaluation metric for image captioning,” 2021.
[32]
K. Huang, K. Sun, E. Xie, Z. Li, and booktitle=NIPS. Liu Xihui, “T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation,” 2023.
[33]
T. Brooks, A. Holynski, and booktitle=CVPR. Efros Alexei A, “Instructpix2pix: Learning to follow image editing instructions,” 2023.
[34]
K. Zhang, L. Mo, W. Chen, H. Sun, and booktitle=NIPS. Su Yu, “Magicbrush: A manually annotated dataset for instruction-guided image editing,” 2023.
[35]
Q. Yu et al., “Anyedit: Mastering unified high-quality image editing for any idea,” 2025.
[36]
Y. Ye et al., “ImgEdit: A unified image editing dataset and benchmark,” 2025.
[37]
J. Mao, J. Huang, A. Toshev, O. Camburu, A. L. Yuille, and K. Murphy, “Generation and comprehension of unambiguous object descriptions , booktitle = CVPR,” 2016.
[38]
C. Schuhmann et al., “Laion-5b: An open large-scale dataset for training next generation image-text models,” 2022.
[39]
C. Wu et al., “Qwen-image technical report,” arXiv preprint arXiv:2508.02324, 2025.
[40]
“GPT-5.2.” 2025.
[41]
G. Team et al., “Gemini: A family of highly capable multimodal models,” arXiv preprint arXiv:2312.11805, 2023.
[42]
J. Cho et al., arXiv preprint arXiv:2310.18235, 2023.
[43]
S. Bai et al., “Qwen3-VL technical report,” arXiv preprint arXiv:2511.21631, 2025.
[44]
N. Carion et al., arXiv preprint arXiv:2511.16719, 2025.
[45]
C. Wu et al., “OmniGen2: Exploration to advanced multimodal generation,” arXiv preprint arXiv:2506.18871, 2025.