the@affilaff1

the@affilaff2

the@affilaff3

the@affilaff4

the@affilaff5

the@affilaff6

the@affilaff7

the@affilaff8

the@affilaff9

the@affilaff10

the@affilaff11

the@affilaff12

the@affilaff13

1 Introduction↩︎

Medical diagnosis fundamentally follows a structured multi-level reasoning pipeline that is inherently multimodal in both inputs and outputs. Physicians systematically observe multimodal raw data, e.g., imaging patterns, patient histories, and symptom descriptions [1], [2], integrate this with medical knowledge, e.g., medical literature, domain expertise, and cross-modal associations [3], [4], and analyse to produce diverse diagnostic outputs: textual reports explaining findings, visual annotations localizing abnormalities, and comparative imagery for treatment planning [5]–[8].

Consider a radiologist examining suspected lung pathology: they process chest X-rays, prior CT scans, and patient history to generate multiple complementary outputs: detailed reports describing findings, visual annotations highlighting specific regions, and comparative visualizations for surgical planning. This exemplifies how a typical medical diagnosis requires unified processing of multimodal inputs to generate diverse multimodal outputs, where neither textual reports alone nor visual annotations alone suffice. However, existing medical AI systems can only understand medical images or produce visual annotations using separate models [9]–[13], rather than unifying them within a single model, as shown in Fig. 1. The limitation primarily originates from three critical levels: (i) Data: Medical datasets remain predominantly single-modal despite clear evidence that multimodal integration substantially improves diagnostic accuracy [14]–[16]. (ii) Learning Paradigm: Although existing medical multimodal models accept multimodal inputs, current approaches essentially apply naive two-stage training strategies, i.e., continual pre-training and instruction tuning, and thus lack a systematic progressive learning paradigm that can effectively capture deep cross-modal relationships between visual and textual information. Recent progress in large language models shows that a well-designed learning paradigm can elicit emergent cross-task generalization at scale, yet such a paradigm remains absent in medical multimodal modeling. (iii) Capability Convergence: While general-domain models have made progress toward unified architectures, the medical domain still lacks truly unified models. For instance, although HealthGPT [11] demonstrates both understanding and generation capabilities for medical tasks, it requires loading different model checkpoints depending on the task type. However, the design compels domain experts to manually select different checkpoints for different tasks, hindering seamless multi-task operation in medical workflows. Hence, this raises a central question for medical multimodal modeling: can image understanding and image generation share a single medical model and mutually reinforce each other, or does joint training inevitably compromise either capability?

To bridge this gap, we cast unified medical multimodal modeling as a three-level alignment problem through the Observation-Knowledge-Analysis (OKA) paradigm, directly mapping it to the three bottlenecks above: data, learning paradigm, and capability convergence. At the Observation Level, we reformat existing single-modal medical datasets across various tasks into multimodal input-output pairs and construct UniMedVL-5M, comprising over 5.6M samples; this directly exposes the model to large-scale cross-modal pairings instead of disjoint single-modal data and address the data silo that bottlenecks current medical multimodal learning. At the Knowledge Integration Level, we design Progressive Curriculum Learning with three stages: foundation training for basic medical vision–language alignment, instruction tuning for task-following on high-quality data, and a final unified multimodal training stage that jointly optimizes understanding and generation with interleaved inputs and outputs. At the Analysis Level, we introduce UniMedVL, a unified medical model that performs both understanding and generation with a single set of parameters trained jointly under the curriculum above.

Experiments provide evidence that, in medical multimodal modeling, aligned multimodal supervision and Progressive Curriculum Learning enable bidirectional transfer between image understanding and generation within a single model. Our experiments suggest two insights: (1) unified medical training captures cross-modal correlations that transfer across medical tasks and benefit both understanding and generation; and (2) the learned unified representation can be effectively adapted to downstream tasks via task-specific fine-tuning. Our contributions are as follows, as illustrated in Fig. [fig:teaser]:

• Medical Multimodal Dataset. We construct and release UniMedVL-5M, a 5.6M-sample multimodal medical dataset covering 8 imaging modalities. It provides a standardized training scheme that jointly supports medical image understanding and generation.

• Progressive Multimodal Learning Paradigm. We propose a three-stage training pipeline with stage-specific data enhancements and a final unified multimodal training stage, offering a reusable recipe for unified medical model training.

• Unified Medical Vision-Language Model. We introduce UniMedVL, a medical vision-language model with a single set of parameters for both medical image understanding and generation. UniMedVL unifies inference across task types and demonstrates competitive performance.

2 Related Work↩︎

2.1 Medical Multimodal Large Language Models↩︎

Early medical MLLMs largely adopted adapter-style fusion between medical vision encoders and general LLMs [17], [18], enabling visual question answering and report generation, but often generalizing poorly to real-world medical tasks. A data-centric effort, HuatuoGPT-Vision, converts PubMed papers into large-scale medical image-caption and VQA data [19]; however, it remains primarily comprehension-oriented with limited medical reasoning. Reasoning-focused systems including BioMedGPT and Med-PaLM 2 improved biomedical reasoning [20], zhang2023biomedgpt?, and a recent study further systematizes medical output evaluation [21]; however, these works do not unify image-level generation with textual reasoning. Recently, [11] proposed HealthGPT as the first medical MLLM for unified multimodal inputs and outputs. It introduces MoE LoRA to mitigate task interference and support diverse tasks. Its unification relies on multiple task-specific models during inference. Thus, these capabilities are not consolidated into a single end-to-end model that performs all tasks in a unified manner.

2.2 Unified Image Understanding and Generation↩︎

Outside the medical domain, unified multimodal research spans several paradigms. Autoregressive models [22]–[24] treat images as discrete tokens in decoder-only Transformers, providing architectural unity but constraining high-resolution synthesis due to long sequences and discrete reconstruction. Dual-encoder designs [25], [26] mitigate the granularity conflict between semantic understanding and pixel-level generation via separate visual pathways, improving task performance at higher inference cost. Hybrid objectives jointly optimize complementary generative losses, as in Transfusion [27] and Show-O [28], while modular approaches [29]–[31] connect frozen MLLMs to diffusion models via learnable connectors, trading end-to-end differentiability for flexibility. Representation advances narrow the semantics–fidelity gap via multi-codebook quantization [32], contrastive-aligned tokenization [33], and unified CLIP semantic spaces [34]. Recent autoregressive methods [35], [36] further improve interleaved generation through deeper fusion. Despite these advances, balancing semantic understanding with pixel-level image generation remains challenging in the medical field, where supervision is fragmented and objectives are heterogeneous.

3 Methodology↩︎

We present UniMedVL through an observation–knowledge–analysis framework, which first curates large-scale medical multimodal observations, then builds task capability through progressive curriculum learning, and finally unifies medical understanding and generation within a single model.

3.1 Observation Level↩︎

At the observation level, we address the data bottleneck by reformulating fragmented medical datasets into aligned multimodal input-output pairs.

Data Source and Modality Coverage. UniMedVL-5M is built through systematic re-curation and re-creation of heterogeneous medical resources into a unified multimodal learning corpus. It spans 8 primary medical imaging modalities and supports understanding, generation, and interleaved generation tasks through standardized multimodal input-output instances. Detailed dataset statistics are provided in Appendix 6.2.

Quality Control Pipeline. The data curation process is formalized as sequential quality filters: \[\label{eq:data95pipeline} \mathcal{D}_{\mathrm{qc}} = \mathcal{F}_{\mathrm{align}} \circ \mathcal{F}_{\mathrm{coarse}}(\mathcal{D}_{\mathrm{raw}})\tag{1}\] where \(\mathcal{F}_{\mathrm{coarse}}\) and \(\mathcal{F}_{\mathrm{align}}\) retain image-text pairs \((x, t) \in \mathcal{X} \times \mathcal{T}\) satisfying progressively stricter criteria. Expert validation is performed on a stratified subset of \(\mathcal{D}_{\mathrm{qc}}\) as an audit.

• Coarse Filtering (\(\mathcal{F}_{\mathrm{coarse}}\)). We apply standard modality-specific preprocessing and light text cleaning/tokenization and retain samples based on the quality indicator: \(\phi(x, t) = \boldsymbol{1}[\min\{H(x), W(x)\} \geq 128] \cdot \boldsymbol{1}[\tau_{\min} \leq \ell(t) \leq \tau_{\max}]\), where \(H(x)\) and \(W(x)\) denote image height and width, and \(\ell(t)\) is the text length in characters with bounds \(\tau_{\min}=10\), \(\tau_{\max}=1024\).

• Medical Alignment (\(\mathcal{F}_{\mathrm{align}}\)). To preserve pathological findings, we compute a composite alignment score. MedGemma-27b [37] generates \(K=5\) candidate captions \(\mathcal{C}(x) = \{c_k\}_{k=1}^K\) per image. The alignment score combines semantic similarity and medical-specific matching: \[\label{eq:align95score} \begin{align} S(x,t) &= \max_{c \in \mathcal{C}(x)} \cos(E(t), E(c)),\\ M(x,t) &= S_{\mathrm{MedSigLIP}}(x, t),\\ S_{\mathrm{final}}(x, t) &= \lambda \cdot S(x,t) + M(x,t) \end{align}\tag{2}\]

  where \(E(\cdot)\) denotes E5-large-v2 embeddings [38], \(S_{\mathrm{MedSigLIP}}\) is obtained via MedSigLIP [37], and \(\lambda=0.5\). We normalize both \(S(x,t)\) and \(M(x,t)\) to \([0,1]\) before combination. We retain the top 50% of pairs ranked by \(S_{\mathrm{final}}\) to obtain a higher-quality subset of training data.

• Expert Validation. Five medical experts independently assess a stratified subset spanning medical modalities, with sampling proportional to each modality’s frequency. Each sample is rated along seven medically grounded dimensions. We aggregate the seven scores into a mean quality metric, \(\bar{Q}=\frac{1}{7}\sum_{j=1}^{7} Q_j\). We note that MedGemma-27b is used only for alignment scoring during filtering, not to generate the retained training targets; the retained pairs remain original image–text pairs from human-validated open datasets. Expert validation serves as a quality audit on a representative subset, but not a guarantee of zero error across the full dataset.

Interleaved Tasks Construction. We curate data from 5 interleaved tasks, including medical image prompt segmentation, super-resolution, counterfactual generation, virtual immunohistochemistry staining, and cross-modal synthesis. Since the original datasets contain only images, we develop a two-stage construction pipeline to formulate them into structured multimodal input-output pairs: Templatization and MLLM Refinement. First, we instantiate structured multimodal input-output pairs by sampling textual prompts and responses from a task-specific template library. Then, we leverage MedGemma-27b to refine the linguistic diversity and medical expressiveness of the input prompts and output explanations. This two-stage design lets us scale interleaved supervision from image-only sources while keeping the textual prompts and explanations clinically faithful. The resulting instances provide aligned multimodal targets that directly support the curriculum described next.

3.2 Knowledge Level: Progressive Curriculum Learning↩︎

At the knowledge level, we address heterogeneous objectives in medical multimodal modeling with Progressive Curriculum Learning. The curriculum first learns medical vision–language alignment, then strengthens instruction following, and couples understanding and generation through interleaved multimodal tasks.

• Stage 1: Foundation Training. We conduct foundation training on the UniMedVL-5M dataset to equip the model with basic medical image understanding and generation capabilities. At this stage, training emphasizes learning general medical vision-language alignments without task-specific constraints.

• Stage 2: Instruction Tuning. In this stage, we conduct instruction tuning on our curated high-quality instruction data to improve the model’s instruction-following capability for solving complex medical tasks. The instruction-formatted medical tasks follow the format \((q, x_v, k) \rightarrow (a_t, a_v)\) where query \(q\), visual input \(x_v\), and knowledge context \(k\), e.g., patient history, imaging modality metadata, or medical guidelines relevant to the query, generate textual \(a_t\) and visual \(a_v\) responses. We implement enhancement strategies for distinct task types: For medical understanding tasks such as medical VQAs, we augment standard responses with existing Distilled Chain of Thought (DCOT) data that explicitly articulate the reasoning pathway from visual observation to medical conclusions. For generation tasks, we employ Caption Augmented Generation (CAG) pipeline to enhance caption quality, which incorporates structured planning steps that guide the visual synthesis process. The details are provided in Appendix 6.4.

• Stage 3: Unified Multimodal Training. We further adapt the stage-2 model using our curated interleaved tasks, where the model jointly processes image inputs and textual prompts to produce corresponding visual content and explanatory text. This stage is designed to establish interleaved multimodal reasoning, which requires our model to integrate understanding and generation within unified sequences. The model consolidates previously learned capabilities while unlocking the ability to generate visual and textual contents simultaneously.

3.3 Analysis Level: UniMedVL↩︎

At the analysis level, UniMedVL uses a single set of parameters to perform both medical understanding and generation.

Task Organization. Model training is systematically organized into three primary task families that reflect the fundamental capabilities required for unified medical multimodal systems: (i) Understanding tasks, which emphasize medical image comprehension, visual question answering, diagnostic reasoning, image captioning, and medical report generation; (ii) Generation tasks, which focus on conditional medical image synthesis and planning-guided text-to-image generation; and (iii) Interleaved tasks, which involve coupled visual-textual inputs and outputs. Unlike single-direction understanding or generation tasks, interleaved tasks require the model to perform bidirectional multimodal grounding, maintain cross-modal semantic consistency and combine both of textual and visual responses within a unified sequence.

Model Architecture Overview. Following [30], we adopt a unified architecture with dual visual encoders and a Transformer backbone. The understanding-oriented encoder \(E_{\mathrm{VIT}}\) extracts semantic tokens \(z_{\mathrm{VIT}} = E_{\mathrm{VIT}}(x_v)\) for multimodal comprehension tasks, while the generation-oriented encoder \(E_{\mathrm{VAE}}\) produces latent representations \(z_{\mathrm{VAE}} = E_{\mathrm{VAE}}(x_v)\) for visual synthesis tasks. The transformer backbone consists of specialized FFN layers for the understanding and generation tasks, with shared self-attention layers. The VIT tokens \(z_{\mathrm{VIT}}\) and VAE tokens \(z_{\mathrm{VAE}}\) are first processed by the specialized FFN layers and then concatenated with text tokens \(x_{\text{text}}\) to form a single token sequence. This sequence is fed into shared self-attention layers to predict the next text tokens and the velocity on the VAE latent features. For generation outputs, the decoder \(D_{\mathrm{VAE}}\) reconstructs the VAE latent features to images.

3pt

Training Objectives. We train a single model for multimodal understanding and conditional image generation. Given an image-text pair \((X,T)\) with \(T=(T_{\mathrm{in}},T_{\mathrm{out}})\), we extract two latent features \(z_{\mathrm{VIT}}=E_{\mathrm{VIT}}(X)\) and \(z_{\mathrm{VAE}}=E_{\mathrm{VAE}}(X)\), and define the unified condition \(C=(T_{\mathrm{in}}, z_{\mathrm{VIT}})\). To motivate the unified training objective, we observe a standard information-theoretic identity: jointly modeling the output distribution \((T_{\mathrm{out}}, z_{\mathrm{VAE}})\) can exploit cross-task correlations that conditionally factorized (independent) training cannot. Specifically, the joint conditional entropy is bounded by the sum of individual entropies, with the gap quantifying the redundancy reduction: \[\label{eq:info95bound} \begin{align} & \left[H(T_{\mathrm{out}}\mid C)+H(z_{\mathrm{VAE}}\mid C)\right] - H(T_{\mathrm{out}}, z_{\mathrm{VAE}}\mid C) \\ & \quad = I(T_{\mathrm{out}}; z_{\mathrm{VAE}}\mid C) \ge 0. \end{align}\tag{3}\] This inequality implies that a unified model can exploit cross-modal correlations (\(I > 0\)) to achieve a lower Bayes-optimal uncertainty than separate models. This perspective suggests that joint training can outperform factorized training when the two outputs share non-trivial conditional dependencies. Full derivations are provided in Appendix 6.9. Motivated by this theoretical lower bound, we train the model with a combined objective of next-token prediction (NTP) \(\mathcal{L}_{\mathrm{NTP}}\) and rectified flow matching \(\mathcal{L}_{\mathrm{flow}}\): \[\label{eq:main95obj} \mathcal{L}(\theta) = \mathcal{L}_{\mathrm{NTP}} + \mathcal{L}_{\mathrm{flow}},\tag{4}\] where the individual loss terms are \[\label{eq:main95obj95terms} \begin{align} \mathcal{L}_{\mathrm{NTP}} &= -\sum_{i=1}^{n} \log p_\theta\!\left(t_{i+1}\mid t_{\le i}, z_{\mathrm{VIT}}\right), \\ \mathcal{L}_{\mathrm{flow}} &= \mathbb{E}_{t,\epsilon}\!\left[\left\|v_\theta(\tilde{z}_{\mathrm{VAE}}, t, C) - v\right\|_2^2\right], \end{align}\tag{5}\] with \(\tilde{z}_{\mathrm{VAE}}\) and \(v\) following the standard rectified-flow construction.

4 Experiments↩︎

4.1 Benchmarks and Baselines↩︎

Evaluation Benchmarks. We evaluate UniMedVL across medical visual understanding and generation benchmarks. For image understanding tasks, we employ VQA-RAD [39], SLAKE [40], PathVQA [41], OmniMedVQA [42], and GMAI-MMBench [43], which cover diverse medical scenarios. For image generation tasks, we split the image–caption pairs in the proposed dataset into 80% for training and 20% for testing. We use the test set to evaluate UniMedVL’s text-to-image generation performance. For interleaved tasks, we utilize the BCI dataset [44] for the virtual immunohistochemistry staining task. The IXI dataset [45] is leveraged to evaluate the super-resolution task, and the BraTS 2023 dataset [46] is used for evaluating the cross-modal synthesis task. We use the ICG-CXR dataset [47] to evaluate the counterfactual generation task.

Baseline Methods. We consider two categories: specialized models and unified multimodal models. For specialized models, we include understanding-only models such as Med-Flamingo [48], LLaVA-Med [18], HuatuoGPT-Vision [19], RadFM [49], GMAI-VL[9], Lingshu[50], LLaVA-v1.5 [51], and InternVL2 [52]. We compare against modality-specialized medical generative models, including RetinaLogos [10] for color fundus photographs, MedSyn [53] for CT, CheXGen [54] for chest X-ray, and PathLDM [55] for histopathology. We compare with image translation models including CycleGAN [56], pix2pix [57], pix2pixHD [58], pyramid pix2pix [44], SRCNN [59], VDSR [60], SwinIR [61], Restormer [62], AMIR [63], ResViT [64], and TransUNet [65]. Additionally, to determine the model performance of medical imaging generation capability, we finetuned LlamaGen-MediTok [66]. For unified multimodal models, we include general models like Janus [25] and Bagel [30], as well as a medical model HealthGPT [11].

Evaluation Metrics. We employ task-specific metrics aligned with medical relevance. For image understanding tasks, we utilize accuracy as the evaluation metric. For image generation tasks, we employ generation FID (gFID) and BioMedCLIP score (CS) to evaluate the quality of synthesized images. For interleaved tasks, we leverage PSNR and SSIM as evaluation metrics for virtual immunohistochemistry staining, super-resolution, and cross-modal synthesis tasks. For counterfactual generation, we follow the experimental setup of ProgEmu [47], using gFID, AUC-ROC, and F1 to evaluate the quality of synthesized images, and BLEU-3, METEOR, and ROUGE-L to assess the quality of the explanatory text.

4.2 Performance of UniMedVL↩︎

Medical Visual Understanding Performance. We evaluate the understanding capabilities of UniMedVL across diverse medical VQA and image comprehension benchmarks. Table 1 compares UniMedVL with existing medical VLLMs and unified multimodal models. Despite supporting both understanding and generation within a single architecture, UniMedVL achieves competitive or superior performance across medical domains. Notably, compared with HealthGPT, which requires separate model checkpoints for different tasks, UniMedVL achieves 85.8% on OmniMedVQA versus HealthGPT-L14’s 74.4% while enabling seamless task switching.

3.0pt

3pt

3pt

3.5pt

Medical Image Generation Performance. Table 2 and Figure 3 evaluate generation quality across eight medical imaging modalities. Comparing UniMedVL-Gen (generation-only training) with UniMedVL reveals that incorporating understanding training substantially improves generation fidelity, reducing the average gFID from 108.40 to 96.29. UniMedVL also achieves BioMedCLIP scores of 0.706 on average across modalities in Figure 3, which indicates strong semantic alignment between generated images and medical text descriptions.

This challenges the conventional assumption that joint training compromises individual task performance and shows that medical multimodal learning benefits from task synergy. To verify generalization beyond the training distribution, we evaluate on a held-out external benchmark of text-to-image generation. As Table 3 shows, UniMedVL achieves the lowest average FID and highest average BioMedCLIP Score across eight modalities. This suggests that the generation gains extend beyond the training distribution. We further compare four modality-specialized generation models in Table 4. Despite its unified design, UniMedVL achieves lower FID than all four models, suggesting that unified training can preserve domain-specific generation quality. Across modalities, UniMedVL keeps FID below 100 on six out of eight medical imaging modalities. This suggests that, in our setting, understanding and generation can reinforce rather than compete within a single model.

[t]
    \centering
    \caption{\textbf{Performance on Medical Image Translation.}
    Bidirectional translation between T$_2$ and FLAIR MRI sequences. 
    $\dagger$ indicates  fine-tuning variant.}
    \label{tab:medical95translation}
    \small
    \setlength{\tabcolsep}{3.5pt}
    \adjustbox{width=0.85\textwidth,center}{
    \begin{tabular}{lccc}
    \toprule
    \textbf{Method} & \textbf{T$_2$ $\rightarrow$FLAIR} & \textbf{FLAIR$\rightarrow$T$_2$} & \textbf{Avg} \\
    \midrule
    ResViT & \textbf{24.97}/0.870 & \textbf{25.78}/\textbf{0.908} & \textbf{25.38}/\textbf{0.889} \\
    pGAN & 24.01/0.864 & 25.09/\underline{0.894} & 24.55/0.879 \\
    pix2pix & 23.15/0.869 & 24.52/0.883 & 23.84/0.876 \\
    A-UNet & 23.69/\underline{0.873} & 24.56/0.891 & 24.13/0.882 \\
    SAGAN & 24.02/0.860 & 25.10/0.893 & 24.56/0.877 \\
    \midrule
    HealthGPT-M3 & 18.88/0.745 & 19.30/0.750 & 19.09/0.748 \\
    \methodname$^{\dagger}$ & 23.99/0.711 & 23.49/0.732 & 23.74/0.722 \\
    \rowcolor{cyan!20}
    \textbf{\methodname} & \underline{24.90}/\textbf{0.881} & \underline{25.23}/0.883 & \underline{25.07}/\underline{0.882} \\
    \bottomrule
    \end{tabular}
    }

Performance on interleaved tasks. In Table 5, for virtual immunohistochemistry staining from H&E to IHC, UniMedVL reports 20.27 PSNR, exceeding HealthGPT-M3 by 28%. On MRI super-resolution, our model obtains 27.29 PSNR and 0.890 SSIM, indicating a clear gain over the unified baseline. For T2-to-FLAIR cross-modal synthesis in Table [tab:medical95translation], UniMedVL achieves an average PSNR of 25.07, narrowing the gap to specialized models [67] while preserving unified multimodal capabilities. Figure 4 presents qualitative examples of these results. The comparison between UniMedVL\(^\dagger\) and the full model further shows consistent gains across tasks, indicating that the complete progressive training paradigm captures cross-modal relationships beyond simple fine-tuning.

Table 6 evaluates counterfactual generation with accompanying explanatory text. UniMedVL\(^\dagger\) obtains 27.17 gFID and higher text-quality metrics than specialized baselines. Its counterfactual check rate of 0.797 AUROC further indicates that unified training can generate medically plausible scenarios while producing coherent textual explanations.

4.3 Ablation Study↩︎

2.2pt

Understanding-Generation Synergy. Table 7 shows that joint training (H-Joint-Base) benefits both directions during the foundation stage, surpassing the single-task variants. On understanding, adding generation training lifts GMAI-MMBench accuracy from 0.505 to 0.593, with similar relative gains of 24.8% on SLAKE and 10.9% on OMVQA. On generation, adding understanding training reduces the average gFID by 11.2% over the generation-only variant in Table 2, indicating that semantic supervision benefits generation fidelity. The lone exception is PathVQA, where a 0.5% dip is recovered through instruction tuning in Stage 2.

-4pt

2.0pt

Progressive Training Stages. Table 8 shows that progressive staging further reinforces the synergy between understanding and generation. One-stage joint training obtains 0.535 on GMAI-MMBench and 123.48 gFID, whereas the three-stage progressive approach reaches 0.608 and 96.29, corresponding to a 13.5% gain on GMAI-MMBench and a 22.0% reduction in gFID, respectively. The most pronounced gain in generation appears at Stage 3, where gFID decreases from 120.04 to 96.29 after introducing interleaved tasks that require simultaneous understanding and generation, indicating that these tasks are a key driver for connecting the two capabilities. While joint training already yields synergy in the earliest stage, the largest generation gain arrives only at Stage 3, where interleaved supervision couples the two capabilities most tightly.

4.0pt

4.0pt

Data Augmentation Strategies. Tables 9 and 10 assess our data enhancement approaches across understanding and generation tasks separately. For understanding tasks, DCOT raises PathVQA from 0.367 to 0.453, with joint training further improving SLAKE to 0.754. For generation tasks, CAG lowers gFID from 118.60 to 108.40, while interleaved joint training delivers the strongest generation quality. These results indicate that data-quality and training-strategy improvements are complementary: DCOT and CAG enhance understanding and generation data respectively, while progressive staging with interleaved tasks provides the optimization curriculum.

5 Conclusion↩︎

We presented UniMedVL, a unified framework for medical image understanding and generation within a single model, built on two medical-specific design choices: the UniMedVL-5M corpus of over 5.6M multimodal medical samples and a three-stage Progressive Curriculum Learning pipeline that progressively activates bidirectional transfer between understanding and generation. Ablation studies further indicate that joint training and progressive staging benefit both understanding and generation. These findings highlight the value of aligning data construction, training curriculum, and model objectives in a unified medical multimodal framework. UniMedVL achieves competitive understanding performance on five medical benchmarks while maintaining competitive generation quality across eight imaging modalities. We position this work as a step toward unified medical multimodal modeling rather than a deployable clinical solution. More broadly, we hope that the released dataset, UniMedVL-5M, will offer a convenient starting point for further study of how understanding and generation can be trained in unified medical settings.

5.0.0.1 Limitations.

The current study is restricted to 2D medical imaging and does not yet extend to volumetric modalities such as 3D CT or MRI volumes. Evaluation relies on standard automatic metrics for reproducibility; however, these do not substitute for clinical validation, and deployment in real-world clinical settings would therefore require further clinical investigation. In addition, as a single model trained across many tasks, UniMedVL does not yet match the strongest task-specific systems on every individual benchmark; we regard narrowing this gap while preserving unification as an important direction for future work.

Acknowledgements↩︎

This work was supported by Shanghai Artificial Intelligence Laboratory.

Impact Statement↩︎

This paper presents work whose goal is to advance the field of Machine Learning in AI4Health. The potential societal consequences of our work include both positive impacts (improved diagnostic accuracy, reduced clinical workload) and considerations that warrant attention (model reliability in clinical settings, potential biases in medical data).

This work does not involve experiments on human subjects, patient interventions, or the collection of private medical data. All datasets used in this study are publicly available, including CheXpertPlus, SLAKE, PathVQA, OmniMedVQA, IXI, BraTS 2023, and other open-access medical datasets. Data sources were used in accordance with their respective licenses, and all materials were de-identified prior to use. The purpose of this research is to advance the scientific understanding of unified multimodal modelling for healthcare data rather than to deploy clinical decision-support systems. No patient-level decisions or clinical predictions were made based on model outputs. Expert audits were limited to quality control of publicly available samples and did not involve identifiable patient information.

To ensure reproducibility, all implementation details, model configurations, and training hyperparameters are provided in the Appendix. The full code and configuration files are available at https://huggingface.co/datasets/General-Medical-AI/UniMedVL-5M. The UniMedVL-5M dataset construction process, including data sources, quality control criteria, and interleaved task synthesis, is fully documented in the Appendix and illustrated in Figure 2. All benchmarks used for evaluation, VQA-RAD, SLAKE, PathVQA, OmniMedVQA, BraTS 2023, and IXI, are publicly available or derived from publicly available sources.

6 Appendix↩︎

Appendix Contents↩︎

6.1 Implementation Details↩︎

6.1.1 Training Hyperparameters↩︎

Detailed Training Strategy Implementation. Our training employs a three-stage curriculum learning approach that implements the Knowledge component within the OKA framework. We use the AdamW optimizer throughout all stages:

• Stage 1 (Foundation Training) establishes basic medical understanding over 85K steps with a learning rate of \(5 \times 10^{-5}\). The data composition prioritizes image-to-text tasks (75%), complemented by text-to-image generation (25%) and pure text data (5%). This stage trains both ViT and LLM components end-to-end while keeping the VAE frozen. The image resolution is restricted with the range from 512-1024 pixels for the generation branch and 378-980 pixels for the understanding branch.

• Stage 2 (Instruction Tuning) extends training to 120K steps with a reduced learning rate of \(2.5 \times 10^{-5}\). The data mixture evolves to balance text-to-image (45%) and image-to-text (40%) tasks, while introducing interleaved multimodal datasets (10%). The ViT encoder is frozen at this stage to preserve learned visual features. Token capacity increases to 20K per sample.

• Stage 3, Unified Multimodal Training, focuses on interleaved generation capabilities over 70K steps with a learning rate of \(1.0 \times 10^{-5}\). This stage significantly increases interleaved dataset usage to 25% while maintaining balanced generation at 35% and understanding at 37% tasks. The expanded token budget, 27K, and broader image resolution range, 32 to 1024 pixels for generation, support interleaved tasks, including medical image super-resolution, modality translation, and counterfactual generation.

Hardware Requirements and Training Infrastructure. Our model training was conducted using 8× A800 GPUs, 80GB memory each, for experimental validation. However, for optimal training efficiency and to fully exploit the model’s capacity, we recommend a minimum configuration of 16× A800 GPUs or equivalent hardware.

Technical Implementation Details. The training employs a unified loss function that balances understanding and generation objectives with a CE:MSE weight ratio of 0.25:1.0. We apply consistent dropout rates across all stages (Text: 0.3, ViT/VAE: 0.05) to prevent overfitting. The EMA coefficient is set to 0.995 for stable model convergence. Throughout training, the VAE remains frozen to maintain stable latent representations.

6.1.2 Pretrained VAE Diagnostic for Medical Imaging↩︎

Rationale for Using Pretrained VAE without Fine-tuning. Our approach leverages a general-purpose pretrained VAE model from FLUX [68] without medical domain-specific fine-tuning. This design choice addresses two core questions: (1) the reconstruction capability of pretrained VAE on medical imaging modalities, and (2) the cost-benefit trade-off of fine-tuning versus preserving existing capabilities. Regarding the first question, we conducted comprehensive reconstruction experiments across eight medical imaging modalities to evaluate performance. For the second question, considering that our training data is not specifically designed for reconstruction optimization, we did not pursue domain-specific fine-tuning to avoid potential degradation of the model’s general-purpose capabilities while maintaining stable latent representations throughout our progressive training stages.

The empirical evaluation demonstrates that the VAE (FLUX) achieves competitive reconstruction performance across eight distinct medical imaging modalities without requiring domain-specific fine-tuning. With a compression factor of \(f_d = 8\), the model consistently delivers low rFID scores, competitive PSNR values, and robust SSIM scores. To evaluate the necessity of domain-specific adaptation, we performed direct end-to-end fine-tuning of the FLUX VAE on medical imaging data (highlighted in red in Table 12). The empirical results demonstrate that domain-specific fine-tuning yields negligible performance improvements for generation tasks across medical modalities, exhibiting inconsistent variations in rFID scores with marginal changes in corresponding metrics. Consequently, these observations validate the deployment of pretrained VAE models without domain-specific fine-tuning for 2D medical imaging generation applications within our experimental framework.

6.1.3 Reconstruction Fidelity for Medically Important Small Lesions↩︎

While Table 12 demonstrates competitive aggregate reconstruction metrics (rFID, PSNR, SSIM) across diverse medical modalities, these metrics may not fully capture the preservation of small but medically critical structures such as polyps in endoscopy, dermatoscopic features in skin lesions, or fractures in radiographs. To address this concern, we conducted targeted qualitative analysis focusing on the reconstruction fidelity of fine anatomical details and pathological findings that are essential for medical interpretation.

We selected representative cases from three imaging modalities where small lesion detection is medically critical: (1) an endoscopy image containing a polyp, (2) a dermoscopy image with skin lesion with globules, and (3) an X-ray image. For each case, we compared the original image with its VAE-reconstructed counterpart, examining both full-field views and magnified regions of interest (ROIs) centered on the lesions.

Figure 7 illustrates that the pretrained FLUX VAE maintains visually discernible fidelity for small pathological features. In the endoscopy image, the polyp’s morphology and surface texture remain well preserved in the reconstruction, with boundary definition comparable to the original. For the dermoscopy image, the characteristic globular patterns, key diagnostic features for distinguishing benign nevi from melanoma—are clearly visible in both the original and reconstructed versions. In the X-ray image, the highlighted region and surrounding anatomical structures maintain structural coherence post-reconstruction. These qualitative observations suggest that the general-purpose VAE also preserves medically relevant fine-grained details that are useful for downstream tasks within our unified framework.

6.2 Dataset Statistics↩︎

6.2.1 Dataset Composition Details↩︎

6.2.2 Medical Domain and Modality Distribution↩︎

6.3 Data Enhancement Pipeline: CAG Implementation↩︎

This section presents the complete prompt templates used in our Caption Augmented Generation (CAG) pipeline for image generation tasks, as described in Section [sec:method]. The CAG pipeline consists of two main stages: (1) structured medical description generation for quality control, and (2) caption fusion that combines original captions with generated descriptions.

6.3.1 Stage 1: Structured Description Generation↩︎

You are a universally expert medical image analyst, proficient in all
imaging modalities and anatomical systems.
Your input is a single medical image, with no supplementary information.
Your only task is to provide a comprehensive, objective, and structured
description at four distinct levels, from the highest overview down to
the most specific and exceptional findings.
You must not offer any diagnostic, interpretive, or clinical advice.

---

Output Structure (Four-Level, Top-to-Bottom — definitions for your
internal guidance; do NOT reproduce these headings in your answer)

LEVEL 1: IMAGE TYPE & GLOBAL CONTEXT
• In one sentence, state the presumed imaging modality (if visually
  clear), main body region(s), and overall image category (e.g.,
  cross-sectional, projectional, histological).
• Example: "This is an axial CT image of the abdomen and pelvis,
  showing cross-sectional anatomy at the level of the lower kidneys."

LEVEL 2: MACRO-ANATOMICAL OVERVIEW
• In 2–4 concise lines, summarize the global distribution and layout
  of major anatomical regions, dominant structures, and any clearly
  visible large-scale abnormalities, masses, or disease patterns.
• Describe anatomical orientation, symmetry, major organ relationships,
  and other visually prominent features.

LEVEL 3: ORGAN / SUBREGION DETAILS — must be the most detailed section
• In 6–12 lines (use complete sentences), describe the visual
  appearance of individual organs, vessels, bones, or other relevant
  subregions.
• Provide precise, granular, reproducible details so that all main
  features can be reconstructed.
• Maintain strict objectivity; do not include diagnostic language.

LEVEL 4: SPECIAL OR INCIDENTAL FINDINGS
• List any unusual devices, postsurgical changes, image artifacts,
  rare morphologic features, or observations not already mentioned above.
• If none are visible, explicitly state: "No distinct pathological
  or incidental findings are visible."

Writing Instructions
1. Write the entire description as one continuous paragraph that
   implicitly follows the LEVEL 1 → LEVEL 4 order—do not include
   level headings, bullet points, or numbered lists in the paragraph.
2. Do not use bullet points elsewhere (except within the examples).
3. For more complex images, the portion corresponding to LEVEL 3 should
   naturally be longer; for simpler cases, keep it proportionally concise.
4. Avoid any clinical judgement or speculation—describe only what is
   directly visible.

6.3.2 Stage 2: Caption Fusion Enhancement↩︎

This stage fuses original captions with Stage 1 generated structured descriptions to create enhanced descriptions for image generation tasks.

You are a universally expert medical image analyst, proficient in all
imaging modalities and anatomical systems.

CRITICAL CONSTRAINT: You must maintain absolute anatomical consistency.
NEVER change, assume, or modify the anatomical location described in the
original caption. Do not make assumptions about different anatomical locations or
transfer descriptions between different body parts.

Your input consists of:
1. A structured, objective, four-level description derived from a locally
   deployed AI model (following a strict hierarchy from global overview
   to specific findings).
2. An original, data-derived textual description containing high-density,
   potentially diagnostic or interpretative information, which may lack
   structured clarity.

Your task is to:
• First, critically review and confirm the completeness of the structured
  description generated by the local model.
• Then, systematically extract and objectively incorporate relevant,
  visually verifiable details from the original data-derived description,
  enhancing information density without including diagnostic, interpretive,
  or clinical judgement.
• Clearly indicate and explicitly include visually evident anatomical
  abnormalities, structural deviations, or incidental observations present
  in the original data but omitted in the structured description.

Output Structure (Four-Level, Top-to-Bottom)
LEVEL 1: IMAGE TYPE & GLOBAL CONTEXT
• In one sentence, state the presumed imaging modality, main body
  region(s), and overall image category.

LEVEL 2: MACRO-ANATOMICAL OVERVIEW
• In 2--4 concise lines, summarize global anatomical distribution,
  dominant structures, anatomical symmetry or deviations, and clearly
  visible large-scale abnormalities.

LEVEL 3: ORGAN / SUBREGION DETAILS -- must be the most detailed section
• In 6--12 complete sentences, describe individual organs, bones,
  vessels, and other relevant anatomical subregions in precise,
  reproducible detail.
• Objectively highlight visually confirmed abnormalities or structural
  deviations derived from the original data description.

LEVEL 4: SPECIAL OR INCIDENTAL FINDINGS
• Explicitly mention unusual devices, postsurgical changes, rare
  morphological features, or visually detectable anomalies present in
  the original description yet absent in the structured description.
• Clearly state the absence of commonly expected baseline anatomical
  or pathological features if definitively not observed in the image.

Writing Instructions
1. Write the final enhanced description as a single, continuous paragraph
   implicitly following LEVEL 1 → LEVEL 4 order--do not include explicit
   level headings, bullet points, or numbered lists.
2. Avoid any clinical judgement, diagnostic language, or speculative
   interpretation--include only details directly verifiable from visual
   inspection.
3. Start your output with "Please generate a realistic [modality] image
   showing" to make it a proper generation instruction.

6.3.3 Stage 3: Thinking-Enhanced Response Generation↩︎

This stage aims to elicit the reasoning process from the medical foundation model (MedGemma-27B-IT) by prompting it to explicitly generate its internal thinking steps. We leverage this specialized medical model to simulate detailed reasoning processes through the structured prompt format. The resulting data, which includes both the explicit thinking traces and the final responses, is then used to train our model.

System: You are a medical image generator. You create [modality] images based
  on clinical descriptions. Your responses should describe what features you
  have generated in the image from the creator's perspective. Use bullet points
  to organize the anatomical structures and clinical features you have included
  in your generated image.
  
  User: Based on this clinical description: "[clinical_description]"
  
  You have been given the corresponding medical image. Please provide a response
  following this format:
  
  Required format:
  <think>Analyzing the clinical description, I need to generate an image that
  captures: 1) The key pathological process described, 2) The anatomical
  structures involved, 3) The specific imaging characteristics for [modality].
  Based on the clinical presentation, I should include [key features reasoning].
  [structured_caption if available]</think>
  
  Here/This is the generated [modality] image that displays:
  • [anatomical structure or clinical finding 1]
  • [anatomical structure or clinical finding 2]
  • [anatomical structure or clinical finding 3]
  
  IMPORTANT:
  1. In the <think> tag, reason through WHAT you need to generate and WHY based
     on medical knowledge
  2. Respond from the GENERATOR perspective - describe what features you have
     CREATED/GENERATED in the image
  3. Use the exact format above with bullet points (•) to list features
  4. Start with 'Here is the generated [modality] image that displays:'
  5. Each bullet point should describe a specific anatomical structure,
     clinical finding, or visual feature that you have included
  6. Do NOT use observational language like 'shows', 'visible', 'can be seen'
     - instead use generative language like 'displays', 'includes',
     'features', 'contains'
  
  Note: The thinking tag should reflect your decision-making process: "I need
  to generate X because Y", "The clinical description indicates I should
  include Z", etc.

The enhanced captions from Stage 2 (if the process "generating" is not generated successfully) and Stage 3 (if the process "thinking" is generated successfully) are sampled and then submitted to the Expert Review system (Section 6.4) for final validation.

6.4 Expert Review Validation System↩︎

This section presents an expert review validation system that evaluates the quality of our UniMedVL-5M dataset construction and two caption generation approaches described in the Data Enhancement Pipeline (Section 6.3):

Simple approach: Caption fusion that combines structured descriptions from Stage 1 with original captions (Stage 2 of CAG pipeline).

Thinking-enhanced approach: Incorporates an additional planning process with <think> tags that integrates reasoning steps before medical image generation (Stage 3 of CAG pipeline). The validation system evaluates both data quality and methodological effectiveness.

6.4.1 Expert Review Framework Overview↩︎

Our expert review validation system is designed around a seven-dimensional medical evaluation framework that assesses medical AI performance.

Our evaluation framework encompasses seven dimensions that assess the synthetic quality of medical image captions. All dimensions are scored on a 0 to 5 scale, except Modality Match which uses a binary 0 to 1 scale. The framework begins with Modality Match, which measures consistency between images and declared medical imaging modalities, followed by Factual Accuracy that evaluates the precision of anatomical structure and pathological finding descriptions. Information Completeness assesses coverage of diagnostically relevant key information, while Position/Quantity Accuracy measures precision in anatomical localization and quantitative assessments. The framework also incorporates Professionalism to evaluate adherence to medical reporting standards, Planning Coherence to assess systematic thinking and logical organization quality, and finally Clinical Reasoning, a Turing-test style judgment, to measure approximation to human expert-level performance.

Expert Validation Protocol: Experts conducted audits of 200 samples across all seven dimensions, with mean quality score \(\bar{Q}\ge 0.85\) consistently achieved across all dimensions.

6.4.2 Evaluation Dimension Analysis↩︎

Figure 9 presents the correlation analysis and comparative results. Figure 9 (a) shows inter-dimensional correlations, while Figure 9 (b) compares the two generation approaches.

6.4.3 Dataset Quality Comparison Analysis↩︎

Figure 10 compares the two generation approaches across all evaluation dimensions. The radar chart (Figure 10 (a)) shows closely aligned performance profiles.

6.4.4 Medical Modality-Specific Analysis↩︎

Figure 11 presents modality-specific performance across eight medical imaging modalities. Figure 11 (a) shows statistical comparisons, and Figure 11 (b) displays detailed performance metrics.

Across the evaluation figures above, the simple approach yields marginally higher scores on caption-fidelity dimensions. The simple approach stays close to the original human-validated captions with minimal extrapolation, whereas the thinking-enhanced approach introduces explicit reasoning traces that target downstream planning and counterfactual generation rather than caption fidelity.

6.5 Downstream Task Results↩︎

6.5.1 Medical Report Generation↩︎

6.5.2 Multi-Round Medical VQAs↩︎

6.5.3 Medical Image Generation↩︎

6.5.4 Interleaved Tasks↩︎

6.6 Additional Experimental Results on Downstream Tasks↩︎

6.6.1 Blinded Expert Study on Generation Realism↩︎

We conducted a blinded expert study to assess the realism of generated images. We randomly selected 100 images (50 real, 50 generated) and asked two experts to distinguish real from generated samples. The true positive rate for generated images was 64%, the true negative rate for real images was 80%, and the balanced accuracy was 72%, providing evidence that UniMedVL generates visually realistic medical images beyond what automatic metrics capture.

6.6.2 Fine-Tuning Performance on Medical VQA↩︎

While the main paper emphasizes unified zero-shot capability, UniMedVL can also adapt effectively to downstream medical VQA settings through task-specific fine-tuning.

Fine-tuning improves performance by +5.06 on VQA-RAD, +16.21 on SLAKE, and +9.11 on PathVQA over zero-shot UniMedVL.

Aligned protocol comparison. To enable direct comparison with prior work using recall for open-set and accuracy for closed-set questions, we additionally evaluated under this protocol:

3pt

Under the aligned protocol, UniMedVL-finetune achieves an overall score of 73.32, comparable to LLaVA-Med at 73.37 and ExGra-Med at 74.75. UniMedVL-finetune shows particular strength on SLAKE with a score of 92.58, outperforming both baselines by a notable margin, while remaining competitive on VQA-RAD and PathVQA. The original gap was largely attributable to protocol differences.

6.6.3 Segmentation Task Comparison↩︎

We compare UniMedVL on the segmentation task with existing text-guided segmentation models, including SAM3 [83] in the general domain and Medical SAM3 in the medical field.

4pt

Compared to Medical SAM3, UniMedVL still exhibits a performance gap overall, 33.51 vs., while it outperforms general-domain SAM3 at 17.96. This is expected, as Medical SAM3 is a specialized segmentation model, whereas UniMedVL is a unified model that simultaneously supports understanding, generation, and segmentation without task-specific checkpoints.

6.6.4 CXR Lung Opacity Image Translation↩︎

6.6.5 Medical Report Generation↩︎

3pt

6.7 Efficiency Analysis↩︎

We measure inference efficiency with single expert activation for a controlled comparison. Comparison with BLIP3-o, Janus, and HealthGPT is provided in Table  [tab:throughput95gen].

Image generation (Table [tab:throughput95gen](a)). UniMedVL (14B) requires 40.03 TFLOPs per image and 28.39 GB peak memory, whereas BLIP3-o 8B requires 142.59 TFLOPs per image and 25.55 GB. This corresponds to approximately 3.6x lower compute per image for UniMedVL, while peak memory rises only about 11%, from 25.55 to 28.39 GB, even though UniMedVL uses a dual-encoder and a transformer backbone with distinct FFN layers for understanding and generation tasks and shared self-attention layers, and has almost twice the parameters. UniMedVL’s compute cost is also close to that of Janus 7.42B (35.56 TFLOPs per image) while providing a larger unified model.

VQA on GMAI-MMBench (Table [tab:throughput95vqa](b)). UniMedVL achieves 25.86 tokens/s with 2.256 TFLOPs per sample and 28.25 GB peak memory, compared with BLIP3-o’s 30.40 tokens/s, 9.307 TFLOPs per sample, and 18.21 GB. Thus, UniMedVL attains an approximately 4.1x reduction in FLOPs per sample with comparable throughput (about 85% of BLIP3-o’s tokens/s), at the cost of higher peak memory due to the dual-encoder design. Compared to another 14B unified model, HealthGPT-L14 (12.57 tokens/s, 3.009 TFLOPs, 29.22 GB), UniMedVL is roughly twice as fast in throughput and more compute-efficient.

(a) Image Generation Throughput

(b) VQA Understanding Throughput (GMAI-MMBench validation set)

6.8 Failure Cases and Analysis↩︎

We present representative failure cases of UniMedVL organized by task type: medical image generation, medical image editing, and medical image understanding, with illustrative examples shown in Figures 23 and 24.

6.8.1 Medical Image Generation↩︎

6.8.1.1 Global appearance and background artefacts.

Although UniMedVL generally produces realistic images across modalities, a characteristic failure mode in text-to-image generation concerns embedded text and annotations (Figure 23). In some synthesised samples, the model hallucinates spurious on-image text or renders partially legible words, labels, or font styles that do not appear in the corresponding real medical images or do not match typical acquisition overlays. These artefacts do not alter the main anatomical content but introduce visually unnatural patterns. As shown in Figure 23, the generated chest X-ray exhibits spurious text overlays (red boxes highlight the artefacts), the ultrasound image contains hallucinated text labels that deviate from standard medical annotations, and the CT scan shows partially corrupted metadata text at the bottom that does not match typical DICOM overlay formatting.

6.8.2 Medical Image Editing↩︎

6.8.2.1 Structure preservation in generation and editing.

For interleaved editing-style tasks (e.g., virtual staining, super-resolution, cross-modal synthesis, counterfactual generation), UniMedVL does not always perfectly preserve all spatial structures outside the region being semantically edited (Figure 24). For instance, in some counterfactual CXR generations, small devices or lines (e.g., catheters) can become slightly blurred or shifted, even when the main pathological change is correctly applied. Figure 24 illustrates these challenges across three representative cases: in the brain MRI cross-modal synthesis tasks (top two rows), the generated images show subtle structural discrepancies in the cerebellar and temporal regions compared to ground truth, and in the chest X-ray counterfactual generation (bottom row), while the model successfully modifies the target pathological region, minor shifts and blurring are occasionally in the cardiac silhouette boundaries.

6.8.3 Medical Image Understanding↩︎

6.8.3.1 Medical demanding quantitative and fine-grained recognition scenarios.

In medical demanding tasks that require precise quantitative reasoning or fine-grained visual recognition, the unified model shows clear limitations across multiple task categories. Table 21 presents UniMedVL’s performance on GMAI-MMBench validation set, revealing particularly low accuracy on surgical video recognition tasks: Surgeon Action Recognition, Surgical Instrument Recognition, and Surgical Workflow Recognition.

6.9 Information-Theoretic Advantages of Unified Understanding and Generation↩︎

6.9.0.1 Setup.

Let \((X,T)\sim p_{\mathrm{data}}\) denote paired training data, where \(X\) is an image and \(T=(T_{\mathrm{in}},T_{\mathrm{out}})\) is the text sequence split into an input prompt \(T_{\mathrm{in}}\) and a target continuation \(T_{\mathrm{out}}\). We form two deterministic latent views of \(X\): \(z_{\mathrm{VIT}}=E_{\mathrm{VIT}}(X)\) (for understanding) and \(z_{\mathrm{VAE}}=E_{\mathrm{VAE}}(X)\) (for generation). The unified condition is \(C:=(T_{\mathrm{in}}, z_{\mathrm{VIT}})\) with embedding \(c:=g(C)\). All entropies and mutual informations below are computed under the joint distribution induced by \(p_{\mathrm{data}}\) and these encoders.

6.9.0.2 Training objective (context).

We optimize understanding tasks via next-token prediction (NTP) and generation tasks via rectified flow matching: \[\label{eq:supp95obj} \begin{align} \mathcal{L}(\theta) &=\mathcal{L}_{\mathrm{NTP}}+\mathcal{L}_{\mathrm{flow}},\\ \mathcal{L}_{\mathrm{NTP}} &=-\sum_i \log p_\theta\!\left(t_{i+1}\mid t_{\le i}, z_{\mathrm{VIT}}\right),\\ \mathcal{L}_{\mathrm{flow}} &=\mathbb{E}_{t,\varepsilon}\!\left[\left\|v_\theta(\tilde{z}_{\mathrm{VAE}},t,c)-v\right\|_2^2\right], \end{align}\tag{6}\] where \(\tilde{z}_{\mathrm{VAE}}\) and \(v\) follow the standard rectified-flow construction.

6.9.0.3 Information-theoretic guarantees.

The following statements are distributional and hold independent of the particular optimizer.

Proposition 1 quantifies the information-theoretic gain of unified modeling. While factorized modeling implicitly assumes \(p(T, z \mid C) = p(T \mid C)\,p(z \mid C)\), the joint objective captures the cross-modal mutual information \(I(T_{\mathrm{out}}; z_{\mathrm{VAE}} \mid C)\). Unified joint modeling is therefore never worse than separate task-specific models, and yields strict improvements whenever the tasks share mutual information—i.e., whenever understanding informs generation or vice versa.

References↩︎