0.0.1 Introduction↩︎

Recent emergence of unified multi-modal models (UMMs) [1], [2] that seamlessly bridge understanding and generation across modalities within a single framework has opened a new frontier in multi-modal reasoning: interleaved Text-Image-Text Chain-of-Though (TIT-CoT), where verbal thinking alternates with intermediate image generation to externalize perceptual steps beyond the reach of language [3]–[5]. The underlying premise, often termed the visual superiority hypothesis [6], is that for physically grounded tasks, visual generation affords a more faithful world model than verbal reasoning, whose expressiveness is inherently bounded by the symbolic nature of language. So far, this premise has been pursued exclusively through supervised fine-tuning (SFT) on curated TIT-CoT traces [7], [8], an approach bottlenecked by the cost and scarcity of high-quality interleaved data. Reinforcement learning (RL) offers a principled alternative: by learning from task rewards rather than curated data, it enables models to autonomously discover novel reasoning strategies that go beyond what static data can teach, as already shown in language [9], [10] and multi-modal reasoning [11], [12].

However, existing formulations leave TIT-CoT half-optimized: policy gradients flow through text tokens alone, while intermediate image generation remains outside the training loop, undermining the very advantage on which interleaved reasoning is premised. COOPER [13] optimizes visual generation, but through a separate supervised stage. While DeepEyes [14] is trained end-to-end with RL, its policy gradients reach only the visual tool calling rather than the image generation process, still leaving visual reasoning outside the optimization loop. Meantime, RL for image generation is well established: DDPO [15] and DPOK [16] cast diffusion denoising as an MDP, and subsequent work extends this to flow matching in FlowGRPO [17] and DiffusionNFT [18]. These formulations provide a ready-made RL toolkit for the visual branch of a unified model, yet none has been integrated into an interleaved reasoning setting. The concurrent work, UniGRPO [19], applies RL to both modalities, but limited to single-turn text-to-image generation. These observations point to a critical missing piece: a unified RL framework that jointly optimizes textual and visual reasoning within an interleaved TIT-CoT trajectory, enabling end-to-end credit assignment across modality boundaries and exploration over the full combinatorial space of interleaved strategies.

To close this gap, we propose BRAID (Bridging inteRleAved multI-modal reasoning as a unified Decision process), which casts multi-turn text-image-text reasoning as a unified MDP. This formulation allows both textual and visual generation turns to be optimized jointly under one RL objective, without requiring separate training stages or modality-specific reward designs. Built on BAGEL [2], BRAID formulates every turn in a \(\{\texttt{text}\!\to\!\texttt{image}\!\to\!\texttt{text}\!\to\!\cdots\!\to\!\texttt{answer}\}\) trajectory as a macro-action. However, unifying the two modalities under a single objective is non-trivial: the text branch yields per-token autoregressive likelihoods while the image branch traverses continuous denoising paths via flow matching, and naively combining them collapses the optimization due to incompatible scales. BRAID addresses the modality boundary by deriving a shared trajectory-level advantage and channeling it into modality-native policy gradients: autoregressive token generation for text and a likelihood-free flow-matching objective for image. Further, BRAID introduces a vision-thinking process reward to address credit assignment over the long-horizon TIT-CoT rollout: a VLM judge scores each intermediate image on its utility to the reasoning trajectory, providing turn-level feedback to sharpen learning at critical visual branches. We hope BRAID serves as a step toward simple, general-purpose RL frameworks for UMMs, and inspires further investigation into principled, modality-agnostic policy optimization paradigms.

Experiments show that BRAID improves BAGEL by +5.73 avg.across seven benchmarks, surpassing GPT-4o with only 7B parameters, with pronounced gains on SAT (+14.00) and V*Bench (+10.76). Notably, Maj@\(n\) shows BRAID’s solution space scales efficiently with sampling budget while baselines plateau, and case studies reveal that the optimized image branch produces faithful intermediates and also transfers strong instruction-following capability to text-to-image generation.

0.0.2 Related Work↩︎

Interleaved Multi-Modal Reasoning. Chain-of-Thought prompting [20], [21] decomposes complex reasoning into intermediate steps, yet pure text limits expressiveness for visually grounded tasks where spatial/geometric relations are more naturally captured by images [22], [23]. Recent unified architectures address this gap by integrating generation and understanding within a single backbone, enabling interleaved text-image reasoning traces with intermediate visual states [24], [25]. These architectures follow two designs: fully autoregressive (AR) models that treat both modalities as discrete tokens under a unified next-token objective [1], [26], and hybrid AR-diffusion models that employ selective activation of modality-specific parameters (AR for text, diffusion for images) [2], [27]. Both lines demonstrate that cross-modal reasoning can be made intrinsic to a single model through dedicated objectives and curated interleaved data [7], [28], [29]. BRAID builds on the hybrid backbone for its superiority in delegating each modality to its best-suited generative mechanism [30]–[32].

RL Fine-tuning. Verifiable-reward RL has emerged as a principled paradigm for enhancing reasoning beyond the ceiling of supervised data, first in LLMs [9], [10] and subsequently in VLMs that are still limited to textual action space [11], [33]. A parallel line adapts RL to image generation by casting diffusion denoising as an MDP [15], [16], [34], with subsequent work generalizing to flow-matching models in FlowGRPO [17], accelerating learning through direct forward-process optimization in DiffusionNFT [18], and unifying these variants under a single policy-gradient framework in DanceGRPO [35]. However, these two lines of work operate within single-modality generation, and neither accommodates the setting where a unified model emits interleaved text-image outputs. BRAID aims to bridge this divide by formulating RL over the joint multi-modal reasoning trajectory.

RL for Interleaved Multi-Modal Reasoning. A growing body of work applies RL to fine-tune unified models. For AR backbones, Emu3.5 [5] scales post-training with verifiable-reward RL, and other studies adopt the same paradigm with a hybrid reward design [36], [37]. For hybrid architectures, prior work typically excludes the visual branch from the RL loop: COOPER [13] trains image generation in a separate supervised stage, and DeepEyes [14] backpropagates policy gradients to the visual tool calling rather than the image generation process itself. Further, the concurrent work UniGRPO [19] does jointly optimize both modalities under a single RL objective, but is limited to single-turn text-to-image generation. In summary, no existing pipeline applies RL to the multi-turn interleaved reasoning setting where each text/image turn conditions subsequent generation. BRAID fills this gap by casting the full interleaved trajectory as a unified decision process and deriving modality-native gradients end-to-end within a single policy.

0.0.3 Preliminaries↩︎

0.0.3.0.0.1 Group Relative Policy Optimization.

GRPO [10] samples, for each prompt \(q\), a group of \(G\) rollouts \(\{\tau_i\}_{i=1}^{G}\) from the behavior policy \(\pi_{\theta_{\mathrm{old}}}\), scores each \(\tau_i\) with a scalar reward \(r(\tau_i)\), and assigns a group-normalized advantage as \(\hat{A}(\tau_i) = \frac{r(\tau_i)-\mu(r(\tau))}{\sigma(r(\tau)) + \kappa}\), where \(\mu(\cdot)\) and \(\sigma(\cdot)\) denote the mean and standard derivation, and \(\kappa\) is a small constant for numerical stability. The policy \(\pi_\theta\) is then updated by a clipped surrogate objective with a KL penalty against a frozen reference model \(\pi_{\mathrm{ref}}\) as \[\label{eq:grpo95obj} \mathcal{J}_{\text{GRPO}}(\pi_\theta) = \mathbb{E}\!\Biggl[ \frac{1}{G}\sum_{i=1}^{G} \frac{1}{|\tau_i|}\sum_{t=1}^{|\tau_i|} \min\bigl( \rho_{i,t}\,\hat{A}_i,\; \mathrm{clip}(\rho_{i,t},1\!-\!\varepsilon,1\!+\!\varepsilon) \,\hat{A}_i \bigr) \Biggr] - \eta D_{\mathrm{KL}}\!\bigl[\pi_\theta\|\pi_{\mathrm{ref}}\bigr],\tag{1}\] where \(|\tau_i|\) is the length of rollout \(\tau_i\), \(\rho_{i,t}=\pi_\theta(\tau_{i,t}|q,\tau_{i,<t})/ \pi_{\theta_{\mathrm{old}}}(\tau_{i,t}|q,\tau_{i,<t})\) is the per-token importance ratio, \(\varepsilon\) is the clipping threshold, and \(\eta\) controls the KL regularization strength.

0.0.3.0.0.2 RL for Image generation.

Image steps are modeled by flow matching [38]. Given context \(c\), the policy parameterizes a conditional velocity field \(v_\theta(x_t, c, t)\) that transports a noisy sample \(\epsilon\sim\mathcal{N}(0,I)\) to a clean image \(x_0\) via the probability-flow ODE along the continuous time axis \(t\in [0,1]\). On interpolated samples \(x_t=(1-t)x_0 + t\epsilon\), flow matching regresses \(v_\theta\) onto the ground-truth velocity: \[\label{eq:fm} \mathcal{L}_{\text{FM}}(\theta) = \mathbb{E}_{t,x_0,x_1,c} \bigl\lVert v_\theta(x_t, c, t)-(\epsilon-x_0)\bigr\rVert^{2}.\tag{2}\]

Applying RL to flow-matching generators requires injecting reward signals into this objective. Rather than estimating the intractable likelihood \(\log p_\theta(x_0|c)\), DiffusionNFT [18] turns Eq. 2 into a reward-aware regression by splitting each sample into a positive and a negative branch weighted by a soft label \(r(x_0, c)\in[0,1]\), i.e., the reward representing the optimality probability: \[\label{eq:nft} \mathcal{L}_{\text{NFT}}(\theta) = \mathbb{E}\Bigl[\, r(x_0, c)\,\bigl\lVert v^{+}_\theta(x_t, c, t)-(\epsilon-x_0)\bigr\rVert^{2} +(1-r(x_0, c))\,\bigl\lVert v^{-}_\theta(x_t, c, t)-(\epsilon-x_0)\bigr\rVert^{2} \Bigr],\tag{3}\] with implicit predictors \(v^{\pm}_\theta:=(1\mp\beta)\,v_{\theta_{\mathrm{old}}}\pm\beta\,v_\theta\) that linearly mix the current policy \(v_\theta\) with a frozen behavior velocity \(v_{\theta_{\mathrm{old}}}\), where \(\beta\) is a hyperparameter. High-reward samples pull \(v_\theta\) toward the data target, while low-reward ones push it away, injecting the RL signal directly into the forward training loss without any likelihood or score estimation.

0.0.4 Method↩︎

Here, we first cast TIT-CoT reasoning as a two-level MDP that unifies text and image generation in (§0.0.4.2). Then, we introduce the unified optimization mechanism that channels the shared trajectory-level advantage into modality-native policy gradients in (§0.0.4.3). Finally, we present the vision-thinking process reward design that enables efficient credit assignment for image turns in (§0.0.4.4).

0.0.4.1 Problem Formulation

We formulate a UMM policy \(\pi_\theta\) that unifies text and image generation within a single model [2]. Given a prompt \(c=(q,\{\mathcal{I}^n_q\}_{i=1}^{M})\) pairing a textual query \(q\) with \(M\!\ge\!1\) images, the model produces an interleaved reasoning trajectory \(\tau\sim\pi_\theta(\cdot\,|\,c)\) as \[\label{eq:traj} \tau = \bigl(T_1,\, I_1,\, T_2,\, I_2,\, \ldots,\, T_K,\, I_K,\, T_{\mathrm{ans}}\bigr),\tag{4}\] which alternates between text chunks \(T_k\) and intermediate images \(I_k\) across multiple turns, and ends with a textual answer \(T_{\mathrm{ans}}\). Both modalities are generated from \(\pi_\theta\) using a shared multi-modal attention backbone [7]. We denote this structure as Text-Image-Text Chain-of-Thought (TIT-CoT).

0.0.4.2 Interleaved Reasoning as a Unified MDP

To optimize text and image turns under a joint RL framework, we formulate the generation of a TIT-CoT trajectory as a two-level MDP as \(\mathcal{M}=(\mathcal{M}_{\mathrm{turn}},\mathcal{M}_{\mathrm{step}})\). The outer level \(\mathcal{M}_{\mathrm{turn}}\) abstracts each reasoning turn as a macro-action that emits either a text chunk \(T_k\) or an intermediate image \(I_k\). The inner level \(\mathcal{M}_{\mathrm{step}}\) unrolls each macro-action into a sequence of micro-actions, i.e., tokens for text generation and denoising paths for image generation, where the turn-level advantage can be consistently propagated to every micro-action within that turn.

Turn-level MDP. The interleaved reasoning over \(2K{+}1\) turns is mapped to the following MDP: \[\begin{align} {3} &s_k \triangleq (c,\, \tau_{<k}) \qquad &&\pi(a_k \mid s_k) \triangleq \pi_\theta(a_k \mid c,\, \tau_{<k}) \qquad &&P(s_{k+1} \mid s_k, a_k) \triangleq \delta_{\,s_k \oplus a_k} \\[6pt] &a_k \triangleq \begin{cases} T_k\in \mathcal{A}_{\text{txt}} \\[2pt] I_k\in \mathcal{A}_{\text{img}} \end{cases} \qquad &&\rho(s_0) \triangleq \delta_{(c,\,\varnothing)} \qquad &&R(s_k, a_k) \triangleq \begin{cases} r(\tau) & \text{if } k \!=\! 2K\!+\!1 \\[2pt] 0 & \text{otherwise} \end{cases} \end{align}\] The state \(s_k\) concatenates the prompt \(c\) with all preceding turns \(\tau_{<k}\), and \(\rho(\cdot)\) is the initial state distribution. The action \(a_k\) is either a text chunk \(T_k\) or an image \(I_k\), sampled from the policy \(\pi_\theta(a_k|s_k)\) whose text and image experts share a unified backbone. The transition \(P\) is the deterministic append \(s_{k+1}\!=\!s_k\!\oplus\!a_k\), the reward \(R\) is a sparse rule-based signal assigned only at the final answer \(T_{\mathrm{ans}}\). By collapsing each text chunk or intermediate image into a single macro-action, this abstraction renders the two modalities homogeneous at the outer level: text and image turns are seamlessly integrated into a single MDP that shares a unified state space, transition dynamics, and reward signal, thereby enabling a coherent RL formulation over the entire interleaved reasoning trajectory.

Step-level MDP. Each turn-level action \(a_k\) is realized by rolling out a lower-level MDP \(\mathcal{M}_{\mathrm{step}}\) that operates at a fine-grained scale. The specific form depends on the modality:

• For a text turn \(a_k\!=\!T_k\), \(\mathcal{M}_{\mathrm{step}}\) is the standard autoregressive token generation: at step \(i\), the micro-state is the running prefix \(s_k^i:=\left(s_k, a_k^{<i}\right)\), the micro-action is the next token \(a_k^i:=T_k^i\), and \(\pi_\theta\) is factorized as \(\pi_\theta(T_k|s_k)=\prod_{i=1}^{|T_k|}\pi_\theta\left(T_k^i\mid s_k,T_k^{<i}\right)\), where \(|T_k|\) is the text length.

• For an image turn \(a_k\!=\!I_k\), \(\mathcal{M}_{\mathrm{step}}\) is the iterative denoising procedure via flow matching: at denoising timestep \(t\!\in\! [0,1]\), the micro-state is the noisy image \(s_k^t:=x_t\), the micro-action is the predicted flow velocity \(a_k^t:=v_\theta(x_t,s_k,t)\), and the state transition follows the probability-flow ODE as \(x_{t_{n-1}}\!=\! x_{t_n}+v_\theta(x_{t_n},s_k,t_n)\cdot\mathrm{d}t\), with the clean image \(I_k=x_0\) as the terminal state, where \(n=N, N\!-\!1,...,1\) is the denoising step along the time axis.

Let \(\mathcal{K}_{\mathrm{txt}}\) and \(\mathcal{K}_{\mathrm{img}}\) index text and image turns, respectively. Because all turns reside in a single MDP, the joint policy over a complete trajectory factorizes naturally along the turn sequence: \[\label{eq:traj-uni} \log \pi_\theta(\tau \mid c) \;=\; \sum\nolimits_{k=1}^{2K+1} \log \pi_\theta(a_k \mid s_k)\,.\tag{5}\] Expanding each macro-action into its step-level form then reveals two modality-native branches: \[\log\pi_\theta(\tau\mid c) \;=\; \underbrace{\sum_{k\in\mathcal{K}_{\mathrm{txt}}}\sum_{i=1}^{|T_k|} \log\pi_\theta\bigl(T_k^i\mid s_k,T_k^{<i}\bigr)}_{\text{text branch}} \;+\; \underbrace{\sum_{k\in\mathcal{K}_{\mathrm{img}}}\sum_{n=1}^{N} \log\pi_\theta\bigl(x^k_{t_{n-1}}\mid x^k_{t_n},\,s_k\bigr)}_{\text{image branch (flow-matching)}}. \label{tjbzgmnr}\tag{6}\] Although the two branches differ in computational mechanism (autoregressive token generation for text and flow-matching denoising for image generation), their structural parallelism ensures that a shared trajectory-level advantage can be consistently back-propagated through both modality-native policy gradients, a property we exploit in the unified RL objective in §0.0.4.3.

0.0.4.3 Unified Policy Optimization

We train the unified policy \(\pi_\theta = (\pi_\theta^{\text{txt}},\, \pi_\theta^{\text{img}})\) by jointly optimizing both branches on grouped rollouts, leveraging the structural parallelism established in Eq. [eq:traj-decomp]: a shared turn-level advantage \(\hat{A}(s_k, a_k)\) is back-propagated through each modality-native surrogate, a GRPO-style clipped objective for the text branch and a DiffusionNFT loss [18] for the image branch, yielding the unified objective: \[\mathcal{J}(\theta) \!=\! \sum_{k\in\mathcal{K}_{\mathrm{txt}}}\!\!\mathcal{J}_{\mathrm{GRPO}}\bigl(\pi^{\mathrm{txt}}_\theta(T_k|s_k);\hat{A}(s_k,T_k)\bigr) \;-\;\lambda\cdot\!\! \sum_{k\in\mathcal{K}_{\mathrm{img}}}\!\mathcal{L}_{\mathrm{NFT}}\bigl(\pi^{\mathrm{img}}_\theta(I_k|s_k);\hat{A}(s_k,I_k)\bigr), \label{eq:joint}\tag{7}\] where \(\lambda\) balances the two modalities. Eqs. 5 7 together realize BRAID’s core principle: despite distinct generative mechanisms, text and image branches are optimized within a unified RL loop, sharing one interleaved trajectory, one turn-level advantage, and one joint backward pass.

To inject \(\hat{A}\) into flow matching without likelihood estimation, we map it to a soft reward label \(r_k=\sigma(\hat{A}(s_k,I_k)/\upsilon)\in[0,1]\) with temperature \(\upsilon\), and plug it into the DiffusionNFT loss of Eq. (3 ): \[\mathcal{L}_{\mathrm{NFT}}(\theta) = \mathbb{E}\Bigl[ r_k\,\bigl\lVert v^{+}_\theta\left(x_t,s_k,t\right)-(\epsilon-x_0)\bigr\rVert^{2} +(1-r_k)\,\bigl\lVert v^{-}_\theta\left(x_t,s_k,t\right)-(\epsilon-x_0)\bigr\rVert^{2} \Bigr], \label{ihczwsou}\tag{8}\] where \(x_0=I_k\) is the clean image, \(\epsilon\) is Gaussian noise, \(x_t=(1-t)x_0+t\epsilon\) is the interpolated sample, and \(v^{\pm}_\theta=(1\!\mp\!\beta)v_{\theta_{\mathrm{old}}} \pm\beta v_\theta\) are implicit predictors as described in §0.0.3. Intuitively, a high \(\hat{A}_k\) steers the velocity field \(v_\theta\) toward the generated image \(I_k\), while a low \(\hat{A}_k\) steers it away, thereby propagating the terminal reward signal to image generation directly via the flow-matching loss.

0.0.4.4 Vision-Thinking Process Reward

The trajectory-level advantage \(\hat{A}(\tau)\) broadcasts a single terminal-reward scalar uniformly to every turn. As the TIT-CoT horizon grows, this undifferentiated signal increasingly obscures which intermediate steps truly matter for the final answer. The issue is amplified for image turns: unlike autoregressive text, image generation must produce coherent, pixel-level content in each non-autoregressive pass, demanding a far sharper learning signal than a sparse, delayed reward can provide.

To address this, we introduce a vision-thinking process reward that scores each intermediate image on its utility to the reasoning trajectory, providing dense credit assignment for image turns. Specifically, we employ a VLM judge ¹ to score each intermediate image \(I_k\) according to a set of complementary criteria \(\mathcal{C}\), yielding a per-turn vector of process rewards: \(\boldsymbol{r}^{\mathrm{vis}}=\bigl[r^{(c)}\bigr]_{c\in\mathcal{C}}\in \mathbb{R}^{|\mathcal{C}|}\). In practice, we instantiate \(\mathcal{C}=\{\texttt{vc},\texttt{vf},\texttt{ru}, \texttt{tr}\}\), covering four salient aspects of a visual thought:

• Visual Correctness \(r^{(\mathrm{vc})}\): whether the intended visual operation is executed accurately;

• Visual Fidelity \(r^{(\mathrm{vf})}\): whether the image is coherent and free of hallucinated content;

• Reasoning Utility \(r^{(\mathrm{ru})}\): whether the image provides useful evidence to the final answer;

• Trustworthiness \(r^{(\mathrm{tr})}\): whether the image is unlikely to mislead a reasonable reasoner.

Decoupled Advantage Estimation. Since the visual process reward is intrinsically multi-objective, its components typically operate on disparate and dynamically shifting scales during training. This complicates the development of a flexible weighting scheme for aggregating the process reward components. To circumvent this issue, we pool all image turns within the group of rollouts into a visual set \(\mathcal{I}_{\mathcal{G}}\), compute the advantages for each reward component independently, and simply merge them with equal weight at the advantage level: \[\hat{A}^{\mathrm{vis}}(s_k,I_k) \;=\; \sum\nolimits_{c\in\mathcal{C}} \frac{r^{(c)}_k-\mu_{\mathcal{I}_{\mathcal{G}}}(r^{(c)})}{\sigma_{\mathcal{I}_{\mathcal{G}}}(r^{(c)})}, \qquad I_k\in\mathcal{I}_{\mathcal{G}}, \label{eq:adv-vis}\tag{9}\] where \(\mu_{\mathcal{I}_{\mathcal{G}}}(\cdot)\) and \(\sigma_{\mathcal{I}_{\mathcal{G}}}(\cdot)\) denote the mean and standard derivation calculated from \(\mathcal{I}_{\mathcal{G}}\). Analogously, we incorporate the terminal reward \(r(\tau)\) with the visual process reward in a decoupled way as \[\hat{A}(s_k,a_k) \;=\; \begin{cases} \hat{A}^{\mathrm{ans}}(s_k, T_k), & \text{for text turns},\\[2pt] \hat{A}^{\mathrm{ans}}(s_k, I_k) + \lambda_{\text{vis}}\cdot\hat{A}^{\mathrm{vis}}(s_k,I_k), & \text{for image turns}, \end{cases} \label{eq:adv-combined}\tag{10}\] where \(\hat{A}^{\mathrm{ans}}(\cdot)\) is the advantage calculated from the terminal reward \(r(\tau)\), and \(\lambda_{\text{vis}}\) balances two advantages. By decoupling advantage estimation, each reward signal is normalized relative to its own baseline before aggregation. This architecture mitigates sensitivity to scale mismatch and temporal fluctuations, facilitating more robust and interpretable control over the process reward integration.

0.0.5 Experiments↩︎

Dataset and Evaluation. Following the literature [7], our training corpus spans a wide range of eight task families: Matterport3D [39], RealSee3D [40], Perspective, DirectionalQuery, \(360{+}x\) [41], PositionedDirection, Jigsaw [42], and VisualSearch [43]. In each task, the dataset is partitioned into SFT and RL pools under three ratios: \(\mathcal{D}^{2{:}1}\), \(\mathcal{D}^{1{:}1}\), and \(\mathcal{D}^{\cap}\). We default to the partial-overlap regime \(\mathcal{D}^{\cap}\), which bridges supervised initialization and on-policy exploration. We evaluate on seven benchmarks spanning two categories: 1) spatial reasoning, including MMSI-Bench [44], SAT [45], MMVP [46], and CV-Bench 3D [47]; 2) visual perception, covering BLINK [48], V\(^{*}\)Bench [49], and CV-Bench 2D [47]. All evaluations are conducted under the VLMEvalKit framework [50]. Appendix 0.0.7.1 shows details of datasets and benchmarks.

Baselines and Training Recipe. We compare our method to two categories of baselines: 1) proprietary and open-source VLMs: GPT-5.4, GPT-4o [51], and Qwen2.5-VL [52]; 2) competitive UMMs: Janus-Pro [30], Chameleon [1], and BAGEL [2]. For RL training, we remove the KL loss term, and set the key hyperparameters: rollout batch size \(64\), update batch size \(32\), \(8\) rollouts per prompt. For DiffusionNFT, we set \(\beta=0.4\) and use \(20\) sampling steps for training and evaluation. All experimental details are documented in Appendix 0.0.7.

0.0.5.1 Main Results

3pt

Comparison among Unified Models. Table ¿tbl:tab:main95results? summarizes the performance of all tested methods on all benchmarks. Among UMMs of comparable scale, BAGEL already establishes a strong baseline (59.46 avg.), substantially outperforming Janus-Pro (46.05) and Chameleon (27.49), which validates its hybrid AR-diffusion architecture as a capable backbone for multi-modal reasoning (even from limited data) and motivates our choice of building upon it. Nevertheless, a notable gap remains between BAGEL and understanding-only VLMs: even the same-scale Qwen2.5-VL-7B (62.11) surpasses it by 2.65 points, with the margin widening further against Qwen2.5-VL-72B (70.53). It suggests that the generative versatility of unified architectures does not automatically translate into strong visual reasoning without dedicated post-training.

Spatial Reasoning and Visual Perception Performance. Effective on-policy exploration presupposes a policy that can already generate reasonable rollouts. Our SFT stage fulfills precisely this prerequisite: it improves over BAGEL by +2.53 points on average, with gains observed on nearly every benchmark, thereby furnishing a reliable initialization for the subsequent RL phase. From this initialization, RL yields an additional +2.73-point average gain over BAGEL, lifting BRAID to 65.19 and surpassing GPT-4o (62.46) despite employing only 7B parameters. Notably, the most pronounced gains concentrate on SAT (+14.00) and V*Bench (+10.76), which demand mental viewpoint transformation or fine-grained visual discrimination – precisely the capabilities where unified models have historically trailed understanding-only counterparts. This confirms that BRAID can narrow such deficits without any architectural change. The sole regression appears on CV-Bench 3D (-1.24), which we attribute to a distributional gap in the SFT data; encouragingly, BRAID recovers part of this loss, reducing the deficit from -1.74 (SFT) to -1.24.

Reasoning Capacity Boundary. Figure 3 examines the reasoning ceiling via Maj@\(n\) and Avg@\(n\) metrics across varying samples. On VStar, BRAID’s Maj@\(n\) curve climbs steadily from 66.5 to 70.7 as \(n\) grows from 3 to 17, while curves of SFT and BAGEL plateau early, suggesting that RL endows the model with a broader and more exploitable solution space. SAT exhibits the same pattern: BRAID attains 61.3 at Maj@17, exceeding SFT (56.7) and BAGEL (52.3) by +4.6 and +9.0, respectively. The Avg@\(n\) curves reinforce this interpretation: BRAID’s mean per-sample quality is consistently higher, ruling out the possibility that its majority-vote advantage stems solely from greater output diversity. Taken together, these scaling trends confirm that RL not only aligns the model with the reward signal but also incentivizes genuinely stronger reasoning capabilities.

0.0.5.2 Ablation Study

To isolate the contribution of each component, we evaluate two ablated variants under the same training recipe: (1) BRAID w/o\(\mathcal{L}_{\text{NFT}}\), which freezes the image generation parameters during RL and only optimizes the text branch, following the design of COOPER [13]; and (2) BRAID w/o\(r^{\text{vis}}\), which removes the vision-thinking process reward and relies solely on the terminal reward \(r(\tau)\) for advantage estimation. Results are reported in Table ¿tbl:tab:ablation? and Figure 4 with key findings as

• Dropping \(r^{\text{vis}}\) (BRAID vs. w/o \(r^{\text{vis}}\)) results in a moderate decline (65.19 vs. avg.), mainly on VStar (\(-\)​4.48) and SAT (\(-\)​2.67), two benchmarks reliant on fine-grained visual reasoning. Training accuracy, however, does not collapse: the shared trajectory-level advantage still couples the two modalities, allowing the terminal reward signal to backpropagate to image turns.

• Excluding the image branch (w/o \(\mathcal{L}_{\text{NFT}}\)) incurs a steeper decline, notably on VStar (\(-\)​6.00) and CV-Bench 2D (\(-\)​4.32). However, its format reward curve converges faster, presumably because the text branch receives an undiluted reward as the sole trainable modality. In contrast, BRAID trades slower early format convergence for a higher accuracy ceiling, confirming that jointly optimizing the image branch produces richer visual intermediates that benefit downstream reasoning.

• \(\mathcal{L}_{\text{NFT}}\) opens the gradient pathway for image-turn optimization, while \(r^{\text{vis}}\) supplies a dedicated signal that steers those gradients toward reasoning-relevant outputs. Neither alone suffices: BRAID outperforms the two ablations by +3.09 and +2.51 on average, confirming that both components are essential. Notably, removing \(\mathcal{L}_{\text{NFT}}\) is more damaging than removing \(r^{\text{vis}}\), underscoring that bringing the image branch into the RL loop (BRAID’s central premise) is the more critical factor.

0.0.5.3 Analysis

Rollout-to-Update Ratio. We study how the rollout-to-update batch ratio affects training stability. Our default (2:1, rollout 64 / update 32) maintains a rollout buffer that stabilizes gradient estimates. We compare this against a more aggressive setting (Ratio = 1:1, rollout and update batch is 64) that immediately consumes all rollouts in a single update. As shown in Table ¿tbl:tab:on95off95policy? and Figure 5, the 1:1 ratio leads to a notable decline of avg. 65.19 to 61.93. The accuracy curve exhibits pronounced oscillations and recovers slowly, indicating that aggressive updates can destabilize the policy. Interestingly, despite the overall degradation, CV-Bench 3D improves to 84.83 under the 1:1 ratio, even surpassing the default 2:1 setting (82.2). We conjecture that higher update frequency encourages aggressive exploration that coincidentally favors this task but destabilizes others; the 2:1 ratio therefore better balances exploration and stable improvement.

Data Allocation Strategy. We compare three data-allocation regimes (detailed in Appendix 0.0.7.1): a disjoint split with equal budget (\(\mathcal{D}^{1:1}\), \(\omega{=}0\)), a disjoint split favoring SFT (\(\mathcal{D}^{2:1}\), \(\omega{=}0\)), and a replay regime (\(\mathcal{D}^{\cap}\), \(\omega{\approx}0.32\)) where the RL pool partially overlaps with SFT data. Results are reported in Table ¿tbl:tab:data95split95result? and Figure 6. The \(\mathcal{D}^{1:1}\) SFT checkpoint already exhibits weakness on several benchmarks (e.g., CV-Bench 2D drops to 65.24), indicating that an insufficient SFT budget yields a suboptimal initialization that constrains downstream RL gains. We therefore did not proceed with RL training from this checkpoint. With adequate SFT initialization (\(\mathcal{D}^{2:1}\)), RL brings consistent gains and reaches 62.63 avg. The replay regime \(\mathcal{D}^{\cap}\) further improves this to 65.19 avg., with pronounced gains on SAT (+9.34 over \(\mathcal{D}^{2:1}\) RL) and VStar (+3.43). This confirms that replaying a portion of SFT data during RL bridges imitation learning and on-policy exploration, enabling the model to revisit familiar prompts under its own policy and thereby consolidate supervised knowledge while continuing to discover novel reasoning strategies.

0.0.5.4 Case Study

Interleaved Reasoning. We qualitatively compare BRAID and its ablations on a visual search task (Appendix 0.0.9). BRAID generates diverse, high-fidelity intermediate images that directly support correct reasoning. In contrast, w/o \(\mathcal{L}_{\text{NFT}}\) produces distorted visuals that induce hallucinated text reasoning, and w/o \(r^{\text{vis}}\) introduces fabricated artifacts. Although all variants reach the correct answer on this instance, only BRAID’s reasoning chain is grounded in faithful visual evidence, consistent with its superior robustness under Maj@\(n\) sampling.

Text-to-Image Generation. Figure 7 shows whether BRAID’s RL training transfers to standalone generation on demo prompts requiring spatial compositionality and numerical accuracy. Without \(\mathcal{L}_{\text{NFT}}\), the image branch fails to follow instructions (cat placed beside the fishbowl; balloon count exceeds five). Removing \(r^{\text{vis}}\) improves spatial relations but still lacks numerical precision. Only BRAID correctly satisfies both prompts, indicating that the vision-thinking reward provides optimization direction that generalizes beyond interleaved reasoning (full comparison in Appendix 0.0.10).

0.0.6 Conclusions, Limitations, and Future Work↩︎

We presented BRAID, a framework that casts interleaved text-image-text reasoning as a unified MDP, enabling joint optimization over both textual and visual generation within a unified policy. By deriving a shared trajectory-level advantage and channeling it into modality-native policy gradients, BRAID propagates reward signals end-to-end across modality boundaries. A complementary vision-thinking process reward, scored by a VLM judge, supplies turn-level feedback that sharpens credit assignment at critical image branches. Experiments on seven spatial-reasoning and visual-perception benchmarks demonstrate that BRAID consistently outperforms various baselines.

Several limitations point to promising future directions. First, BRAID is built upon BAGEL’s hybrid AR-diffusion backbone; extending to fully autoregressive UMMs [4], where discrete image tokens are generated via next-token prediction, would broaden the framework’s applicability. Second, the vision-thinking reward currently relies on an external VLM judge, inducing inference cost and dependency; distilling it into a compact, jointly trained PRM would improve both efficiency and reproducibility. Finally, the current formulation assumes a fixed interleaving pattern; exploring adaptive turn structures and richer credit-assignment mechanisms, such as hierarchical advantage decomposition across turns, could further unlock RL’s potential for interleaved multi-modal reasoning.

Appendix↩︎

0.0.7 Experimental Details↩︎

0.0.7.1 Data Configuration

Following ThinkMorph and its follow-ups [7], our training corpus spans eight task families (Matterport3D, RealSee3D, Perspective, DirectionalQuery, \(360{+}x\), PositionedDirection, Jigsaw, VisualSearch), each contributing both SFT and RL split. To study how the SFT/RL data budget interacts with our two-stage pipeline, we define an overlap ratio \(\omega \triangleq |\mathcal{D}_{\text{SFT}}\cap\mathcal{D}_{\text{RL}}|/|\mathcal{D}_{\text{RL}}| \in [0,1]\), and consider three data-allocation regimes (per-task counts in Table 1): (1) a disjoint regime (\(\mathcal{D}^{2{:}1}\), \(\omega{=}0\)) with a \(2{:}1\) SFT/RL budget, favoring supervised initialization; (2) a disjoint regime (\(\mathcal{D}^{1{:}1}\), \(\omega{=}0\)) with a \(1{:}1\) budget, isolating the effect of equal-sized splits; and (3) a replay regime (\(\mathcal{D}^{\cap}\)) where the RL pool augments the original RL split with additional samples drawn from the SFT pool (\(20{,}000\)/\(15{,}200\), \(\omega{\approx}0.32\)), so that part of the RL prompts revisit SFT examples under on-policy rollouts. We adopt regime (3) as our default.

4pt

0.0.7.2 Evaluation Benchmarks

We evaluate on seven benchmarks spanning two categories. All evaluations are conducted under the VLMEvalKit [50] framework with accuracy as the primary metric.

0.0.7.2.0.1 Spatial Reasoning.

(1) MMSI-Bench [44] is a multi-image spatial intelligence benchmark comprising 1,000 multiple-choice questions across 10 fundamental spatial tasks (e.g., object motion tracking, camera ego-motion, scene reconstruction), curated from over 120,000 images by 3D-vision experts. (2) SAT [45] evaluates dynamic and static spatial reasoning, including egocentric movement, allocentric perspective shifts, depth relations, and action consequence prediction. It contains over 218,000 QA pairs across 22,000 synthetic scenes generated with the ProcTHOR engine, plus 150 real-image QA pairs for out-of-distribution testing. (3) MMVP [46] targets systematic failures of vision encoders by constructing “CLIP-blind pairs”—image pairs that CLIP encodes similarly despite clear visual differences in orientation, counting, color, or viewpoint. It contains approximately 150 image pairs formulated as multiple-choice questions. (4) CV-Bench 3D [47] is the 3D split of the Cambrian Vision-Centric Benchmark, evaluating depth ordering and relative distance understanding via natural language questions derived from OMNI3D annotations.

0.0.7.2.0.2 Visual Perception.

(1) BLINK [48] contains 3,807 multiple-choice questions reformulated from 14 classic computer vision tasks (relative depth, visual correspondence, multi-view reasoning, forensic detection, spatial relations, etc.) that humans can solve “in a blink” but multimodal LLMs find challenging. (2) V\(^{*}\)Bench [49] evaluates fine-grained visual perception in high-resolution, visually crowded images, comprising 191 multiple-choice questions on attribute recognition (color, material) and spatial relationship reasoning for small or occluded objects that require targeted visual search. (3) CV-Bench 2D [47] is the 2D split of the Cambrian Vision-Centric Benchmark, assessing spatial relationship understanding and object counting via questions derived from ADE20k and COCO annotations.

0.0.7.3 Training Recipe

6pt

We build on BAGEL-7B [2] and train with a two-stage pipeline: SFT followed by RL. Both stages use FSDP2 with hybrid sharding, gradient checkpointing, and the VAE frozen throughout. During RL, all other modules (LLM, ViT, understanding head, generation head) are unfrozen and jointly optimized.

For the SFT stage, we follow ThinkMorph [7] and train on curated TIT-CoT trajectories under the \(\mathcal{D}^{2:1}\) data regime (detailed in Section 0.0.7.1). For the RL stage, we adopt the replay regime \(\mathcal{D}^{\cap}\) (\(\omega{\approx}0.32\)) as default, where the RL pool partially overlaps with SFT data to bridge imitation learning and on-policy exploration. We sample rollouts with temperature 1.0 and allow up to 3 interleaved image rounds per trajectory, with a maximum of 2048 thinking tokens per round and 32768 total tokens. Image generation uses 20 denoising steps (for both training and evaluation) with a timestep shift of 3.0, classifier-free guidance scales of 4.0 (text) and 2.0 (image), and a guidance interval of \([0.4, 1.0]\). The noise level for intermediate images is set to 0.7. Table 2 summarizes the key hyperparameters.

0.0.7.4 Vision-Thinking Process Reward Prompt

To instantiate the per-turn visual process reward \(\boldsymbol{r}^{\text{vis}}\), we employ GPT-5.2 as the VLM judge. Given the question, the problem image(s), and the generated intermediate image \(I_k\), the judge follows a three-step protocol (question understanding \(\rightarrow\) image analysis \(\rightarrow\) scoring) and emits four integer scores in \([1,10]\), corresponding one-to-one to the four criteria \(\{r^{(\text{vc})}, r^{(\text{vf})}, r^{(\text{ru})}, r^{(\text{tr})}\}\) defined in Sec. 0.0.4.4. The full system prompt is shown below.

0.0.8 Additional Results↩︎

0.0.8.1 Ablation Study

5pt

Table ¿tbl:tab:ablation? reports the complete numerical results of the ablation study discussed in Section 0.0.5.2, covering all seven benchmarks for BAGEL, SFT, and the two ablated variants. Several per-benchmark patterns are worth noting. On MMSI, all RL variants outperform SFT, indicating that even without \(r^{\text{vis}}\) or \(\mathcal{L}_{\text{NFT}}\), the text-branch RL alone provides gains on multi-image spatial tasks. On CV-Bench 3D, both BRAID and w/o \(r^{\text{vis}}\) recover the SFT regression (82.17 vs.), whereas w/o \(\mathcal{L}_{\text{NFT}}\) does not (81.50), suggesting that optimizing the image branch is necessary to recover 3D spatial understanding. The perception benchmarks (BLINK, VStar, CV-Bench 2D) show the largest gap between the full model and ablations, confirming that fine-grained visual tasks benefit most from the combination of image-branch RL and dense process reward.

0.0.8.2 Analysis of Reasoning Capacity

4pt

Table 3 provides the full Maj@\(n\) and Avg@\(n\) results across \(n\in\{3,5,9,17\}\) on VStar and SAT, corresponding to the curves in Figure 3. BRAID consistently achieves the highest scores at all sampling budgets, with the gap widening as \(n\) increases. Notably, on VStar the Maj@17 gap between BRAID and SFT is +3.15, whereas the Avg@17 gap is +2.34, indicating that BRAID benefits from both higher per-sample quality and greater diversity. On SAT, BRAID’s Avg@\(n\) improves monotonically from 58.11 to 60.75 as \(n\) grows, while SFT and BAGEL plateau or decline, suggesting that RL produces a policy whose samples are more complementary and less redundant.

0.0.8.3 Analysis of Training Recipe

5pt

Table ¿tbl:tab:data95split95result? presents the full results under the three data-allocation regimes discussed in Section 0.0.5.3. The \(\mathcal{D}^{1:1}\) SFT checkpoint underperforms BAGEL on CV-Bench 2D (65.24 vs.), a \(-\)​8.29 drop that we attribute to the reduced SFT budget failing to cover sufficient visual perception data. By contrast, the \(\mathcal{D}^{2:1}\) regime provides a stable SFT initialization that enables effective downstream RL. The replay regime \(\mathcal{D}^{\cap}\) further improves upon \(\mathcal{D}^{2:1}\) RL on all benchmarks except BLINK (tied at 57.76), with the largest gains on SAT (+9.34) and VStar (+3.43). This pattern suggests that replaying SFT prompts under on-policy rollouts is particularly beneficial for tasks requiring multi-step spatial reasoning, where the model can consolidate learned strategies while exploring new ones.

5pt

Table ¿tbl:tab:on95off95policy? reports the complete benchmark results for the rollout-to-update ratio analysis. The 2:1 ratio achieves the best average (65.19 vs.), with pronounced advantages on MMSI (+7.20), SAT (+8.67), and VStar (+8.15). The 1:1 ratio, despite its overall lower performance, achieves the highest score on CV-Bench 3D (84.83) and BLINK (59.07), both exceeding even the 2:1 setting. We hypothesize that the more aggressive update frequency acts as implicit exploration pressure, which benefits benchmarks where the SFT initialization is already close to a local optimum but fails to generalize on tasks requiring stable policy improvement over many steps.

0.0.9 Interleave Reasoning Case↩︎

We present a qualitative comparison of BRAID and its ablated variants on a visual search task from the VStar benchmark.

0.0.9.0.0.1 Analysis.

This case reveals several observations about the role of image quality in TIT-CoT reasoning:

(1) BRAID produces diverse and faithful visual intermediates. The two BRAID rollouts adopt distinct visual strategies: Rollout 1 generates a tightly cropped view of the license plate for precise character reading, while Rollout 2 produces a full-scene image with the plate region highlighted. Both yield high-fidelity images where “OU68 EPD” is clearly legible, directly supporting downstream text reasoning.

(2) Without \(r^{\text{vis}}\), images contain hallucinated artifacts. The w/o \(r^{\text{vis}}\) variant generates an image containing a hallucinated second license plate (“E-EI SV”) alongside the actual plate. Although the model correctly identifies the relevant plate here, the fabricated visual content introduces unnecessary ambiguity that would likely mislead reasoning on harder instances.

(3) Without \(\mathcal{L}_{\text{NFT}}\), images degrade and mislead reasoning. The w/o \(\mathcal{L}_{\text{NFT}}\) variant generates a distorted, blurry crop where the plate characters are garbled. The text model hallucinates “GI01 EPD” from the corrupted visual evidence and arrives at the correct answer through faulty reasoning (incorrectly mapping “GI01 EPD” to option A). The reasoning chain is fundamentally unreliable: the model cannot distinguish correct from fabricated visual evidence.

(4) Correct answers can mask fragile reasoning. All variants arrive at the correct answer, yet only BRAID produces a trajectory where the visual evidence genuinely supports the conclusion. The ablated variants achieve correctness despite their visual intermediates, not because of them, a distinction that manifests as lower robustness under Maj@\(n\) evaluation (Section 0.0.5.1).

0.0.10 Text-to-Image Generation Quality↩︎

Beyond interleaved reasoning, we examine whether the RL training in BRAID also improves standalone text-to-image generation quality. We compare generations from BAGEL, SFT, BRAID w/o \(r^{\text{vis}}\), BRAID w/o \(\mathcal{L}_{\text{NFT}}\), and the full BRAID on three prompts that test different aspects of instruction following: spatial compositionality, numerical accuracy, and creative concept binding.

0.0.10.0.0.1 Analysis.

The generation results in Figure 8 reveal that BRAID’s RL training yields improvements in both image quality and instruction adherence:

(1) Spatial compositionality. For “A cat sitting inside a fishbowl”, BAGEL, SFT, and w/o \(\mathcal{L}_{\text{NFT}}\) all place the cat next to the fishbowl rather than inside it, failing to satisfy the spatial relation specified by the prompt. Both w/o \(r^{\text{vis}}\) and BRAID correctly render the cat inside the bowl, but BRAID additionally preserves higher visual fidelity with a fish coexisting in the scene.

(2) Numerical accuracy. For “Exactly five balloons”, BAGEL generates approximately eight, SFT produces four, w/o \(\mathcal{L}_{\text{NFT}}\) produces an over-saturated chaotic scene with many balloons, and w/o \(r^{\text{vis}}\) generates seven to eight. Only BRAID correctly generates exactly five balloons, demonstrating that the joint RL objective improves precise counting ability.

(3) Creative concept binding. For “Hourglass with galaxies as sand”, BAGEL, SFT, and w/o \(r^{\text{vis}}\) all produce an hourglass with regular sand and merely place it against a galaxy background, failing to bind the “galaxies as sand” concept. The w/o \(\mathcal{L}_{\text{NFT}}\) variant shows a galaxy swirl inside the hourglass, and BRAID further improves this with a clearly visible galaxy serving as the flowing sand material.

These observations suggest that the vision-thinking process reward \(r^{\text{vis}}\) provides optimization direction that transfers beyond the interleaved reasoning setting to general text-to-image generation. The reward signal, which evaluates visual correctness and faithfulness during TIT-CoT training, implicitly teaches the image branch to better follow compositional instructions. Notably, w/o \(\mathcal{L}_{\text{NFT}}\) consistently shows the weakest instruction following among RL-trained variants, confirming that without the DiffusionNFT gradient pathway, the image branch cannot internalize the reward-driven improvements.

References↩︎

---

1. We use GPT-5.2 in all experiments; see Appendix 0.0.7.4 for the full prompt.↩︎