1 Introduction↩︎

Diffusion models, as delineated in recent studies [1]–[7], have exhibited remarkable capabilities in image synthesis, bolstering numerous applications such as text-to-image generation [8], [9], image editing [8], [10]–[12], and image inpainting [10], [13]. A key characteristic of these models is their multi-step generative process, which not only allows for correction of the diffusion path [7] but also enhances controllability [14], [15].

Despite these advancements, the inference process in diffusion models typically involves a specific flow, whereas the training process entails random step selections from multiple flows. This randomness often results in a given training step correlating with diverse flows, creating ambiguity in target identification from the network’s perspective, as depicted in Fig. [fig:OOD](a). We term this phenomenon ‘xFlow’.

xFlow’s emergence during training can hinder the model’s optimization at certain steps, leading to a spectrum of generative issues. Notably, it challenges the model’s ability to generate samples via a straight flow, compromising deterministic sampling across varying step counts, as shown in Fig. [fig:OOD](c). It also complicates predicting later sampling steps from earlier ones, limiting the effectiveness of reward models [16] and guided models [3]. Moreover, in the context of distillation, which typically adopts a progressive approach, xFlow can introduce misleading signals, as evidenced in Rectified Flow [17]. Perhaps most critically, xFlow can lead to the generation of Out-Of-Distribution (OOD) samples or low-quality samples, especially as sampling step size increases, as illustrated in Fig. [fig:OOD](d-e).

In this paper, we propose a novel training strategy aimed at resolving the xFlow challenge in diffusion models. Our method centers on augmenting the input dimensionality to these models, a change that effectively prevents flow crossing. As depicted in Fig. [fig:OOD](a), the issue at hand arises when two flows intersect, creating ambiguity; the input to the network (for instance, a noisy image) remains constant, yet it is associated with multiple potential targets (such as distinct noises originating from different images). To address this, our approach entails predicting the flow itself during the training phase, as shown in Fig. [fig:OOD](b). Notably, we utilize the noise predicted by the network as the flow’s endpoint, incorporating this element into the model’s input. This technique sidesteps the pitfall of using groundtruth noise as input, which would otherwise result in trivial training solutions devoid of substantive learning. For practical implementation, we found ControlNet [18] particularly effective in this context. Additionally, our methodology integrates a bootstrap approach reminiscent of Analog bits [19], which significantly enhances our model’s optimization and effectively narrows the gap between training and inference phases."

To evaluate our approach, we introduce the Inference Flow Consistency (\(\operatorname{IFC}\)) metric, reflecting xFlow severity. We also utilize Inception Score (IS) [20] and Fréchet Inception Distance (FID) [21] for assessing generation quality. Our models, trained from scratch and compared against baselines on Cifar-10 [22], demonstrate not only an avoidance of xFlow but also an enhancement in generation quality. The contributions of this paper include:

• We identify a widespread phenomenon in diffusion models, termed xFlow, leading to non-straight flow during inference that may generate OOD or suboptimal samples.

• We attribute xFlow’s origins to the instability of the target during the training process. Accordingly, we introduce the Non-Cross Diffusion, a novel training and inference pipeline to mitigate the xFlow problem by enhancing input dimensionality.

• Our experiments on both a toy model and the Cifar-10 dataset demonstrate that our method not only improves the proposed \(\operatorname{IFC}\) metric by addressing xFlow, but also significantly enhances other image evaluation metrics, such as IS and FID.

2 Related work↩︎

2.1 Diffusion models↩︎

Diffusion models, as generative models, learn the reverse denoising process from Gaussian noise to image distribution, achieved through either Markov [2] or non-Markov operations [1]. They are favored over other generative models like GAN [23] and VAE [24] due to their training stability and superior generation quality. Subsequent enhancements to these models primarily concern varied network architectures [25], noise schedulers or losses [4], transition from image space diffusion to latent space [5], and improved sampling techniques [26], with little attention to the xFlow during training. Rectified Flow [17] noted the mismatch in sampling across different inference steps, a significant distillation issue, but did not analyze it further. Instead, they proposed a workaround using a 2-rectified flow to fit another model to a non-crossing flow between source and target distributions, which depends on a well-trained diffusion model and requires additional retraining. Our paper is the first to examine xFlow in diffusion and offer solutions.

2.2 Conditional Image Generation↩︎

Conditioning techniques are instrumental in managing generated content [5], [27]–[29]. For diffusion models, score-based model [7] proposes classifier guidance, which is an efficient method to balance controllability and fidelity using the gradients from a classifier, while classifier-free guidance [30], being another important conditioning technique to diffusion models, trains both conditional and unconditional diffusion models, and combining their score to achieve better controllability. ControlNet [18] employs pretrained encoding layers from billions of images as a backbone to learn diverse conditional controls, which is an architecture adopted in this paper. Analog Bits [19] introduces a technique that conditions the model on its own previously generated samples during iterative sampling, akin to our work. However, Analog Bits mainly aims to enhance sample quality by reusing the previous target, while our focus is to introduce a condition in the training flow to prevent crossing issues.

3 Method↩︎

In this section, we start with a brief review of the formulation of DDPM [2]. Next, we show the drawback of baseline flow and analyze the cause of xFlow. Then, we introduce Non-Cross Diffusion to avoid crossing by ascending dimension of input, together with training, inference, and network architecture of Non-Cross Diffusion. Finally, we introduce \(\operatorname{IFC}\) for evaluating the consistency of the inference flow.

3.1 Preliminary↩︎

Given samples from data distribution \(x_0\sim q(x_0)\), DDPM [2] defines a forward noising process \(q\) which produces latent variables \({x_1,\ldots,x_T}\) by gradually adding Gaussian noise with a variance schedule \(\beta_t\in(0,1)\) as follows: \[\begin{gather} q(x_1,\ldots,x_T) \mathrel{\vcenter{:}}= \prod_{t=1}^T q(x_t|x_{t-1}), \\ q(x_t|x_{t-1}) \mathrel{\vcenter{:}}= \mathcal{N}(x_t;\sqrt{1-\beta_t}x_{t-1},\beta_t \mathbf{I}). \label{eq95q} \end{gather}\tag{1}\] With \(\alpha_t\mathrel{\vcenter{:}}= 1-\beta_t\) and \(\Bar{\alpha}_t \mathrel{\vcenter{:}}= \prod_{s=0}^t \alpha_s\), the marginal \(q(x_t|x_0)\) can be derived through Eq. 1 as follows: \[\begin{gather} q(x_t|x_0)= \mathcal{N}(x_t,\sqrt{\Bar{\alpha}_t}x_0,(1-\Bar{\alpha}_t)\mathbf{I}), \\ x_t = \sqrt{\bar{\alpha}_t} x_0+\sqrt{1-\bar{\alpha}_t} \epsilon, \label{eq5} \end{gather}\tag{2}\] where \(\epsilon \sim \mathcal{N}(\mathbf{0},\mathbf{I})\). Using Bayes theorem, we can calculate the posterior \(q(x_{t-1}|x_t,x_0)\) in terms of \(\beta_{t}, \alpha_{t}\) and \(\bar{\alpha}_{t}\). There are many different ways to parameterize \(p_{\theta}\) to approximate the posterior, while DDPM [2] chooses \(p_{\theta}(x_{t-1}|x_t)=\mathcal{N}(x_{t-1};\boldsymbol{\mu}_{\theta}(x_t,t), \sigma_t^2\mathbf{I})\), and propose that predicting \(\epsilon\) works best with a loss function: \[\begin{gather} L_{simple} = E_{t,x_0,\epsilon}[\| \epsilon - \epsilon_\theta(x_t,t)\|^2], \label{loss} \end{gather}\tag{3}\] where \(\boldsymbol{\mu}_\theta(x_t,t) = \frac{1}{\sqrt{\alpha}_t}(x_t - \frac{\beta_t}{\sqrt{1-\bar{\alpha}_t}}\epsilon_\theta(x_t,t))\)

3.2 Understanding Drawbacks of DDPM Flow↩︎

Training stage. Given source distribution \(\pi_0\) (, \(q(x_0)\)) and target distribution \(\pi_1\) (, \(\mathcal{N}(\mathbf{0}, \mathbf{I})\)), we sample two data pairs \((x_0,x_T),(y_0,y_T) \sim \pi_0 \times \pi_1\). During the training stage, assume these two training flows cross at time step \(t\) (, \(x_t = y_t\)). Following Eq. 2 , we have: \[\begin{gather} x_t = \sqrt{\bar{\alpha}_t} x_0+\sqrt{1-\bar{\alpha}_t} \epsilon_x, \\ y_t = \sqrt{\bar{\alpha}_t} y_0+\sqrt{1-\bar{\alpha}_t} \epsilon_y. \end{gather}\] At the crossing point, both flows aim to minimize the loss function as follows during training: \[\begin{gather} L_{simple} = E_{t,x_0,\epsilon}[\| \epsilon - \epsilon_\theta(x_t,t)\|^2], \end{gather}\] For given \((x_0,x_T),(y_0,y_T)\) and cross point \(t\), the loss can be reformulated as follows: \[\begin{gather} L_1 = \frac{1}{2}\|\epsilon_x-\epsilon_\theta(x_t,t)\|^2+\frac{1}{2}\|\epsilon_y-\epsilon_\theta(y_t,t)\|^2,\label{equ:L1} \end{gather}\tag{4}\] where \(\epsilon_x=x_T\) and \(\epsilon_y=y_T\).

To check how the crossing point affects the optimization process, we simplify the target in the formulation of an optimization problem. Since \(\epsilon_\theta(x_t,t)=\epsilon_\theta(y_t,t)=\epsilon_\theta(z_t,t)\), we use the notion of \(\epsilon_\theta\) to represent them all. \[\begin{align} \theta^*&=\arg\min_\theta \frac{1}{2}\|\epsilon_x-\epsilon_\theta\|^2+\frac{1}{2}\|\epsilon_y-\epsilon_\theta\|^2 \\ &=\arg\min_\theta \frac{1}{2}(\epsilon_x^2-2\epsilon_x\epsilon_\theta+\epsilon_\theta^2+\epsilon_y^2- 2\epsilon_y\epsilon_\theta+\epsilon_\theta^2) \\ &=\arg\min_\theta \frac{1}{2}(\epsilon_x^2+\epsilon_y^2)-(\epsilon_x+\epsilon_y)\epsilon_\theta+\epsilon_\theta^2 \\ &=\arg\min_\theta \frac{1}{4}(\epsilon_x^2+2\epsilon_x\epsilon_y+\epsilon_y^2)-(\epsilon_x+\epsilon_y)\epsilon_\theta+\epsilon_\theta^2 \\ &=\arg\min_\theta \|\frac{\epsilon_x+\epsilon_y}{2}-\epsilon_\theta\|^2 \label{qwpusezn} \end{align}\tag{5}\] Hence, with the existence of a crossing point, the optimizing target is equivalent to \(\frac{\epsilon_x+\epsilon_y}{2}\). However, since \(\epsilon_x\sim\mathcal{N}(\mathbf{0}, \mathbf{I})\) and \(\epsilon_y\sim\mathcal{N}(\mathbf{0}, \mathbf{I})\), we have \(\frac{\epsilon_x+\epsilon_y}{2} \sim \mathcal{N}(\mathbf{0}, \frac{\sqrt{2}}{2}\mathbf{I})\), which no longer follows standard normal distribution. This implies that at the crossing point, the model is given an incorrect target, which will lead to ambiguity in data generation (, the denoising process).

Inference stage. Considering the ambiguity exists in a trained model, xFlow, where the generation flow may deviate from the correct direction, results in various failure cases, as illustrated in Fig. [fig:OOD]. We further visualize the inference flow of text-to-image Stable Diffusion in different steps. We save the intermediate latent of each step and visualize the inference flow by using T-SNE. As illustrated in Fig. 2, the visualization demonstrates that Stable Diffusion also shows various inference flows across different steps, which implies the influence of xFlow. The results in Fig. 2 demonstrate that the xFlow phenomenon occurs frequently within stable diffusion, which results in sub-optimal or even OOD synthesis.

Besides, we propose that the consequences of xFlow also depend on the timestep of inference. Specifically, more inference steps, correlating with smaller strides, subtly affect inference flow because the deviation is also smaller, and subsequent steps can correct minor errors. On the contrary, fewer inference steps, leading to larger strides, significantly impact and alter the inference flow due to the crossing point, potentially generating inconsistent or OOD samples. Such phenomena further decrease the determinism of diffusion models.

3.3 Non-Cross Diffusion↩︎

As analyzed in Sec. 3.2, xFlow is caused by incorrect training targets. To solve xFlow, in this section, we introduce a new formulation of diffusion models that can avoid crossing points during training, namely Non-Cross Diffusion.

Given the fact that latent variables are linear combinations of \(x_{0}\) and \(\epsilon\) as in Eq. 2 . We can think of the issue with geometry, where training flows are line segments in 2D coordinates, as shown in Fig. [fig:OOD] (a), with the crossing point as the intersection of two segments. From a basic geometrical concept, , any two distinct lines in a plane can intersect at most once, as long as we can avoid the intersection once, the two segments will never intersect again. Therefore, we aim to eliminate crossing points between any two different training flows, thereby maintaining the integrity and distinctiveness for all of them.

To operationalize this concept, we propose to ascend the dimension of model input. The primary issue with Eq. 4 is that \(\epsilon_\theta(x_t,t)=\epsilon_\theta(y_t,t)\) when \(x_t=y_t\). To rectify this, we can introduce condition \(c^x\neq c^y\) to ensure \(\epsilon_\theta(x_t, c^x, t)\neq\epsilon_\theta(y_t, c^y, t)\) and prevent the training flow from crossing. The challenge is identifying \(c^x\neq c^y\) given a cross point \(t\) and training flow. To solve this, we use \(x_i,y_i\) as conditions for each non-crossing step \(i\) on the training flow, i.e., \(c^x=x_i\) and \(c^y=y_i\). Specifically, given \(x_{t}=y_{t}\), by sampling another point on the flow (, \(x_{i}\) and \(y_{i}\)), we have \(x_{i} \neq y_{i}\), and thus \([x_i,x_t]\neq [y_i,y_t],\forall i\in[0,T]\setminus\{t\}\). This reminds us that any other samples (\(x_i\)) from the same flow can be used for ascending dimensions. This strategy effectively creates a multidimensional space where the likelihood of training flows intersecting is significantly reduced.

Selection of Condition. \(x_{i}\) is effective for ascending dimensions only if it is significantly different from \(x_{t}\). Given the continuity of both linear combination and diffusion models, we propose to ascend the dimension with either initial noise \(x_T\) (, \(\epsilon\)) or the data point \(x_0\). Furthermore, we find the distance between randomly sampled noise is stable while the distance between data points may not. Take image data as an example, for two randomly sampled noise \(n_1,n_2\in \mathbb{R}^{H\times W\times C}\), we have \(E[\|n_1-n_2\|^2] = 2CHW\). Besides, we can only get the initial noise during the inference stage. Therefore, using the initial noise \(x_T\) for dimension ascending is more practical.

3.4 Inference Flow Consistency↩︎

To better evaluate the consistency of the inference flow for image generation, we propose a metric by computing the similarity between intermediate generated image \(\hat{x}_0^t\) in timestep \(t\) and the final generated image \(\hat{x}_0\) based on peak signal-to-noise ratio (PSNR) as follows: \[\begin{gather} \operatorname{IFC}= \frac{1}{T} \sum_{t=0}^{T} PSNR(\hat{x}_0^t,\hat{x}_0). \end{gather}\]

Training Stage. The cornerstone of our training strategy is to circumvent trivial solutions and avoid training collapse. To achieve this, we replace the use of initial noise \(\epsilon\) with predicted noise \(\hat{\epsilon}\). This substitution is critical in refining our model’s predictive accuracy since it can effectively avoid trivial solutions. However, in the initial training phase, the substantial error \(\|\epsilon-\hat{\epsilon}\|^2\) indicates \(\hat{\epsilon}\)’s poor estimation. Hence, a bootstrap strategy is introduced to \(\hat{\epsilon}\) during training, preventing misleading estimation of \(\hat{\epsilon}\) and thus enhancing learning robustness in the early stage.

As illustrated in Fig. 1, our training objective is formulated as follows: \[\begin{gather} \min_\theta E_{t,x_t,\epsilon,}[\|\epsilon -\epsilon_\theta(x_t,\hat{\epsilon}_t,t)\|^2],\end{gather}\] where \(x_t = \sqrt{\bar{\alpha}_t} x_0+\sqrt{1-\bar{\alpha}_t} \epsilon\). During training, we apply the bootstrap as follows: 1) with a fixed probability \(p\), we set \(\hat{\epsilon}_t=\mathbf{0}\) (, Case 1 in Fig. 1); 2) at other cases, \(\hat{\epsilon}_t\) is assigned the value of \(\epsilon_\theta(x_t,\mathbf{0},t)\) (, Case 2 in Fig. 1). We do not back-propagate through estimated noise \(\hat{\epsilon}_t\).

Inference Stage. As illustrated in Fig. 1, during the inference stage, to alleviate the computational costs, we use estimated noise in the previous step instead of the current step as the condition and iteratively predict \(\hat{\epsilon}\) as follows: \[\begin{gather} \hat{\epsilon}_{T} = \epsilon_\theta(\hat{x}_T,\mathbf{0},T), \\ \hat{\epsilon}_t = \epsilon_\theta(\hat{x}_t,\hat{\epsilon}_{t+1},t), t<T. \end{gather}\] When the number of inference steps is large, the discrepancy between \(\hat{\epsilon}_t\) and \(\hat{\epsilon}_{t+1}\) is small, which ensures the performance of our method.

Network Architecture. Inspired by ControlNet [18], to efficiently use \(\hat{\epsilon}_t\), Non-Cross Diffusion employs an additive U-net branch, with \(\hat{\epsilon}_t\) as input. For optimization, modifications are introduced, specifically removing all zero convolution layers and initializing the addictive encoder for \(\hat{\epsilon}_t\) with the original U-net. The output is incorporated into the U-net decoder via addition. The whole network is trained end-to-end from scratch.

A change in training flow direction at a specific timestep yields notable differences in pre- and post-change images, reducing PSNR. This can be effectively assessed for consistency across inference stages using our PSNR-based metric.

4 Experiment↩︎

In this section, we discuss our experimental results on toy examples (Sec. 4.1) and image generation tasks (Sec. 4.2), as well as ablation studies (Sec. 4.3) and further discussion for Non-Cross Diffusion(Sec. 4.4).

4.1 Toy Examples↩︎

In this section, we follow the setting in Rectified Flow [17], drawing a training dataset from Gaussian mixture \(\pi_0\times\pi_1\). Given a sample \(\{ x_0^i,x_1^i\}\) from \((X_0,X_1)\sim \pi_0\times\pi_1\), for baseline model, we train a 3-layer MLP \(v_{\theta}(z,t)\) to transfer from \(\pi_0\) to \(\pi_1\) with l2-loss as follow: \[\begin{gather} \min_{\theta} \|v_{\theta}(x_t^i,t)-(x_1^i-x_0^i) \|_2, \\ x_t^i = tx_1^i+(1-t)x_0^i, t\in [0,1). \end{gather}\] Our method enhances this approach by incorporating an additional dimension with the estimated target as follows: \[\begin{gather} \min_{\theta} \|v_{\theta}([x_t^i,\hat{c}_t^i ],t)-(x_1^i-x_0^i) \|_2, \\ \hat{c}_t^i = \begin{cases} \mathbf{0}& \text{p\leq0.5}\\ v_{\theta}([x_t^i,\mathbf{0}],t)& \text{otherwise} \end{cases} \end{gather}\] with \(p\sim U(0,1)\). The inference process is also similar to our proposed method, , we use the estimated target in the previous step as the condition.

Results. As shown in Fig. 3, the baseline model’s inference flow alters direction at the intersection of two flows due to an erroneous loss function (Eq. [wrong95target]), generating OOD samples. For toy examples, by adding an extra dimension using the estimated result, our method prevents training and inference flow intersection, maintaining consistent inference flow direction and effectively inhibiting OOD sample generation.

4.2 Experiments on Image Generation↩︎

Implementation Details. Our models are trained on Cifar-10 [22] and MNIST [31], with MNIST images resized to \(32\times32\). The fidelity of generated samples is evaluated using IS [20] and FID [21] and inference flow consistency with \(\operatorname{IFC}\). As a baseline, we train iDDPM [4] from scratch with the same UNet. Training for Cifar-10 follows iDDPM except using \(L_{simple}\) only and with 250k steps. For MNIST, training is similar to Cifar-10 but with 100k steps. In this paper, we consider unconditional generations on each dataset (w/o class labels).

Comparison of Sampling Quality. Tables 1 and  2 compare our model’s performance with a baseline model in generating Cifar-10 and MNIST images. Fig. 6 visualizes the generated Cifar-10 images, where our model notably outperforms the baseline, especially at {1000, 100, 50} inference steps. The decreased number of inference steps and increased strides enlarge the discrepancy in estimated noise between steps, introducing bias and performance decline during inference. To counter this, we suggest conditioning on the current step’s estimated noise. This augments image quality generated by our method even at smaller steps in {20, 10, 5}. Besides, our model outperforms the baseline on MNIST at steps {1000, 100, 50}. The adapted sampling strategy also enhances our model’s performance on MNIST as the number of inference steps decreases.

Comparison of Inference Consistency. Table 3 reveals our method’s superior consistency over the baseline on Cifar-10 at inference steps {50, 20, 10, 5}, in terms of \(\operatorname{IFC}\), achieved by preventing xFlow. The impact of xFlow is minimal for larger steps ({1000, 100}), resulting in similar \(\operatorname{IFC}\) between our method and the baseline. Fig. 4 and Fig. 5 illustrate the visualization results of generated images under different steps on Cifar-10 and MNIST. The results further demonstrate higher consistency compared with baselines, indicating a straighter inference flow.

4.3 Ablation Study↩︎

In this section, we verify the effectiveness of each component through ablation studies.

Ablation of bootstrap. Table 4 demonstrates the impact of the bootstrap strategy. We experimented with three settings: without bootstrap (w/o bootstrap, where \(\hat{\epsilon}_t=\epsilon_\theta(x_t,\mathbf{0},t)\) is consistently applied), exponential schedule (exp-schedule, where the probability \(p\) increases exponentially), and fixed probability (fix-probability, where \(p=0.5\)). The findings reveal that lacking a bootstrap strategy can notably degrade model performance.

Ablation of condition. We scrutinize the impact of varying conditions on our method. The \(\hat{x}_0\) condition implies the model utilizes the predicted image as the condition, while the mid. condition uses the midpoint of training flow (\(\frac{\hat{x}_0+\hat{\epsilon}}{2}\)). During inference, each technique employs its corresponding condition. As in Table 4, mid. performs suboptimally, presumably due to proximity to the training flows’ intersection, as mentioned in Sec. 3.3. Benefiting from its stability, \(\hat{\epsilon}\) as a condition leads to a significant FID improvement over \(\hat{x}_0\) condition.

Ablation of architecture. We also examine a Double U-net variant, , doubling the U-net’s input channel to accommodate \(\hat{\epsilon}\) input. As Table 4 shows, the ControlNet-based model enhances FID by 1.42, likely because the two inputs serve distinct roles, and the additive branch enables effective differentiation between them, thereby facilitating training.

Ablation of inference strategy. We also consider several inference strategies applied after training (we utilize a trained \(\hat{\epsilon}\)-conditioned model). These inference strategies include the Zero condition (utilizing \(\mathbf{0}\)), the \(\epsilon\) condition (using initial noise \(\epsilon\)), and the \(\hat{\epsilon}\) condition (employing estimated noise \(\hat{\epsilon}\)). Performance significantly deteriorates under the \(\epsilon\) condition due to \(\epsilon\)-\(\hat{\epsilon}\) discrepancy. Though our model circumvents training ambiguity, the Zero condition marginally compromises performance during inference due to potential redirection at cross points in the inference flow.

4.4 Discussion↩︎

This section offers an alternative perspective to understand our Non-Cross Diffusion. The inclusion of \(\hat{\epsilon}_{t}\) in the model input is seen as a strong conditional constraint, which we propose can reduce the variability of semantic information, thus lessening the severity of xFlow during inference.

Specifically, the \(\hat{\epsilon}_{t}\) condition in Non-Cross Diffusion is highly specific at the pixel level, making it an exceptionally stringent constraint. Consequently, Non-Cross Diffusion effectively mitigates the xFlow issue. An interesting question arises: how would xFlow be influenced by other forms of conditions with varying degrees of strength? To investigate this, we employed the pre-trained ControlNet model for empirical analysis and results.

Fig. [fig:OOD](c) shows text conditional images from Stable Diffusion [5], as well as pose conditional and depth conditional images from ControlNet [18]. The text condition images at steps {5, 10, 25, 50, 250, 500} present a significant shift in the semantic content of the images at each step. In contrast, pose conditional images demonstrate more consistent semantic content across steps: the pose of super-hero remains the same but the background and style still show a large variation. The depth conditional images keep the highest consistency across different steps, despite some variability in details such as the background and pattern. Therefore, we hypothesize that stronger conditions ease the severity of xFlow.

5 Conclusion↩︎

In this study, we have addressed the xFlow phenomenon in diffusion models, characterized by deviations in generative flow that result in semantic inconsistencies and suboptimal image generation. Our novel approach, ‘Non-Cross Diffusion’, innovates in the realm of generative modeling by adopting ordinary differential equation models. Our empirical investigations, including both a toy example and the Cifar-10 image dataset, demonstrate the substantial efficacy of the Non-Cross Diffusion approach. The results show a marked reduction in semantic inconsistencies at various inference stages and significant improvements in the overall performance of diffusion models.

Looking Ahead. The identification of xFlow as a critical issue during inference opens new avenues for research and application optimization. Despite the effectiveness of the proposed Non-Cross Diffusion approach on mitigating xFlow, we acknowledge the challenges associated with retraining large-scale diffusion models such as Stable Diffusion. However, we are optimistic that future research will find ways to integrate these improvements into existing models, potentially circumventing the need for extensive retraining. This paper lays the groundwork for such advancements, aiming to enhance the reliability and quality of diffusion model outputs.

References↩︎