1 Introduction↩︎

Classifier-Free Guidance (CFG) [1] forms the key basis of modern text-guided generation with diffusion models. From Bayes rule, CFG constructs a Bayesian classifier \(\nabla_{\boldsymbol{x}}\log p({\boldsymbol{c}}|{\boldsymbol{x}}) = \nabla_{\boldsymbol{x}}\log p({\boldsymbol{x}}|{\boldsymbol{c}}) - \nabla_{\boldsymbol{x}}\log p({\boldsymbol{x}})\) without training additional external classifiers [2]. In practice, it is common to emphasize the classifier vector direction with some constant \(\gamma\), which corresponds to sharpening the posterior, i.e. \(p({\boldsymbol{x}})p({\boldsymbol{c}}|{\boldsymbol{x}})^\gamma\). While this may potentially distort the sampling distribution [3], it is known that the sample quality along with the text alignment increases.

While CFG was originally devised for adhering to the target condition, more often than not, there are cases where it is desirable to avoid sampling from some conditions. Canonical examples include conditions that describe the poor quality of the image, or conditions that are related to harmful content [4], [5]. Often referred to as Negative Prompting (NP), most implementations simply negate the vector direction of CFG, corresponding to inversely weighting with the classifier probability, i.e. \(p({\boldsymbol{x}})/p({\boldsymbol{c}}|{\boldsymbol{x}})^\gamma\). Nonetheless, such näive implementation often leads to a decrease in the sample quality [6], [7]. Specifically, due to the unboundness of an inverted probability distribution, sampling from \(p({\boldsymbol{x}})/p({\boldsymbol{c}}|{\boldsymbol{x}})^\gamma\) leads to sampling from the low-density region or completely outside of the original density support.

To mitigate these drawbacks, by leveraging the recent advances in guided sampling methods [8]–[12], we reformulate the conditional guidance as an optimization problem with a positive or negative prompt-conditioned contrastive loss [13], [14]. Then, we derive a reverse diffusion sampling strategy with the optimized denoising direction. This results in a simple modification of CFG to the sampling process with little computational overhead. The resulting process, termed ContrastiveCFG (CCFG), optimizes the denoising direction by attracting or repelling the denoising direction for the given condition. Furthermore, the attracting and repelling forces are automatically controlled along the sampling process. Through extensive experiments, we verify that CCFG not only guides the samples to satisfy the wanted conditions, but also successfully avoid undesirable concepts while preserving the sample quality.

2 Related works↩︎

2.1 Classifier-free guidance for diffusion models.↩︎

Diffusion models [15], [16] are a class of generative models that learn the score function [17] of the data distribution, and use this score function to reverse the forward noising process. The forward process, denoted with the time index \(t\), is governed by a Gaussian kernel that the underlying data distribution \(p({\boldsymbol{x}}_0) \equiv p({\boldsymbol{x}})\) eventually approximates the standard normal distribution at time \(t = T\), i.e. \(p({\boldsymbol{x}}_T) \approx {\mathcal{N}}(0, {\boldsymbol{I}})\). The variance preserving forward transition kernel [16] is given as \(p({\boldsymbol{x}}_t|{\boldsymbol{x}}_0) = {\mathcal{N}}({\boldsymbol{x}}_t; \sqrt{\bar\alpha_t}{\boldsymbol{x}}_0, (1 - \bar\alpha_t){\boldsymbol{I}})\). The reverse generative process follows a stochastic differential equation (SDE) [15] governed by the score function \(\nabla_{{\boldsymbol{x}}_t}\log p({\boldsymbol{x}}_t)\). To estimate this score function, one typically uses epsilon matching [16] \[\label{eq:epsilon95matching} \theta^* = \mathop{\mathrm{arg\,min}}_\theta {\mathbb{E}}\left[\|{\boldsymbol{\epsilon}}_\theta(\sqrt{\bar\alpha_t}{\boldsymbol{x}}_0 + \sqrt{1 - \bar\alpha_t}{\boldsymbol{\epsilon}}) - {\boldsymbol{\epsilon}}\|_2^2\right],\tag{1}\] which can be shown to be equivalent to denoising score matching [18], \({\boldsymbol{s}}_\theta({\boldsymbol{x}}_t) = \nabla_{{\boldsymbol{x}}_t} \log p({\boldsymbol{x}}_t) = - \frac{1}{\sqrt{1 - \bar\alpha_t}} {\boldsymbol{\epsilon}}_\theta({\boldsymbol{x}}_t)\). By Tweedie’s formula [19], one can recover the posterior mean \(\hat{\boldsymbol{x}}({\boldsymbol{x}}_t) = {\mathbb{E}}[{\boldsymbol{x}}_0|{\boldsymbol{x}}_t] = \frac{1}{\sqrt{\bar\alpha_t}}({\boldsymbol{x}}_t - \sqrt{1 - \bar\alpha_t}\epsilon_\theta({\boldsymbol{x}}_t))\). Moreover, it is common practice to train a conditional score function conditioned on the text prompt [1] with random dropping to use it flexibly, either as \({\boldsymbol{\epsilon}}_\theta({\boldsymbol{x}}_t,{\boldsymbol{c}})\) or \({\boldsymbol{\epsilon}}_\theta({\boldsymbol{x}}_t):={\boldsymbol{\epsilon}}_\theta({\boldsymbol{x}}_t,\varnothing)\), where \(\varnothing\) refers to the null condition. In practice, a popular way of generating images through reverse sampling is through DDIM sampling [20], where a single iteration can be written as \[\begin{align} \tag{2} \hat{\boldsymbol{x}}_\varnothing({\boldsymbol{x}}_t) &= ({\boldsymbol{x}}_t - \sqrt{1 - \bar\alpha_t}{\boldsymbol{\epsilon}}_\theta({\boldsymbol{x}}_t,\varnothing)) / \sqrt{\bar\alpha_t} \\ {\boldsymbol{x}}_{t-1} &= \sqrt{\bar\alpha_{t-1}}\hat{\boldsymbol{x}}_\varnothing({\boldsymbol{x}}_t) + \sqrt{1 - \bar\alpha_{t-1}}{\boldsymbol{\epsilon}}_\theta({\boldsymbol{x}}_t,\varnothing), \tag{3} \end{align}\] where we defined \(\hat{\boldsymbol{x}}_\varnothing({\boldsymbol{x}}_t)\) as the posterior mean without the conditioning. Iterating equation 2 and equation 3 amounts to sampling from \(p({\boldsymbol{x}})\).

Plugging the wanted condition \({\boldsymbol{c}}^+\) into the conditional epsilon \({\boldsymbol{\epsilon}}_\theta({\boldsymbol{x}}_t,{\boldsymbol{c}}^+)\) to sample from the conditional distribution \(p({\boldsymbol{x}}|{\boldsymbol{c}}^+)\) does not work well in practice, due to the guidance effect being too weak. To mitigate this downside, it is standard to use classifier-free guidance (CFG) [1] at sampling time. The key idea is to use \[\begin{align} \label{eq:cfg} \hat{\boldsymbol{\epsilon}}_{{\boldsymbol{c}}^+}^\gamma({\boldsymbol{x}}_t) := {\hat{\boldsymbol{\epsilon}}_\varnothing}({\boldsymbol{x}}_t) + \gamma({\hat{\boldsymbol{\epsilon}}_{{\boldsymbol{c}}^+}}({\boldsymbol{x}}_t) - {\hat{\boldsymbol{\epsilon}}_\varnothing}({\boldsymbol{x}}_t)), \end{align}\tag{4}\] where we defined \(\hat{\boldsymbol{\epsilon}}_{\boldsymbol{c}}({\boldsymbol{x}}_t) := {\boldsymbol{\epsilon}}_\theta({\boldsymbol{x}}_t,{\boldsymbol{c}})\)¹. Running DDIM sampling with \(\hat{\boldsymbol{\epsilon}}_{{\boldsymbol{c}}^+}^\gamma({\boldsymbol{x}}_t)\) in the place of \(\hat{\boldsymbol{\epsilon}}_\varnothing({\boldsymbol{x}}_t)\) leads to sampling from the gamma-powered distribution \(p^\gamma({\boldsymbol{x}}|{\boldsymbol{c}}^+)\propto p({\boldsymbol{x}})p({\boldsymbol{c}}^+|{\boldsymbol{x}})^\gamma\), a sharpened posterior. In this way, adherence to the condition \({\boldsymbol{c}}^+\) is emphasized.

2.2 Guided sampling methods for diffusion models↩︎

Another popular way of posterior sampling with diffusion models is to use guided sampling, where the noised sample \({\boldsymbol{x}}_t\)[8], [11], [12], [21] or its posterior mean \(\hat{\boldsymbol{x}}_\varnothing({\boldsymbol{x}}_t)\)[10], [22] is added by appropriate guidance terms according to the gradient of a pre-defined energy function \(\ell(\cdot)\) to be minimized. For instance, Decomposed Diffusion Sampling [10] has the form of \[\begin{align} \label{eq:dds} {\boldsymbol{x}}_{t-1} = \sqrt{\bar\alpha_{t-1}}({\hat{\boldsymbol{x}}_\varnothing}- \omega_t\nabla_{{\hat{\boldsymbol{x}}_\varnothing}}\ell({\hat{\boldsymbol{x}}_\varnothing})) + \sqrt{1 - \bar\alpha_{t-1}}{\hat{\boldsymbol{\epsilon}}_\varnothing}, \end{align}\tag{5}\] where \(\omega_t\) is the step size, to generate sample \({\boldsymbol{x}}_0\) with small \(\ell({\boldsymbol{x}}_0)\). [23] presented that CFG can also be seen as such guided sampling, when we replace \(\ell({\hat{\boldsymbol{x}}_\varnothing})\) in equation 5 with the following loss function similar to Score Distillation Sampling (SDS) [24]. \[\begin{align} \label{eq:sds} \ell_{SDS}({\boldsymbol{x}}) &:= \|{\boldsymbol{\epsilon}}_\theta(\sqrt{\bar\alpha_t}{\boldsymbol{x}}+ \sqrt{1 - \bar\alpha_t}{\boldsymbol{\epsilon}}, {\boldsymbol{c}}) - {\boldsymbol{\epsilon}}\|_2^2 \notag\\ &= \frac{\bar\alpha_t}{1 - \bar\alpha_t}\|{\hat{\boldsymbol{x}}_{\boldsymbol{c}}}- {\boldsymbol{x}}\|_2^2 \end{align}\tag{6}\] This leads to the alternative version of CFG called CFG++[23] with \(\gamma := \frac{2\bar\alpha_t}{1 - \bar\alpha_t}\omega_t\), \[\begin{align} \label{eq:cfg95sds} {\boldsymbol{x}}_{t-1} = \sqrt{\bar\alpha_{t-1}}({\hat{\boldsymbol{x}}_\varnothing}+ \gamma({\hat{\boldsymbol{x}}_{\boldsymbol{c}}}- {\hat{\boldsymbol{x}}_\varnothing})) + \sqrt{1 - \bar\alpha_{t-1}}{\hat{\boldsymbol{\epsilon}}_\varnothing}, \end{align}\tag{7}\] which is shown as a weight-rescaled version of CFG where the improvement is especially dominating at the early stage of reverse sampling.

2.3 Negative guidance in diffusion model sampling↩︎

In contrast to CFG that samples toward the \({\boldsymbol{c}}^+\), negative guidance aims to avoid samples that meet the unwanted condition \({\boldsymbol{c}}^-\). Negative guidance can be utilized alone when samples from a broad unconditional distribution, yet excluding certain unwanted features, are desired. When the unconditional distribution is too broad to be meaningful(e.g., text-to-image models), negative guidance is often combined with the positive guidance of CFG.

Inversely proportional distribution-based methods. The simplest case [4], [7], [25], [26], often called negative prompting(NP), negates the guidance direction of CFG in equation 4 such as \[\begin{align} \label{eq:cfg95negative1} \hat{\boldsymbol{\epsilon}}_{{\boldsymbol{c}}^-}^\gamma({\boldsymbol{x}}_t) := {\hat{\boldsymbol{\epsilon}}_\varnothing}({\boldsymbol{x}}_t) - \gamma({\hat{\boldsymbol{\epsilon}}_{{\boldsymbol{c}}^-}}({\boldsymbol{x}}_t) - {\hat{\boldsymbol{\epsilon}}_\varnothing}({\boldsymbol{x}}_t)), \end{align}\tag{8}\] where the goal is to avoid sampling from \({\boldsymbol{c}}^-\). Note that this corresponds to sampling from \(p^{-\gamma}({\boldsymbol{x}}|{\boldsymbol{c}}^-) := p({\boldsymbol{x}})/p({\boldsymbol{c}}^-|{\boldsymbol{x}})^{\gamma}\), a joint distribution inversely proportional to the posterior likelihood. When the goal is to sample from \({\boldsymbol{c}}^+\) while avoiding \({\boldsymbol{c}}^-\), one uses [27] \[\begin{align} \hat{\boldsymbol{\epsilon}}_{{\boldsymbol{c}}^+,{\boldsymbol{c}}^-}^\gamma &:= {\hat{\boldsymbol{\epsilon}}_\varnothing}+ \gamma({\hat{\boldsymbol{\epsilon}}_{{\boldsymbol{c}}^+}}- {\hat{\boldsymbol{\epsilon}}_\varnothing}) - \gamma({\hat{\boldsymbol{\epsilon}}_{{\boldsymbol{c}}^-}}- {\hat{\boldsymbol{\epsilon}}_\varnothing}) \notag\\ &= {\hat{\boldsymbol{\epsilon}}_\varnothing}+ \gamma({\hat{\boldsymbol{\epsilon}}_{{\boldsymbol{c}}^+}}- {\hat{\boldsymbol{\epsilon}}_{{\boldsymbol{c}}^-}})\label{eq:np} \end{align}\tag{9}\] In both cases, pushing away from \({\boldsymbol{c}}^-\) is governed by the negation of the vector direction \(({\hat{\boldsymbol{\epsilon}}_{{\boldsymbol{c}}^-}}- {\hat{\boldsymbol{\epsilon}}_\varnothing})\).

Several works proposed empirical improvements for aggregating positive and negative concepts in equation 9 for the situation where the negative guidance term conflicts with the pre-given positive conditions. This includes the on-the-fly control of the negative guidance scale[28], or canceling the negative guidance component parallel to the CFG term[7].

Utilizing complementary conditions. [25] proposed dynamic negative guidance (DNG) and proposed sampling from \(p({\boldsymbol{x}})p(\neg{\boldsymbol{c}}^-|{\boldsymbol{x}})^\gamma\), where \(\neg{\boldsymbol{c}}^-\) is defined as a hypothetical condition such that \(p(\neg{\boldsymbol{c}}^-|{\boldsymbol{x}})=1-p({\boldsymbol{c}}^-|{\boldsymbol{x}})\). This \(\neg{\boldsymbol{c}}^-\) can also be seen as a union of all possible input conditions except \({\boldsymbol{c}}^-\), as a complementary set of \({\boldsymbol{c}}^-\). Applying the Bayes rule, one can show that \[\begin{align} \nabla_{{\boldsymbol{x}}_t} &\log p({\boldsymbol{x}}_t|\neg{\boldsymbol{c}}^-) = \nabla_{{\boldsymbol{x}}_t} \log p({\boldsymbol{x}}_t) \notag \\ &- \gamma({\boldsymbol{x}}_t, {\boldsymbol{c}}^-) (\nabla_{{\boldsymbol{x}}_t} \log p({\boldsymbol{x}}_t|{\boldsymbol{c}}^-) - \nabla_{{\boldsymbol{x}}_t} \log p({\boldsymbol{x}}_t)), \end{align}\] where \(\gamma({\boldsymbol{x}}_t,{\boldsymbol{c}}^-) := \frac{p({\boldsymbol{c}}^-|{\boldsymbol{x}}_t)}{1 - p({\boldsymbol{c}}^-|{\boldsymbol{x}}_t)}\), which can be approximated during the reverse diffusion process to adjust the guidance scale dynamically.

3 Contrastive Classifier-Free Guidance↩︎

3.1 Motivation↩︎

The downside of the NP term in equation 8 can be explained in two different aspects: the sampling distribution involves the inverted probability \(p({\boldsymbol{c}}^-|{\boldsymbol{x}})^{-\gamma}\), which quickly dominates over \(p({\boldsymbol{x}})\) as \(\gamma\) grows [25] and heavily distorts the sampling distribution off the supports of the marginal data distribution. This can also be seen from the NP term \(({\hat{\boldsymbol{\epsilon}}_{{\boldsymbol{c}}^-}}- {\hat{\boldsymbol{\epsilon}}_\varnothing})\) becoming bigger in regions far from \({\boldsymbol{c}}^-\), unnecessarily pushing further away. Figure 2-(b) visualizes this issue with a toy dataset consisting of two classes, with two modes of the blue class and one mode mixed with blues and reds. When NP sampling was done with the red class, the output distribution for the other two modes was entirely shifted from the original location. The ideal behavior of negative guidance is that the guidance strength decreases to zero as \({\boldsymbol{x}}_t\) becomes irrelevant to the given condition, being equivalent to unconditional sampling.

Meanwhile, sampling from the complementary condition \(\neg{\boldsymbol{c}}^-\) [25] resolved this issue by weighting the NP guidance term. However, the biggest limitation is that its sample faithfully avoids \({\boldsymbol{c}}^-\) only when \(p({\boldsymbol{c}}^-|{\boldsymbol{x}}_t)\approx1\) for all \({\boldsymbol{x}}_t\) that satisfies \({\boldsymbol{c}}^-\). If the conditions are not mutually exclusive, the output distribution still includes the data that agrees with \({\boldsymbol{c}}^-\), as the samples are still retained in the third node as a blue class in Figure 2-(b). This limitation is especially crucial for text-to-image (T2I) models (e.g. StableDiffusion [29]).

3.2 Derivation of CCFG↩︎

Similar to redefining positive CFG as a guided sampling that assimilates the denoising direction with the conditioned noise [9], [23], we implement sampling that negates certain conditions with an appropriate objective function. Considering this task as optimizing the data in the sampling process closer to or further away from a given condition, we propose using contrastive loss [14], [30] as the objective function for positive/negative prompt guidance. Contrastive loss assimilates features with the same context while distancing unrelated features. Among various contrastive loss concepts, we found that Noise Contrastive Estimation (NCE) [13] best suits our situation.

Specifically, NCE parameterizes the data by performing logistic regression to discriminate the observed data from the noise data distribution. With a modeled data distribution \(p_\theta({\boldsymbol{x}})\) parameterized by \(\theta\) and pre-defined noise distribution \(q({\boldsymbol{x}})\), the logit of \({\boldsymbol{x}}\) for the logistic regression is defined as \[\begin{align} l_{\theta}({\boldsymbol{x}})&:=\log p_\theta({\boldsymbol{x}})-\log q({\boldsymbol{x}}), \end{align}\] which formulates the NCE loss as \[\begin{align} \mathcal{L}&_{NCE}:=-y\log\sigma(l_\theta(x)) - (1-y)\log(1-\sigma(l_\theta(x))) \notag\\ &=-y\log\frac{p_\theta({\boldsymbol{x}})}{p_\theta({\boldsymbol{x}})+q({\boldsymbol{x}})} - (1-y)\log\frac{q({\boldsymbol{x}})}{p_\theta({\boldsymbol{x}})+q({\boldsymbol{x}})} \end{align}\] where \(y\) is 1 if \({\boldsymbol{x}}\) is the real data and 0 if \({\boldsymbol{x}}\) is sampled from \(q({\boldsymbol{x}})\). Although NCE mostly updates the model parameter with a given positive or noise data, we optimize a datapoint with a pre-trained model that nicely parameterizes \(p_\theta({\boldsymbol{x}})\).

When we guide the denoising of the noised data \({\boldsymbol{x}}_t\) to satisfy \({\boldsymbol{c}}\), we set \(q({\boldsymbol{x}}):=p({\boldsymbol{x}}_{t-1}|{\boldsymbol{x}}_t,\varnothing)\) and parameterize \(p_\theta({\boldsymbol{x}})\) with the pre-trained diffusion model \(p({\boldsymbol{x}}_{t-1}|{\boldsymbol{x}}_t,{\boldsymbol{c}})\) to optimize \({\boldsymbol{\epsilon}}\) with the corresponding NCE loss:

\[\begin{align} \ell^+({\boldsymbol{\epsilon}}) &:= -\log\frac{p_\theta(\hat{{\boldsymbol{x}}}_{t-1}({\boldsymbol{\epsilon}}))}{p_\theta(\hat{{\boldsymbol{x}}}_{t-1}({\boldsymbol{\epsilon}}))+q(\hat{{\boldsymbol{x}}}_{t-1}({\boldsymbol{\epsilon}}))} \label{eq:pos95contr} \end{align}\tag{10}\] Here, \(\hat{{\boldsymbol{x}}}_{t-1}({\boldsymbol{\epsilon}})\) is a mean prediction of \({\boldsymbol{x}}_{t-1}\) from \({\boldsymbol{x}}_t\) with \({\boldsymbol{\epsilon}}\). DDIM [20] states a closed form representation of \(p({\boldsymbol{x}}_{t-1}|{\boldsymbol{x}}_t)\) with a given noise prediction \({\boldsymbol{\epsilon}}\) as a Gaussian distribution \(\mathcal{N}({\boldsymbol{\mu}}({\boldsymbol{x}}_t, {\boldsymbol{\epsilon}}),\sigma_t^2)\) with a certain choice of variance \(\sigma_t^2\) and \[\begin{align} {\boldsymbol{\mu}}({\boldsymbol{x}}_t,& {\boldsymbol{\epsilon}})=\frac{\sqrt{\bar\alpha_{t-1}}}{\sqrt{\bar\alpha_t}}{\boldsymbol{x}}_t \notag\\ &-\left(\frac{\sqrt{\bar\alpha_{t-1}}\sqrt{1-\bar\alpha_t}}{\sqrt{\bar\alpha_t}} - \sqrt{1-\bar\alpha_{t-1}-\sigma_t^2}\right){\boldsymbol{\epsilon}} \end{align}\] Therefore, \(p({\boldsymbol{x}}_{t-1}|{\boldsymbol{x}}_t,\varnothing)=\mathcal{N}({\boldsymbol{\mu}}({\boldsymbol{x}}_t, {\hat{\boldsymbol{\epsilon}}_\varnothing}),\sigma_t^2)\) and \(p({\boldsymbol{x}}_{t-1}|{\boldsymbol{x}}_t,{\boldsymbol{c}})=\mathcal{N}({\boldsymbol{\mu}}({\boldsymbol{x}}_t, {\hat{\boldsymbol{\epsilon}}_{{\boldsymbol{c}}}}),\sigma_t^2)\). By replacing \(\hat{{\boldsymbol{x}}}_{t-1}({\boldsymbol{\epsilon}})\) with \({\boldsymbol{\mu}}({\boldsymbol{x}}_t, {\boldsymbol{\epsilon}})\), the loss function in equation 10 can be simplified as \[\begin{align} \ell^+({\boldsymbol{\epsilon}})= &-\log\frac{e^{-\frac{\|{\boldsymbol{\mu}}({\boldsymbol{x}}_t, {\boldsymbol{\epsilon}})-{\boldsymbol{\mu}}({\boldsymbol{x}}_t, {\hat{\boldsymbol{\epsilon}}_{{\boldsymbol{c}}}})\|_2^2}{2\sigma_t^2}}}{e^{-\frac{\|{\boldsymbol{\mu}}({\boldsymbol{x}}_t, {\boldsymbol{\epsilon}})-{\boldsymbol{\mu}}({\boldsymbol{x}}_t, {\hat{\boldsymbol{\epsilon}}_{{\boldsymbol{c}}}})\|_2^2}{2\sigma_t^2}} + e^{-\frac{\|{\boldsymbol{\mu}}({\boldsymbol{x}}_t, {\boldsymbol{\epsilon}})-{\boldsymbol{\mu}}({\boldsymbol{x}}_t, {\hat{\boldsymbol{\epsilon}}_\varnothing})\|_2^2}{2\sigma_t^2}}} \tag{11} \\ =& -\log\frac{e^{-\tau_t\|{\boldsymbol{\epsilon}}-{\hat{\boldsymbol{\epsilon}}_{{\boldsymbol{c}}}}\|_2^2}}{e^{-\tau_t\|{\boldsymbol{\epsilon}}-{\hat{\boldsymbol{\epsilon}}_{{\boldsymbol{c}}}}\|_2^2}+e^{-\tau_t\|{\boldsymbol{\epsilon}}-{\hat{\boldsymbol{\epsilon}}_\varnothing}\|_2^2}} \tag{12} \end{align}\] Here, \(\tau_t\) is a coefficient that depends on the noise scheduling and the selection of \(\sigma_t\), working as the “temperature" on the logit [14]. Although \(\tau_t\) can be defined as timestep-dependent based on the diffusion reverse process, we found that setting \(\tau_t\) as a constant is sufficient to yield stable and preferable results, while being simple to implement. A more detailed analysis of the selection of \(\tau_t\) can be found in Appendix 8.3.

If we take the derivative of \(\ell^+({\boldsymbol{\epsilon}})\) with respect to \({\boldsymbol{\epsilon}}\) at \({\boldsymbol{\epsilon}}={\hat{\boldsymbol{\epsilon}}_\varnothing}\) and update \({\boldsymbol{\epsilon}}\) with step size \(\gamma_t\), we can obtain the positive guidance term: \[\begin{align} \hat{\boldsymbol{\epsilon}}_{{\boldsymbol{c}}, \ell^+}^{\gamma_t,\tau_t} &:= {\hat{\boldsymbol{\epsilon}}_\varnothing}- \gamma_t\nabla_{{\boldsymbol{\epsilon}}}\ell^+({\hat{\boldsymbol{\epsilon}}_\varnothing})\\ &= {\hat{\boldsymbol{\epsilon}}_\varnothing}+ \frac{2\omega_t}{1+e^{-\tau\|{\hat{\boldsymbol{\epsilon}}_\varnothing}-{\hat{\boldsymbol{\epsilon}}_{{\boldsymbol{c}}}}\|_2^2}}({\hat{\boldsymbol{\epsilon}}_{{\boldsymbol{c}}}}- {\hat{\boldsymbol{\epsilon}}_\varnothing}) \label{eq:guidance95pos} \end{align}\tag{13}\] Here, \(\omega_t:=\tau_t\gamma_t\) governs the overall guidance strength, which works similarly to the guidance scale in naive CFG.

Similarly, to utilize the same objective function to avoid \({\boldsymbol{c}}\), we set \(q({\boldsymbol{x}})\) and \(p_\theta({\boldsymbol{x}})\) the same as above but treat \({\boldsymbol{x}}_t\) as a sample from \(q({\boldsymbol{x}})\). When we perform gradient descent on \({\hat{\boldsymbol{\epsilon}}_\varnothing}\) to minimize the following loss term, we get \[\begin{align} \ell^-({\boldsymbol{\epsilon}}) &:= -\log\frac{q(\hat{{\boldsymbol{x}}}_{t-1}({\boldsymbol{\epsilon}}))}{p_\theta(\hat{{\boldsymbol{x}}}_{t-1}({\boldsymbol{\epsilon}}))+q(\hat{{\boldsymbol{x}}}_{t-1}({\boldsymbol{\epsilon}}))} \\ &= -\log\frac{e^{-\tau_t\|{\boldsymbol{\epsilon}}-{\hat{\boldsymbol{\epsilon}}_\varnothing}\|_2^2}}{e^{-\tau_t\|{\boldsymbol{\epsilon}}-{\hat{\boldsymbol{\epsilon}}_{{\boldsymbol{c}}}}\|_2^2}+e^{-\tau_t\|{\boldsymbol{\epsilon}}-{\hat{\boldsymbol{\epsilon}}_\varnothing}\|_2^2}} \end{align}\] which results in the following update: \[\begin{align} \hat{\boldsymbol{\epsilon}}_{{\boldsymbol{c}},\ell^-}^{\gamma_t,\tau_t} &:= {\hat{\boldsymbol{\epsilon}}_\varnothing}- \gamma_t\nabla_{{\boldsymbol{\epsilon}}}\ell^-({\hat{\boldsymbol{\epsilon}}_\varnothing}) \\ &= {\hat{\boldsymbol{\epsilon}}_\varnothing}- \frac{2\omega_t e^{-\tau_t\|{\hat{\boldsymbol{\epsilon}}_\varnothing}-{\hat{\boldsymbol{\epsilon}}_{{\boldsymbol{c}}}}\|_2^2}}{1+e^{-\tau_t\|{\hat{\boldsymbol{\epsilon}}_\varnothing}-{\hat{\boldsymbol{\epsilon}}_{{\boldsymbol{c}}}}\|_2^2}}({\hat{\boldsymbol{\epsilon}}_{{\boldsymbol{c}}}}- {\hat{\boldsymbol{\epsilon}}_\varnothing}). \label{eq:guidance95neg} \end{align}\tag{14}\] Explicit derivation of equation 13 and  equation 14 can be found in Appendix 8.2.

Algorithm [alg:contrastive95cfg] congregates the two scenarios and performs deterministic² DDIM sampling to satisfy or remove the given condition \({\boldsymbol{c}}\). Compared to the conventional CFG, an additional weighting coefficient was added to the guidance term of each step. We plot the objective functions and their guidance term of CCFG in the right side of Figure 3 as the difference between \({\hat{\boldsymbol{\epsilon}}_\varnothing}\) and \({\hat{\boldsymbol{\epsilon}}_{{\boldsymbol{c}}}}\) changes. As \(\|{\hat{\boldsymbol{\epsilon}}_{{\boldsymbol{c}}}}-{\hat{\boldsymbol{\epsilon}}_\varnothing}\|\) increases, the positive guidance term approximates a linear function as the original CFG. On the other hand, the negative contrastive loss and its guidance term flatten out to 0 as \(\|{\hat{\boldsymbol{\epsilon}}_{{\boldsymbol{c}}}}-{\hat{\boldsymbol{\epsilon}}_\varnothing}\|\) gets bigger, while the negative guidance of NP diverges. This demonstrates the instability of the NP term, and how CCFG can resolve this issue by canceling the gradient guidance when \({\hat{\boldsymbol{\epsilon}}_\varnothing}\) from the current sample \({\boldsymbol{x}}_t\) is sufficiently unrelated to the condition. The behavior of the objective function is also changed by the choice of \(\tau_t\), as described in Appendix 8.3.

Overall, we list the advantages of the proposed CCFG as follows: First, it faithfully removes unwanted concepts compared to complementary condition-based methods such as DNG. Concurrently, contrary to NP, it minimizes the influence on the sampling odds of unrelated data. Finally, the negative guidance of CCFG is a stand-alone method, while many empirical adjustments of NP require the presence of positive prompts to be applicable [7], [28].

4 Experimental Results↩︎

We tested how CCFG steers the sample to satisfy or exclude certain conditions, along with its effect on the sample quality. We then thoroughly compared its performance to the following baselines: For scenarios where only one of the positive and negative conditions is used, we tested the conventional CFG, CFG++ [23], NP, and DNG. In more complex scenarios involving both positive and negative conditions, we also tested Perp-Neg [7], SLD [28] and SAFREE [31] as empirical improvements of NP in the presence of positive conditions. We conducted experiments across a range of scenarios where CFG is applicable, from simple class-conditioned image generation to text-to-image (T2I) generation.

4.1 Class-conditioned models↩︎

Positive-only and negative-only guidance. To show a clear effect of concept negation and its trade-off between sample quality, we first employed MNIST [32] and CIFAR10 [33], which comprise only 10 classes(e.g., 10% chance of unwanted class in unconditional sampling) where each class is clearly distinctive from one another. We sampled 10,000 images with a positive or negative guidance for a randomly selected class, and calculated their classification accuracy to quantify the alignment to the given positive or negative class. The classification was done by a separate external classifier. Fréchet Inception Distance (FID) [34] between the sampled images and the real data was measured to quantify the overall sampled image quality. See Appendix 9.2 for more details of the experimental settings.

Figure 4 shows the curve of the classification rate against FID with varying guidance scales. For the positive guidance, the curve of CCFG almost lies on the curve of the original CFG, indicating that positive CCFG behaves similarly to the widely-used guidance term. The curve of the negative CCFG is located at the bottom left in both MNIST and CIFAR10, outperforming the NP and DNG. In contrast to NP, which corrupts the image quality and DNG, which fails to thoroughly remove the unwanted classes, CCFG achieve similar or better precision with higher sample quality.

Guidance with positive and negative prompts. Beyond datasets with categorically distinct classes (e.g., MNIST, CIFAR10), large-scale class-conditional settings such as ImageNet-1k [35] contain many fine-grained pairs whose features are strongly entangled (e.g., malalmute vs. siberian husky). This motivates evaluating guidance schemes for a task that (i) pull samples toward a desired class while (ii) repelling them from a closely related, potentially confounding class. We curated 100 fine-grained class pairs with strong visual proximity \((c^+, c^-)\) from ImageNet-1k(See Appendix 9.2 for the construction) and generated 100 images for each pair, applying positive guidance toward \(c^+\) while applying negative guidance to \(c^-\). We report (i) Positive accuracy: fraction of samples classified as \(c^+\) by an external classifier; (ii) Negation ratio: fraction not classified as \(c^-\); and (iii) Overall accuracy: the product of the two, simultaneously capturing the alignment and exclusion. We report Aesthetic Score [36] over all generated samples to quantify perceptual quality.

The table in Figure 4 summarizes the trade-off between class alignment, successful negation, and perceptual quality. CCFG attains the highest overall accuracy across pairs, indicating the most faithful target-class alignment while reliably excluding the forbidden class. NP, whose inverted weighting can over-repel, shows degradation of the positive accuracy and the perceptual quality. DNG, whose usage of complementary-conditions can under-repel when classes overlap, shows the lowest negation ratio. Perp-Neg, while demonstrating the highest negation ratio, suffers minimum positive accuracy compared to SLD, DNG and CCFG. Meanwhile, CCFG obtains the highest positive accuracy and near-perfect negation without sacrificing perceptual quality, demonstrating that CCFG’s contrastive weighting intrinsically attenuates negative guidance when the sample is far from \(c^-\), preventing unnecessary distributional shift.

4.2 Text-conditioned models↩︎

Now we test the performance of CCFG in the context of T2I generation, where the conditioning signal is much complex and intertwined. The main results were obtained from StableDiffusion 1.5(SD1.5) and SDXL [37], where additional experiments on a flow-matching model of StableDiffusion 3 [38] are in Appendix 8.4. See Figure 1 and Figure 13 for the examples of T2I generation scenarios with CCFG using positive and negative prompts.

Guidance with positive prompts only. Tab. 1 lists the performance of CFG and CCFG on GenEval and DPGBench, the text-to-image generation benchmarks with given text prompts to follow. CCFG showed similar or better performance to CFG in most of the subtasks, and achieved a slightly higher overall score in both benchmarks and a larger model scale on SDXL.

Guidance with positive and negative prompts. Figure 5 illustrates the result of different negative sampling algorithms from SD1.5 with several hand-crafted positive and negative prompt pairs. We observed that due to the characteristic of the overlapping text condition, the calculated guidance scale of DNG was insufficient to faithfully remove the concept in the negative prompt(e.g., cane in case (e)). The exaggerated negative guidance of NP induces sample bias(e.g., negative prompt “strawberries" removed any red fruit in case (b)) and even hampers the affinity to the positive prompt(e.g.,”tree" and “traveler" in cases (d) and (e)). We also observed that the quality of the images can be heavily degraded, with unrecognizable contents and artifacts for cases (a) and (c). While Perp-Neg and SLD can integrate negative and positive guidance to reduce conflicts with the positive prompt, they may still fail to suppress unwanted features without careful hyperparameter search. CCFG has successfully eliminated the concept of negative prompts while maintaining positive prompts and image quality. See Appendix 8.4 for a quantitative analysis of these case studies.

For a more thorough evaluation beyond case studies, we tested the behavior of CCFG and other baselines on carefully designed benchmarks. Nonetheless, very few benchmarks provide or evaluate the output image with both positive and negative prompts; most focus on how positive prompts are suppressed by the opposing negative prompts [28].Therefore, we designed to build a negative prompt that “overlaps but doesn’t conflict" with its corresponding positive prompt, based on source and target prompt pairs from known image editing benchmarks. Specifically, for given source and target prompt pairs in PIE-Bench [39], we compare the two prompts and utilize their mismatching objects, features, or style. This disagreement is utilized as a negative prompt, and its corresponding positive prompt is set to one of the two original prompts that doesn’t contain the disagreement. We refer to this dataset as PIE-Bench-Neg, where a more detailed construction process can be found in Appendix 9.3.

We used various text-alignment metrics of CLIP cosine similarity [40], ImageReward [41], and HPS-v2 [42] to evaluate the faithfulness to the given positive or negative prompts. However, these metrics have the limitation of conflating image quality, prompt alignment, and human preference in a single value(e.g., the score with negative prompts can decrease by not the removal of unwanted features, but simply image corruption). To address this limitation, we adopt the VLM-as-a-judge framework [43], [44] to perceptually evaluate the prompt alignment on a scale of 0 to 1 (See Appendix 9.3 for details). To solely quantify the quality of the image, we measured the Aesthetic Score of the samples.

Tab. 2 shows the performance of CCFG and other baselines on our PIE-Bench-Neg benchmark. DNG shows the highest alignment scores with the negative prompt due to weak negative guidance, and NP shows degradation in both alignment with the positive prompt and the Aesthetic score. The drawback of these two methods led to lower VLM\(_{all}\), which is the accuracy for both positive prompt alignment and negative prompt removal according to the VLM. Meanwhile, CCFG had a better trade-off between two different guidance, and achieves more output images that perceptually satisfy the given conditions with a better image quality; overall, it showed comparable or better performance with empirical baselines of Perp-Neg and SLD. We conclude that CCFG stands as a flexible and competitive method in the T2I generation task, on top of its broader applicability for negative-only guidance as discussed in Section 4.1.

5 Conclusion↩︎

In this work, we proposed CCFG, a sampling method to guide diffusion samples to satisfy or repel the given condition by optimizing the denoising direction with contrastive loss. Through the experiments on various datasets, we showed that CCFG resembles the positive guidance of the traditional CFG and resolves the downsides of the previous negative guidance methods, resulting in high-quality samples while faithfully aligning or avoiding specific attributes. Despite its effective computational overload and performance, one limitation could be the absence of an analytic closed-form sampling distribution corresponding to the proposed CCFG sampling. It would be a possible future work to propose negative guidance that allows a probabilistic interpretation while preserving the suggested advantages of CCFG. We believe that CCFG can be easily integrated and benefit various downstream applications and modalities that require negative sampling.

Acknowledgements↩︎

This work was supported by the National Research Foundation of Korea under Grant RS-2024-00336454. This research was supported by the AI Computing Infrastructure Enhancement (GPU Rental Support) User Support Program funded by the Ministry of Science and ICT (MSIT), Republic of Korea (RQT-25-120217). This research was supported by the Advanced GPU Utilization Support Program funded by the Government of the Republic of Korea (Ministry of Science and ICT) (02-26-01-0404).

Impact Statement↩︎

This paper presents work on image generative models, and we acknowledge that generative models can produce misleading, biased, or not-safe-for-work visual content, which may amplify misinformation, prejudice, or privacy violations.

6 CCFG on flow-matching models↩︎

Our derivation of CCFG leverages the distribution of the next denoised sample with different conditions, which came from the stochastic sampling process, On the other hand, recent text-to-image models such as StableDiffusion 3 [38] are constructed within the flow-matching framework, where generation is modeled as the solution of an Ordinary Differential Equation(ODE) driven by a learned velocity field. Unlike diffusion models based on stochastic forward–reverse processes, flow-matching models employ deterministic dynamics of equation 15 over the time interval \([0, 1]\) that transport Gaussian noise toward the data distribution.

\[d{\boldsymbol{x}}_t = {\boldsymbol{v}}_{\theta}({\boldsymbol{x}}_t, t)dt, \quad {\boldsymbol{x}}_0 \sim \mathcal{N}(0, {\boldsymbol{I}}). \label{eq:sd395ode}\tag{15}\]

That said, according to the Fokker-Planck equation, there exists a set of stochastic differential equations that has the same marginal distribution \(p({\boldsymbol{x}}_t)\) as equation 15 ,

\[\begin{align} d{\boldsymbol{x}}_t &= \left({\boldsymbol{v}}_{\theta}({\boldsymbol{x}}_t, t)+\frac{t\sigma_t^2}{2(1-t)}\left({\boldsymbol{v}}_{\theta}({\boldsymbol{x}}_t, t)-\frac{1}{t}{\boldsymbol{x}}_t\right)\right)dt + \sigma_td\mathbf{B}_t, \quad {\boldsymbol{x}}_0 \sim \mathcal{N}(0,\mathbf{I}), \label{eq:sd395sde} \end{align}\tag{16}\]

where \(\sigma_t\) is an arbitrary choice of time-dependent diffusion coefficient. The stochasticity in equation 16 enables the same analogy that we derived CCFG in diffusion models in Section 3.2. Algorithm 6 describes the deterministic sampling process on flow-matching models with positive and negative prompt guidances by CCFG.

We’ve conducted the same experiments as those done in Section 4.2, where we tested CCFG on GenEval, DPGBench, and PIE-Bench-Neg. Note that since other baselines in the main manuscript were not proposed for the flow-matching model, we compared CCFG with a naive CFG, NP, and CFG-zero [45] as an alternative for CFG on flow-matching models. Table 3 and Table 4 show that CCFG achieved both positive prompt alignment and negative prompt removal with a similar trend in diffusion models, demonstrating the generalizability of CCFG across different model types. See Figure 13 for the example of CCFG sampling on StableDiffusion 3.

7 Discussion↩︎

7.1 The behavior of NP on different diffusion sampling scenarios↩︎

So far, we’ve shown the theoretical pitfall of the NP and demonstrated its negative effect on the sample distribution and its quality. Meanwhile, we observed that the criticality of NP’s drawback has been mitigated as the learned data distribution and the nature of the used condition get more complex: the NP term completely shifts the sample distribution out from the marginal support in the toy example of Figure 2, but often produced reasonable images for T2I generation tasks. Indeed, many off-the-shelf implementations for the T2I generation model have used this guidance to this day for concept negation [4], [7], [25], [26].

We insist that this phenomenon occurs for the following reasons: First, as the possible conditions become more diverse and their relationship with the data distribution becomes more complex, it becomes harder for the model to accurately parameterize the entire unconditional and conditional distribution. It is well known that many other T2I models perform relatively poorly for pure unconditional generation, and extra distribution sharpening with CFG is almost essential for acceptable image quality. When this happens, even an unbounded guidance term can enhance the sample, until it overshoots and introduces artifacts. For more simplified scenarios like MNIST, where the model can accurately learn the target data distribution, the downside of NP becomes more evident. The second reason comes from \(p({\boldsymbol{c}}^-|{\boldsymbol{x}})^{-\gamma}\), the factor that NP introduced to the sampling distribution. In a class-conditioned dataset where each class doesn’t overlap too much, \(p({\boldsymbol{c}}^-|{\boldsymbol{x}})\) can be close to 1 if the given image is sampled from class \({\boldsymbol{c}}^-\). This contrasts \(p({\boldsymbol{c}}^-|{\boldsymbol{x}})^{-\gamma}\) with different \({\boldsymbol{x}}\) significantly and assigns a high probability for the region with low \(p({\boldsymbol{c}}^-|{\boldsymbol{x}})\), even for \({\boldsymbol{x}}\) with low marginal probability. On the other hand, a single image can be described by numerous different texts, therefore the magnitude of \(p(c|{\boldsymbol{x}})\) for an image-text pair is small no matter how well \(c\) describes the given image.

Nonetheless, we believe that the fundamental flaws of NP have been already demonstrated with various experiments, and a risk of failure still remains for T2I models.

7.2 Comparison with attention-based guidance methods↩︎

As an alternative to traditional CFG, recent methods [46], [47] have begun to investigate the implementation of negative prompting via manipulation of the attention space, which are often employed for more recent models such as distilled or few-step models. While we acknowledge these approaches, we believe that manipulating attention exploits different aspects of diffusion models from ours, and thus can be applied in parallel and considered separately. Moreover, attention-based methods rely more on empirical observations, while our method was based on probabilistic motivation.

8 Additional results↩︎

8.1 CCFG in perspective of guided sampling on the posterior mean↩︎

In the suggested algorithm of CCFG in Alg. [alg:contrastive95cfg], we applied the suggested contrastive loss to optimize the denoising direction \({\hat{\boldsymbol{\epsilon}}_\varnothing}({\boldsymbol{x}}_t)\), with the same spirit of the conventional CFG that modifies \({\hat{\boldsymbol{\epsilon}}_\varnothing}({\boldsymbol{x}}_t)\) directly. This enabled us to conduct a fair comparison with the conventional CFG using an equal guidance scale.

Meanwhile, recall that the motivation for constructing CCFG was to pose positive and negative prompt sampling as guided sampling similar to CFG++ [10], which optimizes the posterior mean \({\hat{\boldsymbol{x}}_\varnothing}({\boldsymbol{x}}_t)\) with the iteration of equation 7 . In order to follow this scheme, one would have to use the null noise \({\hat{\boldsymbol{\epsilon}}_\varnothing}({\boldsymbol{x}}_t)\) in the renoising process, as shown in lines 9\(\sim\)​10 of Alg. 7.

Here, we show that these two different implementations can be equivalent with appropriate guidance scale scheduling. Considering a single DDIM reverse sampling step, the obtained \({\boldsymbol{x}}_{t-1}\) from \({\boldsymbol{x}}_t\) by Alg. 7 can be written as follows: \[\begin{align} {\boldsymbol{x}}_{t-1} = \frac{1}{\sqrt{\alpha_t}}{\boldsymbol{x}}_t + \Big( \sqrt{1 - \bar\alpha_{t-1}} - \frac{\sqrt{1 - \bar\alpha_t}}{\sqrt{\alpha_t}} \Big){\hat{\boldsymbol{\epsilon}}_\varnothing}- \rho_t \lambda \frac{\sqrt{1 - \bar\alpha_t}}{\sqrt{\alpha_t}} ({\hat{\boldsymbol{\epsilon}}_{{\boldsymbol{c}}}}- {\hat{\boldsymbol{\epsilon}}_\varnothing}). \label{eq:ccfgpp95equiv} \end{align}\tag{17}\] In contrast, in Alg. [alg:contrastive95cfg], the iteration reads \[\begin{align} {\boldsymbol{x}}_{t-1} = \frac{1}{\sqrt{\alpha_t}}{\boldsymbol{x}}_t + \Big( \sqrt{1 - \bar\alpha_{t-1}} - \frac{\sqrt{1 - \bar\alpha_t}}{\sqrt{\alpha_t}} \Big){\hat{\boldsymbol{\epsilon}}_\varnothing}- \omega_t\lambda \left( \sqrt{1 - \bar\alpha_{t-1}} - \frac{\sqrt{1 - \bar\alpha_t}}{\sqrt{\alpha_t}} \right) ({\hat{\boldsymbol{\epsilon}}_{{\boldsymbol{c}}}}- {\hat{\boldsymbol{\epsilon}}_\varnothing}). \label{eq:ccfg95equiv} \end{align}\tag{18}\] Therefore, by setting \[\begin{align} \rho_t = \omega_t\left( 1 - \frac{\sqrt{\alpha_t}\sqrt{1 - \bar\alpha_{t-1}}}{\sqrt{1 - \bar\alpha_t}} \right), \end{align}\] we can show the equivalence between Alg. [alg:contrastive95cfg] and Alg. 7. Hence, for a direct and fair comparison of CCFG against conventional CFG, we chose CCFG with Alg. [alg:contrastive95cfg] as our implementation.

8.2 Derivation of the guidance term from CCFG objective function↩︎

Note that \[\begin{align} f({\boldsymbol{x}};{\boldsymbol{a}},{\boldsymbol{b}},\tau_t):=&-\log\frac{e^{-\tau_t\|{\boldsymbol{x}}-{\boldsymbol{a}}\|_2^2}}{e^{-\tau_t\|{\boldsymbol{x}}-{\boldsymbol{a}}\|_2^2}+e^{-\tau_t\|{\boldsymbol{x}}-{\boldsymbol{b}}\|_2^2}} \\ \Leftrightarrow \nabla_{{\boldsymbol{x}}}f=&-\frac{2\tau_t({\boldsymbol{a}}-{\boldsymbol{b}})e^{-\tau_t\|{\boldsymbol{x}}-{\boldsymbol{b}}\|_2^2}}{e^{-\tau_t\|{\boldsymbol{x}}-{\boldsymbol{a}}\|_2^2}+e^{-\tau_t\|{\boldsymbol{x}}-{\boldsymbol{b}}\|_2^2}} \label{eq:derivative} \end{align}\tag{19}\] When we put \(({\boldsymbol{x}}={\hat{\boldsymbol{\epsilon}}_\varnothing}, {\boldsymbol{a}}={\hat{\boldsymbol{\epsilon}}_{{\boldsymbol{c}}}}, {\boldsymbol{b}}={\hat{\boldsymbol{\epsilon}}_\varnothing})\) and \(({\boldsymbol{x}}={\hat{\boldsymbol{\epsilon}}_\varnothing}, {\boldsymbol{a}}={\hat{\boldsymbol{\epsilon}}_\varnothing}, {\boldsymbol{b}}={\hat{\boldsymbol{\epsilon}}_{{\boldsymbol{c}}}})\) into equation 19 , we get the positive and negative CCFG guidance term for equation 13 and  equation 14 , respectively.

8.3 Analysis on the selection of CCFG hyperparameters↩︎

In the process of simplifying equation 11 into equation 12 , the explicit expression of \(\tau_t\) in terms of \(\sigma_t\) can be obtained as \[\begin{align} \frac{\|{\boldsymbol{\mu}}({\boldsymbol{x}}_t, {\boldsymbol{\epsilon}})-{\boldsymbol{\mu}}({\boldsymbol{x}}_t, {\hat{\boldsymbol{\epsilon}}_{{\boldsymbol{c}}}})\|_2^2}{2\sigma_t^2} &= \frac{\left(\frac{\sqrt{\bar\alpha_{t-1}}\sqrt{1-\bar\alpha_t}}{\sqrt{\bar\alpha_t}} - \sqrt{1-\bar\alpha_{t-1}-\sigma_t^2}\right)^2}{2\sigma_t^2}\|{\boldsymbol{\epsilon}}-{\hat{\boldsymbol{\epsilon}}_{{\boldsymbol{c}}}}\|_2^2 \\ &:= \tau_t\|{\boldsymbol{\epsilon}}-{\hat{\boldsymbol{\epsilon}}_{{\boldsymbol{c}}}}\|_2^2 \\ \Leftrightarrow \tau_t &:= \frac{\left(\frac{\sqrt{\bar\alpha_{t-1}}\sqrt{1-\bar\alpha_t}}{\sqrt{\bar\alpha_t}} - \sqrt{1-\bar\alpha_{t-1}-\sigma_t^2}\right)^2}{2\sigma_t^2} \label{tau95sigma} \end{align}\tag{20}\] where \(0\leq\sigma_t^2\leq1-\bar\alpha_{t-1}\).

Figure 8-(a) shows the objective functions and the corresponding guidance terms of CCFG as \(\tau_t\) changes. In any \(\tau_t\), the guidance term for positive CCFG approximates a linear function for the original CFG, when \(\|{\hat{\boldsymbol{\epsilon}}_{{\boldsymbol{c}}}}-{\hat{\boldsymbol{\epsilon}}_\varnothing}\|\) is sufficiently large or near 0. The guidance coefficient \(\lambda\) for the positive prompt becomes constant when \(\tau_t\) is 0 or diverges to infinity, making positive CCFG equivalent to the original CFG. In the case of the negative guidance, \(\lambda\) becomes 0 when \(\tau_t\) diverges to infinity, and no negative guidance is applied. As \(\tau_t\) goes to 0, \(\lambda\) again becomes constant, making negative CCFG equivalent to NP. In general, the choice of \(\tau_t\) affects more on the negative CCFG since it governs the range of \(\|{\hat{\boldsymbol{\epsilon}}_{{\boldsymbol{c}}}}-{\hat{\boldsymbol{\epsilon}}_\varnothing}\|\) in which the negative guidance is applied sufficiently.

Figure 8-(b) illustrates this behavior in T2I generation with SD1.5, where too large \(\tau_t\) disables proper negative guidance and too small \(\tau_t\) introduces the drawback of NP(e.g. applying too much negative guidance disrupts following the positive prompt).

Figure 9 shows the performance of CCFG on PIE-Bench-Neg with different selection of \(\omega_t\) and \(\tau_t\). The positive prompt alignment(VLM\(_{pos}\)) and negative prompt removal(VLM\(_{neg}\)) improve as the guidance scale \(\omega_t\) increases, while the overall quality of the image(Aesthetic Score) has a sweet spot at a suitable \(\omega_t\). This trend agrees with a well-known observation of traditional CFG. Meanwhile, as illustrated in Figure 8-(b), a smaller \(\tau_t\) can more thoroughly remove the negative concept, but also hinders the positive prompt alignment.

8.4 Text-to-image sampling with positive and negative prompts↩︎

 Figure 10 shows the quantitative results on five T2I case studies in Figure 5 with CLIP cosine similarity score, ImageReward, HPS-v2, and perceptual accuracy judged by VLM. Here, we show the average score over the five cases for each method. We sampled 200 images for each method and case. The ideal behavior as an effective and safe negative sampling method is to maintain alignment with positive prompts while decreasing alignment with negative prompts, compared to when no negative guidance is applied.

DNG showed the smallest alignment metric decrease between its samples and the given positive prompt, but failed to thoroughly rule out the features in the negative prompt; the alignment with the negative prompt was marginally decreased, and VLM often detected the attributes described in the negative prompt. Meanwhile, NP showed the lowest alignment score not only with the negative prompts, but also for the positive prompts. Also, almost 20% of its output was judged to not satisfy the positive prompt by VLM. While Perp-Neg and SLD can orchestrate the negative guidance with the positive guidance to prevent the misalignment between the positive prompt, they also occasionally fail to remove the unwanted features without careful hyperparameter search. Our proposed CCFG removes the undesired properties enough to be undetected by VLM, and still maintains the agreement with the positive prompts.

In Figure 13, we display additional image generation examples by CCFG and given positive-negative prompt pairs from StableDiffusion 1.5, SDXL, and StableDiffusion 3. Leveraging the ability of CCFG for an effective and stable concept negation, we can resolve certain undesirable features such as objects, motions, and multiple composed features. Furthermore, CCFG can help the model generate minority samples and remove potentially harmful objects by negating potential internal bias or inappropriate contents.

8.5 Possible failure cases↩︎

Figure 11 presents several failure cases of CCFG on text-to-image generation tasks; when the negative condition involves multiple attributes simultaneously, CCFG might suppresses only a subset of the undesired aspects. And when the undesired negative prompt describes subtle continuous attributes, the negative guidance strength of CCFG may be weakened in the process of concept removal, resulting in outputs on the borderline of unwanted features.

9 Experimental details↩︎

9.1 Sampling hyperparameters↩︎

Tab. 5 lists the hyperparameter values used for DNG and CCFG for the results in the main manuscript. Apart from the guidance scale \(\omega\) that is shared by all negative sampling methods we’ve tested, DNG requires a prior for the given condition \(p(c)\), an affine transformation weight \(\tau'\) and bias \(\delta\) for its guidance term calculation. We first took the values from the authors of DNG [25] as a basis, then additionally searched and tuned for the best performance on each dataset. We note that DNG only showed acceptable performances with the heavily exaggerated condition prior \(p(c)\) which are far from the reasonable value(e.g. 0.1 for MNIST and CIFAR10).

9.2 Class-conditioned models↩︎

For the experiments about CCFG on MNIST and CIFAR10, we trained DDPM [16] for each dataset from scratch which enables both conditional and unconditional generation to perform CFG, CCFG, and other sampling methods. The external classifier was imported from the following checkpoints, where the ViT-base is trained with the classification of MNIST and CIFAR10, respectively. The diffusion models were trained with 500 noise timesteps. For the inference, we used deterministic DDIM sampling with NFE=50.

For the experiment on ImageNet-1k, We prepared 100 class pairs with strong visual proximity \((c^+, c^-)\) (e.g., Maltese_dog - Shih-Tzu, Italian_greyhound - whippet, Samoyed - Pomeranian, racer - sports_car, beach_wagon - cab, bathing_cap - shower_cap, garden_spider - wolf_spider) according to WordNet hierarchy [48] of ImageNet labels and the following pipeline:

• For each ImageNet-1k class, we retrieve the corresponding WordNet ID (wnid) and its hypernyms.

• We construct a pool of candidate pairs by including all pairs of classes that share an identical hypernym.

• From this pool, we uniformly sample 100 class pairs to obtain the final benchmark set, while ensuring that every class in the 100 class pairs is unique.

The sampling was done at \(256{\times}256\) using an ImageNet-1k–pretrained DiT-XL/2 [49] by deterministic DDIM sampling with NFE=50. For the external classifier, we used a frozen ResNet50 [50] pretrained on ImageNet-1k using torchvision’s default IMAGENET1K_V2 weights.

9.3 Text-conditioned models↩︎

For the SD1.5 and SDXL image sampling, we used DDIM deterministic sampling with 50 NFE and a guidance scale of 7.5. In the flow-matching model sampling with StableDiffusion 3, we used a deterministic sampling with 28 NFE and a guidance scale of 3.0. In the case of DNG and SLD, we used one of the hyperparameter configurations provided by the authors.

To build the PIE-Bench-Neg benchmark, we processed each pair of source and target prompts in PIE-Bench [39] according to the given affiliation that describes the editing task:

• \(\texttt{add\_object}\): Compared to the source prompt(e.g., “a bee illustration on a paper"), the newly added objects in the target prompt are listed in the attribute of \(\texttt{aspect\_mapping}\)(e.g.,”leaves, flower"). These objects become the negative prompt, and the source prompt is the positive prompt.

• \(\texttt{delete\_object}\): The objects from the source prompt(e.g., “a wathet bird sitting on a branch of yellow flowers") removed in the target prompt(e.g.,”a wathet bird sitting on a branch") are listed in the attribute of \(\texttt{aspect\_mapping}\)(e.g., “flowers"). These objects become the negative prompt, and the target prompt is the positive prompt.

• \(\texttt{change\_object, change\_attribute\_content, change\_background}\): The changed appearances from the source prompt(e.g., “fishes and kelp in the ocean") are marked in the target prompt(e.g.,”[sharks] and [flowers] in the ocean"), which becomes the negative prompt(e.g., “sharks, flowers"). The source prompt is used for the positive prompt.

• \(\texttt{change\_attribute\_pose, change\_attribute\_color}\), \(\texttt{change\_attribute\_material}\): These editing tasks require certain features of the objects in the source prompt(e.g., “a brown bear is sleeping with a sign that says no sleep") to be changed, and the mapping between them can be found in key-value pairs of \(\texttt{aspect\_mapping}\)(e.g.,”bear":“black",”sign":“white"). We combine the objects and their target editing features to construct a negative prompt(e.g.,”black bear, white sign"), and the original source prompt is used as the positive prompt.

• \(\texttt{change\_style}\): The target style is described in the attribute of \(\texttt{blended\_words}\)(e.g., “baroque, watercolor"). These styles become the negative prompt, and the source prompt(e.g.,”a cartoon giraffe sitting down on a white background") is the positive prompt.

In total, we prepared 560 positive prompts, paired with the corresponding negative prompt that is correlated but not entirely opposed to the positive prompt.

The performance of CCFG in PIE-Bench-Neg prompts from each of these category is shown in Figure 12. Here, we observed that CCFG is relatively less effective on the series of “change_attribute_XX" categories, which tend to yield relatively lower positive prompt VLM scores(e.g., change_attr_color) or higher negative prompt VLM scores(e.g., change_attr_pose). This can be because the negative prompts from these categories require removing specific attributes of something explicitly contained in the positive prompt, therefore conveying overlapping concepts and making the task more difficult. We also found that having nouns as negative prompts(e.g., change_object) or adjectives(e.g., change_style) shows similar performances.

To evaluate a perceptual agreement of the image with the given prompt, separated from the image quality, we employed a Vision-Language Model of GPT4o-mini [51]. With the VLM, we decompose the given prompt and check the given image’s agreement with each component to eventually rate the image. Prompt [prompt_eval] was used to evaluate the perceptual accuracy of the given image to follow the text prompt using VLM. The last line of the output is interpreted as a fraction between 0 and 1, where 1 is the highest possible agreement score. Similar to how we calculated the overall accuracy in Section 4.1, we multiply the VLM score for the positive and the negative prompt as VLM\(_{_all}\) to quantify how the model output matches for both prompt conditions.

References↩︎