Training-Free Debiasing of Diffusion Models via CLIP-Guided Denoising Optimization
July 01, 2026
Abstract
Text-to-image diffusion models achieve impressive visual quality, yet demographic bias remains a challenge, as neutral prompts consistently produce stereotypical representations across gender and race. Existing approaches remain limited by costly retraining or by inference-time interventions that often degrade image quality and semantic alignment. We propose Text Embedding Steering (TES), a training-free framework that mitigates demographic bias by directly optimizing conditional text embeddings during the diffusion process. We show that a two-stage strategy — early-stage global alignment followed by iterative denoising-time refinement with CLIP-based feedback — enables stable and controllable attribute steering without modifying model parameters. Extensive experiments on Stable Diffusion demonstrate that TES outperforms existing training-free baselines in fairness while maintaining competitive image quality. These results highlight that inference-time text embedding optimization is a practical and scalable solution for fairness-aware generation in diffusion models.
1 Introduction↩︎
Recent advances in text-to-image (T2I) models [1]–[4] have enabled the generation of highly realistic and diverse images from natural language prompts. Models such as Stable Diffusion [1] have demonstrated strong performance across a wide range of applications, including content creation, design assistance, and visual storytelling. As these systems become increasingly integrated into real-world applications, ensuring the reliability and trustworthiness of generated content has emerged as an important challenge. One critical issue is demographic bias. Despite receiving neutral prompts, T2I models often generate stereotypical representations associated with attributes such as gender, race, and age. Such biases can reinforce existing social stereotypes and lead to systematic underrepresentation of certain demographic groups. In real-world applications such as advertising, automated content creation, and educational materials, biased outputs can reinforce stereotypes and influence how users perceive social roles. Moreover, when these models are integrated into downstream pipelines such as data augmentation or content generation systems, bias can accumulate and propagate, reducing the reliability of AI systems.
To address this issue, prior work [5]–[9] has explored model retraining, fine-tuning, or inference-time interventions. While retraining-based approaches can improve fairness, they are computationally expensive and difficult to apply to already deployed models, limiting their practicality. Inference-time methods provide a more practical alternative by mitigating bias during generation without modifying model parameters. However, many existing approaches rely on heuristic prompt engineering or latent-space manipulation, which can be difficult to interpret and control. More recent methods attempt to intervene in internal representations through embedding-level adjustments or learned transformations [7], [10]–[12]. While these approaches show promising results, they often depend on predefined directions or static mappings, limiting their ability to provide dynamic and context-aware adjustments during generation.
In this work, we propose Text Embedding Steering (TES), a training-free framework that optimizes text embeddings during the diffusion process. Our key observation is that demographic attributes are largely determined in the early stages of denoising, where the global structure of the image is formed. Motivated by this observation, TES performs an early-stage embedding update to guide the overall generation direction, followed by iterative refinement using CLIP-based feedback from intermediate clean-image estimates. We evaluate TES on a diverse set of profession-based prompts and show that it effectively reduces gender and race bias while preserving image quality and semantic alignment. Importantly, improvements in metrics correspond to more balanced and realistic representations of social roles, leading to more reliable and trustworthy outputs in real-world applications. An overview of our approach is illustrated in Figure 1. Given an input prompt, our method dynamically updates the text embedding during the diffusion process, enabling continuous and context-aware control of generated attributes.
Our contributions are summarized as follows:
We propose TES, a training-free framework that mitigates demographic bias by continuously optimizing text embeddings without requiring model retraining.
We identify the importance of early-stage intervention and introduce a two-stage optimization strategy for stable and consistent control.
We demonstrate that TES achieves a strong balance between fairness and image quality, and improves the reliability of generated content in practical deployment scenarios.
2 Related Work↩︎
2.1 Bias in Text-to-Image Diffusion Models↩︎
Recent studies have shown that text-to-image (T2I) diffusion models [13]–[18] inherit and often amplify social biases, resulting in demographically skewed or stereotypical visual representations. In particular, profession-related prompts exhibit strong demographic associations, where occupations such as CEO or engineer are predominantly associated with male representations [15]–[17], [19], [20]. Prior work further shows that these biases affect not only demographic frequencies but also generation quality and semantic fidelity across groups [14], [21]. These findings suggest that demographic bias in T2I models is systematic rather than incidental, motivating the need for controllable and reliable debiasing methods.
2.2 Training-Based Debiasing Methods↩︎
Training-based approaches mitigate bias through data rebalancing, fine-tuning, or fairness-aware optimization [6], [7], [22]–[27]. Representative methods include SCoFT [22], Fair Mapping [7], Balancing Act [6], FairRAG [23], AITTI [24], and LightFair [25]. While effective, these methods require access to training data or model parameters, often involving costly fine-tuning procedures. As a result, they are difficult to apply to closed-source or already deployed models, limiting their practical usability and motivating training-free alternatives.
2.3 Training-Free Debiasing Methods↩︎
To overcome these limitations, recent work [28]–[36] has explored training-free debiasing strategies that operate entirely at inference time. These methods can be broadly categorized into three groups. Prompt-based approaches modify input prompts to encourage more balanced generations [28], [29], [31]. While simple and easy to deploy, they are heuristic and often sensitive to prompt wording and sampling conditions. Sampling-based methods instead intervene during the denoising process through mechanisms such as attention guidance, cross-attention editing, switching strategies, or latent guidance [32]–[35]. For example, FairGen [35] discovers demographic control directions through an additional learning stage and applies them during inference. Although this enables plug-and-play debiasing, it still requires a separate adapter training stage. Embedding-based methods instead operate directly in the representation space. FairImagen [12] mitigates demographic bias by projecting prompt embeddings onto a predefined fairness-aware subspace before generation. However, these methods rely on fixed embedding transformations or predefined control directions that remain unchanged throughout sampling.
Unlike previous approaches, TES formulates debiasing as a dynamic optimization problem over conditional text embeddings. Rather than relying on separately trained adapters, or predefined demographic control directions, TES continuously updates the conditional text embedding using CLIP-based feedback from intermediate clean-image estimates, enabling adaptive and context-aware demographic control throughout denoising without retraining or auxiliary modules.
3 Preliminaries↩︎
3.1 Diffusion Models↩︎
Denoising Diffusion Probabilistic Models (DDPM) [37] generate images by progressively transforming Gaussian noise into data samples through a sequence of denoising steps. Starting from a random latent variable \(x_T \sim \mathcal{N}(0, I)\), the model iteratively removes noise to obtain a clean sample. At each timestep \(t\), a neural network \(\epsilon_\theta\) predicts the noise component in the latent variable, and the sample is updated according to a predefined diffusion schedule: \[x_{t-1} = f(x_t, \epsilon_\theta(x_t, t)).\] In text-to-image diffusion models, the denoising network is conditioned on text embeddings obtained from a pretrained text encoder. The conditional noise prediction can be written as \(\epsilon_\theta(x_t, t, e)\), where \(e\) denotes the text embedding that encodes the input prompt. Latent diffusion models such as Stable Diffusion [1] further improve efficiency by performing diffusion in a compressed latent space.
Among different sampling methods, DDPM follows a stochastic reverse process, while Denoising Diffusion Implicit Models (DDIM) [38] provide a deterministic formulation: \[x_{t-1} = \sqrt{\bar{\alpha}_{t-1}} \, \hat{x}_0 + \sqrt{1 - \bar{\alpha}_{t-1}} \, \epsilon_\theta(x_t, t).\] Compared to stochastic sampling, DDIM produces more stable intermediate states and more predictable denoising trajectories. This property is particularly beneficial for optimization-based generation methods, since stable trajectories enable reliable feedback and iterative updates during sampling. Following prior inference-time optimization approaches, we adopt DDIM sampling throughout this work.
3.2 Clean Image Estimation↩︎
Given a noisy latent \(x_t\) and the predicted noise \(\epsilon_\theta(x_t, t)\), the clean sample \(\hat{x}_0\) can be estimated as: \[\label{eq:x0} \hat{x}_0 = \frac{x_t - \sqrt{1 - \bar{\alpha}_t} \, \epsilon_\theta(x_t, t)}{\sqrt{\bar{\alpha}_t}}.\tag{1}\] This formulation provides an estimate of the final denoised image at each timestep. Compared to the noisy latent \(x_t\), the predicted clean sample \(\hat{x}_0\) offers a more interpretable approximation of the underlying image content and often reveals semantic structure before the denoising process is completed. As a result, clean-image estimates have been widely used in diffusion-based guidance and optimization methods as a proxy for image-level supervision. In this work, we leverage \(\hat{x}_0\) as a semantic feedback signal for updating conditional text embeddings during inference. By evaluating intermediate clean-image estimates rather than noisy latents, the optimization objective can operate directly on semantically meaningful visual attributes.
4 Method↩︎
4.1 Overview↩︎
Given a neutral text prompt \(T_{\text{base}}\), such as “a photo of a CEO”, our goal is to generate images that preserve the semantic content of the prompt while reducing demographic imbalance in the generated outputs. Following prior work, we formulate fairness-aware generation as a demographic distribution alignment problem, where the generated demographic distribution is encouraged to match a predefined target distribution. In this paper, we mainly adopt a uniform target distribution across demographic groups, although TES can support arbitrary target distributions depending on the application. Figure 2 illustrates the overall TES pipeline with early-stage alignment and denoising-time refinement.
Let \(G_\theta\) denote a frozen text-to-image diffusion model and \(e_0\) the conditional text embedding produced by the text encoder. Instead of updating the model parameters, TES optimizes only an additive perturbation to the conditional text embedding, \[e_t = e_0 + \Delta e,\] while keeping the diffusion backbone, text encoder, and sampling procedure fixed. TES dynamically updates this embedding throughout denoising using CLIP-based feedback from intermediate clean-image estimates. This enables adaptive demographic control without retraining or auxiliary modules, while preserving the semantic content of the original prompt.
4.2 CLIP-Guided Embedding Optimization↩︎
At each optimization step, TES reconstructs an intermediate clean image and evaluates it using CLIP [39]. Let \(I_t\) denote the decoded clean-image estimate at timestep \(t\). The target attribute used during optimization is determined by the automatic target-assignment procedure, with details provided in the supplementary material. Consequently, the optimization objective is derived from the predefined target distribution rather than from manually specified demographic labels.
Embedding Objective. We define an attribute alignment objective using a target attribute description \(T_{\text{target}}\) and an opposing attribute description \(T_{\text{opp}}\). For gender debiasing, these correspond to descriptions such as “a male person” and “a female person.” Let \(s_{\text{target}}=\mathrm{sim}(I_t,T_{\text{target}})\) and \(s_{\text{opp}}=\mathrm{sim}(I_t,T_{\text{opp}})\) denote the CLIP cosine similarities to the target and opposite attribute prompts, respectively. The attribute loss is \[\mathcal{L}_{\text{attr}} = -\log \frac{\exp(s_{\text{target}}/\tau)}{\exp(s_{\text{target}}/\tau)+\exp(s_{\text{opp}}/\tau)},\] where \(\tau\) is the temperature. To preserve the semantic content of the original prompt, we further optimize \[\mathcal{L}_{\text{prof}}=1-\mathrm{sim}(I_t,T_{\text{base}}).\] To prevent excessive deviation from the original embedding, we additionally regularize the embedding perturbation, \[\mathcal{L}_{\text{reg}}=\|\Delta e\|_2^2.\] The complete objective is \[\mathcal{L}_{\text{total}}=\lambda_{\text{attr}}\mathcal{L}_{\text{attr}}+\lambda_{\text{prof}}\mathcal{L}_{\text{prof}}+\gamma\mathcal{L}_{\text{reg}}.\] This objective encourages the generated image to move toward the predefined demographic distribution while preserving the semantic content of the original prompt and preventing excessive embedding drift.
Intermediate Clean-Image Supervision. Rather than computing CLIP supervision directly from noisy latent representations, TES evaluates the reconstructed clean image estimated at each denoising step. Given the noisy latent \(x_t\) and the predicted noise \(\epsilon_\theta(x_t,t,e_t)\), we first estimate the clean sample \(\hat{x}_0\), decode it into image space, and compute the CLIP-guided objective using the resulting image \(I_t\). Optimization is intentionally deferred from the earliest denoising steps because the reconstructed clean-image estimate remains unreliable under extremely high noise levels. Instead, embedding updates are activated only within a selected timestep window, where the reconstructed image already captures meaningful semantic structure while the global generation trajectory remains sufficiently flexible for effective intervention.
4.3 Two-Stage Optimization↩︎
TES performs embedding optimization in two complementary stages: an early-stage alignment followed by denoising-time refinement.
Stage 1: Early-stage Alignment. The first stage provides a coarse global correction before demographic attributes become firmly established. Rather than directly modifying the diffusion model, TES performs gradient updates on the conditional text embedding using the CLIP-guided objective, \[\Delta e\leftarrow\Delta e-\eta_t\nabla_{\Delta e}\mathcal{L}_{\mathrm{total}}.\] This initial update biases the global generation trajectory toward the desired demographic distribution before fine-grained visual details emerge.
Stage 2: Denoising-Time Refinement. After the initial alignment, TES continuously refines the conditional embedding during subsequent denoising steps. At each optimization timestep, the embedding is updated using the same CLIP-guided objective computed from the reconstructed clean image. This iterative refinement adapts the embedding to the evolving image content while maintaining the demographic alignment established during the first stage, preventing the generation from drifting back toward biased outcomes.
Active Timestep Window. Embedding updates are activated only within a predefined timestep window, \(t\in[t_{\mathrm{start}},t_{\mathrm{end}})\). The window is chosen to intervene early enough to influence the global generation trajectory, while avoiding the earliest steps where the reconstructed clean-image estimate is still dominated by noise. The update-timing analysis in Figure 3 supports this design, showing that early intervention enables more coherent semantic shifts, whereas later updates mainly affect localized details after the trajectory has been established. Thus, the selected window provides a practical trade-off between semantic reliability and controllability.
Learning-Rate Schedule. Within the active window, TES adopts a timestep-dependent learning rate, \[\eta_t=\eta\cdot\phi(t),\] where \(\eta\) denotes the base learning rate and \(\phi(t)\) is a schedule over the active window. Unless otherwise specified, we employ a linearly increasing schedule, \[\phi(t)=\frac{t_{\mathrm{start}}-t}{t_{\mathrm{start}}-t_{\mathrm{end}}}.\] This schedule applies smaller updates when the reconstructed image is still relatively uncertain and gradually increases the update strength as more reliable semantic information becomes available. The influence of the timestep window and learning-rate schedule is analyzed in the ablation study.
Input: base prompt \(T_{\text{base}}\), initial embedding \(e_0\) Initialize: \(\Delta e \gets 0\), \(x_T \sim \mathcal{N}(0, I)\) compute CFG prediction using \(e = e_0 + \Delta e\) update latent \(x_t \rightarrow x_{t-1}\) estimate clean sample \(\hat{x}_0\) using Eq. 1 decode \(\hat{x}_0\) into image \(I\) compute \(\mathcal{L}_{\mathrm{attr}}\) and \(\mathcal{L}_{\mathrm{prof}}\) form total loss \(\mathcal{L}_{\mathrm{total}}\) update \(\Delta e \gets \Delta e - \eta_t \nabla_{\Delta e}\mathcal{L}_{\mathrm{total}}\) Return: generated image
4.4 Integration with Diffusion Sampling↩︎
TES is integrated into the standard classifier-free guidance pipeline without modifying the diffusion backbone, text encoder, or sampler. At each denoising timestep, the current conditional embedding, \(e_t=e_0+\Delta e\), is first used to compute the standard classifier-free guidance prediction, and the latent is updated accordingly. If the current timestep lies outside the active optimization window, sampling proceeds normally. Otherwise, TES reconstructs the intermediate clean image from the updated latent, evaluates the CLIP-guided objective, and updates the embedding perturbation before proceeding to the next denoising step. Consequently, the embedding used at timestep \(t-1\) incorporates the feedback obtained from the reconstructed image at timestep \(t\). Algorithm [alg:method] summarizes the complete inference-time procedure. Unlike one-shot embedding editing, TES continuously refines the conditional text embedding throughout sampling while leaving all model parameters unchanged. As a result, the proposed framework remains fully training-free and can be directly applied to existing diffusion models without additional retraining or auxiliary modules.
5 Experiments↩︎
| Gender | ||||||||
|---|---|---|---|---|---|---|---|---|
| 2-9 Method | Fairness | Quality | ||||||
| Bias-O\(\downarrow\) | Bias-Q\(\downarrow\) | FD\(\downarrow\) | CLIP-T\(\uparrow\) | CLIP-I\(\uparrow\) | DINO\(\uparrow\) | FID\(\downarrow\) | IS\(\uparrow\) | |
| SD[1] | 0.85 (±0.05) | 1.84 (±0.63) | 0.54 (±0.03) | 29.90 (±0.15) | - | - | 259.36 (±4.81) | 1.23 (±0.03) |
| debias VL[40] | 0.43 (±0.09) | 1.44 (±0.48) | 0.28 (±0.05) | 28.20 (±0.22) | 70.01 (±0.96) | 0.49 (±0.02) | 245.11 (±3.72) | 1.35 (±0.03) |
| UCE[41] | 0.90 (±0.04) | 1.67 (±0.71) | 0.59 (±0.02) | 29.41 (±0.13) | 87.94 (±0.86) | 0.71 (±0.02) | 268.52 (±3.92) | 1.22 (±0.02) |
| EntiGen[29] | 0.42 (±0.03) | 2.10 (±0.38) | 0.25 (±0.02) | 29.25 (±0.16) | 69.22 (±1.12) | 0.49 (±0.02) | 255.01 (±3.60) | 1.24 (±0.02) |
| LightFair[25] | 0.33 (±0.10) | 1.40 (±0.28) | 0.21 (±0.05) | 30.82 (±0.19) | 75.29 (±0.99) | 0.63 (±0.02) | 231.46 (±3.30) | 1.35 (±0.02) |
| TES (Ours) | 0.26 (±0.02) | 1.40 (±0.08) | 0.13 (±0.01) | 30.08 (±0.05) | 75.62 (±0.71) | 0.69 (±0.01) | 226.13 (±2.42) | 1.35 (±0.01) |
| Race | ||||||||
|---|---|---|---|---|---|---|---|---|
| 2-9 Method | Fairness | Quality | ||||||
| Bias-O\(\downarrow\) | Bias-Q\(\downarrow\) | FD\(\downarrow\) | CLIP-T\(\uparrow\) | CLIP-I\(\uparrow\) | DINO\(\uparrow\) | FID\(\downarrow\) | IS\(\uparrow\) | |
| SD[1] | 0.63 (±0.01) | 2.06 (±0.35) | 0.21 (±0.01) | 29.90 (±0.15) | - | - | 259.36 (±4.81) | 1.23 (±0.03) |
| debias VL[40] | 0.49 (±0.03) | 1.91 (±0.92) | 0.14 (±0.01) | 28.15 (±0.26) | 67.42 (±0.96) | 0.46 (±0.02) | 242.78 (±4.21) | 1.33 (±0.03) |
| UCE[41] | 0.50 (±0.03) | 1.95 (±0.37) | 0.16 (±0.01) | 29.44 (±0.12) | 80.46 (±1.13) | 0.64 (±0.02) | 250.57 (±4.49) | 1.23 (±0.03) |
| EntiGen[29] | 0.55 (±0.03) | 3.07 (±0.39) | 0.14 (±0.01) | 28.12 (±0.12) | 65.34 (±1.02) | 0.45 (±0.02) | 253.53 (±3.83) | 1.23 (±0.03) |
| LightFair[25] | 0.40 (±0.03) | 1.82 (±0.44) | 0.11 (±0.01) | 30.26 (±0.16) | 77.47 (±1.05) | 0.53 (±0.03) | 230.59 (±6.53) | 1.35 (±0.01) |
| TES (Ours) | 0.15 (±0.02) | 0.86 (±0.04) | 0.09 (±0.01) | 29.49 (±0.07) | 69.87 (±0.35) | 0.65 (±0.01) | 228.52 (±1.08) | 1.36 (±0.01) |
5.1 Experimental Setup↩︎
We conduct all experiments on Stable Diffusion v2.1 [1] using DDIM sampling with 40 inference steps and a classifier-free guidance (CFG) scale of 7.5. Following standard evaluation settings in recent fairness studies on text-to-image diffusion models, we use the neutral prompt template “Photo portrait of a {profession}, a person” and evaluate on six profession categories: {artist, CEO, doctor, nurse, taxi driver, teacher}. We consider two demographic attributes: gender ({male, female}) and race ({white, black, asian}). For each profession, we generate 100 images, resulting in 600 images per evaluation run, and repeat the experiment five times with different random seeds to account for stochastic variation in diffusion sampling. All methods are evaluated under the same protocol, with baseline details provided in the supplementary material. For fairness evaluation, we adopt Bias-O, Bias-Q, and FD. Bias-O measures imbalance in the frequency of generated demographic attributes, while Bias-Q measures disparities in generation quality across attributes. In addition, FD serves as a complementary fairness metric by measuring distributional discrepancy between generated attribute distributions. For quality evaluation, we report CLIP-T, CLIP-I, DINO, FID, and IS. Among them, CLIP-T measures text-image semantic alignment, CLIP-I measures consistency with the original model outputs, DINO captures feature-level similarity, FID evaluates image realism at the distribution level, and IS reflects image quality and diversity. Demographic labels for fairness-related metrics are assigned using the FairFace classifier. Samples without detected faces are excluded only from fairness-related metrics, and the same filtering rule is applied to all methods.
5.2 Quantitative Results↩︎
Table 1 and Table 2 report quantitative results on both gender and race debiasing using Stable Diffusion v2.1. Overall, our method achieves the strongest fairness performance while preserving competitive, and in several cases superior, generation quality. Across both demographic attributes, TES consistently ranks first or second on most fairness and quality metrics, demonstrating a favorable trade-off between demographic alignment and image quality.
Our method substantially improves fairness across all three fairness metrics: Bias-O, Bias-Q, and FD. For gender debiasing, TES reduces Bias-O from 0.85 for the original Stable Diffusion baseline to 0.26, and lowers FD from 0.54 to 0.13. For race debiasing, the improvements are even more pronounced: Bias-O decreases from 0.63 to 0.15, Bias-Q decreases from 2.06 to 0.86, and FD is reduced from 0.21 to 0.09. These results indicate that our method not only reduces the dominance of majority demographic groups in generated samples, but also improves balance in the quality of generated images across attributes.
Compared with strong baselines such as LightFair, our method further improves fairness in both settings. For example, under race debiasing, TES improves Bias-O from 0.40 to 0.15 and Bias-Q from 1.82 to 0.86 relative to LightFair, while also achieving a lower FD. These improvements demonstrate that TES consistently reduces demographic imbalance beyond existing training-free baselines while maintaining competitive generation quality.
Despite these strong fairness gains, our method maintains competitive generation quality. In the gender setting, TES achieves a CLIP-T score of 30.08, which is close to the best-performing baseline and higher than the original Stable Diffusion model. It also achieves the best FID score of 226.13, indicating that the generated image distribution remains highly realistic even after debiasing. Similarly, in the race setting, TES obtains the best FID score of 228.52 and the highest IS score of 1.36, demonstrating that fairness improvement does not come at the cost of reduced visual quality or diversity.
The DINO results further support this observation. TES achieves 0.69 for gender and 0.65 for race, outperforming or matching the strongest baselines in feature-level similarity. Since DINO captures structural and perceptual consistency from image features rather than direct text-image matching, these results suggest that our method preserves the overall semantic and visual coherence of the generated samples while adjusting demographic attributes. In other words, the debiasing effect is not achieved by simply distorting outputs toward target labels, but by steering the generation process in a visually consistent manner.
A notable trend in Table 1 is that existing baselines often exhibit unstable trade-offs between fairness and quality. For instance, some methods reduce Bias-O but still retain relatively high Bias-Q or FD, indicating that demographic frequencies may become more balanced while image quality remains uneven across groups. Other methods preserve CLIP-I or DINO similarity well, but fail to sufficiently reduce demographic imbalance. This inconsistency highlights a common challenge in bias mitigation for diffusion models: improving fairness along one axis often degrades semantic alignment, realism, or consistency along another.
In contrast, TES maintains a more balanced profile across all metrics. Rather than achieving fairness through aggressive post-hoc editing or by relying on fixed control rules, our method continuously adjusts the conditional embedding in response to intermediate generation states. This allows the model to better align demographic control with semantic preservation, resulting in lower fairness discrepancy together with strong CLIP-T, DINO, FID, and IS scores.
5.3 Qualitative Results↩︎
Figure [fig:qualitative] presents qualitative comparisons across multiple professions. The baseline tends to generate images concentrated on dominant demographic groups (e.g., male CEOs or female nurses). In contrast, our method produces more balanced outputs while preserving identity-related features such as facial structure, pose, and overall scene composition. As shown in Figure 4, we further evaluate intersectional attribute control by jointly steering gender and race for highly biased professions, where our method generates a broader range of attribute combinations than the baseline while maintaining semantic consistency, demonstrating its effectiveness in multi-attribute bias mitigation without explicit supervision. Finally, to assess generalization across diffusion backbones, we apply our method to Stable Diffusion v1.5 without modification. As shown in Figure 5, the baseline exhibits similar bias patterns as in Stable Diffusion v2.1, while our method consistently steers generated images toward the target attributes without degrading visual quality, indicating that the proposed approach generalizes effectively in a plug-and-play manner.
5.4 Ablation Study↩︎
| Method | CLIP-T\(\uparrow\) | Bias-O\(\downarrow\) | Bias-Q\(\downarrow\) | FD\(\downarrow\) | CLIP-I\(\uparrow\) | FID\(\downarrow\) |
|---|---|---|---|---|---|---|
| SD | 29.86 | 0.82 | 1.06 | 0.41 | - | 246.61 |
| Initial Only | 30.17 | 0.81 | 5.76 | 0.40 | 87.97 | 236.38 |
| Iterative Only | 30.12 | 0.29 | 1.53 | 0.14 | 77.44 | 225.05 |
| Ours | 30.14 | 0.25 | 1.44 | 0.12 | 75.93 | 224.60 |
Table 3 analyzes the role of each stage in the proposed two-stage optimization framework. Using only the initial update yields the highest CLIP-T and CLIP-I scores, indicating that early correction can effectively steer the global semantic trajectory. However, this strategy fails to achieve stable fairness control: Bias-Q rises sharply to 5.76, while Bias-O and FD remain close to the baseline. This suggests that a single early intervention is insufficient for stable debiasing. In contrast, using only iterative refinement substantially improves fairness, reducing Bias-O from 0.82 to 0.29 and FD from 0.41 to 0.14. This confirms that repeated refinement during denoising is effective for correcting biased attribute emergence. The full two-stage strategy achieves the best overall trade-off by combining both effects: the early-stage update provides global directional correction, while iterative refinement preserves this correction as semantic details emerge. As a result, TES reduces Bias-O to 0.25 and FD to 0.12 while also achieving the best FID. These results show that reliable debiasing requires both coarse early-stage steering and progressive denoising-time refinement.
| Method | CLIP-T\(\uparrow\) | Bias-O\(\downarrow\) | Bias-Q\(\downarrow\) | FD\(\downarrow\) | CLIP-I\(\uparrow\) | DINO\(\uparrow\) | FID\(\downarrow\) |
|---|---|---|---|---|---|---|---|
| Attribute Loss Only | 28.68 | 0.27 | 1.55 | 0.14 | 74.54 | 0.68 | 222.76 |
| + L2 Loss | 28.68 | 0.26 | 1.68 | 0.13 | 74.70 | 0.68 | 225.80 |
| + Profession Loss | 30.13 | 0.25 | 1.22 | 0.13 | 75.96 | 0.69 | 225.12 |
| Ours | 30.14 | 0.25 | 1.44 | 0.12 | 75.93 | 0.69 | 224.60 |
Table 4 examines the contribution of each loss term in the optimization objective. When only the gender objective is used, the model already improves fairness relative to the baseline, but the overall alignment remains weak, as reflected by the relatively low CLIP-T score of 28.68. This indicates that optimizing only for the target demographic direction can correct attribute imbalance to some extent, but does not sufficiently preserve the intended profession semantics. Adding L2 regularization slightly stabilizes the optimization and improves Bias-O and FD, but its effect remains limited. Its role is mainly to prevent excessive drift of the optimized embedding rather than to directly improve fairness or alignment. In contrast, the profession-preserving loss introduces a larger improvement: CLIP-T increases from 28.68 to 30.13, while Bias-O, Bias-Q, CLIP-I, and DINO also improve. This shows that preserving profession semantics is essential for robust debiasing. Overall, the full model achieves the most balanced result across demographic alignment, semantic preservation, and visual quality, demonstrating that effective inference-time debiasing requires jointly optimizing demographic control, semantic consistency, and embedding regularization.
5.5 Discussion↩︎
While the results demonstrate that TES effectively improves demographic balance under our evaluation protocol, TES is designed to steer generated samples toward a predefined demographic distribution rather than to provide a comprehensive fairness solution. Additional implementation details, including inference-time analysis and experimental settings, are provided in the supplementary material. We also include additional qualitative comparisons on Stable Diffusion v2.1, comprehensive evaluations on Stable Diffusion v1.5 against existing methods, and further ablation results. Finally, extending TES to broader intersectional settings with multiple simultaneously varying attributes remains an important direction for future work.
6 Conclusion↩︎
In this paper, we proposed a training-free debiasing framework that operates at inference time by directly optimizing the conditional text embedding of diffusion models. Our method introduces a two-stage strategy that combines early-stage global alignment with iterative refinement during denoising, enabling stable and controllable attribute steering. Extensive experiments demonstrate that our approach consistently reduces demographic bias across both gender and race while preserving image quality and semantic consistency. In addition, qualitative and intersectional results show that the proposed method can control multiple demographic attributes without retraining or architectural modification. Overall, our work highlights the effectiveness of text embedding optimization as a practical and scalable solution for fairness-aware generation in diffusion models.
7 Implementation Details↩︎
This section provides additional implementation details that are omitted from the main paper for brevity. Unless otherwise specified, all experiments follow the settings described in Sec. 5.1 of the main paper.
7.1 Attribute and Semantic Alignment↩︎
At every active optimization timestep, the reconstructed clean image \(\hat{x}_0\) is decoded into the image space and evaluated using CLIP [39]. Both image and text features are \(\ell_2\)-normalized before cosine similarity is computed.
For gender debiasing, the candidate attribute descriptions consist of a target and an opposite attribute (e.g., “male” and “female”). For race debiasing, the candidate set is extended to three classes (“Asian”, “Black”, and “White”), where the optimization objective encourages the generated image toward the assigned target class while suppressing the remaining candidates. The final objective combines the attribute alignment loss, profession-preserving loss, and embedding regularization described in Sec. 4.2 of the main paper.
7.1.0.1 Target Attribute Assignment.
Target attributes are assigned automatically using a lightweight preview stage. For each random seed, a low-cost preview image is generated using a reduced number of diffusion steps. As shown in Table 5, adding the preview stage increases inference time only slightly, from 36.61s to 37.29s per image, while leaving peak GPU memory unchanged. CLIP similarity scores are then computed for all candidate demographic descriptions. Samples are ranked according to these scores and assigned to demographic targets using a balanced allocation strategy. For gender, the ranked samples are divided equally into male and female targets. For race, class-wise quotas are used to obtain a balanced distribution across the three demographic groups. This procedure is performed once before the main optimization and does not require intermediate clean-image reconstruction.
7.2 Optimization Details↩︎
As described in Sec. 5.1 of the main paper, we adopt DDIM sampling with 40 denoising steps for all experiments. Embedding optimization is activated only within the selected timestep window corresponding to 10%–50% of the denoising trajectory (steps 4–19 for 40 sampling steps). In addition, TES performs an initial embedding update before the main optimization stage by rolling out the latent trajectory to 5% of the denoising process and applying a small number of gradient updates with a reduced learning rate. Gradient clipping is applied during both the initial update and denoising-time optimization to improve numerical stability.
7.3 Computational Overhead↩︎
Table 5 reports the computational overhead of TES measured on a single NVIDIA RTX 4090 GPU at \(512\times512\) image resolution and 40 DDIM sampling steps. Wall-clock inference time is
averaged after one warm-up run, and peak GPU memory is measured using torch.cuda.max_memory_allocated(). The preview stage adds only a small overhead, increasing the generation-only runtime from 36.61s to 37.29s per image. The additional cost
mainly comes from repeated CLIP evaluation and gradient-based text-embedding updates, while the diffusion backbone remains frozen.
| Method | Time / Image (s)\(\downarrow\) | Peak GPU Memory (GB)\(\downarrow\) | Relative Time\(\downarrow\) |
|---|---|---|---|
| Stable Diffusion | 3.19 | 3.59 | 1.00\(\times\) |
| TES (w/o Preview) | 36.61 | 9.87 | 11.48\(\times\) |
| TES (w/ Preview) | 37.29 | 9.87 | 11.69\(\times\) |
8 Additional Results↩︎
8.1 Quantitative Results↩︎
8.1.0.1 Generalization to Other Backbones.
To assess whether our method is specific to a particular diffusion model, we additionally evaluate it on Stable Diffusion v1.5 under the same experimental protocol. As shown in Table 8 and Table 9, our approach consistently improves fairness metrics while maintaining competitive generation quality, following the same trends observed in Stable Diffusion v2.1. These results indicate that our method is not tied to a specific backbone, but can be applied to different diffusion models in a plug-and-play manner. Importantly, the relative improvements over baselines remain consistent across both models.
8.1.0.2 Generalization to Unseen Prompt Templates.
We further evaluate prompt robustness by replacing the main prompt template with “a photo of a {profession}”. As shown in Tables 6 and 7, TES consistently improves all fairness metrics over the Stable Diffusion baseline under this alternative prompt formulation. At the same time, the quality metrics remain within a comparable range, indicating that the method does not achieve debiasing by simply sacrificing generation quality. These results suggest that the proposed inference-time optimization generalizes beyond the prompt template used in the main experiments.
8.2 Qualitative Results↩︎
8.2.0.1 Qualitative Results across Diffusion Backbones.
We present qualitative comparisons between the baseline Stable Diffusion model and our method on both Stable Diffusion v2.1 and v1.5. As shown in Figures 8 and 9, the baseline models exhibit clear demographic bias patterns, including gender imbalance and racial concentration across profession prompts such as artist, doctor, teacher, nurse, and CEO. In contrast, our method produces more diverse and balanced demographic representations while preserving the semantic consistency of the given profession. These results qualitatively demonstrate that the proposed inference-time embedding optimization mitigates both gender and racial biases without degrading visual quality, and generalizes across diffusion backbones without architectural modification.
8.3 Prompt and Profession Generalization↩︎
Generalization to Unseen Prompt Templates. The main experiments use the prompt template “Photo portrait of a {profession}, a person”. To evaluate whether our method is dependent on a specific prompt formulation, we additionally replace the template with “a photo of a {profession}”. As shown in Figures 6 and 7, the baseline Stable Diffusion model continues to exhibit pronounced gender and racial biases under the alternative prompt formulation. In contrast, TES consistently produces more balanced gender and racial representations while preserving the intended profession semantics. Furthermore, Figure 10 shows qualitative results on an additional set of profession prompts that are not used in the main experiments, such as fashion designer, hairdresser, lawyer, technician, librarian, and pilot. TES also yields more balanced gender representations on these additional professions, indicating that the proposed inference-time embedding optimization generalizes not only to unseen prompt templates but also to profession categories beyond those used in the main experiments.
| Method | Fairness | Quality | ||||||
| Bias-O\(\downarrow\) | Bias-Q\(\downarrow\) | FD\(\downarrow\) | CLIP-T\(\uparrow\) | CLIP-I\(\uparrow\) | DINO\(\uparrow\) | FID\(\downarrow\) | IS\(\uparrow\) | |
| SD [1] | 0.73 | 1.90 | 0.45 | 29.31 | - | - | 275.13 | 1.26 |
| Ours | 0.19 | 1.65 | 0.10 | 28.88 | 75.47 | 0.66 | 303.99 | 1.43 |
| Method | Fairness | Quality | ||||||
| Bias-O\(\downarrow\) | Bias-Q\(\downarrow\) | FD\(\downarrow\) | CLIP-T\(\uparrow\) | CLIP-I\(\uparrow\) | DINO\(\uparrow\) | FID\(\downarrow\) | IS\(\uparrow\) | |
| SD [1] | 0.54 | 1.60 | 0.17 | 29.31 | - | - | 275.13 | 1.26 |
| Ours | 0.14 | 1.14 | 0.08 | 28.64 | 70.45 | 0.59 | 307.35 | 1.40 |
| Method | Fairness | Quality | ||||||
| Bias-O\(\downarrow\) | Bias-Q\(\downarrow\) | FD\(\downarrow\) | CLIP-T\(\uparrow\) | CLIP-I\(\uparrow\) | DINO\(\uparrow\) | FID\(\downarrow\) | IS\(\uparrow\) | |
| SD [1] | 0.73 (±0.05) | 1.90 (±0.67) | 0.45 (±0.03) | 29.31 (±0.06) | - | - | 275.13 (±6.75) | 1.26 (±0.03) |
| FairD [28] | 0.79 (±0.04) | 3.25 (±1.15) | 0.45 (±0.02) | 28.79 (±0.11) | 75.91 (±0.56) | 0.53 (±0.02) | 269.62 (±4.42) | 1.30 (±0.03) |
| UCE [41] | 0.78 (±0.07) | 1.79 (±0.46) | 0.48 (±0.04) | 28.91 (±0.13) | 82.72 (±0.81) | 0.70 (±0.02) | 273.95 (±5.53) | 1.26 (±0.03) |
| FinetuneFD [42] | 0.38 (±0.07) | 2.31 (±0.35) | 0.22 (±0.04) | 29.34 (±0.13) | 76.17 (±0.68) | 0.57 (±0.01) | 278.21 (±7.53) | 1.24 (±0.02) |
| FairMapping [7] | 0.46 (±0.05) | 2.16 (±0.72) | 0.30 (±0.03) | 29.30 (±0.16) | 76.00 (±0.66) | 0.53 (±0.02) | 278.81 (±5.84) | 1.26 (±0.02) |
| BalancingAct [6] | 0.41 (±0.05) | 1.70 (±0.55) | 0.24 (±0.03) | 29.30 (±0.11) | 77.37 (±0.64) | 0.55 (±0.02) | 272.08 (±5.16) | 1.28 (±0.02) |
| TI [43] | 0.56 (±0.06) | 1.88 (±0.37) | 0.33 (±0.04) | 28.76 (±0.10) | 75.43 (±0.54) | 0.54 (±0.02) | 278.92 (±6.22) | 1.27 (±0.02) |
| AITTI [24] | 0.41 (±0.06) | 1.34 (±0.44) | 0.25 (±0.02) | 29.03 (±0.09) | 77.25 (±0.44) | 0.56 (±0.02) | 267.23 (±5.14) | 1.29 (±0.02) |
| TIME [36] | 0.65 (±0.04) | 1.76 (±0.35) | 0.40 (±0.02) | 28.45 (±0.12) | 73.71 (±0.69) | 0.52 (±0.02) | 279.17 (±4.43) | 1.25 (±0.02) |
| MIST [34] | 0.39 (±0.05) | 1.35 (±0.43) | 0.22 (±0.03) | 29.10 (±0.13) | 76.67 (±0.35) | 0.55 (±0.01) | 254.33 (±4.76) | 1.27 (±0.02) |
| FairSM [33] | 0.65 (±0.04) | 1.83 (±0.21) | 0.43 (±0.02) | 27.98 (±0.11) | 74.23 (±0.59) | 0.52 (±0.02) | 265.60 (±5.04) | 1.26 (±0.03) |
| SANER [44] | 0.52 (±0.02) | 1.65 (±0.34) | 0.35 (±0.02) | 28.13 (±0.08) | 75.28 (±0.77) | 0.52 (±0.02) | 275.34 (±5.40) | 1.25 (±0.04) |
| DEAR [45] | 0.73 (±0.05) | 1.90 (±0.67) | 0.45 (±0.03) | 29.31 (±0.06) | - | - | 275.13 (±6.75) | 1.26 (±0.03) |
| EntiGen [29] | 0.46 (±0.05) | 2.63 (±0.88) | 0.27 (±0.03) | 28.57 (±0.14) | 71.89 (±0.68) | 0.50 (±0.01) | 263.76 (±5.51) | 1.29 (±0.04) |
| ITI-GEN [30] | 0.39 (±0.06) | 1.27 (±0.82) | 0.18 (±0.04) | 28.36 (±0.12) | 68.82 (±0.59) | 0.45 (±0.02) | 246.55 (±5.97) | 1.29 (±0.02) |
| LightFair [25] | 0.30 (±0.08) | 0.99 (±0.55) | 0.17 (±0.04) | 30.57 (±0.16) | 80.09 (±0.76) | 0.63 (±0.02) | 233.53 (±5.50) | 1.30 (±0.03) |
| Ours | 0.09 (±0.01) | 1.88 (±0.10) | 0.05 (±0.01) | 28.68 (±0.10) | 73.93 (±0.35) | 0.64 (±0.01) | 253.04 (±2.20) | 1.36 (±0.01) |
| Method | Fairness | Quality | ||||||
| Bias-O\(\downarrow\) | Bias-Q\(\downarrow\) | FD\(\downarrow\) | CLIP-T\(\uparrow\) | CLIP-I\(\uparrow\) | DINO\(\uparrow\) | FID\(\downarrow\) | IS\(\uparrow\) | |
| SD [1] | 0.54 (±0.02) | 1.60 (±0.67) | 0.17 (±0.01) | 29.31 (±0.06) | - | - | 275.13 (±6.75) | 1.26 (±0.03) |
| FairD [28] | 0.50 (±0.02) | 1.50 (±0.38) | 0.15 (±0.01) | 28.95 (±0.10) | 74.33 (±0.68) | 0.53 (±0.02) | 262.72 (±4.84) | 1.28 (±0.03) |
| UCE [41] | 0.44 (±0.03) | 1.40 (±0.24) | 0.13 (±0.01) | 29.13 (±0.14) | 90.15 (±0.70) | 0.83 (±0.02) | 281.16 (±5.18) | 1.26 (±0.02) |
| FinetuneFD [42] | 0.20 (±0.03) | 1.41 (±0.23) | 0.07 (±0.01) | 29.02 (±0.15) | 74.57 (±0.53) | 0.54 (±0.01) | 270.09 (±5.99) | 1.26 (±0.02) |
| FairMapping [7] | 0.34 (±0.02) | 1.75 (±0.47) | 0.10 (±0.01) | 29.29 (±0.15) | 76.54 (±0.71) | 0.54 (±0.02) | 280.95 (±5.02) | 1.26 (±0.03) |
| BalancingAct [6] | 0.34 (±0.02) | 1.13 (±0.36) | 0.07 (±0.01) | 29.34 (±0.11) | 77.44 (±0.72) | 0.55 (±0.02) | 271.91 (±5.35) | 1.29 (±0.03) |
| TI [43] | 0.47 (±0.03) | 1.45 (±0.27) | 0.14 (±0.01) | 28.67 (±0.17) | 67.96 (±0.84) | 0.43 (±0.04) | 275.20 (±5.07) | 1.25 (±0.03) |
| AITTI [24] | 0.25 (±0.04) | 1.20 (±0.31) | 0.08 (±0.02) | 29.03 (±0.11) | 85.43 (±0.47) | 0.79 (±0.01) | 271.13 (±5.24) | 1.29 (±0.01) |
| TIME [36] | 0.39 (±0.04) | 1.51 (±0.34) | 0.12 (±0.01) | 27.97 (±0.15) | 76.53 (±0.68) | 0.54 (±0.02) | 275.72 (±6.84) | 1.26 (±0.02) |
| MIST [34] | 0.26 (±0.03) | 1.19 (±0.24) | 0.08 (±0.01) | 29.08 (±0.09) | 83.25 (±0.75) | 0.76 (±0.02) | 265.83 (±6.78) | 1.28 (±0.02) |
| FairSM [33] | 0.42 (±0.03) | 1.61 (±0.37) | 0.13 (±0.01) | 28.68 (±0.06) | 72.83 (±0.60) | 0.51 (±0.03) | 271.58 (±5.83) | 1.27 (±0.03) |
| SANER [44] | 0.45 (±0.03) | 1.41 (±0.33) | 0.13 (±0.02) | 28.50 (±0.13) | 73.64 (±0.51) | 0.50 (±0.02) | 273.21 (±6.42) | 1.24 (±0.02) |
| DEAR [45] | 0.54 (±0.02) | 1.60 (±0.67) | 0.17 (±0.01) | 29.31 (±0.06) | - | - | 275.13 (±6.75) | 1.26 (±0.03) |
| EntiGen [29] | 0.37 (±0.04) | 2.88 (±0.63) | 0.09 (±0.01) | 27.97 (±0.14) | 69.56 (±0.74) | 0.47 (±0.01) | 265.94 (±4.25) | 1.31 (±0.04) |
| ITI-GEN [30] | 0.31 (±0.04) | 1.62 (±0.37) | 0.10 (±0.01) | 28.13 (±0.16) | 66.97 (±0.57) | 0.42 (±0.01) | 269.84 (±6.61) | 1.33 (±0.03) |
| LightFair [25] | 0.18 (±0.04) | 1.06 (±0.43) | 0.06 (±0.01) | 31.34 (±0.20) | 86.31 (±0.70) | 0.81 (±0.02) | 259.96 (±7.75) | 1.33 (±0.03) |
| Ours | 0.14 (±0.02) | 1.03 (±0.11) | 0.08 (±0.01) | 28.53 (±0.06) | 68.95 (±0.36) | 0.64 (±0.01) | 253.62 (±1.84) | 1.37 (±0.01) |
9 Fairness Metrics Details↩︎
Following LightFair [25], we adopt the same evaluation protocol and report both fairness and generation quality metrics. In our experiments, we consider six professions: doctor, CEO, taxi driver, nurse, artist, and teacher. For each profession, we generate 100 images, resulting in a total of 600 generated images for evaluation. All metrics are computed over this set. The fairness evaluation is conducted based on predicted attribute labels (e.g., gender and race), while quality is measured using semantic alignment, perceptual similarity, and distributional statistics.
9.1 Fairness Metrics↩︎
We follow the LightFair protocol for fairness evaluation, where generated images are grouped according to predicted attributes, and fairness is measured in terms of both distribution balance and consistency across groups.
Bias-O [25] evaluates whether unspecified attributes are generated with balanced frequency. Generated images are first assigned attribute labels using pretrained classifiers, and the frequency of each attribute is computed per prompt. Bias-O reflects the imbalance among these frequencies. Lower values indicate a more uniform distribution of attributes.
Bias-Q [25] evaluates whether generation quality is consistent across different attribute groups. After grouping images by predicted attributes, we compute a quality score for each group and measure discrepancies across groups. In our setup, the group-level quality is derived from CLIP-based text-image alignment (CLIP-T). Lower values indicate more consistent quality across attributes.
FD (Feature Distance) [46] measures the deviation between generated images and their corresponding baseline images in a feature space. Each generated image is paired with the baseline image produced using the same prompt and random seed, and the feature distance is computed and averaged across samples. Lower FD indicates better preservation of baseline characteristics.
9.2 Quality Metrics↩︎
To evaluate generation quality, we adopt the metrics used in LightFair and provide additional feature-based analysis.
CLIP-T [39] measures text-image semantic alignment. For each generated image, we compute the CLIP score with respect to its corresponding prompt instantiated from a fixed template (“Photo portrait of a {profession}, a person”). The scores are averaged per profession and then across all samples. Higher CLIP-T indicates better alignment between generated images and textual descriptions.
CLIP-I [39] measures image-level consistency with the original diffusion output. Each generated image is paired with its baseline counterpart generated using the same prompt and random seed. We extract normalized CLIP image features and compute cosine similarity. Higher CLIP-I indicates stronger similarity to the original output.
DINO [47] evaluates feature-level similarity using self-supervised visual representations. We extract global features from a pretrained DINOv2 model and compute cosine similarity between generated and baseline images. Higher DINO indicates better preservation of visual structure.
FID [48] measures the distributional similarity between generated images and real images. We compute FID with an Inception-v3 backbone. As the reference distribution, we use 10,000 images from the FairFace dataset, and compare them with the generated samples for each profession, followed by averaging across all professions. Lower FID indicates higher realism.
IS [49] evaluates both image quality and diversity. Generated images are passed through a pretrained Inception network, and the score is computed using a split-based protocol. We report both mean and standard deviation across samples. Higher IS indicates better quality and diversity.