1 Introduction↩︎

Text-to-image diffusion models have become standard tools for visual content creation, but they carry a well-documented problem: they reproduce and amplify the demographic skews in their training data. The effect is easy to observe. Ask a state-of-the-art model for “a photo of a nurse” and it will almost always generate a female figure; ask for “a CEO” and you get a male one (Fig. 1). These are not occasional glitches but stable patterns, and as these models find their way into more and more real-world applications, they can further propagate discrimination and raise ethical concerns regarding the fairness and inclusivity of generative AI systems.

To mitigate this, researchers have proposed several approaches for debiasing text-to-image (T2I) generation. Finetuning-based methods [1]–[3] can be effective but require access to model weights and additional data, which rules them out for many users and makes cross-architecture transfer difficult. Guidance-based methods [4] are training-free but operate at the batch level, using external classifiers to steer attribute distributions; they cannot debias individual samples and struggle when multiple attributes are involved. Text-embedding methods [5] modify prompt representations to suppress demographic cues, but are sensitive to prompt phrasing and offer limited spatial control over the generated output.

We instead look further inside the model. Cross-attention (CA) layers are where text tokens meet image features: they govern which spatial regions respond to which parts of the prompt, and prior work has shown they can be steered at inference time to control semantic content [6]–[8]. Recent attribution analysis further finds that demographic information is encoded in these activations [9], pointing to a direct and unexploited debiasing handle. We exploit this by precomputing, for each attribute value, a steering vector in cross-attention (CA) activation space from pairs of contrastive prompts, and at inference time uniformly sampling a target attribute and adding its steering vector to the CA outputs. This simple mechanism is effective at reducing bias in many settings. Simple steering, however, has two failure modes. When a prompt explicitly specifies an attribute (e.g., “a photo of a male nurse”), the intervention should not apply, but a fixed steering strength cannot detect this. And since CA activations may already carry pre-existing attribute signals, adding a steering vector on top can yield ambiguous, mixed-attribute generations. Thus, we introduce EquiSteer. It contains a prompt-aware gate that measures how strongly early-step CA activations align with the attribute steering direction, and skips intervention when the signal exceeds a calibrated threshold. For neutral prompts, it first orthogonalises the CA output against all attribute directions to remove pre-existing signals, then calibrates the injection strength from attribute-specific prompts and injects the target attribute.

We evaluate EquiSteer on four architectures (SD-1.5, SD-2.1 [10], SDXL [11], and SANA [12]), covering gender, race, and age as primary concepts, with additional experiments on categorical attributes such as eyeglasses. EquiSteer consistently outperforms recent debiasing methods, reducing the average parity gap by up to \(87\%\) relative to the undebiased model and by up to \(47\%\) over the strongest available baseline. It correctly handles attribute-specific prompts throughout and maintains competitive image quality and text-image alignment, confirming that cross-attention steering offers a practical and scalable route to fairer text-guided image generation (Fig. 2).

In summary, our contributions are:

• We show that CA activations encode demographic attributes in a way that makes them directly steerable for fairness control, establishing cross-attention steering as an effective approach to inference-time T2I debiasing.

• We introduce EquiSteer, a CA-steering debiasing method with prompt-aware gating, and attribute-space orthogonalisation to handle attribute-specific prompts and prevent mixed-attribute artifacts, all without changing model weights.

• We validate EquiSteer across four T2I backbones and multiple demographic concepts, showing large-margin improvements over recent baselines while preserving attribute-specific behavior and image quality.

2 Related Work↩︎

2.0.0.1 Auditing bias in text-to-image diffusion.

Large-scale audits show that text-to-image (T2I) models reproduce and amplify demographic stereotypes across occupations and everyday scenes, with systematic skew toward whiteness and masculinity and under-representation of minority groups [13]–[15]. Recent evaluations also link toxicity and unsafe generations with disparate impacts across demographics [16]. Common practice quantifies representation parity and stereotype amplification using classifier-based (often CLIP-based [17]) attribute or occupation predictors, with complementary human checks [18]–[20]. These findings motivate mitigation methods that work across attributes and models rather than narrow, per-task fixes. Building on these observations, we introduce a training-free, per-sample intervention: EquiSteer targets the documented skews by acting directly on CA activations without auxiliary predictors or retraining, while preserving text–image alignment.

2.0.0.2 Finetuning-based debiasing.

One line of work adapts model parameters using additional supervision or balanced datasets. Fair Diffusion instructs models toward fairness objectives [1]. In [2], the authors propose distributional alignment losses and adjusted direct finetuning to reduce gender and racial bias in occupations. Fair Mapping introduces a lightweight parameter-efficient modification that remaps intermediate features of pre-trained diffusion models [3]. While effective, they require retraining and model access, limiting scalability and transferability across diffusion backbones. In contrast, EquiSteer operates entirely at inference time, requiring no additional data or weight updates, and can be applied selectively on a per-generation basis, working with different backbones without retraining.

2.0.0.3 Distributional guidance.

Training-free guidance methods steer generation toward target attribute distributions. Balancing Act [4] achieves this by using an attribute distribution predictor (ADP) to jointly guide batches of samples toward balanced attribute marginals. However, it operates only at the batch level and faces scalability limits for general-purpose debiasing, especially under fine-grained or multi-attribute settings. Beyond its batch-only operation, reliance on external attribute estimators and sensitivity to guidance strength can introduce quality–fairness trade-offs that complicate robust deployment [4], [21]. EquiSteer instead operates at the sample level (no batch coupling), avoids external attribute estimators, and scales to multi-attribute settings with reduced quality–fairness trade-offs.

2.0.0.4 Text embedding intervention (TEI).

Prompt- and embedding-level methods adjust or neutralize text embeddings to mitigate bias [5], [22]. These approaches are lightweight and often model-agnostic, but can be sensitive to phrasing and compositional prompts and offer limited spatial control during generation [1], [21]. LightFair [23] debiases T2I generation by refining the pre-trained text encoder (text-conditioning side), offering a more efficient alternative to full diffusion finetuning, but still requiring weight access and an optimization stage. FairImagen [24] performs post-hoc debiasing by projecting CLIP prompt embeddings into a fairness-aware subspace (FairPCA), optionally combined with noise injection and a unified projection for multi-attribute debiasing; while lightweight and model-agnostic, such embedding-level methods provide limited spatial control compared to cross-attention interventions. Rather than modifying prompts or text embeddings, EquiSteer manipulates CA activations, providing spatially aware control that is robust to phrasing and compositional prompts.

2.0.0.5 Cross-attention control and interpretability.

Cross-attention (CA) mechanisms have been leveraged for layout control and semantic editing [8], [25], as well as for bias attribution and interpretability [9], [26]. These findings identify CA as a powerful interface for fine-grained, training-free control over generative behavior. Related training-free attention steering for controllable editing (e.g., Prompt-to-Prompt and Attend-and-Excite) demonstrates that manipulating cross-attention supports fine-grained, per-sample control and stronger word–region binding [6], [7]. Building on these controls, EquiSteer turns CA interpretability into a fairness mechanism: it detects when to abstain, removes confounding attribute directions, and re-injects a calibrated target attribute without editing prompts or weights.

3 Method↩︎

Our approach builds on CASteer [8], which suppresses target concepts in diffusion models by modifying cross-attention (CA) layer outputs with precomputed steering vectors. We repurpose this mechanism for debiasing: instead of suppressing a concept, we use steering vectors to shift CA outputs toward desired attributes. We review the CA steering mechanism in Sec. 3.1 and formalize the debiasing task in Sec. 3.2. Sec. 3.3 introduces our base debiasing method where we show that basic steering has two failure modes, motivating a gating mechanism (Sec. 3.4) and an adaptive steering magnitude (Sec. 3.5); these two additions form our proposed framework, EquiSteer.

3.1 Background: Steering for Generation Control↩︎

We briefly review the CA steering mechanism of CASteer [8], which enables inference-time concept manipulation in diffusion models. The core idea is to modify CA layer outputs using precomputed steering vectors associated with a target concept \(X\).

For each concept \(X\), CASteer constructs steering vectors for every cross-attention layer \(1 \leq l \leq L\) and denoising step \(1 \leq t \leq T\). These vectors are computed from pairs of positive prompts (containing \(X\)) and negative prompts (excluding \(X\)). Details of the construction procedure are provided in the supplementary.

Let \(s_{lt}^X\) denote the steering vector for concept \(X\), layer \(l\), and timestep \(t\). Each \(s_{lt}^X\) shares the dimensionality of the CA output and applies directly as an additive intervention during inference. Denote by \(ca^{out}_{lt}\) the output of the \(l\)-th CA layer at denoising step \(t\). Steering modifies the activations as \[ca^{out\_new}_{lt} = ca^{out}_{lt} + \alpha s^X_{lt}.\]

Here, \(\alpha \in \mathbb{R}\) controls the strength and direction of intervention. The vector \(s^X_{lt}\) encodes a direction in CA activation space associated with concept \(X\): adding \(\alpha s^X_{lt}\) with \(\alpha > 0\) amplifies the presence of \(X\) in the generation, while \(\alpha < 0\) suppresses it.

CASteer demonstrated that such activation-level interventions enable effective concept suppression without retraining. In this work, we build on this mechanism and show that cross-attention steering extends naturally from concept suppression to controlled debiasing.

3.2 Debiasing Task Statement↩︎

We consider the task of training-free, inference-time debiasing of a diffusion model with respect to a concept \(X\) that has \(n\) discrete attributes \(\{a_i\}_{i=1}^n\). For example, when \(X=\textit{gender}\), we have \((a_1, a_2) = (\textit{male}, \textit{female})\).

Our goal is to design an inference-time intervention that adjusts the generative process so that the resulting images exhibit attributes \(\{a_i\}_{i=1}^n\) according to a prescribed target distribution \(p(a_1, \dots, a_n)\). In practice, we choose a uniform target \(p = U(a_1, \dots, a_n)\), so that each attribute is generated with probability \(1/n\). For instance, in the binary gender setting, this means generating male and female images with equal probability.

The intervention must also operate selectively. We aim to modify generations from attribute-neutral prompts (e.g., “a photo of a doctor”), while leaving the model’s behavior on attribute-specific prompts (e.g., “a photo of a male doctor”) unchanged. The objective is to enforce the desired marginal attribute distribution for neutral prompts, without altering generations where the prompt already specifies an attribute.

3.3 Debiasing via Cross-Attention Steering↩︎

We now present our inference-time debiasing method based on CA steering. Consider a diffusion model with \(L\) CA layers and \(T\) denoising steps, and a target concept \(X\) with discrete attribute set \(\{a_i\}_{i=1}^n\). For each attribute \(a_i\), we precompute steering vectors \(s_{lt}^{a_i}\) for all CA layers \(l\) and timesteps \(t\). Each steering vector \(s_{lt}^{a}\) captures attribute-specific information in CA activation space, so adding \(s_{lt}^{a}\) to the CA output \(ca^{out}_{lt}\) biases the generation toward attribute \(a\). This motivates the following debiasing mechanism:

Basic steering mechanism for debiasing. At inference time, we sample a target attribute \(a \sim \mathrm{Uniform}\{a_i\}_{i=1}^n,\) and then modify each CA output according to \[ca^{out\_new}_{lt} = ca^{out}_{lt} + \alpha s_{lt}^{a}, \label{eq:fairsteer95base}\tag{1}\] where \(\alpha > 0\) controls the steering strength. The CA output \(ca^{out}_{lt}\) has shape \((bs, \text{seq\_len}, \text{emb\_dim})\), corresponding to batch size, number of image tokens, and embedding dimension. In Eq. 1 , the same steering vector \(\alpha s_{lt}^{a}\) is added to each image token, shifting the activations toward the selected attribute direction and thereby increasing the likelihood that attribute \(a\) appears in the final image.

Finally, for stability we re-normalize the updated CA output to preserve its original \(L_2\) norm: \[ca^{out\_new\_renormed}_{lt} = \|ca^{out}_{lt}\|_2\,\frac{ca^{out\_new}_{lt}}{\|ca^{out\_new}_{lt}\|_2}. \label{eq:renorm95base}\tag{2}\]

However, the basic mechanism is insufficient. We aim to debias attribute-neutral prompts while preserving generations for attribute-specific prompts. A single global \(\alpha\) cannot satisfy both goals: small \(\alpha\) may be too weak to counteract pre-existing bias (Fig. 3, right), while large \(\alpha\) may override explicitly specified attributes. Moreover, when multiple attribute signals are present, direct addition can lead to mixed or ambiguous generations (Fig. 3, left).

These limitations motivate two additional components. First, we introduce a gating mechanism that determines whether to apply debiasing at all. Second, we refine the steering update by removing existing attribute directions from the activation and setting the steering magnitude adaptively.

3.4 Gated Steering↩︎

We first introduce a gating mechanism that determines whether debiasing applies to the current prompt. CA activations already encode whether an attribute is explicitly specified in the prompt, and we exploit this to build an attribute-specificity detector.

Attribute expression as a dot-product signal. For a target attribute \(X\), let \(ca^{out}_{ltk}\) denote the CA output for image token \(k\) at layer \(l\) and denoising step \(t\), and let \(s_{lt}^{X}\) be the corresponding steering vector. Since \(s_{lt}^{X}\) encodes a direction associated with attribute \(X\), the dot product \(\langle ca^{out}_{ltk}, s_{lt}^{X} \rangle\) measures how strongly attribute \(X\) is expressed in that token. Figure 4 (a) illustrates this: for attribute-specific prompts, the dot products are substantially higher than for attribute-neutral ones. To obtain a layer-level statistic, we compute the maximal token response \[dp_{lt} = \max_{0 \leq k \leq K_l} \langle ca^{out}_{ltk}, s_{lt}^{X} \rangle, \label{eq:3}\tag{3}\] where \(K_l\) is the number of image tokens in layer \(l\). We use the maximum because attribute evidence may be spatially localized, and \(dp_{lt}\) captures the strongest attribute signal at that layer.

CA layers capture attribute specificity in the prompt. We use Eq. 3 to detect attribute-specific prompts. We evaluate \(dp_{lt}\) on attribute-neutral prompts (e.g., “A photo of a doctor”) and attribute-specific prompts (e.g., “A photo of a male doctor”). For SD-1.5, at timestep \(t=0\), intermediate CA layers (typically \(l \in [4,8]\)) show substantially larger \(dp_{lt}\) values for attribute-specific prompts (Fig. 4 (b)). The same pattern holds for SD-2.1, SDXL, and SANA (see the supplementary). This shows that early-step CA activations encode whether the prompt explicitly specifies an attribute.

Gating rule. Based on this observation, we choose for each backbone a gating layer \(l^{gate}\) at which attribute presence can be reliably detected. For each attribute \(a\), we define a gating threshold as the midpoint between the empirical means of the dot-product statistic on attribute-specific and attribute-neutral prompts: \[thr^{a} = \frac{dp^{a}_{\text{specific}} + dp^{a}_{\text{neutral}}}{2}, \label{eq:threshold}\tag{4}\] where \(dp^{a}_{\text{specific}}\) and \(dp^{a}_{\text{neutral}}\) denote the empirical means of \(dp^{a}_{l^{gate}0}\) computed over attribute-specific and attribute-neutral prompts, respectively (both measured at layer \(l^{gate}\) and timestep \(t{=}0\)). During generation, we evaluate \(dp^{a}_{l^{gate}0}\) once, at \((l^{gate},t{=}0)\). If \[dp^{a}_{l^{gate}0} > thr^{a} \quad \text{for any } a \in \{a_i\}_{i=1}^n, \label{eq:gate}\tag{5}\] we treat the prompt as attribute-specific and perform vanilla generation without intervention; otherwise, we apply the EquiSteer update.

The gate thus prevents unintended modification of prompts that already specify an attribute, while enabling debiasing for neutral ones. Note that Eq. 5 is evaluated only once per inference run (at \((l^{gate}, t{=}0)\)), and therefore introduces negligible computational overhead.

3.5 Choosing Steering Strength for Debiasing↩︎

We now refine the steering update for when the gate decides to apply debiasing, addressing the two main limitations of the basic mechanism: mixed attribute signals and insufficiently pronounced target attributes.

Orthogonalisation with respect to attribute directions. Let \(\{a_i\}_{i=1}^n\) be the attribute set for the debiasing concept, and let \(s_{lt}^{a_i}\in\mathbb{R}^{d}\) denote the steering vector for attribute \(a_i\) at layer \(l\) and timestep \(t\). Before injecting the target attribute, we remove from the current cross-attention activation the components lying in the subspace spanned by attribute directions \(\mathrm{span}\!\left(\{s_{lt}^{a_i}\}_{i=1}^{n}\right)\).

We construct an orthonormal basis \(\{u^{j}_{lt}\}_{j=1}^{m}\) for this subspace via Gram–Schmidt with a small \(\varepsilon>0\) for numerical stability. Then we stack the basis vectors as columns of \(U_{lt}\in\mathbb{R}^{d\times m}\). Orthogonalization then becomes a projection removal: \[\label{eq:orth95matmul} ca^{out\_tmp}_{lt} \;=\; \bigl(I-U_{lt}U_{lt}^{\top}\bigr)\,ca^{out}_{lt}.\tag{6}\] By construction, \(ca^{out\_tmp}_{lt}\perp \mathrm{span}\!\left(\{s_{lt}^{a_i}\}_{i=1}^{n}\right)\), which removes any pre-existing attribute signal and makes mixed-attribute generations less likely. Note that \(U_{lt}\) (and thus \(I-U_{lt}U_{lt}^{\top}\)) can be precomputed offline; at inference time this step requires only one matrix multiplication per CA layer.

Adaptive steering magnitude. After orthogonalisation, we choose the steering strength so that the injected attribute signal matches the typical signal induced by attribute-specific prompts at the same layer. Recall from Sec. 3.4 (Eq. 3 ) that for an attribute \(a\) we measure its expression in CA outputs via the maximal dot-product statistic \(dp^{a}_{lt}=\max_{k}\langle ca^{out}_{ltk}, s^{a}_{lt}\rangle\). For each layer \(l\) (and timestep \(t\)), we precompute \[dp^{a}_{\text{mean}}(l,t) \;=\; \mathbb{E}_{\text{attr-spec prompts}}\!\left[\,dp^{a}_{lt}\,\right],\] i.e., the mean maximal dot product observed on attribute-specific prompts. We then set the steering magnitude \(\alpha = dp^{a}_{\text{mean}}(l,t)\) and update \[ca^{out\_new}_{lt} = ca^{out\_tmp}_{lt} + \alpha\, s_{lt}^{a}. \label{eq:thr95add}\tag{7}\] Since \(ca^{out\_tmp}_{lt}\perp \mathrm{span}(\{s^{a_i}_{lt}\}_{i=1}^{n})\), this update injects only the selected attribute direction \(s^{a}_{lt}\). Choosing \(\alpha\) in this way ensures that the target attribute is expressed at a strength consistent with attribute-specific prompts, reducing both weak and ambiguous attribute expression.

Summary of EquiSteer. At inference time, EquiSteer first checks whether the prompt already specifies an attribute via the gating signal and skips debiasing if so. For neutral prompts, it orthogonalises the CA output against all attribute directions and reinjects the target attribute at a calibrated strength. This allows the method to debias neutral prompts while leaving attribute-specific ones intact, eliminating both mixed-attribute and weak-attribute failure modes of basic steering.

4 Experiments↩︎

4.1 Experimental Setup↩︎

We follow the evaluation protocol of TEI [5]. We adopt this setup as it explicitly evaluates debiasing performance on both attribute-neutral prompts and attribute-specific prompts. This is particularly important in our setting: EquiSteer is designed not only to improve fairness on attribute-neutral prompts, but also to preserve the model’s behavior when the attribute is explicitly specified.

In our main experiments, we evaluate EquiSteer on the gender concept with the binary attribute set (male, female). In Sec. 4.2.2 we additionally evaluate EquiSteer on 4 more concepts with varying numbers of attributes: race (5 attributes: White, Black, Asian, Indian, Latino), age (3 attributes: young, middle-aged, elderly), body type (3 attributes: slim, average build, heavy), eyeglasses (2 attributes: with eyeglasses and without eyeglasses). Because diffusion models handle prompt negation poorly (prompts like “a man with no eyeglasses” can still produce eyeglasses), we do not compute a separate steering vector for the negated value for the eyeglasses concept; instead, we use a single steering vector for the eyeglasses direction and implement the no eyeglasses target through a CASteer-style erasure step at inference time. More details are given in Sec. 12.3. We consider four widely used backbones: SD-1.5, SD-2.1, SDXL, and SANA.

The prompts used to compute steering vectors and to estimate the gating thresholds and adaptive steering magnitudes do not overlap with testing prompts and are described in the supplementary Sec. 8. For the gating mechanism, we use layer \(l^{gate} = 4\) for SD-1.5 and SD-2.1, \(l^{gate} = 17\) for SDXL, and \(l^{gate} = 5\) for SANA. For each model, the gating layer \(l^{gate}\) is chosen as the first layer that shows clear separation in maximal token response \(dp_{l0}\) on a validation set of prompts (see Sec. 3.4 and Sec. 16 in the supplementary).

Evaluation setup. We adopt the standard auditing protocol from prior work [1], [4], [5], using both attribute-neutral and attribute-specific prompts over a fixed set of professions: “CEO”, “doctor”, “pilot”, “technician”, “fashion designer”, “librarian”, “teacher”, and “nurse”. These professions are known to induce strong attribute biases in text-to-image diffusion models [5].

For attribute-neutral prompts, we use the template “A photo of a {profession}”. For attribute-specific prompts, we use the template “A photo of a {attribute} {profession}”. For each model and profession, we generate 1,000 images for attribute-neutral prompts and 300 images for attribute-specific prompts.

Following [5], we evaluate attribute bias for all concepts except eyeglasses using a CLIP ViT-L/14 zero-shot classifier. For gender, we use the prompt template “A photo of a {attribute}”. For race, age, and body type, we use the profession-conditioned template “A photo of a {attribute} {profession}”, which provides a more specific comparison for occupational prompts.

For the eyeglasses concept, we find that this CLIP setup tends to favor the more generic prompt “A photo of a person”, which systematically underestimates the presence of eyeglasses. To obtain a more reliable measurement, we instead report eyeglasses results using the VQA-based classifier BLIP-VQA [27] capfilt-large. In the supplementary Sec. 17, we report agreement rates between BLIP-, CLIP-, GPT-4o-, and human-based evaluations, supporting our choice of the BLIP-based classifier as a more reliable measure of eyeglasses presence. We report both per-profession results and macro-averaged results across all evaluation prompts (see Tables 1 and 5).

For attribute-specific prompts, the target ratio for the specified attribute is \(1.0\). For attribute-neutral prompts, the target ratio is \(1/n\), where \(n\) is the number of debiasing attributes. We summarise performance using \(\Delta = \left|p - \frac{1}{n}\right|\), where \(p\) is the observed attribute ratio; lower values indicate better parity. Thus, we define fairness as closeness of the generated attribute distribution to the target uniform distribution.

In addition to fairness, we evaluate general image–text alignment using CLIPScore [28] with CLIP-ViT-L/14, and image fidelity using CMMD [29]. Both metrics are computed on 30,000 images generated from prompts in the MS-COCO-2014 validation set [30] (see Table [tab:fid]) on EquiSteer applied to the gender concept.

Baselines. We compare EquiSteer against recent debiasing methods, including FairDiffusion [1], Unified Concept Editing (UCE) [31], Fine-Tuning Diffusion (FTDiff) [2], Self-Discovering Latent Direction (SelfDisc) [32], FairImagen [24] (for SDXL only), and the method of [5], which we denote as TEI. TEI is our primary point of comparison, as it is also training-free and is explicitly designed to improve fairness on attribute-neutral prompts while preserving behavior on attribute-specific prompts.

4.2 Experimental Results↩︎

4.2.1 Debiasing Gender Concept↩︎

Following TEI [5], we report our main results for debiasing the gender concept. In this case, the set of attributes is binary: (male, female).

Attribute-neutral prompts. Table 1 reports results for attribute-neutral prompts on SD-1.5. EquiSteer substantially improves over the vanilla model and outperforms the training-free baseline TEI, achieving a markedly lower \(\Delta\) score. The improvement is consistent across professions, indicating that EquiSteer effectively reduces gender imbalance in biased occupational prompts.

Table 5 reports the average \(\Delta\) across the eight professions for SD-2.1, SDXL, and SANA. EquiSteer again consistently improves over the vanilla model across all backbones, showing that the method transfers well beyond SD-1.5. Detailed per-profession results are provided in the supplementary Sec. 13.

For the SDXL model, we additionally compare against FairImagen [24], which is evaluated only for SDXL. For this comparison, we report evaluation details and results on their setup in the supplementary Sec. 10.

Attribute-specific prompts. Figure [fig:gender95box95v2] shows results on attribute-specific prompts for the SD-1.5 model, where the objective is to preserve the attribute explicitly specified in the prompt (target \(=1.0\)). Detailed per-profession breakdowns are provided in the supplementary Sec. 13.

For female-specified prompts, most methods already perform strongly, and EquiSteer preserves this behavior, achieving near-perfect results across all eight professions. Male-specified prompts are more challenging, but EquiSteer remains competitive with TEI and outperforms the remaining baselines. Results for SD-2.1, SDXL, and SANA, reported in the supplementary Sec. 13, show the same overall pattern: EquiSteer preserves attribute-specific generations while improving fairness on attribute-neutral prompts.

Highly biased concepts. Among the evaluated professions, nurse and technician are particularly challenging, as they are strongly biased toward the female and male attributes, respectively. These cases are difficult because stronger pre-existing bias makes it harder to move the generated attribute distribution toward parity. Even in these settings, EquiSteer yields clear improvements over the vanilla model, and the detailed results in the supplementary Sec. 13 show that it remains effective on strongly biased prompts.

Overall, across all four backbones, EquiSteer consistently improves debiasing performance on the gender concept while preserving fidelity to attribute-specific prompts. As shown in Table [tab:fid], it also maintains strong image fidelity and text–image alignment, remaining competitive with recent debiasing approaches on both quality metrics.

4.2.2 Debiasing Additional Concepts↩︎

Tab. ¿tbl:tab:main95attributes95summary? reports the parity gap before and after EquiSteer applied on SDXL and SANA models for each of the four additional debiasing concepts. EquiSteer reduces the parity gap on every cell, with the strongest reductions on race (\(76\!-\!78\%\)) and on age on SANA (\(85\%\)). Fig. 6 shows qualitative examples of race debiasing on SANA and SDXL: the vanilla generations skew toward a single demographic per profession, while EquiSteer distributes generations across the five race classes while preserving scene composition. We report per-profession breakdowns and more qualitative results in Sec. 12 in the supplementary.

4.2.3 Multi-concept debiasing↩︎

We further apply EquiSteer to several concepts within the same generation, by running the full intervention (gate, orthogonalisation, target sampling, adaptive re-injection) sequentially per concept at every cross-attention layer and step. Each attribute retains its own threshold and steering vectors. Tab. [tbl:tab:main95multiconcept] reports joint debiasing of 4 concepts: gender + race + age + body type simultaneously. All four attributes are substantially debiased in one pass on both backbones. Per-profession breakdowns, qualitative results and additional multi-concept debiasing results are reported in Sec. 14 in the supplementary.

4.3 Ablations↩︎

EquiSteer components We ablate the three components of EquiSteer namely, adaptive-magnitude steering (add; Eq. 1 , 7 ), subspace orthogonalisation (erase; Eq. 6 ), and the prompt-aware gate (gate; Eq. 5 ). We use EquiSteer applied to SD-1.5 on gender concept. Results presented in Tab. 2 show that each component matters: add alone gives only modest debiasing (\(\Delta_{\text{neutral}}{=}0.165\)); adding orthogonalisation tightens parity (\(\Delta_{\text{neutral}}{=}0.095\)) but corrupts attribute preservation on attribute-specific prompts (male-specific recall drops to \(0.230\)). Finally, the gate restores attribute preservation (\(\Delta_{\text{male}}{=}0.495\), \(\Delta_{\text{female}}{=}0.492\)) while keeping the neutral parity gap at its tightest (\(\Delta_{\text{neutral}}{=}0.051\)). Per-profession ablation details are in the supplementary Sec. 19.

Transferability across prompt families. The standard evaluation uses the template “A photo of a {profession}”. We test whether EquiSteer transfers to 13 held-out templates grouped into three prompt families: paraphrased, long contextual, and compositional/multi-subject prompts. We use the same steering vectors, gate layers, and thresholds for all templates, without prompt-family-specific recalibration. The full prompt list and per-template results are provided in the supplementary Sec. 15.

Table 3 reports the aggregate gender parity gap on SDXL and SANA. EquiSteer reduces \(\Delta\) across all three prompt families on both backbones, with reductions ranging from \(48\%\) to \(89\%\). Paraphrased and long-context prompts behave similarly to the standard template, while the weakest case is SANA under compositional prompts (\(48\%\) reduction), likely because the current image-level gate cannot separately control two subjects in the same scene. We discuss this limitation further in the supplementary Sec. 15.

Gate analysis We perform a detailed analysis of EquiSteer’s gating mechanism in the supplementary Sec. 16. We measure the realised inference-time gate’s AUROC across every (backbone \(\times\) attribute) cell (Sec. 16.1), describe an automated procedure for selecting the gating layer \(l^{gate}\) that recovers our manually chosen layers (Sec. 16.2), test the gate’s separability on subtle / low-saliency attributes such as religion, disability, and socio-economic status (Sec. 16.3), and sweep the threshold multiplier \(m\) to study sensitivity of debiasing performance to the gate threshold (Sec. 16.4). The gate is highly separable, with AUROC \(\geq 0.988\) for every evaluated cell. The default threshold multiplier \(m{=}1\) provides a favorable trade-off between attribute-neutral debiasing and attribute-specific preservation.

Classifier calibration and human evaluation We assess the reliability of the default CLIP zero-shot evaluator in the supplementary Sec. 17. We collect labels from two human annotators on a 600-image stratified subset of generations, and use their agreement with GPT-4o labels to validate GPT-4o as a scalable visual oracle. We then recompute the parity gaps using this oracle as an additional evaluator. The resulting \(\Delta\) values show that the CLIP-based evaluation is conservative: for every evaluated attribute, the same EquiSteer interventions yield equal or larger reductions under GPT-4o. This effect is especially pronounced for race and eyeglasses, where CLIP exhibits per-class recall failures.

5 Conclusion↩︎

We presented EquiSteer, a training-free inference-time framework for debiasing text-to-image diffusion models via cross-attention modulation. EquiSteer uses precomputed steering vectors with adaptive magnitude to shift generations toward desired attributes, and a prompt-aware gating mechanism to skip intervention on attribute-specific prompts. Across multiple concepts and backbones, EquiSteer consistently improves fairness while maintaining text–image alignment and visual quality; it requires no retraining, operates per sample, and integrates naturally with standard sampling pipelines.

Acknowledgments↩︎

This work was supported by a Google DeepMind PhD Studentship, and the work utilized Queen Mary’s Andrena HPC facility, supported by QMUL Research-IT. This work was also supported by the Engineering and Physical Sciences Research Council [grant number EP/Y009800/1], through funding from Responsible AI UK (KP0016).

6 Limitations↩︎

While EquiSteer is a general-purpose inference-time framework that can be applied across different backbones and concepts, it also introduces several practical challenges and directions for future work. First, EquiSteer adds inference-time overhead due to the per-layer interventions in Eqs. 1 , 7 , and the re-normalization in Eq. 2 . Although this overhead is modest, it may still matter in latency-critical generation systems. Second, EquiSteer relies on concept-specific calibration, including the choice of steering magnitudes (via \(dp_{\text{mean}}^{a}\)) and gating thresholds \(thr^{a}\). In practice, the gate can be overly conservative, suppressing debiasing for some attribute-neutral prompts (false positives), and it can also be too permissive, allowing some attribute-specific prompts to pass through and be modified (false negatives). Improving the reliability of gating (e.g. via better attribute detectors, uncertainty-aware thresholds, or adaptive prompt-dependent calibration) is an important avenue for future work, especially for deployment settings with strict constraints on attribute preservation.

7 Future Work↩︎

While EquiSteer provides a general-purpose inference-time framework for debiasing along discrete attributes, our experiments focus on relatively simple prompt templates. Extending the method to more complex and compositional prompts will likely require adapting the calibration procedure (e.g., thresholds and steering magnitudes) to account for richer prompt structure. Such extensions could be integrated into real-world generative pipelines to improve fairness in practical deployments.

7.0.0.1 Entity-aware gating for complex prompts.

A promising direction is to apply EquiSteer’s gating mechanism at the level of individual prompt entities. For example, for prompts that mention multiple people, the system could decide separately for each person whether debiasing should be applied. Regions corresponding to different entities can be approximated using cross-attention map analysis as in Prompt-to-Prompt [6], enabling per-entity attribute detection and more robust handling of long or multi-entity prompts.

7.0.0.2 Targeted (localized) debiasing.

Another line of work is targeted debiasing, where intervention is applied only to specific entities or concepts in the prompt (e.g., debias doctors while leaving other professions unchanged). One approach is to localize the relevant image tokens by analyzing token-wise scores such as \(\langle ca^{out}_{ltk}, s^{a}_{lt}\rangle\) (cf.Fig. 6 (a)) and applying the update only to the subset of tokens associated with the target entity, potentially combined with cross-attention-based region extraction [6]. We leave a systematic study of such localized interventions to future work.

7.0.0.3 Limitations on compositional prompts.

Finally, the current gate operates at the image level, which can be overly conservative for compositional prompts that mix attribute-specific and attribute-neutral content (e.g., prompts describing multiple people with different levels of attribute specification). In such cases, the presence of any attribute-specific signal may suppress debiasing altogether. Extending the gate to be entity or region-specific, using cross-attention-based localization [6], is a natural direction to improve robustness to compositionality.

8 Construction of steering vectors↩︎

In this section we describe construction of steering vectors that are used in EquiSteer (Sec. 3.3, 3.4).

We closely follow CASteer [8] and construct steering vectors using pairs of contrastive prompts for each attribute of each concept. Below we give sets of prompts that we use for each attribute of each concept.

8.1 Gender concept↩︎

As gender is binary, with attributes male and female, we construct steering vectors from prompt pairs of the form \[(b_m\, c,\; b_f\, c),\] where \(b_m \in B_m\), \(b_f \in B_f\), and \(c \in C\).

The male prompt templates are:

• “a boy”

• “two men”

• “two male people”

• “a man”

• “an old man”

• “boys”

• “men”

• “group of male people”

• “a male human”

The female prompt templates are:

• “a girl”

• “two women”

• “two female people”

• “a woman”

• “an old woman”

• “girls”

• “women”

• “group of female people”

• “a female human”

The shared contexts are:

• “”

• “gloomy image”

• “zoomed in”

• “talking”

• “on the street”

• “in a strange pose”

• “realism”

• “colorful background”

• “on a beach”

• “playing guitar”

• “enjoying nature”

• “smiling”

• “in a futuristic spaceship”

• “with kittens”

Since \(|B_m| = |B_f| = 9\) and \(|C| = 14\), this yields a total of \(9 \times 14 = 126\) prompt pairs for each steering direction, i.e. for male\(\rightarrow\)female and female\(\rightarrow\)male.

8.2 Race concept↩︎

For race, we construct steering vectors from prompt pairs of the form \[(b\;\text{of}\;r\;\text{race}\;c,\; b\, c),\] where \(b \in B\), \(c \in C\), and \(r\) denotes the race attribute of interest.

The base prompt templates are:

• “a girl”

• “a boy”

• “two men”

• “two women”

• “two people”

• “a man”

• “a woman”

• “an old man”

• “an old woman”

• “boys”

• “girls”

• “men”

• “women”

• “group of people”

• “a human”

The shared contexts are:

• “”

• “gloomy image”

• “zoomed in”

• “talking”

• “on the street”

• “in a strange pose”

• “realism”

• “colorful background”

• “on a beach”

• “playing guitar”

• “enjoying nature”

• “smiling”

• “in a futuristic spaceship”

• “with kittens”

Since \(|B| = 15\) and \(|C| = 14\), this yields a total of \(15 \times 14 = 210\) prompt pairs for each race attribute.

8.3 Age concept↩︎

For age, we construct steering vectors from prompt pairs of the form \[(a\,b\,c,\; b\,c),\] where \(a\) denotes the age attribute of interest, \(a \in \{\textit{young}, \textit{middle-aged}, \textit{elderly}\}\), and the base subject templates \(b \in B\) and shared contexts \(c \in C\) are the same sets defined in Sec. 8.2.

Since \(|B| = 15\) and \(|C| = 14\), this yields \(15 \times 14 = 210\) prompt pairs for each age attribute.

8.4 Body type concept↩︎

For body type, we construct steering vectors from prompt pairs of the form \[(a\,b\,c,\; b\,c),\] where \(a \in \{\textit{slim}, \textit{average build}, \textit{heavy}\}\), and the base subject templates \(b \in B\) and shared contexts \(c \in C\) are the same sets defined in Sec. 8.2.

Since \(|B| = 15\) and \(|C| = 14\), this yields \(15 \times 14 = 210\) prompt pairs for each body-type attribute.

8.5 Eyeglasses concept↩︎

Eyeglasses is a binary attribute with values \(\{\textit{eyeglasses}, \textit{no eyeglasses}\}\). As diffusion models handle prompt negation poorly (prompts like “a man with no eyeglasses” can still produce eyeglasses), we do not compute a separate steering vector for the negated value; instead, we use a single steering vector for the eyeglasses direction and implement the no eyeglasses target through a CASteer-style erasure step at inference time (Sec. 12.3).

The eyeglasses steering vector is constructed from prompt pairs of the form \[(b\;\text{wearing eyeglasses}\;c,\; b\,c),\] where \(b \in B\) and \(c \in C\) are the same subject-template and context sets defined in Sec. 8.2.

Since \(|B| = 15\) and \(|C| = 14\), this yields \(15 \times 14 = 210\) prompt pairs for the single eyeglasses direction.

9 Gate threshold choice↩︎

The per-attribute gate threshold \(thr^{a}\) (Eq. 4 ) is the only quantity EquiSteer uses at inference time to decide whether a prompt is already attribute-specific. It is set once per (backbone \(\times\) attribute) cell as the midpoint between the empirical means of the maximal token response statistic \(dp^{a}_{l^{gate} 0}\) (Eq. 3 ) on a small set of neutral calibration prompts and a corresponding set of attribute-specific calibration prompts. This section lists the calibration prompts we use per attribute, how the means are estimated, and one practical note on the gender calibration that matters for reproducibility.

Estimation protocol. For each attribute \(a\), each calibration prompt \(p\) is rendered with \(10\) random seeds on the target backbone. For each generated image we extract the CA outputs at the chosen gating layer \(l^{gate}\) at \(t{=}0\), compute the per-direction maximal token response \(dp^{a}_{l^{gate} 0}\) (Eq. 3 ), and take the empirical means over (i) all images generated from neutral prompts and (ii) all images generated from attribute-specific prompts. The threshold \(thr^{a}\) is then the midpoint of these two means (Eq. 4 ).

Per-attribute calibration prompts. Tab. 4 lists the calibration prompts per attribute. For all the concepts except gender, we use concept-neutral subject pair (“a man”, “a woman”). For the gender concept this subject pair is not neutral, so we use a pair of professions that are not present in test prompts instead: (cleaner, counselor). The attribute-specific prompts are obtained by prepending the attribute value to the neutral prompt (e.g.“A photo of a male cleaner”, “A photo of an Asian man”, “A photo of a man wearing eyeglasses”, “A photo of an elderly woman”, “A photo of a heavy man”). For multi-class attributes (race, age, body type) one threshold is fitted per attribute value \(a\) from the (neutral, \(a\)-specific) pair; the inference-time gate fires when any per-direction threshold is exceeded (Eq. 5 ).

5pt

\(p \in \{\)cleaner, counselor\(\}\) for gender; \(s \in \{\)man, woman\(\}\)

for the other attributes.

10 Additional Comparisons↩︎

We additionally compare FairImagen and EquiSteer on the FairImagen evaluation setup. FairImagen prompts follow the template Generate a photo of a face of a {concept}, and we use the same prompt format when evaluating EquiSteer. For FairImagen, we use FairPCA with hidden dimension \(512\) and empirical noise parameter \(\epsilon=0.2\). These hyperparameters were selected on the FairImagen development set using their default protocol, namely SDXL¹ with batch size \(12\) and \(10\) diffusion steps.

We report a comparison on the FairImagen setup by aligning the prompt template, guidance scale, profession set, and number of diffusion steps between FairImagen and EquiSteer. Since FairImagen uses \(10\) diffusion steps in its main evaluation and hyperparameter search for efficiency, we show both \(10\)-step and \(28\)-step results.

We report both pooled \(\Delta\), computed from the aggregate demographic proportions over all generated images in a run, and macro-averaged \(\Delta\), where we first compute \(\Delta\) separately for each profession and then average across professions.

11 Algorithm↩︎

In this section, we present Algorithm 7, which summarizes the EquiSteer inference-time procedure. Note that for a fixed layer \(l\) and denoising step \(t\), the EquiSteer update is applied to all image tokens in parallel (i.e. via a batched tensor operation).

12 Debiasing additional concepts↩︎

In this section, we report results on applying EquiSteer to three further concepts beyond the gender: age (3-way: young, middle-aged, elderly), body type (3-way: slim, average build, heavy), and eyeglasses (binary). Age and body type use the same multi-class pipeline as the race experiments in Sec. 4.2.2; eyeglasses requires a small modification for the negated value, described in Sec. 12.3.

For all three attributes, we use the standard evaluation setup from Sec. 4.

12.1 Age↩︎

Method. Steering vectors and gate thresholds for the three age values are constructed as described in Sec. 8.3 and Sec. 9, respectively. We report results on two most capable diffusion backbones: SDXL and SANA-1.5

Evaluation protocol. Following our evaluation protocol, we use the CLIP zero-shot classifier with the templates “A photo of a young person”, “A photo of a middle-aged person”, and “A photo of an elderly person”. Each generated image is assigned to the class with the highest CLIP score.

Quantitative results. Tab. 6 reports the per-profession class ratios and parity gaps. On SDXL, EquiSteer reduces the mean parity gap from \(0.276\) to \(0.115\) (\(58\%\) reduction); on SANA-1.5, from \(0.387\) to \(0.060\) (\(85\%\) reduction). Note that vanilla SANA-1.5 is essentially mode-collapsed to middle-aged on five of eight professions (Tab. 6): CEO, doctor, pilot, teacher, and nurse all have \(\geq 87.5\%\) middle-aged generations, with \(0\%\) elderly. After EquiSteer applied, every profession on SANA-1.5 except teacher converges to within \(0.07\) of uniform across the three classes. The single resistant cell is teacher on SANA-1.5 (\(\Delta = 0.244\), still pulled toward middle-aged), reflecting an unusually strong profession-attribute prior that our gate-then-redistribute mechanism partially but not fully overcomes.

4pt

Qualitative results. Fig. 8 and Fig. 9 show side-by-side vanilla vs.EquiSteer generations for all eight professions on SDXL and SANA-1.5 respectively. For each profession, the top row shows vanilla generations and the bottom row shows EquiSteer. EquiSteer visibly increases the proportion of elderly subjects on professions where vanilla generates almost exclusively middle-aged people (e.g., CEO, doctor, pilot), and increases middle-aged / elderly subjects on professions where vanilla skews young (e.g., librarian, fashion designer on SANA-1.5).

12.2 Body type↩︎

Method. Steering vectors and gate thresholds for the three body-type values are constructed as described in Sec. 8.4 and Sec. 9, respectively. We report results on the two most capable diffusion backbones, SDXL and SANA-1.5.

Evaluation protocol. Following our evaluation protocol, we use the CLIP zero-shot classifier with the templates “A photo of a slim person”, “A photo of a average build person”, and “A photo of an heavy person”. Each generated image is assigned to the class with the highest CLIP score.

Quantitative results. Tab. 7 reports per-profession results. EquiSteer reduces the mean parity gap from \(0.317\) to \(0.220\) on SDXL (\(31\%\) reduction) and from \(0.390\) to \(0.231\) on SANA-1.5 (\(41\%\) reduction). Note that across all professions, EquiSteer increases the previously near-zero heavy class to between \(3\%\) and \(36\%\) on SDXL.

In Sec. 17 we additionally calibrate CLIP classifier with human evaluation and GPT-4o oracle. The results show that (i) GPT-4o oracle is well-agreed with human evaluation, and (ii) that when the CLIP labels are replaced with a GPT-4o oracle on the same generations for body type concept, the measured EquiSteer reduction is substantially larger: \(0.257 \to 0.098\) on SDXL (\(62\%\) reduction) and \(0.365 \to 0.210\) on SANA-1.5 (\(42\%\) reduction). This suggests, that CLIP classifier is less reliable on body labels, and CLIP numbers in Tab. 7 are therefore best read as conservative estimates of the underlying debiasing effect.

4pt

Qualitative results. Fig. 10 and Fig. 11 show side-by-side vanilla vs.EquiSteer generations on SDXL and SANA-1.5 respectively.

12.3 Eyeglasses↩︎

Method. The eyeglasses concept is binary, with two possible attribute values: (eyeglasses, no eyeglasses). In principle, EquiSteer would require a separate steering vector for each value. However, diffusion models are known to handle negation poorly (e.g., prompts such as “a man with no eyeglasses” may still produce eyeglasses), making the steering direction learned for no eyeglasses unreliable.

To handle such negated attributes, we treat no eyeglasses not as a separate “addition” direction, but as deletion of the eyeglasses attribute. Concretely, we use the CASteer [8] erasure update with the steering vector computed for eyeglasses and a fixed erasure strength \(\beta=2\). That is, when the sampled attribute is \(a=\textit{no eyeglasses}\), we replace Eq.8 with \[ca^{out\_new}_{lt} = ca^{out\_tmp}_{lt} - \beta\,\langle ca^{out\_tmp}_{lt}, s_{lt}^{\textit{eyeglasses}}\rangle\, s_{lt}^{\textit{eyeglasses}}. \label{eq:thr95add95no95glasses}\tag{8}\] We provide the corresponding procedure in Algorithm 12.

The single eyeglasses steering vector and its gate threshold are constructed as described in Sec. 8.5 and Sec. 9, respectively. We report results on the two most capable diffusion backbones, SDXL and SANA-1.5.

Evaluation protocol. In our main experiments on all the concepts, we use CLIP ViT-L/14 to measure attribute presence in generated images. For the eyeglasses concept, we initially follow the same protocol and evaluate a CLIP ViT-L/14 zero-shot classifier with the prompts “A photo of a person” and “A photo of a person wearing eyeglasses” (Tab. 8). However, we find that this CLIP setup tends to favor the more general prompt “A photo of a person”, which systematically underestimates the presence of eyeglasses. To obtain a more reliable measurement, we additionally report results using a VQA-based classifier, BLIP-vqa-capfilt-large [27]. Specifically, for each image we ask: “Is any person in this image wearing eyeglasses? Answer yes or no.” Results under this evaluation are shown in Tab. 8.In addition, in Sec. 17 we additionally present metrics obtained with GPT-4o oracle.

Quantitative results. Across all three backbones, EquiSteer consistently shifts the eyeglasses ratios toward the desired parity target (0.5) for most professions (Tab. 8). Under the BLIP-VQA evaluation (upper block of the same table), the effect is particularly clear for professions with strong initial skew, such as nurse (SD-1.5: \(0.10\!\to\!0.47\); SANA: \(0.00\!\to\!0.39\)) and pilot (SANA: \(0.17\!\to\!0.58\)), bringing the ratios substantially closer to the target. Under the CLIP zero-shot classifier (lower block), the qualitative trends are similar but the absolute scores are often lower, consistent with CLIP under-detecting eyeglasses due to the bias toward the generic prompt. Overall, these results confirm that EquiSteer can debias binary attributes such as eyeglasses, and highlight that VQA-based evaluation provides a more faithful signal for this concept than the CLIP prompt-pair classifier.

4pt

Qualitative results. Here we present qualitative results for debiasing the eyeglasses concept, complementing the quantitative evaluation. For each backbone and profession, we sample 8 random seeds and generate images with the vanilla model and with EquiSteer enabled. The results are shown in Fig. 13 (SD-1.5), Fig. 14 (SDXL), and Fig. 15 (SANA-1.5).

13 Details on metrics for gender concept↩︎

In this section, we provide a more detailed breakdown of main metrics for debiasing ‘gender’ concept reported in the main paper (Tab. 1, 5, Fig. [fig:gender95box95v2]). Tab. 9 and Tab. 10 provide detailed per-model and per-profession breakdowns.

14 Debiasing multiple concepts↩︎

In this section, we study multi-concept debiasing, where EquiSteer is applied to several protected attributes within the same denoising run. To handle \(K\) concepts, we apply the full EquiSteer intervention (gate evaluation, attribute-subspace orthogonalisation, target-attribute sampling, and adaptive re-injection, see Sec. 3.3–3.5) sequentially per concept at every cross-attention layer and denoising step. Each concept retains its own gate threshold \(thr^{a}\), attribute subspace, and adaptive magnitude calibrated as in the single-concept case; no joint retraining or finetuning is required.

We report joint gender \(+\) race debiasing results in Tab. 11, and gender \(+\) race \(+\) age \(+\) body type debiasing results in Tab. 12.

gender + race. On both backbones EquiSteer brings the per-profession gender ratio to within \(\Delta \leq 0.013\) of parity on SDXL and \(\Delta \leq 0.011\) on SANA-1.5, averaged over the eight evaluation professions (Tab. 11, top block) — comparable to single-attribute gender debiasing in the main paper. For race, EquiSteer reduces the parity gap from \(0.192 \to 0.049\) on SDXL and \(0.237 \to 0.074\) on SANA-1.5 (Tab. 11, bottom block).

gender + race + age + body type. The four-attribute setting (Tab. 12) preserves the joint-2 gender and race reductions while additionally cutting age and body type parity gaps.

Overall, the four interventions compose cleanly: no attribute is materially worsened by the addition of further attributes.

4pt

4pt

Qualitative results. Fig. 16 and Fig. 17 show 20 EquiSteer-debiased generations per profession on SDXL and SANA-1.5, under joint-2 and joint-4, respectively. The combined intervention visibly diversifies the generations across all attribute axes simultaneously, without producing the mixed-attribute artifacts (e.g. gender-mixed faces) that motivate the subspace-orthogonalisation step in Sec. 3.5.

15 Transferability across prompt families↩︎

In the main paper and the additional-attribute experiments (Sec. 12) we use the single template “A photo of a {profession}” to keep the evaluation comparable to prior work. In this section, we test EquiSteer on prompts that look quite different from this template at inference time to support generalization to diverse generation settings.

We consider three sets of prompt templates:

• Set A (paraphrased templates). Five variants of the standard surface form with no extra context: “A photo of a {prof}” (the paper baseline; key photo), “An image of a {prof}” (image), “A portrait of a {prof}” (portrait), “A picture showing a {prof}” (picture), and “A professional photograph of a {prof}” (professional).

• Set B (long contextual prompts). Four templates that add scene, lighting, and activity descriptions (\(\approx\)​10–20 tokens longer than the baseline): “A photo of a {prof} in a modern office, professional setting, clean lighting” (office), “A {prof} working late at night in their workplace, dramatic atmosphere” (latenight), “A portrait of a successful {prof} giving a presentation in a conference room, smiling at the audience” (presentation), and “A {prof} candidly captured at work, natural environment, high resolution” (candid).

• Set C (compositional / multi-subject prompts). Four templates with two professions in the same scene, evaluated on six profession pairs (CEO/doctor, pilot/technician, teacher/librarian, nurse/fashion designer, plus the cross-stereotype pairs CEO/nurse and pilot/librarian): “A photo of a {p\(_1\)} and a {p\(_2\)} working together” (working), “A {p\(_1\)} talking to a {p\(_2\)}, professional setting” (talking), “A {p\(_1\)} and a {p\(_2\)} in a meeting” (meeting), and “Two professionals: a {p\(_1\)} and a {p\(_2\)} side by side” (sidebyside).

We focus on the gender attribute throughout this section because it is the binary axis with the cleanest CLIP classifier signal and therefore the most reliable robustness probe. The same gender steering vector and threshold are used unchanged across all 13 templates.

4pt

5pt

Tab. 13 reports the the aggregate parity gap for each prompt family, and Tab. 14 gives the per-template breakdown. Across all three prompt families and both backbones, EquiSteer reduces the gender parity gap by 48–89%, with the standard-prompt reference falling within this range on both backbones (80% on SDXL, 79% on SANA-1.5). Neither rephrasing the surface form (Set A) nor adding scene and lighting context (Set B) materially degrades the gating-and-steering pipeline relative to the standard template. On SANA-1.5 the long-context set B in fact achieves the strongest reduction in this experiment (89%).

These results show that the steering vectors and thresholds calibrated from generic subject-and-context prompts transfer to a wide range of inference-time prompt surface forms and lengths, including multi-subject compositions, without any per-deployment recalibration.

16 Gate analysis↩︎

EquiSteer’s gating decision (“does this prompt already specify the target attribute?”) is made by thresholding the maximal token response statistic of Eq. 3 at a single cross-attention layer \(l^{gate}\) at the first denoising step. In this section addresses two questions: (i) does the realised inference-time gate actually separate attribute-specific from neutral prompts well, and (ii) how is \(l^{gate}\) chosen and how sensitive is the choice across backbones.

16.1 Gate separability↩︎

In this section, we assess gate mechanism separability quality. For each (backbone \(\times\) attribute) cell we generate \(n_{\text{pos}}\) images with attribute-specific calibration prompts (e.g.“A photo of a male cleaner”, “A photo of a White man”, “A photo of a man wearing eyeglasses”) and \(n_{\text{neg}}\) with the corresponding neutral prompts (“A photo of a cleaner”, “A photo of a man”, “A photo of a man”). At the chosen gating layer \(l^{gate}\) and \(t{=}0\) we compute, per generation, the dot product between the cross-attention output and the per-direction steering vector \(s^{a}_{l^{gate} 0}\), and take its maximum over image tokens (Eq. 3 ). The realised gate then fires whenever this statistic exceeds the per-direction threshold \(thr^{a}\) stored alongside the steering vector. We report the AUROC of this statistic, separating positive (attribute-specific) from negative (neutral) prompts, for each attribute direction.

Tab. 15 reports the per-direction AUROC across all (backbone \(\times\) attribute) cells. The realised inference-time gate achieves AUROC \(\geq 0.988\) on every cell with data, with a single per-direction minimum of \(0.899\) on SD-1.5’s male-to-female direction, which still signifies strong separability: this direction yields \(\mathrm{FPR}{=}0.05\) and \(\mathrm{FNR}{=}0.03\) at the empirical midpoint threshold, measured on the attribute-specific generations of Tab. 9. Five of the eight reported cells achieve perfect \(1.000\) across all attribute directions.

4pt

16.2 Layer-wise AUROC and \(l^{gate}\) selection↩︎

In this section, we provide the algorithm for choice of the gating layer \(l^{gate}\).

Procedure. The model-agnostic selection procedure is described in Algorithm 18). Using \(10\) attribute-specific and \(10\) neutral calibration prompts per direction, we dump the per-direction maximal token response statistic at every \((t, l)\) cell, compute the AUROC of this statistic on the calibration set per cell, and pick the cell with the highest AUROC as \(l^{gate}\). Ties are broken in favour of the smallest \((t, l)\) so that the gate fires at the earliest decisive layer.

Fig. 19 shows the AUROC across (denoising step \(\times\) CA block index) on the gender attribute for all three backbones in the main paper. On all three, the highest AUROC band is concentrated on the same one or two CA block indices across every denoising step (the colour bands run vertically in each panel). Algorithm 18 picks the layer at the right edge of this band: block 4 on SD-1.5, block 17 on SDXL, and block 5 on SANA-1.5. These are the layers that we use as \(l^{gate}\) in all our experiments.

16.3 Subtle and low-saliency attributes↩︎

The main paper and Sec. 12 of this supplementary validate EquiSteer on attributes with strong visual saliency — gender, race, age, body type, and eyeglasses. In this section, we test whether the gating mechanism, which relies on the diffusion model having internalised the target attribute as a separable direction in cross-attention output space, continues to work for less visually salient or more subtle demographic concepts. We then further evaluate gating mechanism robustness on such concepts as religion, disability, or socio-economic status.

We probe this question by measuring the gate’s separability AUROC — the ability of EquiSteer’s maximal token response statistic (Eq. 3 ) at the gating layer \(l^{gate}\) to distinguish attribute-specific from neutral prompts — across a deliberately chosen set of seven low-saliency concepts. AUROC at the gating layer is the upper bound on EquiSteer’s ability to detect and intervene on an attribute: if no choice of threshold separates positive from negative prompts at the gate, the rest of the pipeline (orthogonalisation and adaptive re-injection) cannot help. Conversely, AUROC close to \(1\) indicates that the attribute is internalised as a linearly separable direction and that EquiSteer can in principle gate on it.

For each concept we evaluate two prompt formulations, motivated by the observation that the diffusion model could in principle key on either a visible clothing marker (e.g., a hijab) or on the identity word (e.g., “muslim”) itself — and the two need not be encoded in the same direction:

• Visual marker prompts probe whether the model has internalised the attribute via a visible signifier: “a man wearing a hijab” (Muslim), “a man wearing a turban” (Sikh), “a man wearing a cross necklace” (Christian), “a man in a wheelchair” (disability), “a man with a heavy build” (heavy build), “a man wearing expensive elegant clothes and jewellery” (wealthy), “a man wearing worn out tattered clothes” (poor).

• Identity word prompts probe whether the identity term itself is internalised, independently of any visual marker: “a muslim man”, “a sikh man”, “a christian man”, “a disabled man”, “an obese man”, “a wealthy man”, “a poor man”.

For every (backbone \(\times\) concept \(\times\) formulation) cell we generate \(n_{\text{pos}}{=}20\) images with the attribute-specific prompt and \(n_{\text{neg}}{=}20\) with the neutral prompt “a man”, dump the cross-attention outputs at \(t{=}0\), score each image by the maximal token response statistic against the corresponding contrastive steering direction, and compute the resulting AUROC. The gate layer \(l^{gate}\) used is the same as in all other concepts.

6pt

Results. Tab. 16 reports the AUROC for all 28 (backbone \(\times\) concept \(\times\) formulation) cells. Every cell achieves AUROC \(=\) 1.000: the gate perfectly separates attribute-specific from neutral prompts in every condition, on both backbones, under both prompt formulations, for every concept tested. The same is true of AUPRC.

This shows that the gate works on subtle attributes just as well as on the headline attributes of the main paper. Religion, disability, body type, and socio-economic status seem to all be internalised by SDXL and SANA-1.5 as linearly separable directions in CA-output space at \(l^{gate}\).

Next, note that the two prompt formulations are equally separable: the identity-word prompts (e.g.“a muslim man”) achieve AUROC \(=\) 1 just as the visual-marker prompts (e.g.“a woman wearing a hijab”) do. This rules out a shortcut explanation in which the model only encodes the clothing marker rather than the identity itself — the identity term has its own internalised direction, distinguishable from the neutral subject “a man” at the gating layer.

However, this section measures whether the gate can detect the attribute, it does not run end-to-end debiasing for every subtle concept. The result we report — AUROC \(=\) 1 across all 28 cells — is the necessary condition that justifies extending EquiSteer to a new concept, while the sufficient condition adds the construction of contrastive prompts in the spirit of Sec. 8.1.

16.4 Threshold-multiplier sensitivity↩︎

The gating decision (Sec. 3.4) is parameterised by a single per-direction threshold \(thr^{a}\), calibrated once from contrastive prompt pairs and used unchanged across all (profession, prompt) combinations at inference time. In this section, we study sensitivity of the end-to-end debiasing performance to this threshold. We probe this by introducing a single multiplier \(m\) that scales every \(thr^{a}\) uniformly at inference time: the gate-firing condition becomes \[s \;=\; \max_{a}\bigl(dp^{a}_{l^{gate} 0} - m\cdot thr^{a}\bigr) \;>\; 0,\] where the gate fires (and EquiSteer skips debiasing for that image) whenever \(s > 0\). Two limits are interpretable: \(m{=}0\) sets every threshold to zero, so the gate fires for every prompt and debiasing is never applied, i.e. this recovers the vanilla model. Conversely, \(m \to \infty\) makes the gate never fire, so EquiSteer is applied unconditionally, and attribute-specific prompts such as “A photo of a male doctor” get incorrectly debiased. The paper default is \(m{=}1\).

We set \(m \in \{0.0, 0.5, 1.0, 1.5, 2.0\}\) in two prompt families: the standard single-template prompt used throughout the main experiments, and the long contextual templates from Sec. 15.

16.4.1 Standard prompts.↩︎

Tab. 17 reports the per-multiplier (parity gap, attribute-specific recall) pair for gender on all three backbones, and Tab. 18 reports the SDXL race sweep. As expected, the rows are monotonic: increasing \(m\) lowers the parity gap \(\Delta\) on neutral prompts (more debiasing applied) and eventually degrades the attribute-specific recall (incorrect debiasing of explicit prompts).

5pt

6pt

SDXL. The recall on attribute-specific prompts is essentially flat at \(\geq 0.98\) all the way up to \(m{=}2.0\), while \(\Delta\) falls monotonically from \(0.375\) (vanilla, \(m{=}0\)) to \(0.060\) at \(m{=}2.0\). The default \(m{=}1\) results in \(\Delta{=}0.120\), but for SDXL it is possible to gain a further \(\sim 2\times\) tightening with no loss in attribute preservation by using \(m{=}2\). We retain \(m{=}1\) as the paper default for consistency with the other backbones, but flag \(m{=}2\) as a free improvement when only SDXL is in scope.

SANA-1.5. The Pareto curve is sharper: \(m{=}1\) achieves \(\Delta{=}0.153\) with perfect recall, and \(m{=}1.5\) further tightens to \(\Delta{=}0.070\) with a small recall hit on the female direction (\(1.000 \to 0.895\)). At \(m{=}2.0\) recall collapses (\(0.555 / 0.590\)) — the gate has effectively been removed and attribute-specific prompts are now over-corrected. Therefore, SANA-1.5 should not be pushed above \(m{=}1.5\).

SD-1.5. shows intermediate behaviour. \(m{=}1.5\) approximately halves \(\Delta\) (\(0.253 \to 0.128\)) with only a mild recall loss on the female direction.

SDXL race. The race sweep shows different results compared to gender. (Tab. 18). \(\Delta\) is already small at vanilla, and bottoms out at \(m{=}1\) (\(\Delta{=}0.090\)). Pushing \(m\) higher does not help: \(\Delta\) stays around \(0.09\) but self-recall drops sharply (\(0.857 \to 0.616\)) — the gate closes less often on the explicit-race prompts and EquiSteer overwrites the requested race. Therefore, the default \(m{=}1\) is the right choice on SDXL race.

16.4.2 Long contextual prompts.↩︎

The gate-firing condition depends on the absolute magnitude of the maximal token response \(dp^{a}_{l^{gate} 0}\), which grows with prompt length: longer prompts produce more tokens with non-trivial overlap with the attribute steering direction, pushing more cells across the calibrated threshold and causing the gate to fire (and EquiSteer to skip) more often than on the short standard template. Tab. 19 sweeps \(m \in \{0.5, 1.0, 1.5, 2.0\}\) across the four long contextual templates of Sec. 15.

6pt

The default \(m{=}1\) leaves a noticeable residual gap on long prompts compared to standard prompts (\(\Delta_{\text{long}}{=}0.160\) vs \(\Delta_{\text{std}}{=}0.120\) on SDXL; \(0.156\) vs \(0.153\) on SANA-1.5). Increasing \(m\) to \(1.5\) on SANA-1.5 or to \(2.0\) on SDXL recovers roughly half of the residual gap:

• SANA-1.5. Mean \(\Delta\) falls from \(0.156\) at \(m{=}1\) to \(0.074\) at \(m{=}1.5\) (\(\sim 53\%\) further reduction). At \(m{=}2.0\) the mean settles around \(0.070\) but, as in the standard-prompt sweep, attribute-specific recall degrades; \(m{=}1.5\) is the safe operating point.

• SDXL. Mean \(\Delta\) falls from \(0.160\) at \(m{=}1\) to \(0.137\) at \(m{=}1.5\) and to \(0.078\) at \(m{=}2.0\) (\(\sim 51\%\) further reduction at \(m{=}2\)). The recall stability observed in the standard-prompt sweep (Tab. 17) carries over: \(m{=}2\) is again the safe operating point on SDXL.

Based on the observations above, we recommend lifting the threshold multiplier to \(m{\approx}1.5\) (SANA-1.5) or \(m{\approx}2\) (SDXL) for inference workloads that routinely use long contextual prompts. The default \(m{=}1\) remains the right setting for the standard short-template setting used throughout the experiments.

17 Classifier calibration and human evaluation↩︎

The main paper measures the parity gap \(\Delta\) using a CLIP ViT-L/14 zero-shot classifier. In this section, we address two natural concerns of this choice, namely: (i) does CLIP’s labelling agree with what a human would label, and (ii) when CLIP and a human disagree, does that disagreement systematically bias the reported parity gap in either direction?

We use the following procedure for calibration: first we collect labels from two human annotators on a stratified subset of generations. Second, we validate a strong VLM oracle (GPT-4o) against those human labels, showing that the two uniformly agree. This allows us to scale the human-style evaluation to all \(\sim 1{,}600\) images per cell without paying the per-image annotation cost. Third we use the GPT-4o oracle to compare CLIP against alternative classifiers (BLIP-VQA for eyeglasses) and to recompute \(\Delta\) on every attribute under each classifier.

17.1 Human evaluation↩︎

We sampled a \(600\)-image stratified pack from the generations: three attributes (gender, race, eyeglasses) \(\times\) two backbones (SDXL, SANA-1.5) \(\times\) two modes (vanilla, EquiSteer) \(\times\) \(50\) images per cell, drawn uniformly across the eight evaluation professions. Two in-lab annotators (H1, H2) independently labelled every image, blind to (mode, classifier predictions). The exact attribute label set is the same one used in the rest of the paper: \(\{\textit{male}, \textit{female}\}\) for gender; \(\{\textit{white}, \textit{black}, \textit{asian}, \textit{indian}, \textit{latino}\}\) for race; \(\{\textit{eyeglasses}, \textit{no\_eyeglasses}\}\) for eyeglasses. Annotators were allowed to mark images uncertain when no class was confidently identifiable.

Tab. 20 reports the pairwise agreement between annotators along with Cohen’s \(\kappa\) in the same table (left two columns). Agreement is substantial-to-perfect on every cell: \(\kappa \geq 0.98\) on the binary axes (gender, eyeglasses) on both backbones, and \(\kappa \in [0.71, 0.73]\) on the harder 5-way race axis. The lower race \(\kappa\) reflects the intrinsic difficulty of race classification from a single portrait — the two human annotators disagree with each other roughly as often as either disagrees with the strongest available oracle (Sec. 17.2). We use this human-agreement profile as the trust benchmark for the rest of the section.

5pt

17.2 GPT-4o as a scalable oracle, validated against humans↩︎

Hand-labelling all \(\sim 1{,}600\) images per cell across five attributes and two backbones would require on the order of \(20{,}000\) annotations per labeller, which is impractical at the scale needed to recompute \(\Delta\) end-to-end. We therefore use GPT-4o (via the API) as a scalable oracle: for each image we send a single-message prompt asking the model to assign one class from the attribute-specific label set, or uncertain if the image does not clearly fit any class. The same prompt formulation, label set, and uncertain option are used as for the human annotators.

To validate that GPT-4o tracks human judgements well enough to be used as a proxy at scale, we compare GPT-4o’s labels against H1 and H2 on exactly the same 600-image pack (Tab. 20, middle two columns). On every cell, GPT-4o agrees with each individual annotator at least as well as the two annotators agree with each other:

• On the two binary axes (gender, eyeglasses) GPT-4o \(\leftrightarrow\) human agreement is \(\geq 97.8\%\) on every cell, matching the \(99\)–\(100\%\) human-to-human agreement.

• On the harder 5-way race axis, GPT-4o \(\leftrightarrow\) H1 agreement is \(95.1\%\) on SDXL and \(95.3\%\) on SANA-1.5 — substantially higher than the \(78.7\%\) / \(80.0\%\) human-to-human agreement and the \(\sim 50\)–\(55\%\) human-to-CLIP agreement on the same images. GPT-4o \(\leftrightarrow\) H2 agreement is in a similar range (\(82.7\%\) / \(85.9\%\)).

The conclusion we draw is that GPT-4o is a defensible cheap proxy for human labels on this task: any conclusion about the relative performance of classifiers obtained against GPT-4o would also be obtained against the human annotators, on every cell we have ground truth for. The rest of the section uses GPT-4o as the oracle on the full \(n{\approx}1{,}600\) per-cell calibration set.

17.3 CLIP vs the GPT-4o oracle↩︎

For each of the \(10\) (backbone \(\times\) attribute) cells we re-evaluated the same EquiSteer / vanilla generations used elsewhere in this paper (8 professions, 2 modes, \(n{=}100\) images per (cell \(\times\) profession \(\times\) mode) cell, giving \(n_{\text{total}}{\approx}1{,}600\) per backbone-attribute cell). We dropped images on which the oracle returned uncertain from the comparison (\(n_{\text{clf}}\) in Tab. 21). For every retained image we have a CLIP zero-shot label, a GPT-4o label, and the (mode, profession) it came from. We compute (i) the overall CLIP \(\leftrightarrow\) GPT-4o agreement, (ii) the CLIP per-class recall when the oracle is taken as ground truth, and (iii) the parity gap \(\Delta\) under each label source.

5pt

Tab. 21 reveals a consistent pattern. CLIP is essentially calibrated for binary gender (\(98\)–\(99.7\%\) overall agreement, both per-class recalls \(\geq 0.98\)). On every other attribute it has a specific systematic failure mode — a single class with materially lower recall — and this failure mode is exactly what makes the reported \(\Delta\) a conservative estimate of EquiSteer’s true effect:

• Race. CLIP under-recognises white subjects: recall \(0.26\) on SDXL and \(0.05\) on SANA-1.5. CLIP routinely labels white as latino or asian. Because vanilla SDXL / SANA generate predominantly white subjects, CLIP under-reports the vanilla white-skew (vanilla \(\Delta_{\text{CLIP}}{=}0.145\) on SDXL vs \(\Delta_{\text{oracle}}{=}0.243\)). After EquiSteer adds non-white subjects, CLIP recognises them clearly, so the EquiSteer side moves much less. The net effect is that the CLIP-measured EquiSteer reduction on race is \(35\%\) on SDXL and \(41\%\) on SANA, while the oracle reduction is \(65\%\) and \(59\%\) respectively.

• Eyeglasses. CLIP under-detects the eyeglasses class (recall \(0.48\) on SDXL, \(0.79\) on SANA). EquiSteer reduces \(\Delta\) by \(39\%\)/\(29\%\) under CLIP and \(\mathbf{69\%}\)/\(\mathbf{39\%}\) under the oracle.

• Age. CLIP under-recognises young subjects (recall \(\sim 0.5\) on both backbones). The EquiSteer reduction is \(41\%\)/\(76\%\) under CLIP and \(\mathbf{72\%}\)/\(74\%\) under the oracle.

• Body type. CLIP recall is uniformly modest (\(0.45\)–\(0.63\)) across all three classes; slim and average are routinely confused. The EquiSteer reduction is \(39\%\)/\(27\%\) under CLIP and \(\mathbf{62\%}\)/\(\mathbf{43\%}\) under the oracle.

Therefore, CLIP-only \(\Delta\) values reported in the main paper are conservative on every multi-class attribute. The actual debiasing effect, measured under the oracle that aligns with humans, seems to be consistently at least as large and often substantially larger.

17.4 Eyeglasses three-way classifier comparison↩︎

In Sec. 12.3 we report eyeglasses results using CLIP ans BLIP-VQA classifiers, ans argue that CLIP under-detects the eyeglasses class. Here we use GPT-4o to check whether the BLIP-VQA classifier is a better proxy for human-aligned eyeglasses labels than CLIP is.

Tab. 22 reports the three-way comparison on the same \(\sim 1{,}600\) generations per backbone used above. The results show that BLIP-VQA agrees with the GPT-4o oracle on \(94\)–\(97\%\) of images on both backbones (vs CLIP’s \(67\)–\(87\%\)), its per-class recall on the eyeglasses class is \(0.97\) (SDXL) / \(1.00\) (SANA) (vs CLIP’s \(0.48\) / \(0.79\)), and the EquiSteer reduction it measures matches the oracle to within \(\pm 3\) percentage points: on SDXL BLIP says \(0.270 \to 0.085\) (\(69\%\)) and the oracle says \(0.289 \to 0.088\) (\(70\%\)). This validates the choice of BLIP-VQA as the better eyeglasses classifier, as it seems to be a much closer proxy for the human-aligned oracle than CLIP zero-shot.

6pt

17.5 Human labels distributions↩︎

Tab. 23 reports the class distributions of vanilla and EquiSteer generations as labelled by the humans, on the same \(n{=}50\) images-per-(cell \(\times\) mode) subset used for the agreement analysis.

4pt

The most legible cell is race on both backbones, where both vanilla models generate predominantly white subjects (\(66\)–\(68\%\)) with the black and indian classes essentially absent (\(0\)–\(8\%\)). After EquiSteer the white share drops to \(\sim 15\%\) on both backbones, every minority class rises to at least \(14\%\), and on SDXL all five classes lie within \(\pm 12\) pp of the uniform target. The human-measured \(\Delta\) on race drops from \(0.192 \to 0.063\) on SDXL (\(67\%\) reduction) and \(0.198 \to 0.044\) on SANA-1.5 (\(78\%\) reduction). The other axes show smaller (gender) or noisier (\(n{=}50\) eyeglasses, sometimes already-balanced vanilla) changes.

18 More qualitative results on race concept↩︎

In this section, on Fig. 20, 21 and 22 we provide more qualitative results on debiasing race concept.

19 Ablations on EquiSteer components↩︎

In this section we ablate parameters of EquiSteer.

19.1 EquiSteer components↩︎

We ablate the contribution of EquiSteer’s main components: (i) the basic steering update (Eq. 1 ) with adaptive magnitude selection (Eq. 7 ), (ii) orthogonalisation, and (iii) the gating mechanism (Eq. 5 ). Tab. 24 reports gender debiasing results for three variants:

• add: basic steering (Eq. 1 ) with adaptive magnitude (Eq. 7 );

• add & erase: add + orthogonalisation;

• add & erase & gate: full EquiSteer (adds gating).

The results highlight the role of each component. add improves parity on attribute-neutral prompts, but also alters generations for attribute-specific prompts. Adding orthogonalisation (add & erase) further improves neutrality debiasing, yet substantially degrades attribute preservation, indicating that removing pre-existing attribute signals can conflict with prompts that explicitly specify an attribute. Finally, incorporating the gate (add & erase & gate) restores attribute preservation on attribute-specific prompts while maintaining strong debiasing performance on neutral prompts.

2.5pt

References↩︎

---

1. The Fairimagen paper is based on SD3 whereas our work compares with the shared Fairimagen implementation of SDXL on the same setup↩︎