1 Introduction↩︎

Vision Language Models (VLMs) have demonstrated exceptional zero-shot capabilities across a wide range of multimodal tasks [@deng2009imagenet; @plummer2015flickr30k], reaching the stage of real-world applications. By learning joint representations from web-scale image-text pairs, these models achieve strong cross-modal alignment without task-specific fine-tuning. However, this same training paradigm causes VLMs to internalize social stereotypes [@birhane2021multimodal] present in their training data, leading to biased outputs that reflect and potentially amplify social prejudices [@hall2023genderdisparities; @hamidieh2024implicitbias; @janghorbani2023multimodalbias; @zhao2021racialbiascaptioning; @wolfe2023contrastive; @hausladen2025social]. These biases manifest most critically in person-centric queries, where models produce skewed demographic distributions. For instance, querying “a photo of a doctor” yields disproportionately male images, or certain occupations become strongly associated with specific racial groups. Such behavior poses serious risk of reinforcing discriminatory decision-making.

Existing debiasing approaches for VLMs [@wang-etal-2021-gender; @Chuang_2023_CVPR; @Zhang_2025_CVPR] share common design philosophy: they apply fixed bias correction uniformly across all language queries, regardless of whether individual queries are sensitive to demographic biases. Although conceptually simple, this uniform mitigation strategy causes a fundamental fairness–utility trade-off that we illustrated in Figure 1 (b). Here, we use utility to refer to general-purpose zero-shot performance on downstream tasks (e.g., image classification and cross-modal retrieval).

For bias-sensitive queries, the model’s predictions are often entangled with demographic attributes, and strong debiasing is necessary for fair outcomes. In contrast for bias-insensitive queries, ground-truth semantics are largely orthogonal to demographic attributes, and the model’s original predictions already reflect accurate cross-modal alignment. When a uniform debiasing framework is applied to both types of queries, one of two failure modes inevitably occurs. As we demonstrate in Figure 2, this inflexibility creates a trade-off where existing methods must compromise between fairness in sensitive queries and utility in general tasks, unable to excel at both simultaneously.

We believe that the aforementioned limitation is a consequence of the query-independent design paradigm. Since uniform debiasing methods cannot distinguish between inputs that require mitigation and those that do not, they are constrained to operate in a compromise regime. This observation motivates a paradigm shift toward adaptive debiasing that selectively activates mitigation based on the bias sensitivity of each input. To this end, we propose Reward-Gated Test-Time Adaptation for CLIP (RG-TTA), a reinforcement learning (RL)-based framework designed for selective debiasing. Our key insight is that debiasing should be treated as a per-query decision rather than global transformation, enabling the model to adapt its behavior dynamically based on input characteristics. As illustrated in Figure 3, RG-TTA operates through an episodic test-time adaptation protocol. For each incoming query, we first assess its bias sensitivity by quantifying the alignment discrepancy between the query semantics and a set of demographic attributes. Based on this assessment, an adaptive reward-gating strategy dynamically triggers the fairness-regularized objective only for bias-sensitive queries, ensuring the preservation of the original cross-modal alignment for neutral inputs.

Empirical evaluations on multiple fairness benchmarks–including FairFace [@karkkainen2021fairface], UTKFace [@zhang2017utkface], and the challenging FACET [@gustafson2023facet] dataset–demonstrate that RG-TTA significantly reduces social bias across various demographic attributes. Notably, our framework effectively resolves the fairness–utility trade-off by achieving substantial bias reduction alongside higher accuracy on tasks such as ImageNet-1K [@deng2009imagenet] compared to existing query-independent baselines. By performing episodic optimization with this gated reward, RG-TTA provides a practical design principle for mitigating bias without broadly disrupting the alignment of vision-language models.

2 Related Work↩︎

2.0.0.1 Social debiasing in vision-language models.

Social debiasing for CLIP-style VLMs is broadly categorized into (i) training-based approaches [@alabdulmohsin2024clipthebias; @hirota2025biasgenderbenchmarks; @Zhang_2025_CVPR], which suppress sensitive-attribute signals by introducing additional objectives or modules during learning, and (ii) training-free approaches [@Chuang_2023_CVPR; @gerych2024bendvlm], which keep the foundation model fixed and apply post-hoc adjustments to embeddings or outputs. The former includes joint debiasing methods that align and remove biases across both modalities, while the latter estimates bias directions from attribute prompts and removes them via lightweight projections. However, most existing methods remain query-independent, applying a single globally-defined transformation to all queries. A recent attempt toward query-adaptive debiasing is SANER [@hirota2025saner], which performs selective debiasing by restricting intervention to attribute-neutral text descriptions; yet its adjustment is confined to text features at training time, limiting adaptability to diverse input distributions encountered at deployment. Consequently, such fixed or partially selective rules still struggle to optimize for both objectives simultaneously, forcing a compromise between effective bias mitigation and the preservation of general model utility.

2.0.0.2 Test-Time Adaptation of CLIP via Reinforcement Learning.

Test-time Adaptation (TTA) aims to improve model performance on unlabeled test inputs by performing parameter updates at inference time, enabling models to adapt to distribution shifts without retraining [@sun2020ttt; @wang2021tent]. Early TTA methods relied on auxiliary self-supervised objectives such as entropy minimization [@sun2020ttt; @liu2021tttpp; @wang2021tent]. However, these approaches often suffer from instability from objective mismatch or prediction collapse [@park2025retta], motivating recent interest in utilizing VLM’s internal alignment signal as direct feedback. In particular, treating CLIP similarity as explicit reward and performing RL-based optimization [@zancato2023retrievaltta] at inference time [@zhao2024ttaclipreward] has been shown to reduce the collapse behavior observed in entropy-based updates and to improve zero-shot generalization. We build upon the feedback-driven TTA paradigm and extend it to address social bias in VLMs. Our key contributions are (1) a bias-sensitivity gate that activates debiasing per input, and (2) a reward that combines cross-modal alignment with a bias-subspace debiasing signal. With episodic test-time updates and selective reward gating, RG-TTA mitigates the fairness–utility trade-off, improving fairness and general-purpose utility.

3 Method↩︎

We propose Reward-Gated Test-Time Adaptation (RG-TTA), an RL-based framework for selective debiasing that adaptively updates model parameters during inference. RG-TTA is built on two key components. First, a selective gating strategy evaluates query sensitivity to trigger fairness regularization only when necessary. Second, an adaptive reward function balances CLIP alignment with an attribute-balancing reward, which encourages a uniform representation by favoring under-represented attributes. We update the model using a tractable approximation of the REINFORCE [@williams1992simple] algorithm. By focusing updates on the most relevant candidates through top-\(K\) truncation, this approach ensures that the optimization stays centered on query-specific semantics and limits drift via episodic resets.

3.1 Preliminaries↩︎

3.1.0.1 CLIP.

A pretrained vision-language model (VLM) consists of an image encoder \(f(\cdot)\) and a text encoder \(g(\cdot)\), which project both modalities into a shared embedding space [@radford2021clip]. Given a query \(q\) and a candidate \(y \in \mathcal{Y}\) selected from the opposite modality, we define an alignment score \(S(q,y)\) as the cosine similarity between their embeddings. Specifically, in the text-to-image setting we use \(S(q,y)=\cos\!\big(g(q), f(y)\big)\), whereas in the image-to-text setting we use \(S(q,y)=\cos\!\big(f(q), g(y)\big)\). We use the policy score \(S_\theta(q,y)\) for candidate ranking and parameterizing the policy. We use the reference score \(S_{\mathrm{ref}}(q,y)\) only to compute the CLIP reward; \(S_{\mathrm{ref}}\) is obtained from a fixed CLIP ViT-L/14 model throughout all experiments.

3.1.0.2 Test-time adaptation in vision–language tasks.

Test-time adaptation (TTA [@sun2020ttt; @wang2021tent]) updates a trained model at inference time using a few unlabeled steps. We follow an episodic protocol: each query \(q\) is adapted independently and the parameters are reset before the next query, which mitigates negative transfer. For vision–language retrieval, where \(q\) can be text or an image, we adapt only the query-modality encoder and keep the opposite-modality encoder fixed.

3.2 Selective Gating Strategy↩︎

Applying debiasing uniformly to all queries is unnecessary and can induce parameter drift on bias-insensitive queries. To mitigate the structural fairness–utility trade-off, we introduce a gating mechanism that measures the alignment discrepancy between query semantics and demographic attributes. For each query \(q\), we select the top-1 candidate \(y^*=\arg\max_y S_\theta(q,y)\) as an anchor and assess whether the query–candidate alignment is disproportionately driven by demographic associations by comparing \(y^*\) to the mean attribute similarity over a predefined attribute set \(\mathcal{A}\) of protected-group exemplars. We instantiate \(\mathcal{A}\) in the same modality as the query (e.g., attribute prompts for text and exemplar images for vision), so that \(S_\theta(a,y^*)\) uses the same scoring function; concrete instantiations are given in Sec. 4.4.

The gating logic is based on the following intuition: a large discrepancy suggests that the match is driven by general, attribute-independent semantic alignment, in which case we deactivate fairness regularization (\(\delta(q)=0\)) and focus on refining standard cross-modal alignment to enhance utility. Conversely, if \(y^*\) is close to the attribute alignment distribution (within a threshold \(\epsilon\)), the signal is likely entangled with a specific attribute category, triggering an attribute-balancing reward (\(\delta(q)=1\)). Accordingly, the gate is defined as:

\[\label{eq:saad} \delta(q)=\mathbb{I}\Big[\, S(q,y^*)-\frac{1}{|A|}\sum_{a\in A} S(a,y^*) < \epsilon \Big].\tag{1}\]

Here, \(\mathbb{I}[\cdot]\) denotes the indicator function and \(\epsilon\) is a fixed threshold (\(0.02\)) kept constant throughout the episode to prevent parameter drift.

3.3 Adaptive Reward Function↩︎

In this section, we define a candidate-wise reward \(r(q,y_k)\) for each candidate \(y_k \in \mathcal{Y}_K(q)\), where \(\mathcal{Y}_K(q)\) denotes the candidate set used for adaptation. We describe the construction of \(\mathcal{Y}_K(q)\) in Sec. 3.4. Our reward always includes a CLIP-based alignment signal to preserve general cross-modal matching ability, and adds an attribute-balancing reward only when the gate is activated. This design maintains a balance between alignment and debiasing within each episode.

3.3.0.1 CLIP alignment reward.

We follow prior work in defining a CLIP-based alignment reward [@zhao2024ttaclipreward]. Concretely, for each candidate \(y_k\) we compute a nonnegative alignment score as \(s_k=\max\!\big(S_{\mathrm{ref}}(q,y_k),0\big)\), and use the episode mean \(\bar{s}=\frac{1}{K}\sum_{j=1}^{K}s_j\) as a baseline. The resulting baseline-normalized alignment reward is

\[\label{eq:rclip} r_{\text{clip}}(q,y_k)= s_k-\bar{s}.\tag{2}\]

3.3.0.2 Bias subspace and attribute-balancing reward.

When \(\delta(q)=1\), we incorporate an attribute-balancing reward computed in a predefined bias subspace into the reward. Let \(\mu_c \in \mathbb{R}^{D}\) denote the class-mean embedding for class \(c\) in the shared embedding space, precomputed from a labeled source dataset and kept fixed during test-time adaptation. We further define a reference vector \(\bar{\mu}\), which can be instantiated as the average of \(\{\mu_c\}_{c=1}^{C}\), and construct the set of difference vectors \(\{\mu_c-\bar{\mu}\}_{c=1}^{C}\). We then perform PCA on these differences and extract all nonzero principal components, yielding \(d\) components in total. The resulting orthonormal basis \(U \in \mathbb{R}^{D \times d}\) defines the bias subspace¹

For each selected candidate \(y_k\), let \(z_k\) denote the embedding produced by the frozen encoder of the opposite modality. We then map both candidates and class prototypes into the bias subspace using the projection operator \(P=UU^\top\), yielding \(\tilde{z}_k = P(z_k-\bar{\mu})\) and \(\tilde{\mu}_c = P(\mu_c-\bar{\mu})\). Next, we compute a soft assignment over classes by normalizing Gaussian-kernel similarities with a temperature parameter \(\gamma\). Defining the squared distance between the projected candidate and class prototype as \(d_{kc}=\|\tilde{z}_k-\tilde{\mu}_c\|_2^2\), the soft assignment is given by:

\[\alpha^{(k)}_c= \frac{\exp(- d_{kc}/\gamma)}{\sum_{c'=1}^{C}\exp(- d_{kc'}/\gamma)}. \label{eq:soft-assign}\tag{3}\]

To estimate the attribute distribution at the episode level, we compute the popularity for each class as \(p_c=\frac{1}{K}\sum_{j=1}^{K}\alpha_c^{(j)}\). We then define the attribute-balancing reward for each candidate as \[d(q,y_k)=\sum_{c=1}^{C}\alpha_c^{(k)}\!\left(p_c-\frac{1}{C}\right).\] This score reflects how strongly candidate \(y_k\) is associated with attribute classes that are over- or under-represented in the current episode, encouraging a more balanced set of selected candidates.

3.3.0.3 Final combined reward.

Finally, we define the overall reward for each selected candidate by combining the CLIP alignment reward with the query-conditioned attribute-balancing reward: \[r(q,y_k)= r_{\text{clip}}(q,y_k) - \delta(q)\,\lambda\, d(q,y_k), \label{eq:final95reward}\tag{4}\] where \(\lambda\) is a hyperparameter that controls the strength of the attribute-balancing reward. When \(\delta(q)=0\), the reward reduces to the CLIP alignment term alone. When \(\delta(q)=1\), candidates associated with over-represented classes (i.e., \(p_c > 1/C\)) tend to have larger \(d(q,y_k)\) and thus incur a larger balancing penalty, while candidates linked to under-represented classes (i.e., \(p_c < 1/C\)) tend to have smaller or negative \(d(q,y_k)\) and are relatively favored. As a result, the episode-level attribute distribution is encouraged toward the uniform prior.

3.4 Optimization and Episodic Update↩︎

We optimize the proposed objective at test time via episodic policy-gradient updates [@wang2021tent; @shu2022testtime], treating each input query \(q\) as an episode. We first evaluate the gating indicator \(\delta(q)\) to determine whether to include the attribute-balancing term in the episode reward. Conditioned on this decision, we choose a candidate budget \(K\) (e.g., \(K=10\) when \(\delta(q)=0\) and \(K=1024\) when \(\delta(q)=1\)), and construct a truncated candidate set \(\mathcal{Y}_K(q)\) by selecting the top-\(K\) candidates from the fixed pool \(\mathcal{Y}\) according to the alignment score \(S_\theta(q,y)\). We then compute the candidate-wise reward \(r(q,y_k)\) for each \(y_k \in \mathcal{Y}_K(q)\) and update the query-modality encoder parameters \(\theta\) with a small number of gradient steps. After the episode ends, we reset the parameters to their initial state before processing the next query, mitigating negative transfer across queries. We define the policy \(\pi_\theta(y \mid q)\) over the full candidate pool \(\mathcal{Y}\) using a softmax of the alignment score:

\[\pi_{\theta}(y \mid q) = \frac{\exp\!\big(S_{\theta}(q,y)\big)}{\sum_{y' \in \mathcal{Y}} \exp\!\big(S_{\theta}(q,y')\big)} . \label{eq:policy}\tag{5}\] Although \(\pi_\theta\) is normalized over the full candidate pool \(\mathcal{Y}\) (i.e., the denominator is computed over \(\mathcal{Y}\)), we approximate the policy-gradient objective by summing only over the truncated set \(\mathcal{Y}_K(q)\) for the current query. Concretely, for each episode we minimize the following REINFORCE-style objective: \[\mathcal{L}(q)=-\frac{1}{K}\sum_{y_k\in\mathcal{Y}_K(q)} r(q,y_k)\,\log \pi_\theta(y_k \mid q).\] This top-\(K\) truncation based on \(S(q,y)\) yields a tractable approximation to the full policy-gradient objective while focusing updates on the most relevant candidates for the query.

2.8pt

4 Experiments↩︎

4.1 Datasets↩︎

To comprehensively evaluate both debiasing performance and generalization, we used a diverse set of benchmarks. For fairness evaluation, we consider in-domain settings on FairFace (val) [@karkkainen2021fairface] and UTKFace [@zhang2017utkface], and an out-of-domain setting on FACET [@gustafson2023facet]. To assess whether adaptation preserves general-purpose zero-shot utility, we additionally evaluate on ImageNet-1K [@deng2009imagenet] for image classification and Flickr1k [@plummer2015flickr30k] for retrieval.

4.2 Metrics↩︎

4.2.0.1 Fairness metrics.

Following prior work [@berg-etal-2022-prompt; @seth-etal-2023-dear], we use retrieval-based metrics: MaxSkew@\(k\) and NDKL@\(k\). These metrics quantify the disparity in the distribution of protected attributes (e.g., gender, age, and race) within the top-k retrieved images for neutral queries. MaxSkew measures the maximum representation of a dominant group, while NDKL measures the divergence from uniform distribution. For both metrics, lower values indicate a fairer model.

4.2.0.2 Utility (V–L alignment) metrics.

To ensure that our selective debiasing does not compromise the intrinsic V-L alignment of the pre-trained model, we evaluated zero-shot performance on standard benchmarks. We report Top-1 and Top-5 accuracy on ImageNet-1K[@deng2009imagenet] for classification, and Recall@5 for both Image-to-Text (TR) and Text-to-Image (IR) retrieval on Flickr1k[@plummer2015flickr30k].

4.2.0.3 Alignment and Bias Level Evaluation (ABLE).

Single metrics capture either fairness or utility, but not their trade-off. We use ABLE proposed by Zhang et al.[@Zhang_2025_CVPR] for a holistic assessment. ABLE is defined as the harmonic mean of the zero-shot accuracy and the fairness score: \[\mathrm{ABLE} = \frac{2}{\frac{1}{acc} + \frac{1}{\exp(-\mathrm{MaxSkew}@k)}}\] where \({acc}\) denotes the ImageNet Top-1 accuracy. Higher ABLE indicates a better balance between mitigating social bias and retaining zero-shot accuracy; we report ABLE\(\times\)​100 (%) in tables.

3.5pt

4.3 Baselines↩︎

We compare RG-TTA against the Original CLIP and three representative debiasing baselines using ViT-B/16 and ViT-B/32 backbones. CLIP-clip [@wang-etal-2021-gender] removes bias-correlated embedding dimensions identified via mutual information with attribute labels. Biased-prompts [@Chuang_2023_CVPR] neutralizes bias directions by projecting embeddings using prompt-derived attribute subspaces without retraining. Joint V-L [@Zhang_2025_CVPR] jointly debiases image and text representations to mitigate over-debiasing effects. All methods are evaluated under identical settings for Gender, Age, and Race.

4.4 Implementation Details↩︎

We use the official pretrained CLIP checkpoints [@radford2021clip] with ViT-B/16, ViT-B/32, and ViT-L/14 backbones. Consistent with the episodic TTA protocol, we update only the encoder corresponding to the input query modality (e.g., the text encoder for text-to-image retrieval) while keeping the target modality encoder frozen. Optimization is performed using AdamW [@loshchilov2019decoupled]. Crucially, to balance efficiency and performance, we dynamically adjust the computational budget based on the gating decision \(\delta(q)\): we perform a lightweight update with \(T{=}3\) steps and \(K{=}10\) candidates for bias-insensitive queries (\(\delta(q){=}0\)), while expanding to \(T{=}10\) steps and \(K{=}1024\) candidates for bias-sensitive ones (\(\delta(q){=}1\)). Detailed hyperparameters, including learning rates, gating thresholds, and prompt templates, are provided in Appendix 7.

4.5 Results↩︎

Table 1 summarizes bias-mitigation results across different source datasets (UTKFace and FairFace) for Gender and Age. Overall, our method improves fairness in both in-domain and out-of-domain settings while enhancing ImageNet zero-shot performance. For Gender, we observe a clear in-domain improvement on UTKFace (MaxSkew@\(k\): 0.114→0.051), with particularly large mitigation under distribution shift on FACET (MaxSkew@\(k\): 0.478→0.053). Meanwhile, ImageNet Top-1 accuracy increases from 68.31 to 70.38, and ABLE also rises from 77.39 to 80.87, indicating that fairness gains do not come at the expense of utility (see Table 1 for full results).

This trend is consistent across attributes and source configurations: for Age and Race, we observe substantial in-domain fairness gains while maintaining (or slightly improving) zero-shot utility (Tables 1, 2). Switching the source dataset to FairFace shows the same behavior across in-domain and out-of-domain settings, aligning with our design—gating with episodic resets—that focuses updates on bias-sensitive queries while preventing drift on others.

5pt

5 Discussion↩︎

5.1 Ablation Study↩︎

We ablate key design choices to examine how selective reward-gating mitigates the fairness–utility trade-off.

5.1.0.1 Candidate budget and debiasing strength.

Figure 4 ablates the hyperparameters used when debiasing is active, varying the candidate budget \(K\) and the debiasing strength \(\lambda\) in the \(\delta(q)=1\) regime. We observe that increasing \(K\) generally improves MaxSkew@1000 (notably for Age), suggesting that a larger truncated set yields a more stable estimate of the episode-level attribute distribution and thus a more reliable balancing signal. Varying \(\lambda\) shows a similar trend: weak values yield limited mitigation, while larger values deliver consistent gains with diminishing returns, suggesting performance is not brittle once \(\lambda\) passes a minimal effective threshold.

5.1.0.2 Selective vs.uniform debiasing.

We next examine when to apply the debiasing signal versus how strongly to apply it. Table 3 compares (i) an alignment-only reward, (ii) adding the attribute-balancing term uniformly to all queries, and (iii) activating the balancing term only when the gating indicator \(\delta(q)\) triggers. While uniform debiasing can substantially reduce MaxSkew, it risks unnecessary adaptation and notably degrades zero-shot utility; in contrast, selective activation preserves utility while retaining the fairness gains, yielding a markedly better overall balance.

5.1.0.3 Gating threshold and activation behavior.

We characterize the gate through threshold sensitivity and activation patterns under a dataset-level proxy setting. Figure 5 shows that across a broad range of the threshold \(\epsilon\), fairness improves rapidly and remains largely stable, and utility is stable around the default \(\epsilon{=}0.02\), with noticeable degradation only when \(\epsilon\) is overly large and activates debiasing too aggressively. We also evaluate the gate’s activation behavior (Table 6) under a proxy labeling scheme: treating UTKFace/FACET queries as debiasing-needed and ImageNet-1K queries as debiasing-not-needed, the gate activates for almost all UTKFace/FACET queries while remaining inactive for nearly all ImageNet-1K queries, yielding low false negative/positive rates in this proxy setting.

5.1.0.4 Reward model scale.

A natural concern is that RG-TTA relies on a larger reference model (ViT-L/14) to compute the CLIP reward, which may be undesirable when the backbone itself is smaller. We test whether fairness gains depend on this scale mismatch by replacing the reward model with a same-size variant, matching the backbone (Table 4). The results reveal a clear pattern: fairness gains are largely independent of reward model scale. For ViT-B/16, the same-size reward achieves MaxSkew of 0.050, essentially matching the 0.051 obtained with the larger ViT-L/14 reward, and a similar trend holds for ViT-B/32 (0.057 vs.). The benefit of the larger reward model instead manifests in utility preservation: ImageNet accuracy improves more substantially when using ViT-L/14 as reward (e.g., B/16: 68.31\(\rightarrow\)​70.38 vs.; B/32: 63.39\(\rightarrow\)​69.76 vs.). Importantly, the same-size configuration still improves ABLE over Original CLIP in all cases, confirming that RG-TTA remains practically deployable when an external larger reference model is unavailable, while the larger reward model offers additional utility headroom when available.

5.2 Runtime Analysis↩︎

While RG-TTA introduces per-query test-time optimization, understanding its computational cost is essential for assessing practical deployment. We measure per-query wall-clock time on a single NVIDIA A6000 GPU using ViT-B/16, averaged over 1000 queries.

5.2.0.1 Overhead analysis.

As shown in Table 5, RG-TTA incurs additional latency over Original CLIP due to alignment discrepancy computation and policy gradient updates. The overhead ranges from \(\sim\)​25\(\times\) for bias-insensitive queries (\(\delta(q){=}0\)) to \(\sim\)​104\(\times\) for bias-sensitive queries (\(\delta(q){=}1\)), reflecting the difference in candidate budget \(K\) and update steps \(T\).

5.2.0.2 Practical implications.

Despite this overhead, three factors support the practical viability of RG-TTA. First, both conditions remain sub-second, which is acceptable for fairness-critical batch-processing scenarios such as automated hiring systems or medical image retrieval, where per-query latency of hundreds of milliseconds is tolerable. Second, adaptive budgeting effectively reduces average overhead: as reported in Table 6, the majority of general queries are classified as bias-insensitive, so the average latency in practice approaches \(\sim\)​111 ms rather than the worst-case \(\sim\)​467 ms. Third, RG-TTA is entirely training-free; unlike offline debiasing methods that require per-domain fine-tuning, RG-TTA can be instantly applied to new domains or attributes by replacing the bias subspace, offering a favorable amortized cost in deployment scenarios with evolving fairness requirements.

5.3 Out-of-Domain Analysis on FACET↩︎

On FACET (out-of-domain), our method transfers well for Gender, maintaining lower MaxSkew@\(k\) than prior methods. We hypothesize that Gender is associated with multiple robust visual cues (e.g., face, hairstyle, clothing), which preserve group separation even after bias-subspace projection and lead to more stable debiasing signals. In contrast, Age relies on fine-grained facial cues that are sensitive to distance, resolution, and occlusion; in FACET’s unconstrained images, these cues are weakened, reducing the reliability of the debiasing signal and ultimately limiting gains.

4pt

5.4 Gating Failure Case Analysis↩︎

Figure 6 shows false-negative cases on FACET, where bias-sensitive queries are gated off (\(\delta(q)=0\)) and thus do not trigger fairness regularization. These failures often arise when salient object and scene semantics in the top-1 anchor \(y^*\) overwhelm demographic-correlated cues, causing the semantic–attribute discrepancy to exceed the threshold.

6 Conclusion↩︎

We introduced Reward-Gated Test-Time Adaptation (RG-TTA), a selective test-time debiasing framework for CLIP-style vision–language models. RG-TTA uses an input-dependent reward gate to activate an attribute-balancing term only when necessary, together with episodic updates and parameter resets to limit unintended drift. Experiments across in-domain and out-of-domain fairness benchmarks as well as standard zero-shot utility tasks show that RG-TTA consistently reduces demographic skew while maintaining competitive utility. Overall, our results highlight selective, input-conditioned adaptation as a practical design principle for mitigating bias without broadly disrupting model behavior.

Limitations↩︎

Our approach relies on test-time adaptation (TTA) with per-query parameter updates, which can increase computation and latency, especially when using many update steps or large candidate sets. Because offline-trained debiasing methods amortize cost during training whereas TTA incurs cost online, direct runtime comparisons across these paradigms are not always apples-to-apples and can vary with the deployment scenario. Our experiments focus on a single protected attribute; extending the framework to multiple attributes requires more complex reward/constraint design and may introduce conflicting objectives. The method also assumes access to an external reward signal from a stronger model, which raises availability and cost considerations and may transfer the reward model’s own biases into the adaptation signal. Moreover, our fairness objective implicitly targets proximity to a chosen reference distribution (e.g., uniform), and both evaluation and reward depend on the accuracy and domain robustness of attribute estimators; TTA behavior can further be sensitive to hyperparameters and may be unstable for some queries. Future work will develop more efficient update schemes to reduce online overhead. We will also explore scalable multi-attribute objectives/constraints and robustness techniques to mitigate sensitivity to reward sources and attribute estimators.

Ethics Statement↩︎

This work proposes a selective test-time adaptation (TTA) approach to mitigate distributional biases over protected attributes (e.g., gender, age, and race) that can arise for person-centric queries. Our experiments use publicly available fairness evaluation datasets (e.g., FairFace, UTKFace, and FACET) together with standard utility benchmarks, and quantify bias using distribution-based fairness metrics. Because face images and protected-attribute annotations can be sensitive, our study does not aim to identify individuals and assumes use strictly in accordance with the datasets’ licenses and usage conditions. Our method further relies on an external reward signal from a stronger model (e.g., a fixed CLIP ViT-L/14 reference), which introduces practical considerations about the availability and cost of such signals and raises the possibility that biases present in the reward model could be propagated through the adaptation process. In addition, our fairness objective implicitly assumes a chosen target distribution (e.g., a uniform prior), which may not be appropriate for all tasks or domains. We therefore recommend that any real-world deployment be accompanied by careful auditing of the reward source and attribute estimators for bias and error, and that use in high-stakes decision-making contexts be avoided or subjected to additional, domain-specific validation and oversight.

Acknowledgments↩︎

This work was partly supported by Institute of Information and Communications Technology Planning and Evaluation (IITP) grant funded by the Korea Government (MSIT) (No. RS-2022-II220184, Development and Study of AI Technologies to Inexpensively Conform to Evolving Policy on Ethics), partly supported by the Institute of Information and Communications Technology Planning and Evaluation (IITP) grant funded by the Korea Government (MSIT) [RS-2021-II211341, Artificial Intelligence Graduate School Program (Chung-Ang University)], and partly supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) [RS-2026-25498346].

[@*]

7 Implementation Details↩︎

In this section, we provide comprehensive details regarding the experimental setup, optimization hyperparameters, and the construction of the bias subspace, complementing the summary provided in the main text.

7.1 Computational Environment and Models↩︎

All experiments were conducted on a single NVIDIA A6000 GPU with automatic mixed precision (AMP) enabled to enhance efficiency. We utilized the official pre-trained CLIP checkpoints [@radford2021clip] for ViT-B/16, ViT-B/32, and ViT-L/14 backbones. Consistent with the episodic TTA protocol, the CLIP reward signal was computed using a separate, fixed CLIP ViT-L/14 model to provide stable guidance.

7.2 Modality-Specific Update Strategy↩︎

During the episodic adaptation, we update only the encoder corresponding to the input query modality to align it with the frozen target modality:

• Text-to-Image Retrieval: We adapt the text encoder while keeping the image encoder frozen.

• Image-to-Text Retrieval & Classification: We adapt the image encoder while keeping the text encoder frozen.

7.3 Optimization Hyperparameters↩︎

We used the AdamW optimizer [@loshchilov2019decoupled] with a learning rate of \(1 \times 10^{-4}\) and a weight decay of \(0\). The balancing coefficients for the reward function were set to \(\lambda = 1000\) (debiasing penalty weight). Additionally, we set the Gaussian kernel temperature parameter \(\gamma = 0.25\) to control the sharpness of soft assignments in the bias subspace.

7.4 Adaptive Computational Budgeting↩︎

The Reward Gate (RG) determines the bias sensitivity using a fixed threshold of \(\epsilon = 0.02\). Based on the gate’s output \(\delta(q)\), we dynamically adjust the number of update steps (\(T\)) and the candidate pool size (\(K\)):

• Bias-Insensitive (\(\delta(q)=0\)): We perform a lightweight update with \(T = 3\) steps and \(K = 10\) candidates.

• Bias-Sensitive (\(\delta(q)=1\)): We increase the budget to \(T = 10\) steps and \(K = 1024\) candidates. In this case, candidates are ranked by their CLIP alignment scores to filter the most relevant samples for the update.

3.5pt

7.5 Bias Subspace Construction Details↩︎

The bias subspace is constructed offline using the training split of the dataset:

• Text Queries: We use attribute-specific prompts formatted as “a photo of a {attribute class} person”. One prompt is generated per attribute class.

• Image Queries: We construct the reference set \(A\) by sampling \(M = 5\) images per attribute class uniformly from the UTKFace dataset. To ensure reproducibility and consistency, this reference set is fixed once and reused across all test episodes. The total size of the reference set is \(|A| = C \times 5\), where \(C\) is the number of attribute classes.

8 Additional Results↩︎

8.1 Additional Race Debiasing Results using FairFace↩︎

In the main text (Table 2), we utilized UTKFace as the source dataset for constructing the bias subspace to mitigate Race bias. To verify the robustness of our framework across different source domains, we conducted an additional experiment using FairFace as the source dataset.

The results are presented in Table 7. Consistent with the findings in the main text, our RG-TTA framework demonstrates a superior capability to balance fairness and utility. Regarding fairness, our method effectively mitigates racial bias compared to the Original CLIP; while Joint V-L shows competitive scores on ViT-B/16, our method achieves the best performance on ViT-B/32 (MaxSkew: 0.389). Crucially, in terms of utility preservation, our method consistently outperforms all baselines in zero-shot tasks (ImageNet and Flickr) across both backbones, confirming that our selective routing mechanism successfully prevents the over-debiasing observed in static approaches. Consequently, our method achieves the highest ABLE scores for both backbones (69.62% and 68.74%), proving that it maintains the optimal trade-off between fairness and utility regardless of the source dataset used.

8.2 False-positive gating failures on ImageNet↩︎

Figure 7 illustrates false-positive cases on ImageNet-1K, where bias-insensitive object queries are incorrectly gated on (\(\delta(q)=1\)), triggering unnecessary fairness regularization. These failures typically arise when the queried object strongly co-occurs with humans or human-like features in the top-1 retrieved anchor \(y^*\). For instance, as shown in Figure 7, human figurines on a cake (left) or bystanders in the background (right) provide strong demographic signals that reduce the semantic-attribute discrepancy below the threshold \(\epsilon\). This misleads the gate into treating the object query as bias-sensitive. Although our hybrid reward design minimizes semantic drift even when the gate is mistakenly active, these cases represent a computational inefficiency.

8.3 Attribute-Specified Queries↩︎

A natural question is whether the gating mechanism may overreact to queries that explicitly specify a demographic attribute (e.g., “a photo of a male doctor”), thereby distorting query intent. Since such queries are likely to align with the corresponding attribute prompts in \(\mathcal{A}\), the gate may activate \(\delta(q){=}1\) and apply the attribute-balancing reward, which could in principle override the user’s specified attribute. To quantitatively assess this behavior, we evaluate on FACET (ViT-B/16) using gender-specified queries. We introduce the Attribute Consistency Rate (ACR@\(k\)): the proportion of top-\(k\) retrieved results whose gender matches the query-specified gender. Higher values indicate better preservation of query intent.

As expected, gender-specified queries trigger \(\delta(q){=}1\), since they are close to the corresponding attribute prompts. Nevertheless, RG-TTA limits ACR degradation to only 1.5–2.6 percentage points compared to Original CLIP, because the CLIP alignment reward \(r_{\text{clip}}\) in Eq. 5 assigns higher scores to gender-matching candidates, effectively counterbalancing the attribute-balancing penalty. To verify this, we evaluate a Debiasing Only variant that removes \(r_{\text{clip}}\) and applies the attribute-balancing reward alone: ACR@10 drops sharply by 0.16 (0.692\(\rightarrow\)​0.532), confirming that the CLIP reward is the key component preventing this failure mode.

8.4 Intersectional Debiasing↩︎

Real-world fairness concerns often involve multiple protected attributes simultaneously (e.g., gender and race). RG-TTA accommodates intersectional settings by defining the class set \(\mathcal{C}\) in Eq. 3–4 as a Cartesian product of attributes (e.g., gender \(\times\) race \(\rightarrow\) ), without modifying the pipeline. Rather than introducing separate reward terms per attribute (which can cause multi-objective conflicts), this consolidates multi-attribute balancing into a single reward over cross-product classes.

As shown in Table 9, RG-TTA reduces Gender MaxSkew by 36.8% and Race MaxSkew by 33.7% simultaneously, confirming effective multi-attribute balancing. As a test-time method, only the class definition needs to change—no retraining is required. We acknowledge that exponential growth of cross-product classes (e.g., 2\(\times\)​4\(\times\)​3=24 for three attributes) may degrade the popularity estimate \(p_c\), and a more scalable formulation for high-dimensional intersectional settings remains an open direction.

8.5 Generalization to BLIP↩︎

To verify that RG-TTA generalizes beyond CLIP to other contrastive vision-language models, we apply it to BLIP-base (Salesforce/blip-itm-base-coco) using its image-text contrastive (ITC) branch. We evaluate on UTKFace with gender as the protected attribute.

RG-TTA achieves a 55.1% reduction in MaxSkew and a 61.7% reduction in NDKL on BLIP, confirming applicability to contrastive VLMs beyond CLIP. Notably, no retraining is required; the entire framework operates at test time.

8.5.0.1 Threshold recalibration.

Different architectures produce different similarity score distributions, so the gating threshold \(\epsilon\) does not transfer directly across models. Applying CLIP’s default \(\epsilon{=}0.02\) to BLIP yields 86.2% \(\delta(q){=}1\) on UTKFace and 8.1% on ImageNet (compared to 99.9% / 0.1% on CLIP). However, recalibration only involves adjusting a single scalar with a small validation set—no architectural changes or retraining are needed.

8.6 Generalization Beyond Demographic Bias↩︎

While RG-TTA primarily targets social fairness with demographic attributes, the underlying mechanism—selective debiasing in a bias subspace—is in principle applicable to other types of distributional bias. To briefly probe this generalization, we apply RG-TTA to the Waterbirds benchmark, where bias arises from spurious correlations between bird species and background scenes (water vs.land), rather than from protected demographic attributes. We set the background as the bias subspace attribute and measure the distribution bias in top retrieval results for neutral queries (e.g., “a photo of a bird”).

By simply redefining the bias subspace attribute (demographic \(\rightarrow\) background), RG-TTA reduces MaxSkew by 22.2% and NDKL by 47.8% without retraining. While a thorough investigation in spurious-correlation settings is beyond the scope of this work, this result suggests broader applicability of the proposed selective adaptation principle.

---

1. Projection-based techniques have also been used for group robustness on spurious correlations [@zhu2025project], in a different (training-time) setting and formulation.↩︎