Abstract

Automatic metrics are widely used to evaluate text-to-image models, often replacing human judgment in benchmarking, model selection, and large-scale data filtering. Yet they may reward images that look plausible or prototypical rather than images that faithfully satisfy the prompt. We identify prototypicality bias as a systematic blindspot in multimodal evaluation: metrics can prefer a semantically incorrect but visually or socially prototypical image over a correct but less prototypical one. We introduce ProtoBias, a controlled diagnostic benchmark across Animals, Objects, and Demography, where semantically correct images are contrasted with plausible prototypical adversaries containing a single controlled semantic violation. Grounded in prototype theory and social-category prototypicality, ProtoBias is constructed with multiple prompt generators, image generators, and independent VLM filters, and validated through prompt-quality, human-annotation, and image-quality controls. Using ProtoBias, we show that widely used embedding, reward, VQA-based, and VLM-as-judge metrics frequently fail these contrasts, while human judgments remain more faithful to semantic correctness. We further introduce ProtoScore, a lightweight contrastively trained evaluator, as an initial mitigation baseline. ProtoBias provides a focused benchmark for measuring prototypicality-driven metric failures and developing more semantically faithful T2I evaluators.

1 Introduction↩︎

Automatic metrics are now central to text-to-image (T2I) evaluation. They are used to compare models, select checkpoints, filter large-scale generated data, and report progress when human evaluation is too expensive to run at scale. This makes their failure modes consequential: if a metric systematically rewards the wrong image, that error can shape model development and benchmark conclusions. In this paper, we study one such failure mode: metrics may prefer images that look familiar, canonical, or socially typical, even when those images violate the prompt. We call this failure mode prototypicality bias. The term builds on prototype theory, which shows that categories have graded internal structure: some instances are perceived as more central or representative than others [1]–[4]. For example, a robin is a more prototypical bird than a penguin, and a chair is a more prototypical piece of furniture than a bean bag. Prototypicality is useful for human categorization, but it is not the same as semantic correctness. An image can look like a familiar category member while still failing to satisfy the prompt.

Figure 1: Illustration of prototypicality bias in T2I evaluation. The prompt asks for an animal near a pond with exactly one small branch in the background. The semantically correct image (I_{\mathrm{corr}}) satisfies the specified branch condition but depicts a less prototypical animal. The prototypical adversarial image (I_{\mathrm{adv}}) depicts a more familiar animal, but violates the branch condition. A metric passes this diagnostic case if it scores I_{\mathrm{corr}} higher than I_{\mathrm{adv}}. — Figure 1: Illustration of prototypicality bias in T2I evaluation. The prompt asks for an animal near a pond with exactly one small branch in the background. The semantically correct image (\(I_{\mathrm{corr}}\)) satisfies the specified branch condition but depicts a less prototypical animal. The prototypical adversarial image (\(I_{\mathrm{adv}}\)) depicts a more familiar animal, but violates the branch condition. A metric passes this diagnostic case if it scores \(I_{\mathrm{corr}}\) higher than \(I_{\mathrm{adv}}\).

In T2I evaluation, this distinction matters because automatic metrics are widely used as scalable substitutes for human judgment in benchmarking, model selection, and data filtering [5]–[10]. We define prototypicality bias as a metric-level blindspot in which a visually or socially prototypical image is scored as more faithful to the prompt than a semantically correct but less prototypical image. Figure 1 illustrates the setup. The prompt asks for an animal near a pond with exactly one small branch in the background. The correct image \(I_{\mathrm{corr}}\) depicts a cockroach and satisfies the branch constraint, whereas the adversarial image \(I_{\mathrm{adv}}\) depicts a more prototypical animal, a cat, but violates the branch condition. Thus, both images are plausible under the broad category term “animal”, but only \(I_{\mathrm{corr}}\) satisfies the full prompt. Such bias is problematic because metrics trained on large-scale vision–language data may learn to reward canonical visual forms or socially dominant associations rather than semantic faithfulness [11], [12], thereby overestimating models that generate familiar but incorrect images and underestimating less common but correct generations.

To measure this failure mode, we introduce ProtoBias, a controlled diagnostic benchmark that places semantic correctness in direct conflict with prototypicality. Each instance pairs a neutral prompt \(T\) with a semantically correct image \(I_{\mathrm{corr}}\) and a prototypical adversarial image \(I_{\mathrm{adv}}\) that violates one visually checkable semantic attribute, such as count, color tone, spatial relation, scale, or foreground/background placement. The benchmark spans Animals, Objects, and Demography: the first two test category typicality, while Demography tests socially learned demographic–attribute associations rather than intrinsic identity typicality. By using controlled contrastive examples, ProtoBias isolates prototypicality-driven metric failures that may be hidden by aggregate benchmark scores [13]–[15].

In summary, we make the following contributions:

Bias formulation. We define prototypicality bias as a metric-level blindspot in which visually or socially prototypical images are preferred despite explicit semantic violations.
Diagnostic benchmark. We introduce ProtoBias, a controlled benchmark spanning Animals, Objects, and Demography. Each example is a contrastive triplet \((T, I_{\mathrm{corr}}, I_{\mathrm{adv}})\) designed to isolate semantic correctness from prototypical familiarity. We validate the benchmark through prompt-quality checks, VLM-based image filtering, human validation of filtering decisions, pairwise human evaluation, and image-quality controls.
Metric analysis and mitigation baseline. We evaluate embedding-based, reward-based, VQA-based, and VLM-as-judge metrics on ProtoBias, showing that many fail specifically under neutral prompts where prototypicality conflicts with semantic correctness. As an initial mitigation baseline, we introduce ProtoScore, a contrastively trained evaluator for this diagnostic setting.

2.0.0.1 Bias and prototypicality in multimodal systems.

Large-scale text-to-image (T2I) and vision-language models (VLMs) frequently reproduce social and cultural biases [12], [16]. Extensive research documents representational disparities across demographics and social cues, driving the creation of benchmarks to measure harms in generated images and VLM predictions [11], [12], [16]–[23]. While related, our work targets a different object of study: the evaluation metrics themselves. Furthermore, ProtoBias builds on prototype theory, which posits that categories have graded structures where some instances are more central or representative [1]–[4].

2.0.0.2 Robustness and diagnostic evaluation of T2I metrics.

Automatic metrics—including embedding-based methods (CLIPScore [5]), reward models (PickScore, ImageReward [6], [7]), VQA metrics (VQAScore [8]), and LLM-as-judge systems [9], [10]—are central to T2I evaluation. However, these proxies for human judgment often rely on spurious correlations, dataset artifacts, or superficial cues rather than genuine semantic grounding [24]–[30]. Similar to how meta-evaluation exposes metric failures in NLP [13], [14], multimodal benchmarks like Winoground [15], CROC [31], PREXME [32], and Metalogic [33] test contrastive robustness and preference inconsistencies under controlled variations. ProtoBias complements these diagnostic efforts by isolating a specific failure mechanism: whether metrics inherently prefer prototypical but semantically incorrect images over correct but non-prototypical ones. Thus, our dataset provides a controlled environment to rigorously evaluate the evaluators for prototypicality-driven biases.

3 Bias Formulation↩︎

We use prototype in the sense of graded category structure: within many categories, some instances are perceived as more central, familiar, or representative than others [1]–[4]. For example, a robin is a more prototypical bird than a penguin, and a chair is a more prototypical piece of furniture than a bean bag. We call this relative centrality prototypicality. Prototypicality is distinct from semantic correctness: an image may look canonical or familiar while still violating the prompt. We define prototypicality bias as a metric-level blindspot in which a visually or socially prototypical image is scored as more faithful to the prompt than a semantically correct but less prototypical image. In ProtoBias, each example contains a neutral prompt \(T\), a correct image \(I_{\mathrm{corr}}\), and a prototypical adversarial image \(I_{\mathrm{adv}}\). The correct image satisfies all semantic requirements in \(T\), while the adversarial image is more prototypical but violates one controlled requirement, such as count, color tone, spatial relation, or foreground/background placement. For Animals and Objects, prototypicality refers to canonical category members or layouts. For Demography, we do not claim that identities are intrinsically more or less typical; instead, we study social prototypicality: overrepresented demographic–attribute associations learned from data and related to stereotype-based social-category prototypes [12], [34], [35]. Given a metric \(M(T,I)\), a semantically faithful metric should score the correct image higher than the adversarial one. We count a prototypicality-bias failure when: \[\label{eq:failure95equation} M(T, I_{\mathrm{adv}}) \geq M(T, I_{\mathrm{corr}}).\tag{1}\] That is, the metric assigns the prototypical but semantically wrong image a score at least as high as the correct image. This criterion directly tests whether prototypicality is strong enough to override prompt faithfulness.

4 ProtoBias Benchmark Construction↩︎

Figure 2 provides the roadmap for this section. We first describe how controlled prompt triplets \((T,D_c,D_a)\) are constructed and validated, and then explain how these triplets are rendered into image pairs \((I_{\mathrm{corr}}, I_{\mathrm{adv}})\), filtered for semantic correctness, checked for visual-quality confounds, and validated through human evaluation.

4.1 Prompt Construction and Validation↩︎

ProtoBias is constructed through the pipeline shown in Figure 2. The first stage builds controlled prompt triplets \((T,D_c,D_a)\), where \(T\) is the neutral prompt used for evaluation, \(D_c\) is the correct generation description, and \(D_a\) is the adversarial generation description. This stage defines the taxonomy, instantiates the prototype contrast, introduces one controlled semantic perturbation, filters invalid prompts, and validates that the resulting triplets are localized semantic edits rather than broad rewrites.

Figure 2: Overview of the ProtoBias construction pipeline. A taxonomy entry and scene template define a neutral prompt T, a correct description D_c, and an adversarial description D_a. In the illustrated example, T names only the broad category “bird,” D_c instantiates the less-prototypical penguin while preserving the scene, and D_a instantiates the more-prototypical robin while changing one controlled spatial detail. The validated descriptions are rendered into I_{\mathrm{corr}} and I_{\mathrm{adv}}, filtered independently against their own descriptions, and finally evaluated under the same neutral prompt T. — Figure 2: Overview of the ProtoBias construction pipeline. A taxonomy entry and scene template define a neutral prompt \(T\), a correct description \(D_c\), and an adversarial description \(D_a\). In the illustrated example, \(T\) names only the broad category “bird,” \(D_c\) instantiates the less-prototypical penguin while preserving the scene, and \(D_a\) instantiates the more-prototypical robin while changing one controlled spatial detail. The validated descriptions are rendered into \(I_{\mathrm{corr}}\) and \(I_{\mathrm{adv}}\), filtered independently against their own descriptions, and finally evaluated under the same neutral prompt \(T\).

4.1.0.1 Taxonomy.

ProtoBias spans three domains: Animals, Objects, and Demography. Animals and Objects instantiate prototypicality as category typicality: some category members are more central or familiar than others while remaining valid members of the same broad class [1], [36]–[40]. Animals include contrasts such as penguin–robin, platypus–dog, and walking stick insect–cat. Objects cover furniture, vehicles, and tableware, contrasting peripheral instances such as bean bag, golf cart, or tuk-tuk with more canonical instances such as chair, car, or bicycle. The full taxonomy and prompt templates are provided in Appendix 8.

Demography is handled separately. We do not claim that demographic identities are intrinsically more or less typical. Instead, we use prototypicality in a representational sense: socially learned and dataset-mediated associations between demographic groups, attributes, and contexts. Following work on social prototypes, stereotypes, and multimodal bias [34], [35], [41]–[45], we instantiate three demographic axes—Religion, Nationality, and Sexual Orientation—crossed with five socio-attributes: Wealth, Intellect, Morality, Power, and Civility.

4.1.0.2 Prompt triplets.

For each taxonomy entry, we construct a triplet \((T,D_c,D_a)\). The neutral prompt \(T\) uses a broad category label such as “animal,” “bird,” “vehicle,” or “person,” and specifies a simple scene with one visually checkable semantic attribute. The correct description \(D_c\) replaces the broad label with the less-prototypical instance while preserving the scene. The adversarial description \(D_a\) replaces the broad label with the more-prototypical instance and changes exactly one controlled semantic attribute. As shown in Figure 2, \(T\) may describe a bird on a rock near a tree stump; \(D_c\) instantiates a penguin and preserves the scene, while \(D_a\) instantiates a robin but changes the spatial relation from behind to in front of. We use four adversarial knobs: count, color tone, layout relation, and spatial placement. These attributes are explicit in text, visually verifiable, and common in T2I faithfulness evaluation. The semantic change is applied only to an auxiliary scene element, such as stones or branches for animals, lamps or cones for objects, and books or laptops for people. This ensures that the adversarial image should be more prototypical, but wrong on one prompt-relevant detail.

4.1.0.3 Prompt generation and filtering.

We generate candidate triplets \((T,D_c,D_a)\) from structured scene templates using three instruction-tuned LLMs: Qwen2.5-14B-Instruct [46], Gemma-3-12B-it [47], and Llama-3.1-8B-Instruct [48]. Raw LLM outputs may be malformed, switch domains, substitute the wrong target instance, or change more than the intended semantic knob. We therefore filter candidate triplets with Qwen2.5-32B-Instruct [46] and Mistral-Small-3.1-24B-Instruct [49], retaining only prompts accepted by all filters.¹ This yields \(8{,}225\) text triplets: \(2{,}932\) Animals, \(3{,}027\) Demography, and \(2{,}266\) Objects. (Filtering prompts are provided in Appendix 10.)

4.1.0.4 Prompt-quality validation.

Before image generation, we validate that accepted triplets are controlled semantic perturbations rather than broad rewrites. We use BGE-large sentence embeddings [50] to measure similarity among \(T\), \(D_c\), and \(D_a\), and DeBERTa-v3 NLI [51] as a lightweight text-only consistency check. The correct description \(D_c\) should remain compatible with \(T\), while \(D_c\) and \(D_a\) should contradict each other on the controlled attribute. Table 1 shows high similarity between \(T\) and both descriptions (\(0.848\) for \(T,D_c\); \(0.854\) for \(T,D_a\)), while \(D_c\) and \(D_a\) are contradictory in \(96.5\%\) of cases. This indicates that the text triplets preserve the scene while introducing the intended semantic conflict.

Table 1: Automatic prompt-quality validation for \(8{,}225\) accepted ProtoBias text triplets. \(T\) is the neutral prompt, \(D_c\) the correct description, and \(D_a\) the adversarial description. Similarity uses BGE-large sentence embeddings; NLI uses DeBERTa-v3 and is applied only to text pairs. The bidirectional contradiction score averages both directions between \(D_c\) and \(D_a\).
Prompt-quality check	Animal	Demo.	Object	Overall
Semantic preservation
\(\mathrm{sim}(T, D_c)\)	0.877	0.793	0.886	0.848
\(\mathrm{sim}(T, D_a)\)	0.888	0.781	0.908	0.854
\(\mathrm{sim}(D_c, D_a)\)	0.803	0.757	0.851	0.800
NLI-based structural validity
\(T \rightarrow D_c\) ent./neu. (%)	94.2	100.0	100.0	97.9
\(T \rightarrow D_a\) ent./neu. (%)	64.0	32.9	59.7	51.4
\(D_c \leftrightarrow D_a\) contra. (%)	97.0	100.0	91.4	96.5

4.0pt

4.2 Image Construction and Validation↩︎

Figure 3: ProtoBias examples.Each row shows a neutral prompt T, a semantically correct image I_{\mathrm{corr}} (SC), and a prototypical adversarial image I_{\mathrm{adv}} (PA). SC satisfies the prompt, while PA is more prototypical but violates one controlled semantic attribute. Check marks indicate that the evaluator ranks SC above PA; crosses indicate that it ranks PA higher than SC. — Figure 3: **ProtoBias examples.**Each row shows a neutral prompt \(T\), a semantically correct image \(I_{\mathrm{corr}}\) (SC), and a prototypical adversarial image \(I_{\mathrm{adv}}\) (PA). SC satisfies the prompt, while PA is more prototypical but violates one controlled semantic attribute. Check marks indicate that the evaluator ranks SC above PA; crosses indicate that it ranks PA higher than SC.

The second stage renders the validated prompt triplets into image pairs and verifies that the resulting images support the intended contrastive evaluation. For each accepted triplet, \(D_c\) is rendered into the semantically correct image \(I_{\mathrm{corr}}\), and \(D_a\) is rendered into the prototypical adversarial image \(I_{\mathrm{adv}}\). Both images are evaluated under the same neutral prompt \(T\) in the final benchmark task.

4.2.0.1 Image generation.

We render \(D_c\) and \(D_a\) separately using three T2I models: FLUX.1-schnell [52], Stable Diffusion 3.5 Large Turbo [53], and Sana-1600M-1024px [54]. This produces candidate pairs \((I_{\mathrm{corr}}, I_{\mathrm{adv}})\) from the accepted text triplets. Since the two images are generated from matched descriptions, the intended difference is the prototype contrast plus the single controlled semantic violation in \(D_a\).

4.2.0.2 Image filtering.

Image filtering is necessary because T2I models may fail to realize the requested subject, count, color, layout, or spatial relation. Without this step, later metric failures could be caused by invalid generations rather than prototypicality bias. We filter images with two independent VLMs, Qwen2.5-VL-32B-Instruct [46] and InternVL3-38B-Instruct [55]. Each VLM receives one image-description pair and assigns a 1–10 semantic-correctness score: \(I_{\mathrm{corr}}\) is checked only against \(D_c\), and \(I_{\mathrm{adv}}\) only against \(D_a\). Scores \(\geq 8\) are treated as valid because they indicate that the main subject and controlled attributes are present with at most minor ambiguity. We retain only consensus-valid pairs where both images pass both VLM filters. The final benchmark contains \(45{,}400\) image pairs: \(18{,}397\) Animals, \(14{,}019\) Demography, and \(12{,}984\) Objects. Crucially, image filtering is non-contrastive. It never compares \(I_{\mathrm{corr}}\) and \(I_{\mathrm{adv}}\) under the neutral prompt \(T\); it only verifies that each image realizes its own generation description. Thus, filtering removes invalid generations without encoding the final benchmark decision. The final evaluation remains contrastive: a metric receives the same neutral prompt \(T\) and must rank \(I_{\mathrm{corr}}\) above \(I_{\mathrm{adv}}\). (See filtering prompts in Appendix 10.)

4.2.0.3 Human validation of image filtering.

We validate the automated filtering on a post-hoc subset of \(900\) images, balanced across domains and accept/reject decisions. A trained human evaluator scores each image against its own generation description using the same 1–10 rubric. The human and VLM retain/reject decisions show substantial agreement, with weighted Cohen’s \(\kappa=0.778\) overall and \(14.11\%\) binary disagreement. Accepted images also receive higher semantic-alignment scores than rejected images from both the human evaluator and the VLM filter. Score-level analysis further shows strong correlation between the human score and the mean VLM score (Pearson \(r=0.718\), Spearman \(\rho=0.759\) overall), while the two independent VLM filters agree strongly with each other (Pearson \(r=0.881\)). These results support using VLM filtering as a scalable quality-control step for benchmark construction. Full domain-level results are reported in Appendix 11.

4.2.0.4 Image-quality control.

We also check whether the retained \(I_{\mathrm{corr}}\) and \(I_{\mathrm{adv}}\) images differ systematically in visual quality, since such differences could confound metric failures. Because T2I images have no unique ground-truth reference, we use two no-reference image-quality metrics as perceptual sanity checks, not semantic alignment metrics: MUSIQ [56], where higher scores indicate better perceived quality, and NIQE [57], where lower scores indicate more natural images with fewer distortions. As shown in Table 2, the differences are negligible: MUSIQ is \(76.34\) for \(I_{\mathrm{corr}}\) and \(76.21\) for \(I_{\mathrm{adv}}\) (\(\Delta=0.13\)), while NIQE is \(4.93\) and \(4.91\), respectively (\(\Delta=0.02\)). The fraction of pairs where \(I_{\mathrm{corr}}\) has better quality is also near chance under both metrics (\(52.47\%\) for MUSIQ; \(50.02\%\) for NIQE). Thus, later metric failures are unlikely to be driven by systematic visual-quality differences.

Table 2: No-reference image-quality comparison between \(I_{\mathrm{corr}}\) and \(I_{\mathrm{adv}}\) across the whole ProtoBias. Small absolute differences and near-chance \(I_{\mathrm{corr}}\)-preference rates indicate that the two image sets are broadly comparable in visual quality.
Metric	Mean \(I_{\mathrm{corr}}\)	Mean \(I_{\mathrm{adv}}\)	\(\Delta\)	\(I_{\mathrm{corr}}\) Better (%)
MUSIQ \(\uparrow\)	76.34	76.21	0.13	52.47
NIQE \(\downarrow\)	4.93	4.91	0.02	50.02

5pt

4.2.0.5 Pairwise human evaluation.

Finally, we collect human preferences for the final benchmark task. This study uses a separate subset of \(900\) test samples, distinct from the images used for validating the filtering stage, and is balanced across domains (\(300\) Animals, \(300\) Demography, \(300\) Objects). Six annotators participated, including PhD researchers, a postdoctoral researcher, and master’s-level graduates from multiple countries; annotator details and agreement statistics are reported in Appendix 12. For each item, annotators see the neutral prompt \(T\) and the two images in randomized left–right order, and choose whether \(I_{\mathrm{corr}}\), \(I_{\mathrm{adv}}\), or neither image better matches the prompt. These annotations provide the human reference for metric–human agreement and for the human row in the pairwise-accuracy analysis. Figure 3 shows representative ProtoBias examples from all three domains. In each row, \(I_{\mathrm{corr}}\) satisfies the neutral prompt, while \(I_{\mathrm{adv}}\) is more prototypical but violates one controlled semantic attribute. The check marks and crosses indicate whether each evaluator prefers the semantically correct image or fails by assigning the prototypical adversarial image an equal or higher score.

5 Experimental Setup↩︎

This section describes how we evaluate metrics on ProtoBias. We specify the data split, the metric families, the pairwise scoring rule, and the training setup for ProtoScore.

5.0.0.1 Evaluation Data

We split the filtered ProtoBias image pairs into disjoint train and test partitions at the prompt-triplet level, ensuring that variants of the same underlying triplet do not appear across splits. The train split (\(12k\) samples) is used only for training ProtoScore, while all reported metric evaluations are conducted on the test split. A balanced subset of \(900\) test examples, with \(300\) examples per domain, provides the human reference labels used for agreement analysis. This subset is distinct from the \(900\) images used to validate image filtering.

5.0.0.2 Evaluated Metrics

We evaluate metrics from four families. First, we evaluate embedding-based metrics: CLIPScore [5], AlignScore [58], and BLIPv2Score [59]. Second, we evaluate human-feedback-based metrics: PickScore [6], ImageReward [7], and HPSv2 [60]. Third, we evaluate the VQA-based VQAScore [8]. Finally, we evaluate VLM-as-judge baselines using GPT-5 [10] and Qwen3-VL-4B [61]. The VLM-as-judge baselines are prompted to judge only semantic correctness, following Appendix 12. All metrics produce scalar scores \(M(T,I)\), which we normalize to \([0,1]\) for reporting.

5.0.0.3 Pairwise Evaluation Protocol

For each example, a metric receives the neutral prompt \(T\) and scores both images. Following Eq. 1 , we count a prototypicality-bias failure when \(M(T,I_{\mathrm{adv}}) \geq M(T,I_{\mathrm{corr}})\). Pairwise accuracy is the fraction of examples where the metric avoids this failure and assigns a higher score to \(I_{\mathrm{corr}}\). All reported values in Table 3 are computed on the same annotated human-reference subset. Pairwise accuracy measures how often a metric ranks \(I_{\mathrm{corr}}\) above \(I_{\mathrm{adv}}\) under the neutral prompt \(T\). Cohen’s \(\kappa\) measures chance-corrected agreement between binary metric decisions and human choices.

5.0.0.4 Training ProtoScore

We train ProtoScore as a contrastively trained evaluator by adapting Qwen3-VL-4B-Instruct into a pairwise image–text scorer. For each training example, the model receives the same neutral prompt \(T\) paired with \(I_{\mathrm{corr}}\) and \(I_{\mathrm{adv}}\), and is trained to assign a higher score to the semantically correct image: \(s(T,I_{\mathrm{corr}}) > s(T,I_{\mathrm{adv}}).\) We optimize a scalar scoring head with a pairwise softplus ranking loss [62], [63] using margin \(0.5\). The base model is trained on the ProtoBias train split with LoRA adapters (\(r=16\), \(\alpha=32\), dropout \(0.05\)) and a learned score head. We use AdamW with learning rate \(1{\times}10^{-5}\), weight decay \(0.01\), cosine scheduling with \(3\%\) warmup, gradient clipping at \(1.0\), and bfloat16 mixed precision. We train for one epoch on a single H100 GPU for approximately six hours, with batch size \(1\), gradient accumulation over \(16\) steps, \(512{\times}512\) images, and a maximum of \(192\) text tokens. ProtoScore is intended as an initial mitigation baseline for this diagnostic setting, not as a general-purpose replacement for existing T2I evaluation metrics.

6 Results & Analysis↩︎

Table 3: Main results on ProtoBias.For automatic methods, \(\kappa\) is Cohen’s chance-corrected agreement between the metric’s pairwise choice and the human majority preference on the \(900\)-example annotated test subset.For the Human row, \(\kappa\) is the average pairwise quadratic weighted Cohen’s \(\kappa\) among annotators.Acc. is pairwise accuracy under the neutral prompt \(T\): a method is correct if \(M(T,I_{\mathrm{corr}}) > M(T,I_{\mathrm{adv}})\).For the Human row, Acc. is the fraction of annotated examples where the human majority prefers \(I_{\mathrm{corr}}\).Higher is better for both metrics. Bold indicates the best automatic method in each column; human accuracy is shown as an upper reference.
Method	Animals		Demography		Objects
2-3 (lr)4-5 (lr)6-7	\(\boldsymbol{\kappa}\)	Acc.	\(\boldsymbol{\kappa}\)	Acc.	\(\boldsymbol{\kappa}\)	Acc.
Human semantic-correctness preferences
Human	0.705	0.91	0.648	0.86	0.691	0.82
Embedding-based text–image metrics
CLIPScore	0.001	0.25	-0.005	0.28	0.058	0.32
AlignScore	0.033	0.51	-0.064	0.34	0.011	0.32
BLIPv2Score	0.057	0.33	-0.061	0.39	0.028	0.43
Human-feedback-based reward metrics
PickScore	0.001	0.32	-0.014	0.34	0.126	0.31
ImageReward	0.084	0.52	0.137	0.48	0.192	0.62
HPSv2	0.024	0.25	-0.006	0.38	0.125	0.38
VQA-based metric
VQAScore	0.028	0.51	-0.007	0.47	0.065	0.55
VLM-as-judge baselines
GPT-5	0.108	0.56	0.176	0.51	0.265	0.54
Qwen3-VL-4B	0.045	0.27	0.040	0.38	0.011	0.32
Contrastively trained evaluator (Ours)
ProtoScore	0.182	0.95	0.232	0.91	0.178	0.98

10.5pt

Having validated the construction of ProtoBias, we evaluate whether existing T2I metrics remain faithful to the neutral prompt \(T\) when a plausible prototypical distractor is available. Table 3 reports two quantities. The primary metric is pairwise accuracy: a method is correct only if it assigns \(I_{\mathrm{corr}}\) a higher score than \(I_{\mathrm{adv}}\) under the same neutral prompt \(T\). We also report Cohen’s \(\kappa\), measuring chance-corrected agreement between each metric’s pairwise choice and the human majority preference on the \(900\)-example annotated subset. For the Human row, accuracy is the fraction of examples where the human majority selects \(I_{\mathrm{corr}}\), and \(\kappa\) is the average pairwise quadratic weighted Cohen’s \(\kappa\) among annotators.

6.0.0.1 Human evaluation.

The human study confirms that the benchmark decision is interpretable despite the prototypical distractor. Human majority preferences select \(I_{\mathrm{corr}}\) in \(0.91\), \(0.86\), and \(0.82\) of Animal, Demography, and Object pairs, respectively. Human–human agreement is also substantial, with average pairwise quadratic weighted Cohen’s \(\kappa=0.705\), \(0.648\), and \(0.691\) across domains. Full annotator information, instructions, and the pairwise agreement matrix are reported in Appendix 12.

6.0.0.2 Existing metrics fail under prototypical distractors.

Most standard metrics perform far below the human reference. Embedding-based metrics are especially brittle: CLIPScore reaches only \(0.25\), \(0.28\), and \(0.32\) accuracy across Animals, Demography, and Objects, while BLIPv2Score remains below \(0.43\) in all domains. Human-feedback reward metrics also fail to resolve the contrast reliably: PickScore stays near or below chance (\(0.32\), \(0.34\), \(0.31\)), and HPSv2 performs similarly on Animals and Demography. ImageReward is the strongest reward-based baseline, but still reaches only \(0.52\), \(0.48\), and \(0.62\), well below human performance. Thus, metrics trained for broad alignment or preference prediction do not necessarily preserve the controlled semantic distinction in ProtoBias.

6.0.0.3 Semantic judges improve only partially.

More explicitly semantic evaluators reduce the gap but do not eliminate it. VQAScore reaches \(0.51\), \(0.47\), and \(0.55\), while GPT-5 reaches \(0.56\), \(0.51\), and \(0.54\). Qwen3-VL-4B performs poorly as a zero-shot judge, with accuracies of \(0.27\), \(0.38\), and \(0.32\). These results show that the failure is not limited to older embedding metrics: even VQA-based and VLM-as-judge evaluators can be pulled toward a visually plausible prototype when the correct image is less typical.

6.0.0.4 Metric–human agreement remains weak.

The \(\kappa\) values show that low accuracy is accompanied by weak agreement with human semantic preferences. CLIPScore, VQAScore, and Qwen3-VL-4B have \(\kappa\) values close to zero across domains. GPT-5 gives the strongest baseline agreement, especially on Objects (\(\kappa=0.265\)), but remains far below human–human agreement. This suggests that existing metrics may contain partial semantic signal, yet often fail to convert it into the correct contrastive choice when prototypicality and prompt faithfulness conflict.

6.0.0.5 Contrastive training reduces the failure but does not solve alignment.

ProtoScore achieves high pairwise accuracy (\(0.95\), \(0.91\), \(0.98\)), showing that targeted supervision can reduce prototypicality-driven failures. This suggests that the benchmark captures a learnable contrastive signal, but not necessarily general human-aligned semantic judgment. Yet its agreement with human preferences remains modest (\(\kappa=0.182\), \(0.232\), \(0.178\)); in Objects, GPT-5 has higher \(\kappa\) despite lower accuracy. We therefore view ProtoScore as a mitigation baseline, not a full solution. External transfer results of ProtoScore are reported in Appendix 15.

Overall, ProtoBias shows that current T2I metrics can over-reward prototypical patterns when semantic correctness requires a less typical but prompt-faithful image.

7 Conclusion↩︎

We introduce ProtoBias, a controlled benchmark for testing whether T2I evaluation metrics remain faithful when prototypicality offers a shortcut. Across Animals, Objects, and Demography, it pairs a less typical but semantically correct image with a plausible prototypical adversary that violates one controlled prompt constraint. Our results show that embedding-based, reward-based, VQA-based, and VLM-as-judge metrics often fail this contrastive test despite extensive prompt, image, human-validation, and quality controls. This makes prototypicality bias measurable rather than anecdotal. While ProtoScore shows that contrastive training can reduce this failure, its limited human agreement indicates that benchmark-specific gains are not a complete solution. ProtoBias therefore provides a focused diagnostic for future evaluators: robust metrics should reward prompt faithfulness, not proximity to familiar prototypes.

Limitations↩︎

ProtoBias is designed as a controlled contrastive benchmark, which helps isolate prototypicality bias but cannot capture the full complexity of open-ended T2I evaluation. Its taxonomy covers Animals, Objects, and Demography, but remains necessarily selective: it includes only a subset of categories, attributes, and prototypical associations. In particular, demographic prototypicality is treated as a representational and dataset-mediated phenomenon, and some associations may reflect Western-centric, language-specific, or model-specific patterns rather than universal human judgments. Although we validate prompt quality, image filtering, human preferences, and image quality, the benchmark relies on generated images and automated filtering, so residual artifacts or imperfect realizations may remain. Human evaluation uses a balanced subset with a limited annotator pool, which supports reliability analysis but cannot represent all cultural perspectives on typicality, stereotypes, or semantic salience. Finally, ProtoScore is included as an initial mitigation, not as a general-purpose replacement for T2I metrics; its behavior may depend on the contrastive training distribution and should be tested beyond ProtoBias. Future work should broaden the taxonomy, expand culturally diverse annotation, and evaluate whether prototypicality-aware metrics transfer to more open-world prompts and user-facing T2I systems.

8 Taxonomy & Prompt Templates↩︎

8.1 Taxonomy↩︎

8.1.0.1 Animals.

The Animals taxonomy is organized into three subcategories: birds, mammals, and other animals. Each subcategory defines non-prototypical to prototypical pairs based on perceptual familiarity rather than biological correctness. Bird pairs include penguin, ostrich, kiwi, shoebill, cassowary, flamingo, pelican, and toucan contrasted with small canonical birds such as robin or sparrow. Mammal pairs include platypus, echidna, bat, pangolin, armadillo, and dolphin contrasted with dog, cat, or horse. The remaining animal group includes mosquito, cockroach, walking stick insect, praying mantis, jellyfish, and octopus contrasted with common mammals. Across all pairs, the environment remains identical and realistic, with semantic perturbations introduced only through auxiliary elements.

8.1.0.2 Demography.

The Demography taxonomy is defined over the hypernym person and spans three social axes: religion, nationality, and sexual orientation. Religion includes Christian as privileged and Muslim and Jewish as disadvantaged groups. Nationality includes American as privileged and Nigerian, Indian, and Mexican as disadvantaged groups. Sexual orientation includes heterosexual as privileged and LGBTQ+ as disadvantaged. These identities are crossed with five socio-attributes: wealth, intellect, morality, power, and civility, each with positive and negative realizations. Prototypical cases associate privileged identities with positive attributes, while non-prototypical cases invert these associations. Visual realizations specify clothing, appearance, and symbols consistent with each identity, while environments remain neutral and realistic.

8.1.0.3 Objects.

The Objects taxonomy is divided into furniture, vehicles, and tableware. Furniture pairs include bean bag, hammock, futon mattress, chaise lounge, and floor cushion contrasted with chair, bed, sofa, or dining chair. Vehicle pairs include e-scooter, unicycle, tuk-tuk, golf cart, and segway contrasted with motorcycle, bicycle, or car. All object scenes are minimal and category-appropriate, such as indoor rooms, outdoor paved areas, or plain tabletops, with semantic perturbations introduced only through supporting elements.

8.2 Prompt Templates↩︎

We use structured prompt templates (see Table 12, 14, 13) to generate controlled triplets \((T, I_{\mathrm{corr}}, I_{\mathrm{adv}})\) across all domains. Each template enforces strict lexical and structural consistency between the neutral text, the semantically correct non-prototypical instance, and the prototypical adversarial instance, ensuring that differences arise only from the intended semantic perturbation.

9 Construction Models, Infrastructure, and Runtime↩︎

Table 4 summarizes the model families and compute used for constructing ProtoBias. Prompt generation, prompt filtering, image generation, and image filtering are separated to reduce single-model dependence at each stage. All runtimes are approximate wall-clock times measured on H100 GPUs under our cluster setup.

Table 4: Construction models and approximate runtime for ProtoBias. The table reports wall-clock time under our H100 cluster setup. Prompt generation and filtering operate on text triplets; image generation renders the accepted descriptions; image filtering validates each generated image independently against its own description.
Stage	Models	GPUs	Wall time	Output / purpose
Prompt generation	Qwen2.5-14B-Instruct [46]; Gemma-3-12B-it [47]; Llama-3.1-8B-Instruct [48]	3 H100	\(\sim\)3 h per generator	Candidate triplets \((T,D_c,D_a)\) from structured templates.
Prompt filtering	Qwen2.5-32B-Instruct [46]; Mistral-Small-3.1-24B-Instruct [49]	3 H100	\(\sim\)1 h per filter	Removes malformed JSON, domain errors, target substitutions, and multi-knob changes.
Image generation	FLUX.1-schnell [52]; Stable Diffusion 3.5 Large Turbo [53]; Sana-1600M-1024px [54]	9 H100	\(\sim\)3–4 h per generator	Renders \(D_c\) and \(D_a\) into candidate image pairs \((I_{\mathrm{corr}},I_{\mathrm{adv}})\).
Image filtering	Qwen2.5-VL-32B-Instruct [46]; InternVL3-38B-Instruct [55]	36 H100	\(\sim\)1 h per filter	Scores each image against its own generation description and retains consensus-valid pairs.

4.5pt

10 Prompt and Image Filtering Prompts↩︎

Tables 15–17 provide the LLM prompt-filtering instructions used during benchmark construction. These filters validate generated triplets before image generation, checking that \(T\), \(D_c\), and \(D_a\) preserve the intended scene structure, apply the correct prototype substitution, and introduce only the specified controlled semantic change. Separate instructions are used for Animals/Objects and Demography because demographic triplets require additional checks for group mapping and the absence of identity terms in the neutral prompt.

Tables 18–20 show the VLM image-filtering instructions used during benchmark construction. Each generated image is evaluated independently against its own generation description, rather than contrastively against the neutral prompt \(T\). This keeps filtering separate from the final ProtoBias decision and prevents the filter from encoding the benchmark label.

11 Human Validation of the Filtering Stage↩︎

We validate the automated image-filtering stage on \(900\) image–description examples, balanced across domains and automated accept/reject decisions. A trained human evaluator scored each image against its own generation description using the same 1–10 semantic-correctness rubric as the VLM filters.

Table 5 shows substantial retain/reject agreement: the overall disagreement rate is \(14.11\%\), and weighted Cohen’s \(\kappa\) is \(0.778\). Table 6 shows that filter-accepted samples receive higher semantic-alignment scores than rejected samples from both the human evaluator and the VLM filter. Table 7 further shows strong score-level correspondence between the human score and mean VLM score (Pearson \(r=0.718\), Spearman \(\rho=0.759\)), as well as strong agreement between the two VLM filters (Pearson \(r=0.881\)). Together, these results support VLM filtering as a scalable quality-control step while leaving final benchmark decisions to the contrastive evaluation protocol.

Table 5: Human validation of the automated image-filtering stage. \(\kappa\) denotes weighted Cohen’s agreement between human and filter judgments. RR denotes retention rate, i.e., the percentage of samples judged as semantically aligned by the human evaluator and by the VLM filter, respectively. Disagree denotes binary retain/reject disagreement.
Domain	#Samples	\(\kappa\)	Human RR (%)	VLM RR (%)	Disagree (%)
Animals	300	0.784	55.33	50.00	11.00
Demography	300	0.721	43.67	50.00	18.00
Objects	300	0.828	57.33	50.00	13.33
Overall	900	0.778	52.11	50.00	14.11

9pt

Table 6: Average semantic-alignment scores on a 1–10 scale for samples accepted and rejected by the automated filter. Accepted samples receive higher scores from both the human evaluator and the VLM filter across all domains.
Domain	Human		VLM Filter
2-3 (lr)4-5	Accepted	Rejected	Accepted	Rejected
Animals	8.74	6.17	9.16	6.21
Demography	8.82	5.21	8.41	5.71
Objects	8.79	6.91	9.27	6.81
Overall	8.78	6.10	8.95	6.24

6pt

Table 7: Score-level correlations for image-filter validation. Human–VLM correlations compare the human 1–10 semantic-alignment score with the mean VLM filter score. VLM–VLM correlation compares the two independent filtering models on the same images.
Domain	Human–VLM \(r\)	Human–VLM \(\rho\)	VLM–VLM \(r\)
Animals	0.730	0.787	0.919
Demography	0.671	0.698	0.820
Objects	0.752	0.793	0.906
Overall	0.718	0.759	0.881

4.5pt

12 Human Preference Annotation↩︎

We conduct a pairwise human preference study to verify that the intended ProtoBias decision is clear to human annotators. The study uses \(900\) test examples, balanced across domains (\(300\) Animals, \(300\) Demography, \(300\) Objects). This subset is distinct from the \(900\) image–description examples used for human validation of the filtering stage in Appendix 11. Each example consists of the neutral prompt \(T\), the semantically correct image \(I_{\mathrm{corr}}\), and the prototypical adversarial image \(I_{\mathrm{adv}}\).

Six annotators participated: PhD researchers, a postdoctoral researcher, and master’s-level graduates from multiple countries, all with fluent English proficiency. Aggregate annotator information is shown in Table 8. For each item, annotators saw \(T\) and the two images in randomized left–right order, and selected one of three labels: \(I_{\mathrm{corr}}\) better matches the prompt, \(I_{\mathrm{adv}}\) better matches the prompt, or tie/no clear preference.

12.0.0.1 Annotation rubric.

Annotators were instructed to judge only semantic alignment with \(T\). They checked whether each image satisfies the entities, attributes, counts, colors, spatial relations, foreground/background placement, and other explicitly stated prompt details. They were instructed not to prefer an image because it is more common, realistic, aesthetically pleasing, or prototypical unless such properties are required by the prompt. If both images were similarly valid or similarly invalid, annotators could choose tie/no clear preference.

12.0.0.2 Annotator agreement.

Table 9 reports pairwise quadratic weighted Cohen’s \(\kappa\) between annotators. Agreement ranges from \(0.405\) to \(0.874\), with an average pairwise agreement of \(\bar{\kappa}=0.681\). Most annotator pairs show moderate-to-substantial agreement, supporting the use of the majority preference as the human reference for metric–human agreement analysis.

Table 8: Aggregate information about annotators participating in the human preference evaluation.
Annotators	Age groups	Background
6	18–24 (1), 25–30 (3) 31–45 (1), 45–60 (1)	PhD, postdoc, master’s
Countries
Russia (2), Italy (1), India (1), Germany (1), UAE (1)

4.5pt

Table 9: Pairwise quadratic weighted Cohen’s \(\kappa\) agreement between human annotators in the preference study.
	ann1	ann2	ann3	ann4	ann5	ann6
ann1	–
ann2	0.648	–
ann3	0.753	0.874	–
ann4	0.405	0.645	0.497	–
ann5	0.648	0.826	0.874	0.645	–
ann6	0.723	0.679	0.813	0.589	0.749	–

6pt

13 LLM-as-Judge Prompt↩︎

To evaluate text-image semantic alignment using LLM-as-Judge systems, we adopt a strict scoring prompt designed to isolate semantic correctness while suppressing prototypical or aesthetic biases. The prompt instructs the model to rate a single image against a given text on a four-point ordinal scale, focusing exclusively on semantic fidelity. Check Table 21 for the prompt.

14 Runtime Comparison↩︎

Table 10: Runtime on 900 samples (1,800 text-image pairs) from the ProtoBias test set. All runtimes are measured under the same hardware and inference setup.
Metric	Model	Runtime (\(\approx\))
CLIPScore	CLIP-L-14	5.4 min
BLIPv2Score	BLIPv2	7.2 min
AlignScore	Align-base	6.4 min
PickScore	CLIP-H-14	1.7 min
ImageReward	BLIPv2	7.8 min
HPSv2	CLIP-H-14	28.4 min
VQAScore	CLIP-FlanT5-XXL	10.2 min
LLM-as-Judge	GPT-5	329 min
LLM-as-Judge	Qwen3-VL-4B	25.3 min
ProtoScore	Qwen3-VL-4B	3.7 min

6pt

To compare evaluation cost, Table 10 reports runtimes on 900 samples (1,800 pairs) from ProtoBias. Embedding and reward-based metrics are fast, whereas LLM-as-Judge methods are significantly more expensive.

15 External Benchmark Evaluation↩︎

We additionally evaluate ProtoScore on standard alignment benchmarks to test whether contrastive training on ProtoBias preserves performance beyond our diagnostic setting. These results are intended as transfer checks rather than the main contribution of the paper. Winoground [15] evaluates compositional image–text matching with Text, Image, and Group scores, where Group requires all matching decisions to be correct. TIFA160 [64] evaluates agreement with human-rated text–image pairs; following prior work [8], [65], we report the results in Table 11.

Table 11: External benchmark transfer results. Winoground reports Text, Image, and Group scores, where Group requires all matching decisions for an example to be correct. TIFA160 reports pairwise accuracy against human ratings. Higher is better for all columns. ProtoScore transfers strongly to Winoground, but remains below VQAScore and ImageReward on TIFA160, indicating that ProtoBias training improves compositional contrastive grounding without universally dominating preference-style alignment benchmarks.
Metric	Model	Winoground			TIFA160
3-5		Text	Image	Group
CLIPScore	CLIP-L-14	27.8	11.5	7.8	54.1
BLIPv2Score	BLIPv2	43.3	21.3	17.5	57.5
AlignScore	Align-base	0.35	0.11	0.08	–
PickScore	CLIP-H-14	23.8	12.5	6.8	59.4
ImageReward	BLIPv2	42.8	15.3	12.8	67.3
HPSv2	CLIP-H-14	11.5	7.8	4.0	55.2
VQAScore	CLIP-FlanT5-XXL	60.0	57.5	46.0	71.2
ProtoScore	Qwen3-VL-4B	69.2	67.4	64.2	68.4

5.5pt

16 Additional ProtoBias Examples↩︎

To complement the representative examples shown in the main paper, we provide additional examples from ProtoBias across Animals, Objects, and Demography. Each example contrasts a semantically correct but less prototypical image (\(I_{\mathrm{corr}}\)) with a prototypical but semantically incorrect adversarial image (\(I_{\mathrm{adv}}\)) generated from the same prompt. These examples further illustrate the diversity of semantic perturbations used in the benchmark, including differences in count, color, spatial layout, and socially learned prototypes. (See Figure: ¿fig:fig:animal?, ¿fig:fig:demo?, ¿fig:fig:object?)

fig:

Table 12: Prompt template used to generate animal prompt-triplets.
Animal Prompt Template
Instruction:
You are generating triplet prompts for an animal bias-stress dataset.
Inputs:
hypernym = {hypernym} (non-exhaustive: animal, bird, mammal)
non_proto = {non_proto} (non-prototypical animal, e.g., penguin)
proto = {proto} (prototypical animal, e.g., robin)
knob = {knob} (one of: count, color, layout_relation, spatial)
knob_description = {knob_description}
extra_object = {extra_object} (natural element, e.g., rock, tree, pond)
environment_hint = {env_hint}

Goal:
Produce three single-sentence descriptions of the same simple natural scene. The only allowed differences are: (i) hypernym → non_proto → proto, and (ii) a small, explicit knob change applied only in the adversarial sentence, and (iii) the knob change applies only to the same extra object.
Critical Rules: TEXT: uses the hypernym as subject and instantiates a clear knob value using the extra object. count: exact numerals (e.g., “exactly two rocks”) color: concrete color tone (e.g., “blue water”) layout_relation: precise relation (e.g., “to the left of the animal”) spatial: explicit foreground/background placement CORRECT: identical to TEXT, but replace the hypernym with the non-prototypical animal. ADVERSARIAL: identical to TEXT, but replace the hypernym with the prototypical animal and change only the knob value: count: change by \(\pm 1, 2\) color: switch to another natural color layout_relation: flip relation (e.g., left → right) spatial: background \(\leftrightarrow\) foreground Scene must remain natural, minimal, and renderable (no humans, text, or buildings). Maximum 30 words per sentence. No meta language or explanations.
Output Format:
{
"text": "...",
"correct": "...",
"adversarial": "..."
}

6pt

Table 13: Prompt template used to generate object prompt-triplets.
Object Prompt Template
Instruction:
You are generating triplet prompts for an OBJECT bias-stress dataset.
Inputs:
subcategory = {subcategory} (furniture \| vehicle \| tableware)
non_proto = {non_proto} (non-prototypical object, e.g., bean bag)
proto = {proto} (prototypical object, e.g., chair)
knob = {knob} (one of: count, color_tone, layout_relation, spatial, scale_size)
knob_description = {knob_description}
extra_object = {extra_object} (supporting element, e.g., lamp, plate, cone)
environment_hint = {env_hint}

Goal:
Create three single-sentence scene descriptions referring to the same setup. Differences should appear only in the main object identity (hypernym \(\rightarrow\) non_proto \(\rightarrow\) proto) and a small, realistic knob-based mismatch in the adversarial sentence.
Rules: TEXT: uses the subcategory hypernym as the subject (“a piece of furniture”, “a vehicle”, or “a tableware item”) and describes a simple, neutral, realistic scene consistent with {environment_hint}. It must mention {extra_object} and instantiate a clear knob condition: count: exact number of {extra_object} (e.g., “two cones near it”) color_tone: specific surface/element color (e.g., “on a gray carpet”) layout_relation: precise placement (e.g., “to the left of a lamp”) spatial: explicit background/foreground placement scale_size: relative size cue (e.g., “next to a small table”) CORRECT: copy TEXT exactly, but replace the hypernym with {non_proto}. Keep all other details and knob values identical. ADVERSARIAL: copy TEXT exactly, but replace the hypernym with {proto}. Modify only the knob condition according to {knob_description}. All other wording must remain identical. Keep scenes simple, natural, and easy to visualize (no people, text, brands, food for tableware, or cluttered interiors). Maximum 30 words per sentence. No meta explanations.
Output Format:
{
"text":"...",
"correct": "...",
"adversarial":"..."
}

6pt

Table 14: Prompt template used to generate demography prompt-triplets.
Demography Prompt Template
Instruction:
You are generating triplet prompts for a demography bias-stress dataset.
Inputs (filled by caller):
group_category = {group_category} (religion \| nationality \| sexual_orientation)
socio_attr = {socio_attr} (wealth \| intellect \| morality \| power \| civility)
pole = {pole} (positive \| negative)
attr_token = {attr_token} (e.g., rich, poor, intelligent, uneducated)
disadvantaged_key = {disadvantaged}
advantaged_key = {advantaged}
disadvantaged_desc = {disadv_desc}
advantaged_desc = {adv_desc}
knob = {knob} (count \| color_tone \| layout_relation \| spatial)
knob_description = {knob_description}
extra_element = {extra_element}
environment_hint = {environment_hint}

Non-Prototype vs. Prototype Mapping (Critical): If pole = positive: Non-Prototype: {attr_token} + disadvantaged group (use {disadv_desc}) Prototype: {attr_token} + advantaged group (use {adv_desc}) If pole = negative: Non-Prototype: {attr_token} + advantaged group (use {adv_desc}) Prototype: {attr_token} + disadvantaged group (use {disadv_desc})
Scene Construction Rules: TEXT: Must begin exactly with: “A {attr_token} person ...” One neutral scene consistent with {environment_hint} Mention exactly one {extra_element} encoding a clear knob: count, color_tone, layout_relation, or spatial No group labels; use only the hypernym “person” CORRECT: Copy TEXT verbatim Replace “person” with the non-prototype description Do not change the knob ADVERSARIAL: Copy TEXT verbatim Replace “person” with the prototype description Modify only the knob for the same extra element
Global Constraints: One sentence per field, max 30 words Single person, no added people, text, or brands Environment and objects remain fixed
Output Format:
{
"text": "...",
"correct": "...",
"adversarial": "..."
}

6pt

Table 15: LLM prompt-filtering instruction used to validate animal triplets. The filter checks whether \((T,D_c,D_a)\) forms an exact controlled perturbation with only the intended prototype substitution and knob change.
Animal Prompt-Filtering Instruction
Instruction:
You are a strict validator for a bias-stress triplet dataset. Your task is to verify that transformations are exact and controlled.
Valid Triplet Rules: Structure: The triplet must contain text, correct, and adversarial. Each field must be one sentence with at most 30 words. TEXT: Uses only a generic subject, e.g., “a bird” or “an animal”, and defines a clear knob condition. CORRECT: Must be identical to text, except that the generic subject is replaced with non_proto. No other wording changes are allowed. ADVERSARIAL: Must be identical to text, except that the generic subject is replaced with proto and exactly one knob value is changed. Knob validation: The adversarial change must match the specified knob: count: change number only, e.g., two \(\rightarrow\) three. color or color_tone: change color only, e.g., blue \(\rightarrow\) green. layout_relation: change relation only, e.g., left/right/front/behind. spatial: change background \(\leftrightarrow\) foreground. No extra changes: Any additional wording change makes the triplet invalid. Plausibility: The scene must be clean, simple, realistic, and visually renderable. Normalization Rule: Singular and plural forms refer to the same object, e.g., “cone” vs. “cones”. This is not considered an object change. For count changes, “one cone” \(\rightarrow\) “two cones” is valid because the object remains the same. Output Format:
{
"valid": true/false,
"reason": "very short reason"
}

6pt

Table 16: LLM prompt-filtering instruction used to validate object triplets. The filter checks that only the intended object substitution and controlled semantic knob change are introduced.
Object Prompt-Filtering Instruction
Instruction:
You are a strict validator for a bias-stress triplet dataset. Your task is to verify that transformations are exact and controlled.
Valid Triplet Rules: Structure: The triplet must contain text, correct, and adversarial. Each field must be one sentence with at most 30 words. TEXT: Uses only a generic object-category subject, e.g., “a vehicle”, “a piece of furniture”, or “a tableware item”, and defines a clear knob condition. CORRECT: Must be identical to text, except that the generic subject is replaced with non_proto. No other wording changes are allowed. ADVERSARIAL: Must be identical to text, except that the generic subject is replaced with proto and exactly one knob value is changed. Knob validation: The adversarial change must match the specified knob: count: change number only, e.g., one \(\rightarrow\) two. color or color_tone: change color only, e.g., gray \(\rightarrow\) beige. layout_relation: change relation only, e.g., left \(\rightarrow\) right. spatial: change background \(\leftrightarrow\) foreground. scale_size: change size only. No extra changes: Any additional wording change makes the triplet invalid. Plausibility: The scene must be clean, simple, realistic, and visually renderable. Normalization Rule: Singular and plural forms refer to the same object, e.g., “cone” vs. “cones” or “cushion” vs. “cushions”. This is not considered an object change. For count changes, “one cone” \(\rightarrow\) “two cones” is valid because the object remains the same. Output Format:
{
"valid": true/false,
"reason": "very short reason"
}

6pt

Table 17: LLM prompt-filtering instruction used to validate demography triplets. The filter checks group mapping, absence of group identity terms in the neutral prompt, and the presence of a controlled visual knob change in the adversarial description.
Demography Prompt-Filtering Instruction
Instruction:
You are a validator for demography bias-stress triplets. Your task is to decide whether the triplet is usable for a demography bias-stress dataset. Be lenient about grammar and wording, but strict about correct group mapping and the presence of a clear visual change in the adversarial description.
Rules: Structure: The triplet must contain text, correct, and adversarial. Each field must be one sentence. Minor grammar issues are acceptable. TEXT: Must describe a generic person using attr_token, e.g., “A rich person” or “An uneducated person”. It must not include specific group identity terms and must describe a simple visual scene. Group terms: Group identity terms are forbidden in text, but required and expected in correct and adversarial. Do not reject correct or adversarial because they contain religion, nationality, or orientation terms. Group mapping: Use the metadata fields pole, disadvantaged_group, and advantaged_group. If pole = positive, correct must use the disadvantaged group and adversarial must use the advantaged group. If pole = negative, correct must use the advantaged group and adversarial must use the disadvantaged group. The metadata mapping should not be reinterpreted. CORRECT: Must replace the generic “person” with the mapped group/person description. Clothing, appearance, and group descriptors are expected. The main scene should remain similar to text. Exact string matching is not required. ADVERSARIAL: Must use the mapped prototype group/person description and contain at least one clear visual change compared with correct or text. Knob validation: The primary adversarial change must match the specified knob: count: number must change, e.g., one \(\rightarrow\) two. color_tone: color must change, e.g., gray \(\rightarrow\) beige. layout_relation: position must change, e.g., left \(\rightarrow\) right. spatial: background \(\leftrightarrow\) foreground. scale_size: size must change. Additional variation: Minor variation is allowed only if the main knob change is clearly present and no conflicting or multiple major changes occur. Reject if: text contains specific group identity terms; group mapping is wrong; the adversarial has no clear visual change; the scene changes completely; or the text is broken, nonsensical, unsafe, or not visualizable. Output Format:
{
"valid": true/false,
"reason": "very short reason"
}

6pt

Table 18: Domain-specific VLM image-filtering prompt for animal examples. The prompt checks whether the generated image realizes its own expected description.
Animal Image-Filtering Prompt
Domain: animal.
Expected Description:
{description}
Metadata:
generic text = {text}
correct target = {correct}
adversarial target = {adversarial}
image side = {side}
non-prototype animal = {non_proto}
prototype animal = {proto}
knob = {knob}
extra object = {extra_object}
Validation:
Validate the image against the expected description.
Strict Rules: The animal species or type must match the expected description. The scene must contain the described extra object if one is specified. Count words must be exact. If the description says two stones, exactly two visible stones are required. Color or lighting changes must be visibly present when specified. Spatial relations must match exactly from the viewer’s perspective. Foreground/background changes must be clear if specified. Size/scale changes must be visible if specified. Reject if the image adds another main animal, changes the main scene, omits the main animal, or makes the key attribute unverifiable. If uncertain about any required attribute, mark invalid. Scoring: 10: exact semantic match. 8–9: all required attributes visible, only minor ambiguity. 5–7: one important attribute is wrong or unclear. 1–4: wrong subject or major mismatch. Output Format:
{
"valid": true/false,
"score": 1-10
}

6pt

Table 19: Domain-specific VLM image-filtering prompt for object examples. The prompt checks object identity, supporting elements, and controlled visual attributes.
Object Image-Filtering Prompt
Domain: object.
Expected Description:
{description}
Metadata:
generic text = {text}
correct target = {correct}
adversarial target = {adversarial}
image side = {side}
non-prototype object = {non_proto}
prototype object = {proto}
knob = {knob}
extra object = {extra_object}
Validation:
Validate the image against the expected description.
Strict Rules: The main object must match the expected object in the description. Supporting or extra objects must be present if specified. Count must be exact. Color and tone must match the expected description when specified. Layout relations such as left, right, beside, in front of, behind, above, and below must match from the viewer’s perspective. Foreground/background placement must be clear if specified. Size/scale must match when specified. Reject if the image contains brand text, captions, labels, extra main objects, missing required objects, or an unverifiable key attribute. If uncertain about any required attribute, mark invalid. Scoring: 10: exact semantic match. 8–9: all required attributes visible, only minor ambiguity. 5–7: one important attribute is wrong or unclear. 1–4: wrong object or major mismatch. Output Format:
{
"valid": true/false,
"score": 1-10
}

6pt

Table 20: Domain-specific VLM image-filtering prompt for demography examples. The prompt verifies only visible descriptors and explicitly forbids inferring social identity or attributes from appearance alone.
Demography Image-Filtering Prompt
Domain: demography.
Expected Description:
{description}
Metadata:
generic text = {text}
correct target = {correct}
adversarial target = {adversarial}
image side = {side}
group category = {group_category}
socio attribute = {socio_attr}
pole = {pole}
disadvantaged group = {disadvantaged_group}
advantaged group = {advantaged_group}
gender = {gender}
attribute token = {attr_token}
knob = {knob}
extra element = {extra_object}
Validation:
Validate the image against the expected description.
Strict Rules: The image must show one main person only unless the description explicitly says otherwise. The visible person, clothing, accessories, and scene must match the expected description. Do not infer religion, nationality, sexual orientation, wealth, morality, intellect, power, or civility from appearance alone. If the expected description contains visible clothing, accessory, or group descriptors, verify only those visible descriptors. Count must be exact. Color or lighting must match when specified. Spatial/layout relations must match from the viewer’s perspective. Foreground/background placement must be clear if specified. Reject if there are extra people, missing required visible elements, wrong visible attributes, text/logos, or the key attribute is unverifiable. If uncertain about any required visible attribute, mark invalid. Scoring: 10: exact semantic match. 8–9: all required visible attributes present, only minor ambiguity. 5–7: one important visible attribute is wrong or unclear. 1–4: wrong person/scene or major mismatch. Output Format:
{
"valid": true/false,
"score": 1-10
}

6pt

Table 21: Prompt used for LLM-as-Judge evaluation with GPT-4o and GPT-5. The instruction mirrors the guidelines provided to human annotators, ensuring consistent evaluation criteria across human and automated judgments.
LLM-as-Judge Prompt
Instruction:
You are a strict expert judge of text–image alignment.
Task:
Given a text prompt and one image, rate the image from 1 to 4 based only on how well it matches the text description.
Rules: Focus purely on semantic correctness. Ignore artistic style, beauty, lighting, realism, resolution, and aesthetics. Do not reward images for being more realistic or typical. Interpret LEFT/RIGHT and spatial relations from the viewer’s perspective.
Key Checks: Is the main element correct? Is the extra element correct? Are attributes and relations correct (color, count, position, foreground/background)? Is the overall meaning of the text preserved?
Rating Guide (1-4): 4 - Perfect match: All elements and attributes are correct. 3 - Mostly correct: Main element correct, minor ambiguity in one detail. 2 - One major mistake: Main element correct, but extra element or attribute is wrong. 1 - Poor match: Main element wrong or multiple critical mismatches.
Output Format:
{
"score": <integer 1–4>
}

6pt

Table 22: No caption.
Image Filtration Instruction Prompt
Instruction:
You are given ONE image and the description used to generate it.
Task:
Score ONLY semantic alignment between the image and the description.
Important Rules: Do not check aesthetics, realism, lighting, style, and artistic quality. The main element must exactly match the description. The extra element must be present and correct. Attributes must be exact: Count must match exactly (e.g., “two pebbles” means exactly two). Color must match exactly. Spatial relations (left/right, foreground/background, position) must match from the viewer’s perspective. If any attribute is wrong, the score must be \(\leq\) 7.
Scoring (1–10): 10: Perfect match. All elements and attributes are exact. 8–9: Very minor ambiguity, but no attribute violations. 5–7: One clear attribute error. 1–4: Major mismatch.
Output Format:
{
"score_1to10": <integer from 1 to 10>
}

6pt

References↩︎

[1]

E. Rosch, “Cognitive representations of semantic categories,” Journal of Experimental Psychology: General, vol. 104, no. 3, pp. 192–233, 1975.

[2]

E. Rosch and C. B. Mervis, “Family resemblances: Studies in the internal structure of categories,” Cognitive Psychology, vol. 7, no. 4, pp. 573–605, 1975, doi: https://doi.org/10.1016/0010-0285(75)90024-9.

[3]

E. Rosch, C. B. Mervis, W. D. Gray, D. M. Johnson, and P. Boyes-Braem, “Basic objects in natural categories,” Cognitive Psychology, vol. 8, no. 3, pp. 382–439, 1976, doi: https://doi.org/10.1016/0010-0285(76)90013-X.

[4]

G. L. Murphy, The big book of concepts. The MIT Press, 2002.

[5]

J. Hessel, A. Holtzman, M. Forbes, R. L. Bras, and Y. Choi, “CLIPScore: A reference-free evaluation metric for image captioning.” 2022, [Online]. Available: https://arxiv.org/abs/2104.08718.

[6]

Y. Kirstain, A. Polyak, U. Singer, S. Matiana, J. Penna, and O. Levy, “Pick-a-pic: An open dataset of user preferences for text-to-image generation.” 2023, [Online]. Available: https://arxiv.org/abs/2305.01569.

[7]

J. Xu et al., “ImageReward: Learning and evaluating human preferences for text-to-image generation.” 2023, [Online]. Available: https://arxiv.org/abs/2304.05977.

[8]

Z. Lin et al., “Evaluating text-to-visual generation with image-to-text generation.” 2024, [Online]. Available: https://arxiv.org/abs/2404.01291.

[9]

W. Ge et al., “MLLM-bench: Evaluating multimodal LLMs with per-sample criteria,” 2311.13951v3, Nov. 2023, [Online]. Available: http://arxiv.org/abs/2311.13951v3.

[10]

A. Singh et al., “OpenAI GPT-5 system card.” 2026, [Online]. Available: https://arxiv.org/abs/2601.03267.

[11]

L. Girrbach, S. Alaniz, G. Smith, T. Darrell, and Z. Akata, “Person-centric annotations of LAION-400M: Auditing bias and its transfer to models,” 2510.03721v1, Oct. 2025, [Online]. Available: http://arxiv.org/abs/2510.03721v1.

[12]

Y. Wan et al., “Survey of bias in text-to-image generation: Definition, evaluation, and mitigation,” 2404.01030v3, Apr. 2024, [Online]. Available: http://arxiv.org/abs/2404.01030v3.

[13]

T. He et al., “On the blind spots of model-based evaluation metrics for text generation,” in Proceedings of the 61st annual meeting of the association for computational linguistics (volume 1: Long papers), Jul. 2023, pp. 12067–12097, doi: 10.18653/v1/2023.acl-long.674.

[14]

Y. Chen and S. Eger, “MENLI: Robust evaluation metrics from natural language inference.” 2023, [Online]. Available: https://arxiv.org/abs/2208.07316.

[15]

T. Thrush et al., “Winoground: Probing vision and language models for visio-linguistic compositionality.” 2022, [Online]. Available: https://arxiv.org/abs/2204.03162.

[16]

H. Luo et al., “BIGbench: A unified benchmark for evaluating multi-dimensional social biases in text-to-image models,” 2407.15240v6, Jul. 2024, [Online]. Available: http://arxiv.org/abs/2407.15240v6.

[17]

A. Narayanan, V. R. Khazaie, and S. Raza, “Bias in the picture: Benchmarking VLMs with social-cue news images and LLM-as-judge assessment,” 2509.19659v1, Sep. 2025, [Online]. Available: http://arxiv.org/abs/2509.19659v1.

[18]

J. M. Contreras, “Automated evaluation of gender bias across 13 large multimodal models,” 2509.07050v1, Sep. 2025, [Online]. Available: http://arxiv.org/abs/2509.07050v1.

[19]

R. Nair, B. Tokas, and H. Kerner, “A woman with a knife or a knife with a woman? Measuring directional bias amplification in image captions,” 2503.07878v4, Mar. 2025, [Online]. Available: http://arxiv.org/abs/2503.07878v4.

[20]

H. Seo et al., “Exposing blindspots: Cultural bias evaluation in generative image models,” 2510.20042v1, Oct. 2025, [Online]. Available: http://arxiv.org/abs/2510.20042v1.

[21]

M. N. Said et al., “Deconstructing bias: A multifaceted framework for diagnosing cultural and compositional inequities in text-to-image generative models,” 2505.01430v1, Apr. 2025, [Online]. Available: http://arxiv.org/abs/2505.01430v1.

[22]

C. Raj, B. Wei, A. Caliskan, A. Anastasopoulos, and Z. Zhu, “VIGNETTE: Socially grounded bias evaluation for vision-language models,” 2505.22897v1, May 2025, [Online]. Available: http://arxiv.org/abs/2505.22897v1.

[23]

Y. Jiang, Z. Li, X. Shen, Y. Liu, M. Backes, and Y. Zhang, “\(\texttt{ModSCAN}\): Measuring stereotypical bias in large vision-language models from vision and language modalities,” 2410.06967v1, Oct. 2024, [Online]. Available: http://arxiv.org/abs/2410.06967v1.

[24]

X. Dai, S. Karimi, and B. Fang, “A critical look at meta-evaluating summarisation evaluation metrics,” 2409.19507v1, Sep. 2024, [Online]. Available: http://arxiv.org/abs/2409.19507v1.

[25]

Y. Hirota et al., “Bias in gender bias benchmarks: How spurious features distort evaluation,” 2509.07596v2, Sep. 2025, [Online]. Available: http://arxiv.org/abs/2509.07596v2.

[26]

W. Ye et al., “MM-SpuBench: Towards better understanding of spurious biases in multimodal LLMs,” 2406.17126v1, Jun. 2024, [Online]. Available: http://arxiv.org/abs/2406.17126v1.

[27]

Y. Hwang, D. Lee, K. Min, T. Kang, Y. Kim, and K. Jung, “Fooling the LVLM judges: Visual biases in LVLM-based evaluation,” 2505.15249v1, May 2025, [Online]. Available: http://arxiv.org/abs/2505.15249v1.

[28]

Y. Wang et al., “Understanding and benchmarking the trustworthiness in multimodal LLMs for video understanding,” 2506.12336v2, Jun. 2025, [Online]. Available: http://arxiv.org/abs/2506.12336v2.

[30]

T. Lee et al., “AHELM: A holistic evaluation of audio-language models,” 2508.21376v2, Aug. 2025, [Online]. Available: http://arxiv.org/abs/2508.21376v2.

[31]

C. Leiter, Y. M. Asano, M. Keuper, and S. Eger, “CROC: Evaluating and training T2I metrics with pseudo- and human-labeled contrastive robustness checks,” arXiv preprint arXiv:2505.11314, 2025, doi: 10.48550/arXiv.2505.11314.

[32]

C. Leiter and S. Eger, “PrExMe! Large scale prompt exploration of open source LLMs for machine translation and summarization evaluation,” in Proceedings of the 2024 conference on empirical methods in natural language processing, Nov. 2024, doi: 10.18653/v1/2024.emnlp-main.641.

[33]

Y. Shen, Y. Shu, H. Paik, and Y. Sui, “MetaLogic: Robustness evaluation of text-to-image models via logically equivalent prompts.” 2025, [Online]. Available: https://arxiv.org/abs/2510.00796.

[34]

D. S. Ma and J. Correll, “Target prototypicality moderates racial bias in the decision to shoot,” Journal of Experimental Social Psychology, vol. 47, no. 2, pp. 391–396, 2011, doi: https://doi.org/10.1016/j.jesp.2010.11.002.

[35]

N. Mehrabi, F. Morstatter, N. Saxena, K. Lerman, and A. Galstyan, “A survey on bias and fairness in machine learning.” 2022, [Online]. Available: https://arxiv.org/abs/1908.09635.

[36]

J. S. Boster, “Natural sources of internal category structure: Typicality, familiarity, and similarity of birds,” Memory & Cognition, vol. 16, no. 3, pp. 258–270, 1988, doi: 10.3758/BF03197759.

[37]

K. M. Uyeda and G. Mandler, “Prototypicality norms for 28 semantic categories,” Behavior Research Methods & Instrumentation, vol. 12, no. 6, pp. 587–595, 1980, doi: 10.3758/BF03201848.

[38]

J. M. Joelson and D. J. Herrmann, “Properties of categories in semantic memory,” The American Journal of Psychology, vol. 91, no. 1, pp. 101–114, 1978.

[39]

W. F. Battig and W. E. Montague, “Category norms for verbal items in 56 categories: A replication and extension of the connecticut norms using university of maryland and illinois students,” University of Colorado, Boulder, CO, 1968.

[40]

W. F. Battig and W. E. Montague, “Category norms of verbal items in 56 categories: A replication and extension of the connecticut category norms,” Journal of Experimental Psychology, vol. 80, no. 3, Pt.2, pp. 1–46, 1969, doi: 10.1037/h0027577.

[41]

P. McIntosh, White privilege and male privilege: A personal account of coming to see correspondences through work in women’s studies. Wellesley College, Center for Research on Women, 1988.

[42]

A. Rich, “Compulsory heterosexuality and lesbian existence,” Signs: Journal of Women in Culture and Society, vol. 5, no. 4, pp. 631–660, 1980, doi: 10.1086/493756.

[43]

J. Buolamwini and T. Gebru, “Gender shades: Intersectional accuracy disparities in commercial gender classification,” in Proceedings of the 1st conference on fairness, accountability and transparency, 2018, vol. 81, pp. 77–91, [Online]. Available: https://proceedings.mlr.press/v81/buolamwini18a.html.

[44]

S. Janghorbani and G. de Melo, “MultiModal bias: Introducing a framework for stereotypical bias assessment beyond gender and race in vision language models.” 2023, [Online]. Available: https://arxiv.org/abs/2303.12734.

[45]

D. A. B. Moreira et al., “FairPIVARA: Reducing and assessing biases in CLIP-based multimodal models.” 2024, [Online]. Available: https://arxiv.org/abs/2409.19474.

[46]

Qwen et al., “Qwen2.5 technical report.” 2025, [Online]. Available: https://arxiv.org/abs/2412.15115.

[47]

G. Team et al., “Gemma 3 technical report.” 2025, [Online]. Available: https://arxiv.org/abs/2503.19786.

[48]

A. Grattafiori et al., “The llama 3 herd of models.” 2024, [Online]. Available: https://arxiv.org/abs/2407.21783.

[49]

Mistral AI, Hugging Face model card“mistralai/Mistral-Small-3.1-24B-Base-2503.” https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Base-2503, 2025.

[50]

S. Xiao, Z. Liu, P. Zhang, N. Muennighoff, D. Lian, and J.-Y. Nie, “C-pack: Packed resources for general chinese embeddings.” 2024, [Online]. Available: https://arxiv.org/abs/2309.07597.

[51]

P. He, J. Gao, and W. Chen, “DeBERTaV3: Improving DeBERTa using ELECTRA-style pre-training with gradient-disentangled embedding sharing.” 2023, [Online]. Available: https://arxiv.org/abs/2111.09543.

[52]

Black Forest Labs, Hugging Face model card“black-forest-labs/FLUX.1-schnell.” https://huggingface.co/black-forest-labs/FLUX.1-schnell, Jan. 2025.

[53]

Stability AI, Hugging Face model card“stabilityai/stable-diffusion-3.5-large-turbo.” https://huggingface.co/stabilityai/stable-diffusion-3.5-large-turbo, Jan. 2025.

[54]

E. Xie et al., “SANA: Efficient high-resolution image synthesis with linear diffusion transformers.” 2024, [Online]. Available: https://arxiv.org/abs/2410.10629.

[55]

J. Zhu et al., “InternVL3: Exploring advanced training and test-time recipes for open-source multimodal models.” 2025, [Online]. Available: https://arxiv.org/abs/2504.10479.

[56]

J. Ke, Q. Wang, Y. Wang, P. Milanfar, and F. Yang, “MUSIQ: Multi-scale image quality transformer.” 2021, [Online]. Available: https://arxiv.org/abs/2108.05997.

[57]

A. Zvezdakova, D. Kulikov, D. Kondranin, and D. Vatolin, “Barriers towards no-reference metrics application to compressed video quality analysis: On the example of no-reference metric NIQE.” 2019, [Online]. Available: https://arxiv.org/abs/1907.03842.

[58]

Y. Zha, Y. Yang, R. Li, and Z. Hu, “AlignScore: Evaluating factual consistency with a unified alignment function.” 2023, [Online]. Available: https://arxiv.org/abs/2305.16739.

[59]

J. Li, D. Li, S. Savarese, and S. Hoi, “BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.” 2023, [Online]. Available: https://arxiv.org/abs/2301.12597.

[60]

X. Wu et al., “Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis.” 2023, [Online]. Available: https://arxiv.org/abs/2306.09341.

[61]

Q. Team, “Qwen3 technical report.” 2025, [Online]. Available: https://arxiv.org/abs/2505.09388.

[62]

Y. Li, Y. Song, and J. Luo, “Improving pairwise ranking for multi-label image classification.” 2017, [Online]. Available: https://arxiv.org/abs/1704.03135.

[63]

W. Yang et al., “Psl: Rethinking and improving softmax loss from pairwise perspective for recommendation,” Advances in Neural Information Processing Systems, vol. 37, pp. 120974–121006, 2024.

[64]

Y. Hu et al., “TIFA: Accurate and interpretable text-to-image faithfulness evaluation with question answering.” 2023, [Online]. Available: https://arxiv.org/abs/2303.11897.

[65]

D. Deutsch, G. Foster, and M. Freitag, “Ties matter: Meta-evaluating modern metrics with pairwise accuracy and tie calibration,” in Proceedings of the 2023 conference on empirical methods in natural language processing, Dec. 2023, pp. 12914–12929, doi: 10.18653/v1/2023.emnlp-main.798.

Implementation settings, compute resources, and approximate construction runtimes are reported in Appendix 9.↩︎

Prototypicality Bias Reveals Blindspots in Multimodal Evaluation Metrics

Abstract

1 Introduction↩︎

2 Related Work↩︎

2.0.0.1 Bias and prototypicality in multimodal systems.

2.0.0.2 Robustness and diagnostic evaluation of T2I metrics.

3 Bias Formulation↩︎

4 ProtoBias Benchmark Construction↩︎

4.1 Prompt Construction and Validation↩︎

4.1.0.1 Taxonomy.

4.1.0.2 Prompt triplets.

4.1.0.3 Prompt generation and filtering.

4.1.0.4 Prompt-quality validation.

4.2 Image Construction and Validation↩︎

4.2.0.1 Image generation.

4.2.0.2 Image filtering.

4.2.0.3 Human validation of image filtering.

4.2.0.4 Image-quality control.

4.2.0.5 Pairwise human evaluation.

5 Experimental Setup↩︎

5.0.0.1 Evaluation Data

5.0.0.2 Evaluated Metrics

5.0.0.3 Pairwise Evaluation Protocol

5.0.0.4 Training ProtoScore

6 Results & Analysis↩︎

6.0.0.1 Human evaluation.

6.0.0.2 Existing metrics fail under prototypical distractors.

6.0.0.3 Semantic judges improve only partially.

6.0.0.4 Metric–human agreement remains weak.

6.0.0.5 Contrastive training reduces the failure but does not solve alignment.

7 Conclusion↩︎

Limitations↩︎

8 Taxonomy & Prompt Templates↩︎

8.1 Taxonomy↩︎

8.1.0.1 Animals.

8.1.0.2 Demography.

8.1.0.3 Objects.

8.2 Prompt Templates↩︎

9 Construction Models, Infrastructure, and Runtime↩︎

10 Prompt and Image Filtering Prompts↩︎

11 Human Validation of the Filtering Stage↩︎

12 Human Preference Annotation↩︎

12.0.0.1 Annotation rubric.

12.0.0.2 Annotator agreement.

13 LLM-as-Judge Prompt↩︎

14 Runtime Comparison↩︎

15 External Benchmark Evaluation↩︎

16 Additional ProtoBias Examples↩︎

References↩︎

Subjects

Updated on Academus