1 Introduction↩︎

Artificial Intelligence (AI), particularly through the development of neural networks and machine learning algorithms, has catalyzed transformative changes across diverse domains. From optimizing logistics to predicting market trends, AI’s ability to process massive volumes of data and find overlooked patterns has led to more accurate predictions, quicker responses, and informed decision-making. However, the integration of AI into critical decision-making processes also brings significant ethical and practical challenges to the forefront, especially with respect to fairness, transparency, and accountability.

The sphere of influence of AI algorithms spans a wide spectrum of domains, from the financial sector to the criminal justice system. In finance, AI-driven algorithms have revolutionized risk assessment, enabling lenders to make informed decisions about creditworthiness with unprecedented accuracy. Similarly, in law enforcement, the ability of AI to analyze patterns and anomalies has taken predictive policing and resource allocation to new levels of efficiency [1].

However, as AI’s role in shaping decisions expands, a number of ethical complexities also unfold. A fundamental challenge lies in ensuring that the algorithms that guide decision-making are grounded in fairness, transparency, and accountability. The opacity of deep neural networks can render decision processes inscrutable, with outcomes seemingly coming from a “black box” [2]. This lack of transparency introduces a layer of unpredictability, potentially eroding trust and understanding, posing a challenge for practical AI applications.

Furthermore, algorithms can unintentionally perpetuate biases present in the data on which they are trained, leading to decisions that reflect and even amplify societal inequalities. Cases of algorithmic discrimination, from biased hiring practices to discriminatory criminal sentencing, have highlighted the urgent need to address these issues. The EU AI Act ¹, which is a landmark regulatory framework designed to ensure the safe and ethical deployment of AI technologies, recognizes the profound implications for fundamental rights and privacy.

In light of these critical issues, this work aims to explore the impact of bias on the learning process of computer vision algorithms, particularly face biometric systems, placed at the top of the list of High-Risk AI Systems of the EU AI Act, due to the personal information they process, such as identity, gender, ethnicity, and age.

The existing literature on bias analysis is mainly focused on the inputs \(\mathcal{X}\) [3]–[5] and the outputs \(O\) [6]–[8]. This study goes beyond proposals that seek to model bias through performance measures [9], [10]. The exclusive focus on performance is insufficient to identify and comprehend the origins of bias, hindering efforts to mitigate it effectively. Traditional output-based evaluations treat the model as a “black box”, failing to capture how and where bias is encoded during the learning process.

To address this gap, the primary objective of this research is to propose a multi-level framework that investigates the internal mechanisms of Convolutional Neural Networks (CNNs) to uncover the origins of demographic bias. Specifically, our aim is to address the following research questions.

1. How do data biases alter the distribution of the learned latent space?

2. In what ways do neuron activation levels correlate with class under-representation?

3. Can bias be systematically detected directly from the learned parameters of convolutional filters, completely independent of input data?

By adopting a holistic view of the model at all possible levels –geometry of embeddings, neuron activations, and convolutional filters (or kernels) – we aim to better understand biased learning (see Fig. 1). The key contributions of this work are summarized as follows:

• A comprehensive taxonomy of bias effects in the learning process: We present a multi-level taxonomy outlining how bias propagates through the data-driven learning process of CNNs. We (i) show that the learned latent space is less effectively distributed in biased models, (ii) reveal latent correlations between demographic bias and activation levels, and (iii) demonstrate that networks encode bias-related patterns within their convolutional filter parameters.

• Three novel bias detection methods: Based on our taxonomy, we propose three specific bias detection approaches: (i) SpaceBias (Level 2), which quantitatively assesses bias based on the representation of the learned latent space; (ii) ActivationBias (Level 3), an efficient method to detect bias by examining activation levels; (iii) WeightBias (Level 4) a method to detect bias directly from the model’s weights.

• Extensive experimentation in face biometrics and MNIST: We provide reproducible experiments using the public DiveFace [11] and MNIST [12] datasets. To validate our Level 4 methodology, we trained and evaluated more than 127,000 models with varying degrees of bias.

Delineation from prior conference work↩︎

Preliminary versions of the subsets of this study were presented at IAPR ICPR 2021 [13] (InsideBias, the Level-3 activation analysis) and at the AAAI SafeAI Workshop 2021 [14] (IFBiD, the Level-4 weight-based detector). The journal version differs from those conference papers along three axes:

1. A new method (entirely new in this paper), SpaceBias, the Level-2 latent-space analysis based on neighbor-probability distributions and the Kolmogorov–Smirnov test, was not part of either conference papers.

2. Extension of InsideBias. ActivationBias extends InsideBias with deeper analyses of activations and a Mann–Whitney U test for interpretable detection.

3. A unified multi-level framework with MNIST experiments. The most important new contribution is the integration of all three levels into a single taxonomy (Fig. 1) with consistent notation, dataset, and protocol.

The remainder of the paper is structured as follows: Section 2 summarizes previous work. Section 3 presents our methodology. Section 4 describes the evaluation procedure in Face Biometrics and MNIST. Section 5 presents the experimental results in Face Biometrics and MNIST. Finally, Section 6 summarizes the main conclusions.

2 Previous Work↩︎

Bias in computer-vision models has been studied predominantly from an input–output perspective that treats the network as a black box. [15] probe a model with a hand-selected set of adversarial examples to decide whether it is biased, extending the example-based critique of [16], while [17] compare attribution maps (Grad-CAM, Score-CAM, Integrated Gradients, LRP-\(\epsilon\)) as bias indicators and report that they disagree and occasionally mislead. Such analyses establish that a model is biased but reveal little about where the bias resides. In contrast, the four levels of our taxonomy (Fig. 1) conceptualize bias as a phenomenon that progressively emerges and becomes encoded throughout the neural network hierarchy. We organize the discussion around these four levels, contrasting each with its closest prior work.

Level 1 (output). Most fairness research reads bias from prediction disparities and group metrics. [18] survey the problem at large; [19] systematize fairness definitions and [20] survey how they are measured, while [21] review mitigation across pre-, in-, and post-processing. In face biometrics, [22] document recognition biases beyond demographics, and [10] propose an equity index tailored to operational face systems. Most remedies act on the data: [23] balance training with synthetic faces, and [24] construct a causal benchmark that manipulates input attributes. These input/output remedies say little about how bias is represented internally—a methodological gap that [25] make explicit. Level 1 is thus our established baseline, and we move inward.

Level 2 (learned representations). Prior work tends to shape the embedding rather than read it: [26] learn fair representations by disentanglement, and [27] show that even balanced data leave gender bias in image embeddings. Earlier attempts to read bias from representations either remove it adversarially [7] or examine CNN features against dataset bias [4], typically qualitatively. SpaceBias instead makes the latent geometry quantitative and training-free, characterizing the penultimate space with neighbor-probability distributions and a Kolmogorov–Smirnov test.

Level 3 (layer activations). Intermediate activations have drawn comparatively little attention as a bias signal, even as evidence accumulates that networks lean on shortcuts: [28] frame shortcut learning in general terms, [29] show that overparameterization amplifies spurious correlations, and [30] detect such features at ImageNet scale. ActivationBias targets this regime directly, providing a dynamic characterization of how strongly different groups activate the deep, task-specific layers and testing the gap with a Mann–Whitney \(U\) statistic.

Level 4 (learned parameters). [31] show that networks internalize dataset shortcuts in their weights, so that retraining only the last layer can undo a bias the features still encode. WeightBias pushes this further: rather than retrain, it recovers the type of bias from the convolutional filters alone, with no access to inputs or outputs.

Across the four levels, our contribution is to move from whether biased outcomes occur to where, when, and how bias forms and is stored inside the network.

3 Methodology: Analyzing Bias at Multiple Levels↩︎

In this section, we outline a systematic methodology (illustrated in Figure 2) for exploring the intricacies of neural network architectures, focusing on how biases manifest across various levels of the network, from embeddings and activations to convolutional filters.

Bias at the input-output level (Level 1) is the most commonly studied effect. This level captures how biases in the distribution of input data propagate through the network and influence the output of the model. We do not focus on it in this work; we go beyond the output to analyze the hidden structures and mechanisms that influence bias.

We begin our analysis with embeddings (Level 2) — the rich, condensed representations learned by the network. These embeddings often encapsulate biases present in training data, providing fertile ground for detecting bias. By dissecting these embeddings, we aim to understand how biases are represented in the latent space, where data are compressed into a more abstract form. This analysis is closely related to Input-Output bias, but focuses on understanding how biases influence the internal representations of the network rather than the final output.

Moving beyond the latent space, we investigate layer activations² (Level 3). This approach examines how the network responds to different inputs at each layer, allowing us to observe how these responses vary across different sets of data.

At the deepest level, we explore bias in the weights of the network (Level 4). Unlike the previous levels, this analysis focuses solely on the learned convolutional filters, disregarding the inputs and outputs. This level of analysis aims to demonstrate how bias is embedded even in the most fundamental parameters of the network, as they play a crucial role in shaping the network’s understanding and representation of features.

3.1 Detecting Bias in the Learned Latent Space (Level 2)↩︎

The mathematical description of what a neural network does in each layer is to transform elements in the input space into a target space by means of a function. This mapping is learned during the training of the neural network, where the model adjusts the weights to optimize performance on a specific task. The space in which each layer of the network transforms is called the latent space, the latent feature space, or the embedding space. Elements in this space (embeddings) capture the essential semantic relationships between entities, enabling efficient downstream tasks such as classification. By analyzing the relationships between embeddings, we can identify patterns that may indicate underlying biases.

A model is biased with respect to a specific variable when it behaves differently depending on the variable’s value. We posit that a biased latent space is one in which a particular group of data or class is poorly distributed within it. We focus on the space of the layer prior to the classification layer. This is the representation space to which the application of a linear transformation generates the classification probabilities. This means that if this space is not well modeled, probably neither is the model’s output.

Elements in this representation space are \(n\)-dimensional vectors, which can be interpreted geometrically as single points in \(\mathbb{R}^n\). Direct visualizations in three dimensions will be a reduction in which valuable geometric information may be lost. Therefore, to study in low dimensions how points are distributed in this \(n\)-dimensional space, we use neighbor probabilities, which are the basis of the popular Stochastic Neighbor Embedding [32] and Neighborhood Components Analysis [33].

The probability that point \(i\) chooses \(j\) as its neighbor is: \[\label{eqn:pij} p_{ij}= \frac{\text{exp}(-||\boldsymbol{x}_i-\boldsymbol{x}_j||^2)}{\sum_{k\neq i} \text{exp}(-||\boldsymbol{x}_i-\boldsymbol{x}_k||^2)}\tag{1}\]

In this equation, \(\boldsymbol{x}_i\) and \(\boldsymbol{x}_j\) represent the points \(i\) and \(j\) as vectors. The numerator represents the affinity between the points \(i\) and \(j\), while the denominator is a normalization term that ensures the probabilities sum to one over all possible neighbors \(k \neq i\).

Under this stochastic selection rule, we can compute the probability \(p_i\) that point \(i\) will be correctly classified within a set of \(N\) neighbors. Let \(C_i = \{j \in \mathcal{N} \mid c_i = c_j\}\) denote the set of points in the neighborhood \(\mathcal{N}\) that share the same class as \(i\). Then:

\[\label{eqn:pi} p_i= \sum_{j \in C_i} p_{ij}\tag{2}\]

Here, \(p_i\) is the sum of the neighbor probabilities \(p_{ij}\) for all neighbors \(j\) that belong to the same class as \(i\). This sum represents the probability that point \(i\) will be correctly classified based on its proximity to other points of the same class within the neighborhood. By measuring the probability of all points belonging to their respective classes, we obtain a probability distribution. This probability distribution provides an approximation of how the \(n\)-dimensional space is structured over a distance of N neighbors. Figure 3 illustrates this distribution of probabilities \(p_i\).

The fact that the probability distributions of different groups are diverse indicates that the layout of the latent space for these groups is dissimilar. The sense in which these spaces are dissimilar is that some groups have the samples of both classes more intermingled than the samples of other groups. The direct consequence is that it is more difficult for a classifier (final dense layer) to separate between the two classes. Since the final classification layer operates on these embeddings, separability there bounds what that head can achieve. The real head is a linear softmax, so \(p_i\) measures \(NN\)-separability, which is tight for our method and strongly indicative (not identical) for the linear classifier.

To quantitatively compare these probability distributions, we use the the two-sample Kolmogorov–Smirnov (\(KS\)) test. The \(KS\) statistic, is defined as the maximum vertical distance between the empirical cumulative distribution functions (eCDFs) of two samples, denoted by \(F_{1,n_1}(z)\) and \(F_{2,n_2}(z)\). The \(KS\) test is a statistical method employed to assess the similarity of the cumulative distributions of two sets. It is a non-parametric test, which means that it does not make any assumptions about the underlying distribution of the data. It tests the hypotheses \(H_0 : F_1 = F_2\) versus \(H_1 : F_1 \neq F_2\), where \(F_n\) denotes the empirical cumulative distribution function (eCDF) for \(n\) independent and identically distributed (i.i.d.) observations. Essentially, it determines whether the differences between the two probability distributions are significant or not.

The \(KS\) statistic is given by:

\[\label{eqn:Dn} D_n= \sup_{z} \big |F_{1,n_1}(z) - F_{2,n_2}(z)\big |\tag{3}\]

The significance level \(\alpha\) represents the probability of making a Type I error, which occurs when we incorrectly reject a true null hypothesis. In other words, it quantifies the risk of concluding that there is a significant difference between two samples when, in reality, there is none. A common choice for the significance level is \(\alpha = 0.05\), corresponding to a 95% confidence level. This means that if we were to perform the same experiment many times, approximately \(95\)% of the time, we would correctly fail to reject the null hypothesis when it is true. In our experiments, we use the slightly more conservative \(\alpha = 0.01\) (\(99\)% confidence) because the KS test is high-powered at our sample sizes (\(n_1, n_2 = 6{,}000\) per group). For large samples, the null hypothesis (\(H_0\)) is rejected with confidence level \(\alpha\) if:

\[\label{eqn:null} D_n > \sqrt{-\text{ln}(\frac{\alpha}{2}) \cdot\frac{n_1+n_2}{2n_1n_2}}\tag{4}\]

3.1.0.1 Why the \(KS\) test

The clustering probabilities generated in the latent space are highly non-normal, heavily skewed, and multimodal. Therefore, parametric tests that assume normality (such as t-tests or ANOVA) are mathematically invalid for this context. We selected the two-sample \(KS\) test because it is a non-parametric test that makes no assumptions about the underlying data distribution beyond continuity, which is appropriate for probabilities \(p_i \in [0,1]\). Furthermore, because bias in the latent space alters the concentration of probability mass rather than simply shifting a unified mean, the \(KS\) test’s sensitivity to both the location and the shape of the eCDF makes it the ideal metric to quantify these specific structural discrepancies. Alternatives such as the Anderson–Darling test were not chosen because they overweight tail differences, which are less relevant when \(p_i\) lies in a bounded interval. The Wasserstein-1 distance was also considered, but the \(KS\) test provides a closed-form significance threshold that is more interpretable in a multi-comparison setting.

3.1.0.2 Definition 1 (Latent-space bias)

Let \(\phi:\mathcal{X} \to \mathbb{R}^d\) be a neural network mapping inputs to the penultimate latent space, and let \(\mathcal{V}\) be a variable (e.g. categorical attribute). For each input \(\boldsymbol{x}_i \in \mathcal{X}\) with neighborhood size \(N\), let \(p_i(\boldsymbol{x}_i) \in [0,1]\) denote the clustering probability defined in Eq. 2 . Let \(F_v\) denote the empirical cumulative distribution function (eCDF) of \(\{p_i(\boldsymbol{x}) : \boldsymbol{x} \in \mathcal{V} = v\}\). We say that \(\phi\) is biased in the latent space with respect to \(\mathcal{V}\) at confidence level \(\alpha\) if there exists \(v_1, v_2\) such that the Kolmogorov–Smirnov test rejects \(H_0: F_{v_1} = F_{v_2}\) at level \(\alpha\) (Eq. 4 ). The magnitude of the latent-space bias indicator is: \[\label{eqn:LBI} \Gamma_v(\phi; v_1, v_2) \;=\; D_n\bigl(F_{v_1},\, F_{v_2}\bigr) \;\in\; [0, 1].\tag{5}\] Larger \(\Gamma_v\) indicates a more strongly biased latent geometry; \(\Gamma_v = 0\) corresponds to identically distributed clustering probabilities, i.e.no detectable latent-space bias under a sample of \(n_1 + n_2\) observations.

3.2 Detecting Bias in the Activations of the Layers (Level 3)↩︎

Convolutional Neural Networks are composed of a large number of stacked filters. These filters are trained to extract the richest information for a predefined task. As the input, for example, an image, transverses the network, the filters are activated. Without loss of generality, we present a method to establish a relationship between activation and bias for Convolutional Neural Networks (CNNs). Similar ideas can be extended to other neural learning architectures.

In a convolutional layer of a CNN, the feature maps of the previous layer are convolved with the filters (also known as kernels) and put through the activation function to form the output feature map. The output \(\boldsymbol{A}^{[l]}\) of layer \(l\) consists of \(m^{[l]}\) feature maps of size \(n_1^{[l]} \times n_2^{[l]}\), where \(m^{[l]}\) is the number of filters in layer \(l\). The \(i^{\textrm{th}}\) feature map of layer \(l\), denoted as \(\boldsymbol{A}_i^{[l]}\) is computed as:

\[\label{eqn:feature95map} \boldsymbol{A}_i^{[l]} = g^{[l]}\left(\sum_{j=1}^{m^{[l-1]}} \boldsymbol{f}_{ij}^{[l]} * \boldsymbol{A}_j^{[l-1]} + \boldsymbol{b}_i^{[l]}\right)\tag{6}\]

where \(g^{[l]}\) denotes the activation function of layer \(l\), \(*\) is the convolutional operator, \(\boldsymbol{b}^{[l]}_i\) is a bias vector in layer \(l\) for the \(i^{\textrm{th}}\) feature map and \(\boldsymbol{f}_{ij}^{[l]}\) is the filter connecting the \(j^{\textrm{th}}\) feature map in layer \((l-1)\) with the \(i^{\textrm{th}}\) feature map in layer \(l\). The average activation of the \(i^{\textrm{th}}\) feature map at layer \(l\) is calculated as:

\[\label{eqn:average} \overline{A_i^{[l]}} = \frac{1}{n_1^{[l]} \cdot n_2^{[l]}} \sum_{h=1}^{n_1^{[l]}} \sum_{w=1}^{n_2^{[l]}} A_{i}^{[l]} (h,w)\tag{7}\]

where \((h,w)\) are the spatial coordinates of the output \(A_{i}^{[l]}\). The activation, \(\hat{\lambda}^{[l]}\), is calculated as the maximum of \(\overline{A_i^{[l]}}\) for all feature maps in the layer \(l\):

\[\label{eqn:activation} \lambda^{[l]} = \max_{i}\left(\overline{A_i^{[l]}}\right)\tag{8}\]

We have evaluated both the average and the maximum, but the maximum resulted in a better estimator. Our intuition is that the maximum works best because it is exactly the mechanism used in the MaxPooling layers of the network.

We then compare per-group distributions of the activation \(\lambda^{[l]}\) (Eq. 8 ) for every possible set of samples \(\mathcal{D}_{v_k}\) of a variable’s values \(v_k\). For each pair of groups \(v_1, v_2\) we apply the two-sample Mann–Whitney \(U\) test and report the probability of superiority: \[\label{eqn:theta} \hat{\theta}_{v_1 v_2} \;=\; \frac{U_{v_1 v_2}}{n_{v_1} n_{v_2}} \;=\; \mathbb{P} \bigl(\lambda^{[l]}(\mathcal{D}_{v_1}) > \lambda^{[l]}(\mathcal{D}_{v_2})\bigr), \qquad \mathcal{D}_{v_1} \sim (\mathcal{V}{=}v_1),\; \mathcal{D}_{v_2} \sim (\mathcal{V}{=}v_2),\tag{9}\] which is the probability that a randomly drawn sample from group \(v_1\) activates more strongly than one from group \(v_2\). A value of \(\tfrac12\) indicates stochastically equal activations; deviations towards \(0\) or \(1\) indicate that one group consistently activates less or more.

3.2.0.1 Definition 2 (Activation bias)

Let \(\phi\) be a convolutional neural network with \(L\) convolutional layers, and let \(\mathcal{V}\) be a variable (e.g., categorical attribute). For a pair of values \(v_1, v_2\), let \(\hat{\theta}_{v_1 v_2}\) be the probability of superiority, estimated by the normalized Mann–Whitney \(U\) statistic (Eq. 9 ). We say that \(\phi\) is biased in the activations at layer \(l\) with respect to \(\mathcal{V}\) if there exist \(v_1, v_2\) such that the Mann–Whitney test rejects \(H_0: \theta_{v_1 v_2} = \tfrac12\). The magnitude of the Activation Bias Indicator is \[\label{eqn:ABI} \Lambda^{[l]}_v \;=\; \bigl|1- 2\,\hat{\theta}_{v_1 v_2}\bigr| \;\in\; [0, 1],\tag{10}\] which is the absolute rank-biserial correlation between the two groups’ activations. Larger \(\Lambda^{[l]}_v\) indicates more strongly separated activation distributions; \(\Lambda^{[l]}_v = 0\) corresponds to stochastically equal activations (\(\hat{\theta}_{v_1 v_2} = \tfrac12\)), i.e., there is no detectable activation bias at layer \(l\), while the sign of \((\hat{\theta}_{v_1 v_2} - \tfrac12)\) identifies the favored group.

3.3 Detecting Bias in Parameters of the Convolution Filters (Level 4)↩︎

We analyze the parameters \(\mathbf{\Omega}\) of Convolutional Neural Networks \(\phi(\cdot)\). The aim is to find patterns in the convolutional filters of the model associated with biased outcomes. The purpose of convolution is to detect the filter pattern in the image, producing a high signal when present and a low signal when not present. Following that reasoning, a high activation signal indicates that the image contains the pattern (that is why stronger activations are usually related to the detection of highly discriminative features [34]). Therefore, when a model has not successfully learned the patterns associated with certain classes, these patterns are not detected (activated) by the network, which leads to poor performance on those classes. This outlines the connection between the failure to activate certain patterns and poor performance (bias).

The training process of Neural Networks is usually not deterministic and the resulting parameters \(\mathbf{\Omega}\) depend on several elements: training data, learning architecture (e.g., number of layers, number of neurons per layer, etc.), training hyper-parameters (e.g., loss function, number of epochs, batch size, learning rate, etc.), initialization parameters, and optimization algorithm.

Due to the non-deterministic nature of the network training process \(\phi(\cdot |\mathbf{\Omega})\), the same training subset is likely to give rise to different \(\mathbf{\Omega}\). The reason for this is that since the solution space is very large, the solution (which is iteratively approximated) typically arrives at a local minimum that depends on the initialization, the particular training configuration and the order of the data [35]. In CNNs, this translates into the fact that no two networks are identical in terms of the learned convolutional filters. The challenge, therefore, is to be able to detect patterns in the filters in different positions and configurations. The problem is analogous to detecting patterns in images, where one can be in different parts of an image.

Our proposed approach involves allowing another neural network to uncover bias patterns. Supervised neural network systems are particularly useful for this endeavor as their objective is to identify correlations between input patterns and output. We designed the biased pattern detector as a Neural Network \(\psi(\cdot)\) represented by its parameters \(\mathbf{\Theta}\), concisely described as \(\psi(\cdot | \mathbf{\Theta})\). In this approach, we train the pattern detector using a dataset of biased and unbiased models \(\phi(\cdot)\). These models \(\phi(\cdot |\mathbf{\Omega})\) are trained on data related to a specific task \(T\), such as image classification or face recognition, learning the parameters \(\mathbf{\Omega}\) that optimize their performance on this task. They are trained with subsets \(\mathcal{S}_1, ..., \mathcal{S}_n \subset \mathcal{D}\) from the entire database \(\mathcal{D}\). Each subset \(\mathcal{S}_i\) is intentionally biased in some way. For example, a subset might over-represent or under-represent certain classes or demographic groups. This means that each model \(\phi(\cdot)\) is trained using a different subset \(\mathcal{S}_i\) of the entire dataset \(\mathcal{D}\), which introduces specific biases embedded in its learned parameters \(\mathbf{\Omega}\).

Training \(\psi(\cdot)\) on the parameters \(\mathbf{\Omega}\), it learns to identify if certain biases are present in the parameters themselves. This setup allows the detector to identify biased patterns by analyzing the learned parameters of task-specific models, rather than directly analyzing the task outcomes.

Designing the detector \(\psi(\cdot)\) involves exploring various neural network architectures. The choice of architecture requires careful consideration of how well it can process the input parameters \(\mathbf{\Omega}\) from the models \(\phi(\cdot |\mathbf{\Omega})\). The effectiveness of the detector depends on how well it is tailored to handle these input parameters, and its design must be informed by the characteristics of the parameters \(\mathbf{\Omega}\). (See 10 for more details.)

4 Experimental Set-up: Two Applications↩︎

We instantiate the methodology of §3 in two computer vision applications, chosen so that they vary the task, the data domain, and the bias mechanism while keeping the detection methodology constant:

• Application 1 (§4.1): Face biometrics. Gender classification of face images, with demographic under-representation as the bias mechanism. The bias variable is ethnicity (Asian, African/Indian, Caucasian).

• Application 2 (§4.2): Colored MNIST. Digit classification of synthetic colored-MNIST images, with a skewed color distribution as the bias mechanism. The bias variable is the color of the image (red, green, or blue).

The two applications play complementary roles. The ResNet face models are our analysis instrument: a small, controllable set, deep enough to make the effect interpretable, on which we study how bias reshapes the latent space (Level 2) and the activations (Level 3). Detection is then validated at scale on two independent model populations —the tiny face models and the colored-MNIST models— each containing thousands of biased and unbiased networks with known bias labels. Running all three detectors on the MNIST model population lets us measure detection accuracy and, because the two domains differ in task, data, and bias mechanism, assess how much of the methodology is general rather than face-specific. Colored MNIST plays one further role that faces cannot afford: because it is inexpensive to train, we use it to sweep the severity of the bias and measure how detection degrades as the training distribution moves from heavily skewed toward balanced.

The two roles also reflect how the detectors work. The Level 2–3 detectors are training-free statistical tests applied to a single model, whereas the Level 4 detector is a learned model trained on a population. Because per-model tests are run individually and this is computationally costly, Level 2–3 detection experiments use a fixed subsample of \(100\) models per configuration in each domain, which is sufficient for a stable accuracy estimate; the Level 4 detector is trained on the full populations.

4.1 Application 1: Face Biometrics↩︎

We train neural networks for gender classification from face images. Since establishing ground-truth genetic sex is not possible, we use gender as a proxy for sex, as a simplified instance of a real-world problem.

4.1.1 Database↩︎

We use the DiveFace dataset [11], a collection of facial images categorized into six classes by gender and ethnicity. Ethnicity attributes include not only skin color, but also more complex anthropomorphic facial features. The dataset contains 72,000 images: three images for each of 24,000 identities, balanced at 12,000 per gender and ethnicity. Users are grouped by gender (male or female) and by three categories related to ethnic physical characteristics: Asian, African / Indian, and Caucasian.

In all protocols below, models are trained from scratch with Glorot-uniform initialization [36]; no ImageNet [37] or VGGFace [38] pretraining is used, so that observed biases can be unambiguously attributed to the DiveFace training distribution. Training and test images are equally divided between males and females.

4.1.2 ResNet Face Models: Analysis of the Latent Space and Activations (Levels 2–3)↩︎

For the analysis of the latent space (Level 2) and the activations (Level 3), we use a compact ResNet architecture [39], based on residual connections – a representative baseline for facial attribute detection [40]. It comprises over 370K parameters in 22 convolutional layers; the layer-by-layer specifications are given in 8.

We train five models that differ only in the demographic distribution of their training data, yielding five ResNet face models in total. Table 1 reports the per-group gender-classification accuracy along with the training protocol. The database was split 75% / 25% into training and test sets; the 18,000-image test set is distributed equally across the three ethnic groups and none of the test users appear in the training set. Both biased and unbiased models are trained with the same number of samples (18,000); only the Unbiased (+) model is trained with three times more data (54,000). Each biased model over-represents one group (90% of the images), producing the skewed per-group performance visible in the table.

4pt

4.1.3 Tiny Face Models: Detection in the Parameters (Level 4)↩︎

Validating the learned weight detector of Level 4 requires a large population of task models with known bias labels. For this we use tiny face models (a standard CNN, \(\approx\)​100K parameters; 8), small enough to train tens of thousands of models. Using the same task and bias mechanism as the ResNet models, we train 36,000 biased models under three protocols, each over-representing one ethnic group (Table 2). To keep detector training and evaluation independent, the dataset is split into two equal, non-overlapping halves: the bulk of the models are trained on Partition 1 and the remainder on Partition 2.

Note the gap between model sizes: the ResNet models are nearly perfect (94–98%), whereas the tiny models range between roughly 80% and 90% depending on the group. This is because the architecture of the tiny models lacks the representational capacity to capture the subtle facial features needed for accurate gender classification.

4.2 Application 2: Colored MNIST↩︎

The second application is digit classification on a controlled colored-MNIST benchmark whose bias mechanism is a skewed color distribution. Holding the methodology of §3 constant while changing the task (digit vs.gender) and the data domain (synthetic 28\(\times\)​28 RGB vs.natural face crops) tests whether the multi-level framework captures something general about how CNNs encode bias. With thousands of biased and unbiased models under a single architecture, this population is our second independent detection testbed: agreement between it and the tiny face models, across domains that differ in task, data, and bias mechanism, is what lets us argue the methodology is not face-specific.

4.2.1 Dataset construction↩︎

We used the canonical MNIST partitions (60,000 training images and 10,000 test images) and split the training set into two equal halves of 30,000. Every image is converted to RGB and assigned one of three colors – red, green, or blue – by coloring the background while leaving the digit strokes white; all partitions (training partition 1, training partition 2, and the test partition) are balanced, with each color equally represented within every digit class. Bias is introduced only through the training color distribution.

4.2.2 Architecture and training protocol↩︎

Unlike the face application, where we deliberately varied the size and type of architecture across methodologies, here we standardize on a single architecture for all three levels: since the task is simple, a small network reaches good performance and can be trained thousands of times cheaply. The colored-MNIST models use a small CNN with three convolutional layers and approximately 30\(K\) parameters (¿sec:appx:mnist95arch?). As in the face application, models are trained from scratch with Glorot-uniform initialization, and the dataset is split into two non-overlapping halves to keep detector training and testing independent.

We train models under three biased families—majority red, green, or blue—each at four levels of severity, from the most skewed (I: 99%/0.5%/0.5%) to the least (IV: 60%/20%/20%), together with an unbiased family (33%/33%/33%). Table 3 lists the twelve biased configurations and the unbiased one. We train \(7,000\) models per configuration, split across the two training partitions (5,000 and 2,000); the Level 2–3 detection experiments draw a \(100\)-models-per-configuration subsample, while the Level 4 detector is trained on the full population. The four severity levels let us report detection not as a single number but as a function of how biased the data are; for a head-to-head comparison with faces, we reference the configuration closest to the tiny-face protocol (II, 90%/5%/5%).

5 Findings and Interpretation↩︎

We present results level by level. At each level, we first analyze how bias manifests, using ResNet face models, and then evaluate detection at scale. At-scale statistical detectors of Levels 2–3 are validated on the colored-MNIST population, where the controlled severity sweep additionally measures how detection degrades as the training distribution moves from heavily skewed (I) toward balanced (IV). Every level pairs a disparity measure (for understanding) with a decision rule (for detection): a Kolmogorov–Smirnov (KS) test at Level 2, the rank-biserial correlation of a Mann–Whitney \(U\) test at Level 3, and a learned classifier at Level 4.

5.1 Level 2: Latent Space (SpaceBias)↩︎

The space generated by a model is high dimensional, which is why using the clustering algorithm from [32] can help us understand how the space is distributed. Visualization algorithms like t-SNE or PCA that project the space into three dimensions do not allow us to capture the complex relationships adequately. While with t-SNE or PCA it is observable that images from the same group are clustered together, our aim is to investigate whether the space is poorly distributed for underrepresented groups. We use nearest-neighbor probabilities to estimate the structure of the latent space, that is, the foundation algorithm of t-SNE. To quantify the difference in this structure, we use the Kolmogorov-Smirnov (KS) statistical test between the probability distributions.

5.1.1 Analysis (ResNet face models)↩︎

We first inspect the distribution of clustering probabilities \(p_i\) in the embedding space immediately before the classification layer. Figure 4 shows these distributions, and their empirical cumulative distribution functions (eCDF), for Caucasian-biased and unbiased ResNet models at \(N=100\). In the unbiased model, the three demographic groups are almost indistinguishable, whereas in the Caucasian-biased model, the favored group (Caucasian) is clearly displaced from the other two: its \(p_i\) values are systematically higher, so its eCDF separates from those of the Asian and African/Indian groups, which remain superimposed. A more concentrated distribution of \(p_i\) near one indicates that an image’s neighbors in the latent space more reliably share its class; the favored group enjoys this better-structured neighborhood, while the underrepresented groups do less.

To quantify these differences as a function of the neighborhood size, Figure 5 reports, for each ResNet model, both the \(KS\) statistic and its \(p\)-value for the three pairwise group comparisons as \(N\) grows from \(2\) to \(100\). Two complementary readings emerge. The KS statistic behaves as an effect size: in each biased model the comparisons that involve the favored group grow with \(N\) to \(0.08\)–\(0.18\), while the comparison between the two non-favored groups stays small. This relationship is not symmetrical, since the inherent anthropometric differences vary depending on the specific demographic groups being compared. The Caucasian-biased model is the clearest case: the Asian-vs-African/Indian comparison is essentially null (\(D_n\!\approx\!0\)), confirming that the two underrepresented groups are modeled almost identically, whereas both comparisons against Caucasians are large. In the unbiased and unbiased (+) models all three statistics remain below \(\sim\!0.025\) regardless of \(N\), indicating a latent space that is distributed comparably across the three groups. The \(p\)-value panel is consistent with this and equally discriminative: the favored-group comparisons of the biased models are overwhelmingly significant, with \(p\)-values reaching \(10^{-40}\) and below, whereas in the unbiased and unbiased (+) models none of the comparisons is significant. Here, a \(p\)-value below the threshold indicates a rejection of the null hypothesis (\(H_0\)), which means that the underlying distributions are statistically distinct. Significance therefore tracks the imposed bias, and the separation between the two regimes is sharp.

5.1.2 Detection at scale (colored MNIST)↩︎

We have established that the \(KS\) statistic on \(p_i\) separates biased from unbiased latent geometries. We turn that statistic into a usable bias detector by thresholding its value. Concretely, for each model \(\phi\) we compute the three pairwise \(KS\) tests between the demographic groups and declare the model biased if at least one of the values \(\Gamma\) falls above the chosen threshold. For each model, we compute the clustering probability \(p_i\) in the penultimate latent space, separated by color group (red, green, blue), and apply the \(KS\) test as in §3.1.

We used 300 biased models (100 each bias) and 100 unbiased models trained with partition 1 of MNIST. We fix \(N=100\) for the headline result. As shown in Figure 6, the \(KS\) decision is stable in the range \(10 \le N \le 100\), confirming that the result is not specific to one choice of \(N\). We use \(\alpha = 0.01\) on the raw \(p\)-values together with Bonferroni correction over the three pairwise comparisons, giving an effective per-test threshold of \(\alpha_{\text{eff}} = 0.01/3 \approx 0.0033\).

We threshold the KS statistic rather than its \(p\)-value for two reasons. The statistic is a bounded effect size that is largely insensitive to sample size, so a single threshold transfers across domains with different test-set sizes (faces, \(\sim\)​6,000 per group; MNIST, \(\sim\)​3,300 per group), whereas a \(p\)-value threshold would need recalibration for each. And, as the sweep below shows, the statistic stays clearly discriminative for mild bias, whereas the favored-group \(p\)-value drifts back toward non-significance as the skew weakens. We fix \(N=100\) and set the threshold \(\Gamma\) on the largest pairwise statistic so that the false-positive rate in the unbiased models remains at \(5\%\), which gives \(\Gamma\ge0.06\). In ResNet face models, the same criterion cleanly separates biased models (favored-group comparison \(\geq 0.12\)) from unbiased and unbiased (+) ones (\(\leq 0.025\)).

In colored MNIST, the favored-color \(KS\) statistic at \(N=100\) falls steadily from about \(0.17\), \(0.15\) and \(0.12\) to \(0.07\) for severities I–IV (Figure 6), against about \(0.02\) for unbiased models; the comparison between the two minority colors remains small throughout, mirroring the Asian-vs-African/Indian behavior on faces. At the \(5\%\)-false-positive operating point (\(\Gamma\ge0.06\)), every model biased at severity levels I–III is detected, and \(87\%\) of the mildly biased severity IV models are caught (Table 4). The choice of operating point matters only at this mild-bias boundary: a more permissive threshold raises severity IV recall to \(94\%\) at the cost of a higher false-positive rate, while a stricter one trades recall for specificity (Figure 10). We return to the interpretation of these false positives at the end of §5.2.3.

The statistical test result does not say which group is favored, but that they are different. To see which group is better represented in the latent space, we need to look at the probability distribution. A better representation of the latent space will have the clustering probabilities \(p_i\) more concentrated around 1, as in Figure 4 (left).

5.2 Level 3: Activations (ActivationBias)↩︎

Through this experiment, we observe that the activations of the final convolutional layers serve as indicators of the model’s bias. Minority groups, defined by fewer samples during training, exhibit lower activation levels \(\lambda\) in the final convolutional layer, whereas groups with a larger number of samples demonstrate higher activation levels \(\lambda\). The activation signal differs especially at the end of the network, where the learned patterns are more abstract and task-specific.

5.2.1 Analysis (ResNet face models)↩︎

All activation results on faces use the independent \(25\%\) test partition (\(6{,}000\) images per ethnicity). Figure 7 reports the group-average activation in every convolutional layer of ResNet models for each demographic group, shown as \(10^{\lambda}\) (with \(\lambda\) from Eq. 8 ) to spread the wide dynamic range and keep the deep-layer differences legible. This monotonic rescaling is only for display; the Activation Bias Indicator and the Mann–Whitney rank statistics are computed on \(\lambda\). The per-layer profile shows that the difference in activation between well- and poorly-represented groups is not homogeneous across layers in the Biased models, with the final layers of the networks experiencing a particularly prominent trend: the ethnic group prioritized during training, with a higher proportion of data, consistently yields stronger activation in the final layer.

To interpret this trend, the same figure includes an out-of-domain reference: the average activation generated by \(1{,}000\) random ImageNet images [37], processed identically to the face crops. ImageNet images activate the early layers more strongly than faces because those layers respond to generic low-level features (edges, colors, and textures) present in any natural image; their activation, however, decays in the deeper layers, which encode the higher-level features specific to the task (gender-relevant facial structure) that ImageNet images lack, and the face images overtake ImageNet precisely there. Because a convolution yields a high response only when the input matches the learned filter and a low one otherwise  [34], the activation magnitude measures how well the input fits the patterns the network has actually learned for the task: deep-layer activation reflects the fit of the pattern relevant to the task, rather than a generic property of the image. In light of this, the lower deep-layer activations of the underrepresented groups are not a coincidental correlation, but an interpretable signal that the network has learned their task-relevant patterns less effectively, which in turn may contribute to the performance gaps reported in Table 1.

This reading is consistent with the behavior of the early layers. As Table 1 shows, lower activation in the initial layers does not necessarily translate into lower overall performance: it appears to be compensated by a higher activation in the later layers, which are more closely related to the high-level features of the task \(T\) (gender recognition). The bias therefore localizes to the deep, task-specific layers rather than to the early, generic feature extractors. Ultimately, classification draws on the interplay of all activations [41] rather than on any single layer; what biased models alter is the balance of that interplay in deep layers.

In contrast to Biased models, Unbiased and Unbiased (+) models (Figure 7) exhibit reduced disparities between groups, particularly in the final layers. The Unbiased (+) model, trained with three times more data, shows markedly more uniform activations across demographic groups. These results indicate a coherent relationship between the amount of training data, the performance reported in Table 1, and the activations of Figure 7: as the training distribution becomes balanced (and larger), the activation disparity in the deep layers shrinks.

Figure 8 examines the activations of the final layer directly: for each biased model, the activation distribution of the favored group is shifted towards higher values relative to the two underrepresented groups, whereas the three distributions coincide for the unbiased models. This is the distributional counterpart of the per-layer trend in Figure 7, and is the quantity summarized by the Activation Bias Indicator used for detection.

Because activation is the convolution of the input with the learned filters followed by max pooling—an operation maximized when the input pattern matches the filter and sharpened further by the pooling step—a pattern that occurs frequently in the training data is strongly learned and strongly activated. This is exactly what we observe: the majority group during training attains the highest activation in the final layer, indicating that the network holds filters that closely match the patterns of that group. This direct link between learned filters and activation also motivates the parameter-level analysis of Level 4.

Finally, this signal can be used to detect bias. Bias, however, is intrinsically relative: it describes a relationship between two groups and has no absolute value, so an acceptable level of inequality must be fixed before a model is declared biased—a threshold we make explicit when we turn the activation gap into a detector.

5.2.2 Severity sweep (colored MNIST)↩︎

Figure 9 (left) reports the group-average final-layer activation by digit color for the red-biased family across the four severities and for the unbiased model (the other two colors are given in the appendix and are equivalent). The gap between the majority color and the two minority colors is largest at severity I and shrinks monotonically through severities II and III to severity IV, and vanishes for the unbiased model, where all three colors activate equally. Figure 9 (right) shows the same effect at the level of distributions: under severity I the majority color’s activations are clearly displaced upward, whereas under the unbiased model the three colors’ distributions coincide.

5.2.3 Detection at scale (colored MNIST)↩︎

For detection we use the Activation Bias Indicator, the Mann–Whitney \(U\) comparison of two groups’ final-layer activations by its rank-biserial correlation (Eq. 10 ), and flag a model biased when its largest pairwise \(|\Lambda|\) exceeds a threshold \(\rho\). We use this effect size rather than the test’s p-value because it is bounded, comparable across domains, and directly measures the magnitude of the activation gap. As at Level 2, \(\rho\) trades mild-bias recall against the unbiased false-positive rate, and the full sweep of \(\rho\) is given in the appendix: strongly and moderately biased models (severities I–III) are detected at \(90\%\) or above across the useful range, while severity-IV recall and the unbiased false-positive rate decline together as \(\rho\) grows. Setting \(\rho\) to the same \(5\%\) false-positive budget gives \(\rho=0.32\), at which the detector reaches \(99\), \(98\), and \(90\%\) recall for severities I-III and \(30\%\) for severity IV (Table 4).

At a matched false positive rate, the two training-free detectors are not equivalent: the latent-space test (Level 2) recovers far more of the mild severity IV bias than the activation test (Level 3)—\(86\%\) versus \(30\%\) at \(5\%\) false positives—indicating that, for weak bias, the structure of the embedding space is a more sensitive signal than the magnitude of the final-layer activations.

A final remark concerns the unbiased false positives. Because these detectors measure a property of the learned representation rather than of the training data, the reported false-positive rate is an upper bound on the true error rate: balanced training data does not guarantee a balanced representation. A balanced-data model may still develop a mildly asymmetric internal representation; therefore, some flagged unbiased models may be genuinely, if weakly, biased rather than detector errors, which is why we hold the budget at a conservative \(5\%\) rather than at the more permissive thresholds that would capture additional severity IV models.

12pt

5.3 Detecting Bias in Parameters of the Convolution Filters (Level 4)↩︎

At this level, a secondary network \(\psi\) is trained to classify the bias of a task model only using its convolutional filters, without access to inputs or outputs. The detector therefore requires a population of task models for its own training and evaluation.

We devised various bias detection networks \(\psi\) that maintain the overall architecture whilst modifying the Convolutional Block (Figure 11). The variants outlined here are the outcome of an exploration across a large space of possibilities, and they represent the most informative and noteworthy. The Convolutional Block variants analyzed are:

• MLP: Flatten \(\rightarrow\) Dense(\(r\))

• \(1\times1\) +conv: Conv2D (\(1\times1\)) \(\rightarrow\) MaxPooling2D (\(d\times d\)) \(\rightarrow\) Conv1D (1) \(\rightarrow\) MaxPooling1D (\(r\)) \(\rightarrow\) Flatten.

• \(1\times1\) +max: Conv2D (\(1\times1\)) \(\rightarrow\) MaxPooling3D (\(d\times d\times k\)) \(\rightarrow\) Flatten.

• \(d\times d\) +conv: Conv2D (\(d\times d\)) \(\rightarrow\) Conv1D (1) \(\rightarrow\) MaxPooling1D (\(k\)) \(\rightarrow\) Flatten.

• \(d\times d\) +max: Conv2D (\(d\times d\)) \(\rightarrow\) MaxPooling1D (\(r\)) \(\rightarrow\) Flatten.

• \(1\times1\times1\) +max: Conv3D (\(1\times1\times1\)) \(\rightarrow\) MaxPooling3D (\(d\times d \times c\)) \(\rightarrow\) Flatten.

• \(d\times d \times c\): Conv3D (\(d\times d \times c\)) \(\rightarrow\) Flatten.

where \(d\times d\) represents the dimension of the input filter weights, \(c\) denotes the number of input channels, and \(r\) refers to the number of input filters. Convolutions are followed by a ReLU activation function, and there is always \(0.1\) dropout afterwards (we have seen that it works best among the values: 0.0, 0.1, 0.2 and 0.3).

5.3.0.1 Face biometrics

In tiny face models, the weight detector recovers the type of demographic bias (three-way: Asian, African/Indian, or Caucasian preferred) from the filters alone, reaching up to \(90\%\) accuracy, with the architectures that convolve with a kernel the size of the input weights performing best (d\(\times\)d-conv \(87\%\), d \(\times\) d-max \(90\%\); Figure 15 and the ablation in the appendix). This establishes that the demographic bias is encoded in the learned parameters.

5.3.0.2 Colored MNIST

In colored MNIST, the detector is trained on the four-way task of identifying the bias condition—red-biased, green-biased, blue-biased, or unbiased (chance = 25%)—with one detector trained per severity, using the same two architectures that performed best on faces. Table ¿tbl:tab:l495mnist95sweep? reports the per-class accuracy. For severities I–III, both architectures are near-perfect on all four classes. At the mildest severity,  IV, the three biased colors are still recovered well (\(93\)–\(96\%\)), but the unbiased class is the one that suffers: its accuracy drops to \(78\%\) (d\(\times\)d-max) and \(71\%\) (d \(\times\) d-conv) because some mildly biased models are now confused with unbiased ones. This is the weight-level counterpart of the mild-bias confusion seen at Levels 2–3, and carries the same reading: at severity IV the learned parameters of a biased model are only weakly distinguishable from those of a balanced-data model.

A cross-severity transfer experiment makes the structure of this signature explicit. A detector trained to separate mildly biased (severity IV) models from unbiased ones transfers upward to the strongly biased regime: evaluated on severity-I biased and unbiased models it reaches \(97\%\) accuracy. The reverse completely fails: a detector trained in severity I becomes completely blind to mild bias when tested on severity IV, classifying every model as unbiased (100% detection of unbiased models, 0% for biased models). The weight signature of strong bias is thus an amplified version of that of mild bias: a detector tuned to the subtle signature also recognizes the obvious one, whereas a detector tuned to the obvious signature collapses to predicting “unbiased” once the signature is faint. In practice, this argues for training a weight-based detector on the mildest bias one intends to catch.

This result demonstrates the existence of identifiable patterns associated with bias within the weights of a neural network. The fact that a neural network \(\psi\) can accurately classify the weights of another neural network \(\phi\) based on the type of bias implies that there is a discernible pattern encoded in the weights of network \(\phi\) related to bias.

6 Conclusions↩︎

The ability to detect and quantify biases in neural networks has significant implications for the field of computer vision and AI ethics. It enables developers to create fairer and more inclusive AI systems by identifying and mitigating biases that could lead to discriminatory outcomes. This study presents a robust methodology for the detection and quantification of biases in Convolutional Neural Networks (CNNs). The methodology is then applied to the context of facial biometrics in the gender classification task and Colored-MNIST digits. It is based on the examination of embeddings, layer activations, and convolutional filters, meaning that it focuses on the internal workings of neural networks rather than just their output.

Traditional methods for bias detection, like dataset audits and performance evaluation, while useful, have limitations in terms of scope and data requirements; they often rely on performance metrics across groups, requiring extensive labeled datasets. In contrast, our approach focuses on the internal representations and learned parameters of neural networks, offering two advantages: i) a more granular insight into where and how biases manifest within the network, enabling targeted interventions, and ii) the ability to detect biases with fewer data samples, making it more efficient and practical, particularly in scenarios where large balanced datasets are not available.

First, our exploration of the learned latent space prior to the classification layer provides deeper insights into how biases affect the distribution of representations. We demonstrate that biased models exhibit a poor distribution of underrepresented groups within the latent space, which directly impacts classification accuracy.

Second, examining activations of the network layers further illustrates how bias manifests within the model. We observe that groups underrepresented in the training data consistently exhibit lower activation levels in the final convolutional layers compared to majority groups.

Finally, our analysis of convolutional filters reveals that CNNs encode demographic biases within their filters. This encoding is apparent from the high accuracy of our bias detection model, which can classify biases looking at the convolutional filters alone. Although this method offers independence from the input data, it requires training thousands of models and is less transferable to other architectures or tasks.

The methodologies of activations and latent space are applicable to various neural network architectures and can be extended to different types of bias. Building on our detection methodology, which facilitates the understanding and debugging of biases, it is possible to devise strategies to mitigate identified biases. These strategies could potentially be incorporated into the training loss function and the evaluation phases of neural network development.

Across the three levels, bias leaves a consistent and detectable trace in the internals of the network: a less uniformly distributed latent space (Level 2), lower activation of the underrepresented groups in the final convolutional layers (Level 3), and a recoverable signature in the convolutional filters themselves (Level 4). The colored-MNIST severity sweep ties the levels together: at every level, the disparity and with it the detector’s discriminative power, contracts smoothly as the training distribution moves from skewed toward balanced, vanishing in the unbiased models.

Table 5 summarizes key qualitative characteristics of traditional methods and those proposed here. Our methods complement existing approaches by filling gaps in bias detection where traditional methods may fall short. Although our methods are not universally superior, they contribute to the field by providing alternative tools for bias detection, particularly in scenarios with limited data or where deeper insights into network behavior are required. The table highlights these comparative strengths and weaknesses, situating our work within the broader landscape of bias detection methods.

7 Limitations and Future Work↩︎

Several limitations should be noted. (1) The analysis focuses on convolutional architectures; extending the framework to transformer-based architectures is left for future work. (2) The proposed indicators depend on methodological choices (neighborhood size N, significance level, and detection thresholds); we characterize their sensitivity (Section 5.3 and Appendix B), but a single setting need not be optimal for every model or task. (3) Level-2 and Level-3 signals are statistical properties of internal representations and should be read as indicators correlated with, rather than proof of, discriminatory output behavior. (4) The Level-4 weight-based detector requires training a population of task models, which is computationally demanding; we mitigate this with small, inexpensive models and cross-severity transfer, but the cost still limits applicability to very large models.

In future work, we will exploit recent advances in AI interpretability to evaluate biases [42] and adapt the bias analysis-detection methodologies developed here to LLMs [43] and multimodal setups combining language and vision models [44].

8 Model Architectures↩︎

This section details the architectures of the three convolutional neural networks (CNNs) utilized in this study: a small CNN for MNIST digit classification, a lightweight CNN for Gender Classification, and a custom Residual Network (ResNet) for robust Gender Classification.

Colored-MNIST Classification Model↩︎

The MNIST model employs a standard Sequential CNN architecture. Notably, while standard MNIST images are grayscale, this model is configured to accept three-channel inputs (\(28 \times 28 \times 3\)), an adaptation for RGB-formatted digit images. The network sequentially applies convolutional layers with increasing filter counts to extract hierarchical features, followed by aggressive spatial downsampling via max pooling. A dropout layer is introduced before the final classification head to mitigate overfitting. It contains a total of 29,162 parameters.

Gender Classification Face Biometrics (Tiny)↩︎

The “Tiny” gender classification model is a lightweight, strictly sequential CNN designed for processing \(120 \times 120 \times 3\) facial images. It features a deeper architecture than the MNIST model, using five convolutional blocks. The initial layer uses a larger \(5 \times 5\) receptive field to capture broad spatial features, while subsequent layers rely on standard \(3 \times 3\) kernels to extract finer biometric details. The spatial dimensions are heavily compressed through five consecutive max-pooling operations prior to classification. It comprises 106,242 parameters.

Gender Classification Face Biometrics (ResNet)↩︎

The ResNet-based gender classification model is a truncated custom variant of the standard Residual Network architecture, utilizing “bottleneck” blocks. It contains 371,682 total parameters (366,626 trainable) and operates on \(120 \times 120 \times 3\) input images. The network relies heavily on Batch Normalization and skip connections to optimize gradient flow and allow deeper feature extraction without degradation.

The architecture is divided into three distinct macroscopic stages:

• Stem: The network begins with a \(7 \times 7\) Convolution (32 filters) with zero padding, followed by Batch Normalization, ReLU activation, and MaxPooling2D operation. This reduces the initial spatial resolution to \(30 \times 30\).

• Residual Stage 1: Contains three bottleneck blocks that operate at \(30 \times 30\) spatial resolution.

  • The first block is a convolutional block with a \(1 \times 1\) skip connection (expanding the channel depth from 32 to 128).

  • The subsequent two are identity blocks. Each bottleneck uses a progression: \(1 \times 1\) (32 filters) \(\rightarrow\) \(3 \times 3\) (32 filters) \(\rightarrow\) \(1 \times 1\) (128 filters).

• Residual Stage 2: Contains four bottleneck blocks.

  • The first is a convolutional block for downsampling that reduces the spatial dimensions to \(15 \times 15\) and expands the channel dimension to 256.

  • The remaining three are identity blocks. The bottleneck structure here follows a progression: \(1 \times 1\) (64 filters) \(\rightarrow\) \(3 \times 3\) (64 filters) \(\rightarrow\) \(1 \times 1\) (256 filters) progression.

• Classification Head: A Global Average Pooling2D layer maps the \(15 \times 15 \times 256\) tensor to a flat 256-dimensional vector, feeding into a final Dense layer of 2 units (softmax) for binary classification.

Summary of ResNet Macro-Architecture:

8.1 Implementation Details and Reproducibility↩︎

8.1.0.1 Training configuration

All task models (ResNet face, tiny face, and colored-MNIST) were trained from scratch with Glorot-uniform initialization using the Adam optimizer (learning rate 0.01, [schedule, e.g., constant / reduce-on-plateau, factor [..], patience [..]]), batch size 32, for 10-40 epochs (tiny face and colored-MNIST - ResNet face), minimizing the categorical cross-entropy loss.

8.1.0.2 Preprocessing

Faces were detected and aligned using RetinaFace[45], resized to 120×120×3, and normalized; no augmentation was used. Colored-MNIST images were converted to RGB at 28×28×3 by coloring the background (red, green, or blue) while leaving the digit strokes white, with pixel values normalized.

9 Robustness to Evaluation Sample Size (Face Biometrics)↩︎

The training-free detectors are computed from a finite set of evaluation images, so it is worth asking how much data they require. On the five ResNet models we recompute the Level-2 and Level-3 signals from random subsets of the test set—\(100\) and \(1{,}000\) images per ethnicity (\(50\) and \(500\) per gender)—and repeat each setting over \(100\) independent random draws to obtain the distribution of outcomes.

9.0.0.1 Latent space (Level 2)

Figure 12 shows the Level-2 analysis on the \(100\)-image draws (mean \(\pm\) standard deviation over the draws). At this size the Kolmogorov–Smirnov \(p\)-value has no power: for every model, including the biased ones, the \(p\)-values of all three pairwise comparisons remain above the \(0.01\) significance line at every neighborhood size, so the test does not reject. The \(KS\) statistic still orders the models, but the unbiased baseline increases from \(\sim\!0.02\) on the full test set to \(\sim\!0.05\)–\(0.08\), eroding the margin on which the threshold at-scale (\(\Gamma=0.06\)) relies. Increasing the subset to \(1{,}000\) images contracts the spread and restores the separation (Figure 13).

9.0.0.2 Activations (Level 3)

Figures 14 (a) and (b) show the per-group average final-layer activation \(\lambda^{[l]}\) for the two sizes of subsets. The group means are essentially unchanged between them – the favored group is the highest in every biased model, and the three groups coincide for the unbiased and unbiased (+) models—while the standard deviation across draws contracts by roughly \(\sqrt{10}\) from the \(100\)- to the \(1{,}000\)-image subsets, the \(1/\sqrt{n}\) concentration expected of a sample mean. The same contraction holds for the \(KS\) statistic, so neither signal is intrinsically more stable than the other; both simply sharpen with more data.

9.0.0.3 Automatic detection

Table 9 reports, for each subset size, the fraction of the \(100\) draws in which each model is flagged biased by the Activation Bias Indicator at the \(\Lambda \geq 0.2\) and \(\Lambda \geq 0.1\) operating points. With \(100\) images the more strongly biased models are detected in a majority of draws (\(60\)–\(80\%\)) but the weakly biased Asian model is not (\(24\%\)), and the two unbiased models flag at a similar low rate (\(10\)–\(14\%\)). With \(1{,}000\) images, every biased model is detected (\(83\)–\(100\%\)), and the contrast between the two unbiased models emerges: the standard unbiased model is flagged in \(52\%\) of draws while the unbiased (+) model is flagged in only \(2\%\). Because the estimate is tight at this size, this is not sampling noise but evidence that the standard unbiased model retains a small, real representational asymmetry that the additional balanced data of the unbiased (+) model removes.

12pt

Both signals concentrate with sample size, as expected for finite-sample statistics: the spread of the Level-2 \(KS\) statistic contracts with the evaluation set size and that of the Level-3 activation \(\lambda\) sharpens with it. At \(100\) images per group, the disparity is already visible—the favored group’s average activation is reliably the highest in every biased model—but the automatic tests are not yet dependable. The \(KS\) \(p\)-value lacks the power to reject at this size, and the \(\Lambda \geq 0.1\) rule flags the strongly biased African/Indian model in \(87\%\) of draws, but the weakly biased Asian model in only \(24\%\), against \(10\)–\(14\%\) for the two unbiased models—too close to separate reliably. By \(1{,}000\) images the detectors are reliable and the Asian model is recovered in \(77\%\) of draws, the Afr./Indian and Caucasian models 100%.

At \(1{,}000\) images, the activation detector also resolves a subtler effect. The standard unbiased model is now flagged in \(52\%\) of draws, whereas the unbiased (+) model – trained on three times as much balanced data – is flagged in only \(5\%\). At this sample size, the estimate is tight, so the \(52\%\) is not sampling noise: the standard unbiased model carries a small but real representational asymmetry despite its demographically balanced training data, and the additional balanced data of the unbiased (+) model remove it. Balance in the data therefore does not guarantee balance in the representation – the representational counterpart of bias encoded in the learned parameters (§5.3)—and this is the concrete justification for the conservative false-positive budget adopted in the at-scale detection experiments.

10 Bias Detection Network Ablation↩︎

Figure 16 shows the classification accuracy of various bias detection networks \(\psi\) as a function of the number of samples used for training.

The best performance is obtained with architectures that implement a convolution with the kernel of the size of the input weights (\(d\times d\)-conv and \(d\times d\)-max), reaching \(87\%\) and \(90\%\) classification accuracy, respectively. The improvement in the architecture with the most parameters (\(d\times d\times c\), with 5.7 million) stalls at \(3\)K training samples and above.

Performance scales consistently with the number of training samples across most architectures, including the simple MLP. However, some architectures, such as \(1 \times 1 \times 1\)-max and \(d \times d \times c\), do not appear to benefit significantly from more training samples. Nevertheless, it should be noted that these architectures perform comparably to others when trained with limited data.

The hypothesis that seems to best explain this behavior is that, since there are so many parameters \(\mathbf{\Theta}\), the solution space is so large that the choice of a better architectural configuration occurs automatically, leaving unnecessary parameters unchanged as if they were not present [46]. With little training data, the model \(\psi(\cdot | \mathbf{\Theta})\) adjusts very quickly to those data (losses are practically nil) and in just a couple of epochs it no longer needs to adjust those weights \(\mathbf{\Theta}\). However, when the number of training samples increases, more parameters are needed to be modified to better correlate the training data, thus losing the generalization capability.

We have dealt with many more architectures that are not worth describing here: using two dense layers at the end, adding a dense layer after each convolution, replacing convolutions with dense layers, playing with dropout, etc.; all resulting in worse performance than the learning architectures reported here.

References↩︎

---

1. https://artificialintelligenceact.eu/↩︎

2. We refer to activation as the output of the activation function after each convolution.↩︎