1 Introduction↩︎

As the field of artificial intelligence (AI) expands, systems built with machine learning (ML) models increasingly assist decision-makers in high-risk settings, including healthcare [1], criminal justice [2]–[5], hiring [6]–[8], welfare eligibility [9]–[11], immigration decisions [12], and credit [13]–[15]. In many of these settings, ML models serve as decision support to a human decision-maker, commonly described as AI-assisted decision-making. While AI assistance can be desirable for human decision-makers [16], relying on ML model predictions introduces several risks. One core issue, and the focus of this article, is that ML model predictions are uncertain. In machine learning, uncertainty may be aleatoric, arising from intrinsic randomness in the data-generating process, or epistemic, arising from limits in the model’s knowledge due to design choices and finite data [17]–[20]. Even when ML models achieve similar aggregate predictive performance, uncertainty is often unevenly distributed across individuals and groups [16], [21]–[24]. If decision-makers treat certain and uncertain predictions alike, it risks decision-making that disproportionately harms particular individuals and groups.

Predictive uncertainty has been discussed in the literature on bias mitigation [25]–[27], but most bias detection and mitigation approaches do not align cleanly with the legal requirements from the United Kingdom (UK) [28]–[30]. Selective abstention is no exception. When applied, model predictions are withheld from the human decision-maker if the uncertainty exceeds a given threshold [23], [31]–[34]. Selective abstention rests on the assumption that unaided human judgment is preferable to AI-assisted judgments in uncertain instances. As we show, this assumption is frequently unwarranted and, more importantly, the intervention itself can generate legally cognisable harms. We adopt a legally-informed approach to analyse the risk of applying selective abstention as an uncertainty-based algorithmic intervention in AI-assisted decision-making.

To complement this analysis, we discuss an alternative uncertainty-based algorithmic intervention: selective friction. Under this approach, the ML model’s prediction is always shown to the human decision-maker, but uncertain predictions are accompanied by a yellow flag indicating uncertainty. In our article, we call the selective abstention and selective friction approaches: selective interventions. We investigate them in the context of unlawful discrimination and restrictions on solely automated decision-making under United Kingdom (UK) law. First, we consider the risk of discrimination where predictive uncertainty is unequally distributed across particular groups. Second, we identify the risk that channelling some decisions into unaided human judgment, while leaving others AI-assisted, will exacerbate unfairness by reintroducing cognitive biases that fall disproportionately on under-represented groups. Neither uncritical reliance on the model nor inattentive human review is sufficient.

Meaningful human involvement is treated as a primary safeguard for AI-assisted decision-making in legal and policy frameworks across the UK [35]–[38] and internationally [39]–[41]. A growing literature questions its reliability, particularly where decision-makers lack the training, time, or contextual information needed to evaluate AI outputs critically [42]–[45]. Where human involvement fails to perform that protective function, the legal frameworks that depend on it may themselves be inadequate. This article asks three questions:

1. How are demographic groups differentially impacted by selective interventions in AI-assisted decision making?

2. Under what conditions may unlawful discrimination arise from selective interventions?

3. To what extent can unlawful discrimination be mitigated by selective interventions?

Our article makes two principal contributions:

1. We provide the first doctrinal analysis of selective interventions under UK law. Through two AI-assisted case studies, consumer credit and risk of reoffending, we demonstrate that selective abstention carries substantial legal risks under UK non-discrimination and data protection law.

2. We argue that selective friction is legally preferable to selective abstention as a means of structuring human involvement in AI-assisted decisions, conditional on implementation and empirical evidence. We identify the conditions under which friction is likely to improve decision quality.

The argument proceeds as follows. Section 2 formally defines the two selective interventions, states our relevant assumptions, including how we use entropy to measure uncertainty, and provides background to human-AI collaboration. Section 3 applies these interventions to two case studies and examines the consequences for affected groups across four decision scenarios. Section 4 presents the legal analysis, where we assess both interventions under the relevant UK laws. Section 5 concludes.

2 Problem Setting and Selective Interventions↩︎

In this section, we provide formal definitions and assumptions for our problem setting and two uncertainty-based algorithmic interventions, and provide an overview of relevant literature on human-AI decision-making.

2.1 Preliminaries↩︎

We consider binary classification tasks. We assume access to a dataset, \(D = \{(x_1, y_1), \ldots, (x_n, y_n)\}\), of instances \(x_i \in X\) and true labels \(y_i \in \{0,1\}\). Our predictive ML model is represented by the function \(f: X \longrightarrow \hat{Y}\), where \(\hat{Y} = \{0, 1\}\). The prediction of the classifier for a given \(x\) with true label \(y\) is \(\hat{y} = f(x)\). To represent a human decision-maker who reviews and works with the ML model, we use the function \(r: X \longrightarrow \{0, 1\}\). The prediction of the (unaided) human decision-maker for a given \(x\) is \(\overline{y} = r(x)\). When the human decision-maker observes an ML model prediction, we denote their aided decision as \(\tilde{y} = r(x; f(x))\) for a given input.

A key part of our work is considering how human decision-makers could react when notified that a model’s prediction is uncertain. To this end, we assume that the model \(f\) thresholds an underlying probabilistic learner, as this allows for the model to express degrees of belief, and hence uncertainty, in outcomes. We have \(f(x) = \mathop{\mathrm{arg\,max}}_j \hat{p}(y=j|x)\) where \(\hat{p}(y=j|x)\) is the learnt probability that \(x\) is classified with label \(y=j\). In the binary case, \(\hat{p}(y=0) = 1-\hat{p}(y=1)\) so we write \(\hat{p}(y=0) = \hat{p}\) without loss of information. Unless specified, we use \(\hat{p}\) in this way in the remainder of this article. For convenience, we drop the conditioning on \(x\) from our notation. We note that \(\hat{p}\) is only an approximation of the true distribution \(p\); some implications of the latter are discussed below. The uncertainty of the prediction is quantified using Shannon entropy [46], a common measure for probabilistic classification uncertainty [19], [47]. Shannon entropy (measured in bits) is defined as: \(H(\hat{y}) = H(\hat{p}) - \sum_{j} \hat{p}(y=j)\log_{2}\hat{p}(y=j)\). A plot of \(H(\hat{p})\) on the y-axis against \(\hat{p}\) shows a concave curve, with entropy values in the range 0 to 1. When the probability \(\hat{p}\) of prediction \(\hat{y}\) is certain (\(\hat{p}=0\) or \(\hat{p} = 1\)) the entropy is the minimum \(H(\hat{y}) = 0\) due to the absence of doubt or uncertainty. In addition, the maximum entropy or uncertainty is \(H(\hat{y}) = 1\), when the probability of a given prediction is \(\hat{p} = 0.5\). This curve is visualised in Figure 1.

2.1.0.1 Calibration Assumption and Limitations

A model is well calibrated for class \(k\) over data \(X\) if, for the data \(x\in X\) where the model predicts class \(k\) with probability \(\hat{p}\), then \(k\) is the true label \(100\times\hat{p}\%\) of the time. The predicted probabilities are meaningful for class \(k\) because they faithfully align with the actual outcomes. We assume the probabilistic learner \(\hat{p}\) is well calibrated for both binary classes over the entire population \(X\), but not for subgroups of interest \(X_{D}\) which represent particular demographic groups. The latter concept is referred to as within-group calibration [48]–[51] and forms the calibration fairness notion of this work.¹ We note models are rarely assessed for calibration across the population, let alone within-group calibration. Common probabilistic classifiers such as neural networks can be accurate while poorly calibrated [53], and even where population-level calibration holds, within-group calibration is known to fail in safety-critical domains [54]. Where this occurs, unfaithful probability estimates can produce different intervention rates across demographic groups.

2.1.0.2 Operationalising Uncertainty Thresholds

To operationalise selective interventions, we must set an uncertainty threshold above which the system intervenes, either by abstaining and withholding the prediction, or by applying friction and flagging the prediction as uncertain. These interventions are discussed further below. Let us assume that the decision-maker chooses an entropy boundary \(\tau\), above which predictions are treated as sufficiently uncertain to trigger intervention. Thus, model predictions with entropy values such that \(\tau \leq H(\hat{y})\) will result in model abstention or in a yellow flag for selective friction. These thresholds should be carefully chosen (e.g., by using an AUROC curve) and be domain and use case specific; describing best practices for choosing them is outside of the scope of the article. We call \(\tau \leq H(\hat{y})\) the intervention range, which, if true in our article, triggers a selective intervention. We note that \(\tau\) has a corresponding prediction probability region \([\hat{p}_{\tau-\mathrm{low}},\hat{p}_{\tau-\mathrm{high}}]\) where \(H(\hat{p}_{\tau-\mathrm{low}})=H(\hat{p}_{\tau-\mathrm{high}})=\tau\). A prediction with \(\hat{p}\) in this region would trigger an intervention. The relation is visualised in Figure 1.

2.2 Selective Abstention↩︎

A direct intervention strategy is algorithmic abstention, also known as selective classification, learning to defer, refusal mechanism, or algorithmic resignation [33], [55]–[61]. The primary motivation is to safeguard against erroneous predictions in high-uncertainty cases and to defer such cases to human judgment [32], [56], [62]–[65]. While intuitive to defer to humans to manage uncertainty, it is far from a neutral fix. Selective classification can magnify accuracy disparities across groups, and may even reduce accuracy for groups that already perform worst on the full input distribution [21], because predictive uncertainty is concentrated in under-represented groups [21], [27], [34], [57], [66]. Where uncertainty is concentrated in under-represented groups, abstention can amplify existing disparities [57], [63].

For our model of selective abstention, if the entropy falls within the intervention range, \(\tau \leq H(\hat{y})\), selective abstention is triggered and the decision-maker does not see the model prediction. The decision-maker must then make an unaided decision, \(\overline{y} = r(x)\). If the model is more certain about the prediction, so \(H(\hat{y}) < \tau\) the decision-maker receives the prediction and selective abstention is not triggered. The intervention creates a two-track decision process: some individuals receive AI-assisted decisions, while others, those whose predictions happen to be uncertain, receive unaided human decisions. If the uncertainty distribution correlates with protected characteristics, this two-track structure produces differential treatment that is relevant to the discrimination analysis (Section 4).

2.3 Selective Friction↩︎

Friction-based approaches introduce deliberate pauses in AI-assisted decision-making, such as delayed predictions, additional access steps, or disclosures imposing cognitive load [67]–[71]. Related to behavioural nudges [72]–[74] and microboundaries [75], [76], they create moments of reflection without removing human autonomy. Uncertainty representation is a design decision [77], [78]. Displaying uncertainty information is not on its own sufficient: people struggle to interpret it [79]–[81], sometimes leading to suboptimal decisions [82], [83].

We define our selective friction model such that it communicates uncertainty with a yellow flag if the entropy value falls within the intervention range for a prediction. We assume that the decision-maker is trained on how to use the system, including why uncertainty matters and how the system will provide a yellow flag if the ML model is uncertain about a prediction. In the case of selective friction, the decision-maker always sees the prediction. Their final decision, denoted \(\tilde{y}\), is aided by both the model prediction and a yellow flag noting higher uncertainty. Selective friction preserves human autonomy while introducing a moment of reflection. We assume the selective friction is not adversarially manipulated: the communicated uncertainty estimates are well-calibrated and not altered to misguide humans. Selective friction effectiveness depends on how decision-makers interpret and weigh: the friction, the data, and the model prediction to make their decision. By including a yellow flag, we aim to encourage informed scrutiny of the data and reduce the risk of over-reliance on ML models.

2.4 The Benefits and Limitations of Human-AI Decision-Making↩︎

Human involvement is a widely adopted safeguard across sectors when deploying ML models, but its effects are mixed. This human safeguard is intended to address ML limitations such as bias and ambiguity [16], [84], and to bring principles like fairness and reflexivity into the pipeline that ML models alone cannot satisfy [43], [63]. In turn, ML models can enhance human skills and autonomy [70], [85]; the case for collaboration is strongest in high-stakes domains [25], [61], [84], [86]. However, the empirical record is less reassuring. A persistent concern is inappropriate reliance on ML models [86]–[88]: humans frequently over-rely on model recommendations even when incorrect [70], [82], [86], [89]–[93]. Paradoxically, when the model provides explanations, the human’s acceptance of the model increases regardless of correctness [43], [82], [93]–[96]. Under-reliance also produces errors of omission and humans struggle to calibrate trust appropriately [43], [84]. Appropriate calibration of trust is difficult because humans bring their own cognitive biases [97], [98] and have difficulty evaluating a model’s accuracy and their own [43], [86], [99]–[102]. Section 4 examines the legal consequences of these patterns.

3 Applying Selective Interventions to Case Studies↩︎

We apply the two interventions to two case studies in which ML models are commonly deployed for decision-making: consumer credit (private sector, financial services) and risk of reoffending (public sector, criminal justice). Selective interventions are triggered when \(\tau \leq H(\hat{y})\). For each case study, we examine the consequences for affected groups across four decision scenarios under selective friction (Table 1). In both case studies, there is no contemporaneous ground-truth against which the prediction can be evaluated. The model is asked to predict an event that has not yet occurred, and may never occur regardless of the decision taken. The uncertainty signalled by a flag is the model’s self-reported uncertainty about its own prediction. For abstention, the decision-maker tasked with the unaided judgment has no better access to the future outcome than the model. For friction, warnings convey information about the model’s epistemic state, not the prediction’s match with reality.

Further, we do not argue that automated decisions are preferable to human ones, or that delay for human review is inherently undesirable. Human involvement is treated, both legally and institutionally, as a safeguard against the limits of algorithmic prediction. Our question is what design of the human-AI interface is most likely to deliver that safeguard, and whether the alternative designs satisfy the legal requirements that the safeguard exists to instantiate.

3.1 Case Study 1: AI-Assisted Consumer Credit Predictions↩︎

Consumer credit has long been characterised by inefficiencies and inequalities [13], [103]. The shift toward AI-assisted credit decisions, driven by ML advances and new data sources [13], [103]–[105], has the capacity to address some limitations of conventional scoring while magnifying others. Historical discriminatory lending practices can be perpetuated through biased training data [13], [106], [107], particularly where alternative or non-traditional data is used [103], [108], [109], with disproportionate effects on under-represented borrowers [33], [110], [111].

A subtle but significant issue is the potential uneven distribution of predictive uncertainty across demographic groups in credit risk assessments [112]. Historical discrimination and under-representation in lending data, for instance, may lead to noisier or incomplete credit histories for certain loan applicants, resulting in higher levels of predictive uncertainty [110], [111]. Model miscalibration for specific demographic groups can compound this effect, producing systematic errors in entropy values that do not reflect genuine uncertainty but rather a correctable modelling failure (see Section 2.1.0.1). When stakeholders set entropy thresholds to trigger selective interventions, certain groups are likely to be disproportionately flagged as ‘uncertain’, experiencing either more frequent abstentions or more burdensome secondary reviews [112]. Whether this disproportionate flagging constitutes unlawful discrimination is the central question addressed in Section 4.

We consider a hypothetical ML model that outputs binary predictions where the loan applicant is predicted \(\hat{y}(x)=1\) (low risk of default) or \(\hat{y}(x)=0\) (high risk of default). We assume that applicants predicted as high risk are denied a loan, while those predicted as low risk are granted one.

3.1.1 Selective Abstention↩︎

Selective abstention, while aiming to mitigate errors from uncertain predictions, creates a two-track decision process that has uneven distributional consequences. Consider the error structure. If a loan applicant is predicted as low risk but is actually high risk and ultimately defaults (false negative), granting the loan could cause significant financial harm [109], [113], [114]. Conversely, a low-risk applicant mistakenly labelled as high-risk (false positive) is unfairly denied access to credit. Error rates often vary across demographic groups, and groups that disproportionately receive uncertain predictions may experience higher rates of both false positives and false negatives.

The critical issue with abstention is not merely that errors occur, but that certain groups are systematically channelled into unaided human decision-making while others receive the consistency and speed of AI-assisted decisions. Historical or representation bias in credit data means that applicants from under-represented or historically marginalised groups are more likely to receive uncertain predictions [110]–[112]. These applicants are diverted into manual review more and encounter longer processing times, and are exposed to the variability of individual loan officers. This creates a concrete procedural disadvantage: two applicants with materially similar creditworthiness may receive different decision processes purely because one belongs to a group for which the model has less training data. As we argue in Section 4, this procedural asymmetry is a cognisable detriment under the Equality Act 2010. Abstention also discards information: in marginal cases near the entropy threshold, the model’s uncertain prediction can still anchor a more accurate decision.

3.1.2 Selective Friction↩︎

3.1.2.1 Scenarios 1 and 2: Loan denied.

Scenarios 1 and 2 produce adverse outcomes for the applicant and are the cases most likely to raise discrimination concerns. In Scenario 1, the human lender agrees with the ML model’s uncertain prediction that the applicant is a high credit risk. The lender may be over-reliant on the model despite the yellow flag, failing to engage critically with the application. The model’s uncertain prediction could itself result from negative bias towards the applicant, particularly if they are from an under-represented group not well represented in the training data [8], [115], [116]. If the lender defers to the model uncritically, the yellow flag has failed to serve its intended function and the algorithmic bias passes through to the final decision.

Scenario 2 is the most troubling from a discrimination perspective. Here the model predicts a positive outcome (low risk) with uncertainty, but the human lender overrides to deny the loan. The lender’s independent judgment results in a negative outcome that could be informed by information not considered by the model, but could equally be informed by personal prejudice or an unwarranted aversion to uncertain ML predictions. Friction can produce worse outcomes than the unaided model for members of protected groups, where the override correlates with protected characteristics. This is the central risk of the design. If such overrides correlate with protected characteristics, selective friction actively harms the groups it is intended to protect. Empirical evidence on lender overrides shows that the majority are either no better than random or systematically favour already advantaged groups [117]. This finding should give pause to anyone who assumes that human involvement is inherently corrective.

3.1.2.2 Scenarios 3 and 4: Loan granted.

Scenarios 3 and 4 result in positive outcomes for the applicant, but the quality of the decision-making process varies. In Scenario 3, the lender agrees with the model’s uncertain positive prediction. This could reflect sound judgment, using additional contextual information to confirm the model’s assessment, or it could reflect uncritical acceptance of the model despite the yellow flag. The outcome is initially favourable, but it is contingent: a misclassified applicant who later defaults will suffer delayed harm, and the bank will absorb financial loss, turning the initial benefit into a long-term burden for both parties [29], [109], [113].

Scenario 4 arises when the lender overrides an uncertain negative prediction to grant the loan. In principle, this is the override most likely to benefit under-represented applicants: where uncertainty stems from data sparsity or historical bias against the applicant’s subgroup, the lender can supply contextual knowledge the model lacks. On the available evidence, however, the case for Scenario 4 overrides as a corrective mechanism is weak.  [117] find that the majority of lender overrides are no better than random or systematically favour already-advantaged groups. A small subset of decision-makers can recognise data-sparsity cases and outperform the model on accuracy and equity, but identifying those decision-makers ex ante is itself unsolved. The default expectation should therefore be that Scenario 4 overrides do not systematically benefit under-represented groups, and the design of the intervention alone cannot alter that expectation.

3.2 Case Study 2: AI-Assisted Risk of Reoffending Predictions↩︎

Our second case study is drawn from the public sector: AI-assisted risk of reoffending predictions in the criminal justice system. This case study differs from consumer credit in two respects. First, the decision-maker is executing a public function, which engages the UK’s public sector equality duty (PSED) under section 149 of the Equality Act 2010, an obligation that does not apply to private credit institutions. Second, the consequences of an adverse decision in terms of sentencing is qualitatively different from a credit denial. In criminal justice contexts, the unaided human decision is the constitutional and procedural baseline, and any algorithmic intervention should be justified against that baseline. The case study does not assume that algorithmic assistance is preferable to unaided human decision-making. It asks how, where such assistance is used, selective interventions affect the lawfulness of the resulting decisions and the protection of affected groups. These differences allow us to test whether the risks of selective interventions generalise to a setting with different institutional incentives and regulatory settings.

Once an individual has received a criminal justice sanction, the police and the justice system have an interest in understanding the likelihood that they will reoffend. Measuring this risk is difficult and sensitive. Data used may include criminal history, mental health records, and employment status [118]. For a UK-based example system, West Yorkshire Police use Corvus IOM Case [119], [120], a procured system that produces a likelihood-of-reoffending score mapped to low, medium, and high gradings [121]. The force’s policy treats the score as decision support rather than as a decision, and acknowledges gendered patterns in offending behaviour, including women’s disproportionate experience of gender-based violence as a driver of offending. This suggests that models trained on aggregate data may misestimate risk for women where features do not capture gendered vulnerability and coercion.

The COMPAS literature in the United States offers a partial parallel, though doctrinal transfer to the UK is limited. The Correctional Offender Management Profiling for Alternative Sanctions (COMPAS) tool, used by judges and parole officers across the US, outputs a risk score between 1 and 10 [122]. COMPAS satisfies between-group calibration for gender and ethnicity [123], though scholars have questioned whether calibration is the morally appropriate fairness measure, preferring equality of false positive and false negative rates [52]. Studies have shown that the algorithm overpredicts the actual risk of reoffending for Black [122] and Hispanic [124] individuals and underpredicts the risk for White individuals [122]. While the tool distinguishes some recidivism across gender, it still “systematically overclassif[ies] women in high risk groupings” [125].

The Wisconsin Supreme Court in State v Loomis upheld the use of COMPAS at sentencing; the tool could be used provided it was not the sole basis for a decision and that judges exercised independent judgment [126]. However, the court required five written warnings to accompany any COMPAS report, including that the tool’s proprietary nature prevents disclosure of how risk scores are calculated and that some studies have raised questions about disproportionate classification of minority offenders as high risk [126], [127]. Loomis is instructive for two reasons. First, it mirrors the logic of selective friction: the judge retains the prediction but is warned about its limitations. Second, the court’s warnings are unlikely to create meaningful judicial scepticism because it ignores judges’ limited ability to evaluate risk assessment tools, and fails to account for the anchoring effects that bias judges toward the score they have been shown [127]. In practice, risk of reoffending is typically a regression or multi-class classification problem. For simplicity, we scope it to binary classification, where the individual is either predicted \(\hat{y}(x) = 1\) (low reoffending risk) or \(\hat{y}(x) = 0\) (high reoffending risk). We assume that individuals predicted as high risk are given a strict sentence from a judge, and those predicted as low risk are given a light sentence.

3.2.1 Selective Abstention↩︎

Similarly, selective abstention diverts uncertain cases to unaided human decision-making, and the distributional consequences of this diversion are uneven. Certain groups, particularly women and people from racialised communities, may disproportionately trigger abstentions due to higher levels of predictive uncertainty stemming from under-representation and systemic issues in training data [3], [128], [129]. The consequence is that these individuals are more likely to have their sentence decisions made by the judge without any model input. Whether this helps or harms the individual depends on the quality of the judge’s unaided judgment, which is itself an empirical question.

The harms of an adverse decision in this context are substantial and qualitatively different from a credit denial. An individual given strict sentencing may experience heightened surveillance and supervision after leaving jail. For individuals who did not commit violent offences, and particularly for those from minority groups, strict sentences may feel disproportionate and may reinforce patterns of over-policing that are well documented in the UK criminal justice literature [130]. The inconsistency introduced by selective abstention is particularly concerning in this context because it compounds existing concerns about inconsistent treatment across demographic groups.

3.2.2 Selective Friction↩︎

3.2.2.1 Scenarios 1 and 2: Strict Sentencing.

In Scenario 1, the judge agrees with the uncertain ML model’s prediction that the individual is high risk, and the individual is given a strict sentence. The judge may be over-relying on the model despite the yellow flag, particularly under time pressure or where the judge lacks training in interpreting uncertainty information. The model’s uncertain prediction may itself reflect bias: if the training data under-represents the individual’s demographic group, the prediction is less reliable, and uncritical acceptance by the judge transmits that unreliability into the final decision. In Scenario 2, the model predicts low risk with uncertainty, but the judge overrides to give the individual a strict sentence. As with consumer credit, this is the scenario of greatest concern. Judges may override based on information the model does not capture, or they may override based on stereotypes or personal biases [87], [97], [98]. The risk is heightened in the criminal justice context, where well-documented biases in sentencing may predispose judges to assess individuals from certain groups as higher risk than the evidence warrants [130]. Selective friction could have the perverse effect of empowering biased human overrides in cases where the model, despite its uncertainty, would have produced a more favourable outcome.

3.2.2.2 Scenarios 3 and 4: Light sentencing.

Scenarios 3 and 4 result in the individual being given a light sentence by a judge. In Scenario 3, the judge agrees with the model’s uncertain positive prediction, which may reflect appropriate contextual judgment or uncritical acceptance. In Scenario 4, the judge overrides an uncertain negative prediction to deselect the individual. The judge may have important contextual information about rehabilitation progress, family circumstances, or the nature of the original offence, and in some cases this will produce a better decision than the model alone. The same caveat applies as above: evidence suggests that overrides do not systematically benefit under-represented groups [117]. Documented patterns of disparity in sentencing decisions mean that the same cognitive and institutional factors that produce biased Scenario 2 overrides are also present in Scenario 4. Friction is intended to prompt critical scrutiny rather than deferential acceptance; its effectiveness depends on training, institutional culture, and the flag base rates. Otherwise, the warning will lose its behavioural force.

4 Legal Implications of Selective Interventions↩︎

This section provides a legal analysis of both interventions under UK non-discrimination and data protection law.

4.1 Unlawful Discrimination↩︎

Discriminatory ML models have been identified in a variety of contexts; however, many analyses do not connect algorithmic unfairness with unlawful discrimination under the Equality Act [6], [29], [30], [131]. Unlawful discrimination is defined by three elements: (1) protected context, (2) protected characteristic, and (3) prohibited conduct [132]. Under the Equality Act, relevant protected contexts include the “provision of a service to the public”, encompassing financial services [132], and the exercise of public functions, which covers policing and criminal justice decisions [132]. Protected characteristics are age, disability, gender reassignment, marriage and civil partnership, pregnancy and maternity, race, religion or belief, sex, and sexual orientation [132]. However, marriage and civil partnership, and age for individuals under 18, are not protected in the context of service provision [132]. Discrimination in these protected contexts based on the listed characteristics is prohibited under two conduct categories: direct discrimination and indirect discrimination. We analyse each in turn, applying them to both selective interventions as defined in Section 2 and examined in the case studies in Section 3.

4.1.1 Direct Discrimination↩︎

Direct discrimination occurs when an individual, because of a protected characteristic(s) or an associated proxy, is treated less favourably than a real or hypothetical comparator [132], [133], [134]. Direct discrimination admits no justification defence except for narrow statutory derogations [132]. Intent and justifications are irrelevant. As Lady Hale stated, “however justifiable it might have been, however benign the motives of the people involved, the law admits of no defence” [135], [136]. On the better view, neither intervention amounts to direct discrimination, though the question remains open [131]. The more plausible route to liability is indirect discrimination.

4.1.1.1 Selective Abstention.

Although the entropy threshold that triggers selective abstention is formally neutral, its effects may fall unevenly across protected groups because predictive uncertainty is shaped by historically biased or uneven training data. That does not, however, make the threshold directly discriminatory. The better view is that entropy is not a sufficiently close proxy for a protected characteristic to satisfy the requirement of treatment because of that characteristic. UK law has generally required a tight connection between the proxy criterion and the protected characteristic, where it has an “exact correspondence” such as pregnancy being unique to the female sex [135], [137]. That connection is unlikely to exist here, because entropy reflects multiple interacting factors, including data quality, feature conflict, and model fit, rather than group membership as such.  [131] have argued that some algorithmic outputs may amount to direct discrimination where the model’s reliance on protected-characteristic proxies is sufficiently tight. On their account, the question is not whether the proxy is uniquely correlated with the protected characteristic, but whether the algorithm treats the characteristic as a “but for” cause of the outcome. We accept this analysis in principle but it is unlikely to extend to entropy. Entropy is a property of the model’s epistemic state, not a feature of the input, and its correlation with protected characteristics is mediated through data quality, sample size, and feature availability. The connection is too attenuated to satisfy “because of” disparate treatment. The stronger claim is therefore not direct discrimination, but indirect discrimination.

4.1.1.2 Selective Friction.

The same analysis applies to selective friction. Any correlation between yellow flags and protected characteristics is better understood as reflecting the unequal distribution of predictive uncertainty across groups than as treatment because of a protected characteristic. Without a closer proxy relationship, selective friction is therefore unlikely to amount to direct discrimination.

4.1.2 Indirect Discrimination↩︎

Where a facially neutral provision, criterion, or practice (PCP) disproportionately disadvantages individuals with a protected attribute, it is unlawful indirect discrimination [132]. Indirect discrimination “aims to achieve equality of results in the absence of such justification” [138]. Such PCP is discriminatory if it applies to persons with the protected characteristic or someone who still suffers the same disadvantage, and puts, or would put, them at a particular disadvantage when compared to those without such attributes [132].

4.1.2.1 Selective Abstention.

Under the Equality Act, the analysis proceeds in three stages [132], [138]. First, the claimant must identify the PCP. The abstention rule, triggering the withholding of predictions when \(\tau \leq H(\hat{y})\), constitutes a PCP. It is a criterion applied uniformly to all predictions regardless of the individual’s characteristics. Second, the PCP must be shown to put the claimant at a particular disadvantage. The threshold \(\tau \leq H(\hat{y})\) and the entropy value \(H(\hat{y})\) are facially neutral: neither is defined by reference to protected characteristics, although empirical work demonstrates a correlation between high model uncertainty and under-represented demographic groups [16], [21], [23]. In consumer credit predictions, historical or representation bias in credit data leads to applicants from under-represented or historically marginalised groups receiving more uncertain model predictions; they are therefore diverted into manual review more often, encounter longer processing times, and are exposed to the variability of individual loan officers rather than more standardised model output [110]–[112]. An analogous pattern arises in risk of reoffending predictions, where under-representation of women and racialised communities in training data likely produces higher entropy for those groups’ model predictions [122], [124], [125]. The procedural asymmetry of being diverted into a less consistent decision process satisfies the requirement of particular disadvantage [138], [139]. Where the disparity is attributable to correctable calibration failure rather than irreducible error (Section 2.1.0.1), the case is strengthened: the impact is an artefact of a modelling choice that could be corrected.

Third, unlike direct discrimination, indirect discrimination may be justified if the PCP is a proportionate means of achieving a legitimate aim [132]. The question is whether the measure is appropriate and reasonably necessary to achieve that aim, taking into account whether less discriminatory alternatives are available [140], [141]. The aim must be identified and shown to be legitimate. A plausible aim here is the reduction of erroneous adverse decisions arising from high-uncertainty predictions. The measure must then be appropriate to the aim, meaning rationally connected to achieving it. Selective abstention satisfies this requirement only if diverting uncertain cases to unaided human judgment in fact produces fewer erroneous decisions overall, which the available evidence on overrides does not support without qualification. The measure must be reasonably necessary. Less discriminatory alternatives must be considered, including recalibration, group-conditional thresholds, and selective friction. Each is less restrictive than abstention in at least one respect: none diverts cases into a separate decision track, and none discards the model’s prediction. A decision-maker may respond that these alternatives are too slow or detrimental to the aim. That argument is open, but it is for the controller to make out on the evidence; the availability of these alternatives places a substantial evidential burden to show that abstention is necessary rather than merely beneficial.

4.1.2.2 Selective Friction.

Selective friction similarly constitutes a PCP under the Equality Act framework. The central question is whether it places individuals with protected characteristics at a particular disadvantage compared to those without. Unlike selective abstention, which defers decision-making entirely, selective friction transparently communicates uncertainty with a yellow flag, allowing decision-makers to weigh the model’s uncertainty alongside other factors. However, transparency alone may not reduce bias; it can worsen disparities if decision-makers misinterpret yellow flags, particularly for individuals from protected groups. At the second stage, the analysis is more finely balanced than for abstention. The yellow flag is applied to predictions within the intervention range, and, as with abstention, this range correlates with under-represented demographic groups. However, the nature of the disadvantage is different. Under selective friction, the individual is not diverted into a different decision process. They receive the same AI-assisted process as all other individuals, but with additional information. The potential harm arises when decision-makers, acting under time pressure or without sufficient training, misinterpret uncertainty warnings as definitive indicators of unreliability rather than as contextual cues to be weighed judiciously [43], [67], [80], [84]. Whether selective friction systematically worsens outcomes for protected groups relative to no intervention must be established on the evidence of each deployment.

If selective friction is shown to put protected groups at a particular disadvantage, the justification analysis proceeds differently from abstention. The legitimate aim is the same, and friction has a stronger claim to proportionality: it preserves the decision-maker’s access to the prediction while prompting additional care, avoiding the information loss that abstention creates. Friction is less restrictive than abstention in that it preserves the prediction and avoids the two-track structure. Whether it is proportionate in its own right depends on whether it produces particular disadvantage in deployment, which is an evidential question. This does not mean selective friction is immune from challenge. If evidence in a specific deployment demonstrates that yellow flags systematically trigger biased overrides on credit or sentencing decisions, friction in that deployment may fail the proportionality test. The controller would then need to consider further alternatives: more informative friction (e.g., contextual explanations), decision-maker training, or mandatory documentation of override reasons. The lawfulness of selective friction is not settled in the abstract. The success and lawfulness of selective friction therefore depends on the design of both the ML model and the human-AI decision-making system in which it operates.

4.2 The Public Sector Equality Duty↩︎

UK public authorities are subject to the public sector equality duty (PSED) under section 149 of the Equality Act 2010, which requires due regard to the need to eliminate discrimination and advance equality of opportunity [132]. This is relevant to Case Study 2, where the decision-maker is exercising a public function. In R (Bridges) v Chief Constable of South Wales Police [142], the Court of Appeal held that the South Wales Police breached the PSED by failing to take reasonable steps to enquire whether their automated facial recognition software contained bias on racial or sex grounds. The court emphasised that the purpose of the duty is “to ensure that a public authority does not inadvertently overlook information which it should take into account” [142]. Proof of actual bias was not required; the failure to investigate was itself a breach. The duty applies at the procurement stage, not only at deployment. A public sector body adopting a tool without interrogating the supplier on within-group calibration is unlikely to satisfy the due-regard standard. The duty is proactive and ongoing: it requires investigation before deployment and continued monitoring thereafter. For private sector actors deploying ML models to predict credit risk, no equivalent proactive duty exists; the legal framework depends more heavily on the indirect discrimination provisions discussed above. Although outside the scope of this analysis, public bodies are also subject to additional administrative law constraints: to take all relevant considerations into account, the prohibition on unlawful fettering of discretion, the requirement of procedural fairness, and, in some contexts, the duty to give reasons [143], [144].

4.3 Solely Automated Decision-Making↩︎

Additionally, data protection law imposes requirements that impact the design of selective interventions.

4.3.1 The DUAA Framework↩︎

The GDPR prohibits solely automated decision-making that produce legal or similarly significant effects on individuals unless one of three exceptions applied [35]. However, the UK GDPR has recently been amended by the Data (Use and Access) Act 2025 (DUAA), under which the framework has fundamentally changed [145]. Solely automated decisions are now permitted by default, subject to procedural safeguards rather than a general prohibition [146]. A decision is “based solely on automated processing” where there is “no meaningful human involvement in the taking of the decision” [146]. The Secretary of State is conferred powers to define the circumstances that constitute meaningful human involvement [146]. The DUAA’s reforms restrict the prohibition on solely automated decisions to those based entirely or partly on special categories of personal data [146]. Automated decisions based on non-special-category personal data, including most consumer credit predictions and risk of reoffending predictions, are now permitted by default, subject to safeguards under Article 22C to: (a) provide information about such decisions; enable (b) representations; (c) human intervention; and (d) contesting [146]. The DUAA framework has been criticised for introducing ambiguity and creating incentives for tokenistic human involvement [145], [147]–[149].

Article 15 of the UK GDPR was also amended: the right to “meaningful information about the logic involved” now applies only to decisions falling within Article 22C [146]. The Court of Justice of the European Union recently clarified that Article 15 requires provision of information that is sufficiently complete and contextualised to enable verification of the accuracy of the decision and identify an objectively verifiable consistency and causal link between the method used and the result [150]. While the decision settled considerable academic debate about the right to explanation [151]–[153], it may have diminished significance in the UK.

4.3.2 Selective Interventions and Meaningful Human Involvement↩︎

The DUAA’s threshold for triggering safeguards is the absence of “meaningful human involvement”. It raises the significant question: does selective friction satisfy this threshold, such that the decision is treated as non-automated and the Article 22C safeguards are not triggered? On its face, selective friction appears to involve meaningful human involvement. The human sees the prediction, is warned about its uncertainty, and exercises judgment. If this is sufficient, the decision falls outside the scope of Article 22A, and the safeguards in Article 22C do not apply. However, there are doubts about whether friction-assisted review constitutes meaningful involvement in practice. Decades of research on automation bias demonstrate that human decision-makers frequently defer to algorithmic recommendations without genuine independent evaluation [43], [90], [154], [155]. Persistent agreement with the model despite the flag is consistent with rubber-stamping. Whether this should remove the decision from the scope of Article 22A is unsettled, and likely turns on government guidance. The formal presence of human involvement may be sufficient to avoid the DUAA’s safeguards despite functionally solely automated decision-making.

This creates a tension. The DUAA framework incentivises firms to include a human in the process, because doing so may remove the decision from the scope of Article 22A and the associated safeguards. Selective friction is well suited to this: it places the human in the process and provides them with information, which is precisely what “meaningful human involvement” might plausibly require. However, if the human’s involvement is in practice perfunctory, the friction operates as a compliance device rather than a real safeguard.

Under selective abstention, the analysis is different. For predictions that fall outside the intervention range, the decision-maker receives the prediction without any warning. If the decision-maker routinely approves these predictions without independent evaluation, those decisions may in substance be solely automated. Selective abstention does not resolve this problem; it redirects it. The uncertain predictions are diverted to unaided human judgment (which may satisfy meaningful involvement), but the certain predictions may pass through with only rubber-stamp approval. The DUAA’s vagueness on what constitutes meaningful involvement means that neither intervention provides a clear answer. Until the circumstances of meaningful involvement are defined or interpreted, the practical scope of the safeguards remains uncertain. Selective friction is better designed to facilitate genuinely informed human judgment than selective abstention, because it provides the human with both the prediction and information about its reliability. Whether this design intention translates into meaningful involvement is variable (see Section 3). To reduce tokenistic compliance, legally significant uses should require active, recorded human review. Logs should capture the model’s entropy, whether an intervention was triggered, and whether and why the human agreed with the prediction. Requiring reasons for (dis)agreement is likely the strongest safeguard against rubber‑stamping, though which forms of documentation improve decision quality needs further study.

4.3.3 Connecting the Frameworks↩︎

The data protection and discrimination frameworks interact in two intriguing ways. First, the DUAA’s narrowing of the safeguards for solely automated decisions increases the importance of the Equality Act as a source of accountability. Where the DUAA’s safeguards are not triggered (because meaningful human involvement is present, or because the data is not special-category data), the Equality Act’s prohibition on indirect discrimination and the PSED remain available as alternative legal bases for challenging discriminatory outcomes. The proportionality analysis under indirect discrimination, which considers whether less discriminatory alternatives exist, applies regardless of whether the DUAA’s safeguards are engaged. Second, by incentivising firms to include a human in the loop, the DUAA encourages the very design pattern (human-AI collaboration) that, as Section 3 demonstrates, can reintroduce human biases. A firm that inserts a reviewer to avoid Article 22A’s safeguards may simultaneously increase the risk of indirect discrimination, particularly through Scenario 2 overrides (Table 1). The two legal frameworks pull in different directions: UK GDPR rewards human involvement as a compliance mechanism, while the Equality Act requires that involvement does not produce disproportionately adverse outcomes for protected groups. Selective interventions do not dilute the safeguards under the UK GDPR; however fewer decisions will be triggered, increasing the need for selective interventions to operate with transparency and accountability.

5 Conclusion↩︎

Selective abstention has attracted attention to manage uncertainty in ML systems and improve decision-making. Our analysis shows, however, that abstention does not remove the risk of discriminatory decision-making. Uncertainty-based interventions can still impose disproportionate burdens on protected groups and may therefore give rise to unlawful discrimination. Through the AI-assisted case studies, consumer credit and risk of reoffending, we compare two selective interventions as alternative responses to uncertainty. Our claim is not that any intervention based on uncertainty is necessarily lawful, but that its legal and ethical implications depend on how uncertainty is operationalised and presented to decision-makers. We argue that selective friction is legally preferable to selective abstention. By preserving access to the model’s prediction while signalling uncertainty, selective friction better supports meaningful human judgment, and is more likely, though not certain, to satisfy relevant legal requirements. Whether it also leads to better decisions in practice is an empirical question, and depends on the design of the friction, adequate calibration of the underlying uncertainty estimate, and role of human decision-makers. To our knowledge, this is the first doctrinal analysis of uncertainty-based interventions under UK law. Our analysis also identifies two priorities for future work: empirical study of whether uncertainty flags produce different override rates across demographic groups; and technical analysis of whether demographic-group calibration or other methods reduce the disproportionate impact of selective interventions.

References↩︎