1 Introduction↩︎

State-of-the-art large language models (LLMs) silently “correct” African American English (AAE). Given the AAE context “But I ain’t doing no dishes,”, five out of six LLMs we tested prefer the Standard American English (SAE) continuation “I’m going to stay in my room” over the valid AAE alternative “Imma stay in my room” (Fig. 1).

We call this behavior dialect preference bias: the systematic tendency of LLMs to favor SAE over valid varieties such as AAE, even when the user is writing in AAE. Treating AAE as something to be corrected is not a neutral stylistic choice but a form of linguistic discrimination: AAE is a rule-governed variety with its own consistent grammar and phonology, spoken by over 30 million people [1]. The downstream consequences are well-documented: LLMs disproportionately misclassify AAE text [2], respond to AAE speakers with more stereotyping and condescension [3], and degrade on tasks with non-standard dialect inputs [1], [4]. As LLMs move into hiring, healthcare, and content moderation, this bias actively disadvantages millions of users.

3.5pt

Existing work on AAE dialect bias has three key limitations (Table 1). First, most benchmarks generate AAE synthetically from SAE via rule-based or LLM-driven translation [4]–[7], producing AAE that underestimates real-world dialect effects [1], [7]. Second, real-AAE studies are narrow in scope: they evaluate either classification consistency [2], generative responses [3], or reward-model scoring [8], but none combine discriminative and generative auditing, identify which linguistic features trigger bias, or offer mitigation. Third, the few existing mitigation methods require retraining [4], architectural changes [9], [10], or auxiliary translation pipelines that erase the dialect altogether [11].

We address these limitations as follows. Starting from a corpus of tweets by native AAE speakers [12], we construct Real-AAE, a corpus of 17,479 AAE/SAE/AAE_back triplets via translation and back-translation, validated both automatically and by three native AAE speakers. This AAE-first construction preserves dialect features that synthetic AAE misses.

We then audit six instruction-tuned LLMs (14B to 70B) along two axes: discriminative consistency, via our new conditional Dialect Group Invariance (cDGI) metric, which isolates model bias from translator artifacts; and generative preference, via perplexity, log-probability, and forced-choice continuation. A feature-level localization analysis further pinpoints which AAE markers trigger bias.

Finally, we introduce, to our knowledge, the first application of activation steering to dialect bias: a training-free method that uses causal tracing to identify bias-relevant layers and injects a learned dialect direction at inference. Our analysis yields three main findings: (i) all six models prefer SAE continuations even under AAE context, defaulting to “correcting” the dialect; (ii) syntactic constructions are universal bias triggers every tested models; and (iii) Steering reduces bias 5–20\(\times\) more than prompting, while preserving SAE fluency.

1.0.0.1 Contributions.

• We release Real-AAE, the largest real-AAE parallel corpus to date: 17,479 AAE/SAE/AAE_back triplets from natural tweets, 2 to 6 times larger than prior real-AAE resources, validated both automatically and by three native AAE speakers.

• We introduce an audit framework combining cDGI, which controls for translator-induced drift, with generative-preference metrics, and a feature-level localization analysis that identifies the AAE markers most strongly triggering bias.

• We propose the first activation-steering method for dialect debiasing: training-free, test-time, and applicable across model scales. Across six SOTA LLMs (14B to 70B), it mitigates dialect bias 5 to 20 times more effectively than prompting while preserving SAE fluency.

2 Related Work↩︎

Dialect Bias in NLP: Dialect bias has been documented across natural language processing tasks, including language identification [12], hate-speech and toxicity detection [13]–[15], automatic speech recognition [16], text generation [17], and downstream decision-making about employability, criminality, and personality [18]. Most notably, [18] show that dialect prejudice persists in instruction-tuned LLMs even after overt racial biases are reduced, motivating dialect-specific audits of frontier models.

AAE Bias Benchmarks: Resources for evaluating AAE bias divide along a methodological axis. Synthetic benchmarks transform SAE into AAE via rule-based or LLM-driven translation [4]–[7], an approach shown to underestimate real-world dialect effects [1], [7]. Real-AAE resources instead draw text directly from AAE speakers [2], [3], [8], [12], but each targets a single evaluation mode: classification consistency [2], generative responses [3], or reward-model scoring [8]. The closest prior work, [2], introduces Dialect Group Invariance (DGI) on a real-AAE corpus for sentiment classification. We extend this along multiple dimensions: Real-AAE, a corpus 2–6\(\times\) larger; additional discriminative and generative metrics; feature-level localization of bias triggers; and a test-time mitigation (Table 1).

Dialect Bias Mitigation: Prior mitigation approaches fall into three categories. Training-time methods augment training data with dialectal examples [4], but can degrade SAE accuracy. Architectural methods add dialect-specific modules, such as the LoRDD adapter [9] or the encoder-based DialectGen [10]. Inference-time methods translate AAE inputs to SAE before passing them to the LLM [11], which avoids retraining but introduces auxiliary models and erases the dialect rather than mitigating bias against it. Our approach instead intervenes directly on the target model’s activations at inference, requiring no retraining, architectural changes, auxiliary modules, or dialect erasure.

Activation Steering: Our method builds on activation steering, a mechanistic-interpretability technique that controls model behavior by adding learned directions to hidden states at inference. The method has been applied to general behavioral control [19]–[21], refusal behavior [22], and persona traits [23]. [24] extend it to social bias mitigation, but not to dialect bias. To our knowledge, ours is the first application of activation steering to dialect debiasing.

3 Benchmark Construction↩︎

To audit AAE dialect bias without conflating it with translator artifacts, we construct Real-AAE, an AAE-first parallel corpus of (AAE, SAE, \(\text{AAE}_{\text{back}}\)) triplets grounded in naturally-occurring AAE tweets (Fig. 2). Unlike synthetic benchmarks that derive AAE from SAE, our pipeline starts from real AAE and translates outward, preserving dialectal features that synthetic generation may miss.

3.0.0.1 Source data and filtering:

We build on the TwitterAAE corpus [25], which identifies tweets from African American accounts via demographic inference from geolocation and network analysis. To ensure sufficient content for sentiment analysis and continuation modeling, we retain tweets with at least 10 words and strip non-ASCII characters, user mentions, and URLs, yielding 17,495 tweets.

3.0.0.2 Translation and back-translation:

We translate each AAE tweet into SAE using Gemini-3-flash, then back-translate the SAE into AAE using the same model (denoted \(\text{AAE}_{\text{back}}\)). The back-translation serves two purposes: (i) it provides a content-preservation check, since AAE and \(\text{AAE}_{\text{back}}\) should remain semantically aligned if translation did not introduce drift; and (ii) it enables our cDGI metric (§4.1), which conditions on translation-stable examples to isolate model bias from translator artifacts. We manually inspect sample translations and define filtering criteria to remove pairs with added explanations, semantic distortions, or translation artifacts, yielding our final 17,479 triplets.

3.0.0.3 Automatic validation:

For content preservation, we compute BERTScore F1 between AAE and \(\text{AAE}_{\text{back}}\) using Twitter-RoBERTa,² chosen to match the social-media domain of our inputs. Both mean and median F1 are \(\mathbf{0.95}\) (standard deviation \(0.02\)), indicating strong semantic alignment. For sentiment preservation, we compare AAE/SAE label agreement using the same Twitter-RoBERTa model; agreement is highest for positive and negative cases, with most disagreements concentrated in neutral examples (full results in Appendix 8).

3.0.0.4 Human validation:

We further validate a 1,000-example subset with three native AAE speakers from the West Coast and Northeast/Mid-Atlantic, all of whom self-reported strong familiarity with written AAE. Annotators judged each example along four dimensions: sentiment preservation, semantic equivalence, original AAE naturalness, and \(\text{AAE}_{\text{back}}\) naturalness. Three-way exact agreement was 93.0% (sentiment preservation), 83.0% (semantic equivalence; \(\kappa = 0.468\)), 81.0% (original AAE naturalness; \(\kappa = 0.488\)), and 77.5% (\(\text{AAE}_{\text{back}}\) naturalness; \(\kappa = 0.204\)). The sentiment \(\kappa\) was \(-0.024\), reflecting label imbalance rather than disagreement: 98% of judgments were yes, inflating expected-chance agreement.

4 Methods↩︎

We develop methods to audit, localize, and mitigate dialect preference bias. §4.1 defines our evaluation metrics, combining a new conditional invariance measure (cDGI) with model-likelihood-based generative-preference metrics. §4.2 introduces a feature-level localization analysis that pinpoints which AAE markers most strongly trigger bias. §4.3 presents our mitigation method, dialect activation steering.

4.1 Bias Evaluation Metrics↩︎

We evaluate dialect bias along two complementary axes: discriminative consistency, via the existing DGI metric [2] and our new cDGI metric; and generative preference, via perplexity, a log-probability bias score, and forced-choice continuation computed from model likelihoods.

4.1.1 Conditional Dialect Group Invariance (cDGI)↩︎

DGI [2] measures AAE–SAE classification agreement, but it cannot separate the model’s dialect bias from translator artifacts: an apparent inconsistency may reflect semantic drift in the LLM-produced SAE translation or in \(\text{AAE}_{\text{back}}\). We therefore introduce conditional DGI (\(\mathrm{cDGI}\)), restricted to examples satisfying two conditions: (i) the predicted label is preserved under back-translation, controlling for translator semantic drift; and (ii) \(\text{AAE}_{\text{back}}\) was judged natural by our native-AAE annotators, preventing \(\text{AAE}_{\text{back}}\) drift toward SAE from inflating the conditioning set. Letting \(y^{\mathrm{AAE}}, y^{\mathrm{SAE}}, y^{\mathrm{AAE_{back}}}\) denote model predictions on the AAE, SAE, and back-translated AAE inputs respectively, \(\mathrm{cDGI}\) is defined as \[\mathrm{cDGI} = \Pr\!\left( y^{\mathrm{AAE}} = y^{\mathrm{SAE}} \;\middle|\; y^{\mathrm{AAE}} = y^{\mathrm{AAE_{back}}} \right),\] computed over examples meeting condition (ii). \(\mathrm{cDGI}\) thus isolates the model’s dialect sensitivity from both translator noise and dialect erasure in the conditioning set.

4.1.2 Perplexity Analysis↩︎

To measure generative preference, we compute conditional perplexity (\(\mathrm{PPL}\)) for all four context\(\rightarrow\)continuation combinations: SAE\(\rightarrow\)SAE, SAE\(\rightarrow\)AAE, AAE\(\rightarrow\)SAE, and AAE\(\rightarrow\)AAE. Lower \(\mathrm{PPL}\) indicates the model finds a continuation more natural given the context. A dialect-invariant model should prefer dialect-matched continuations: lower \(\mathrm{PPL}\) for SAE\(\rightarrow\)SAE than SAE\(\rightarrow\)AAE, and lower for AAE\(\rightarrow\)AAE than AAE\(\rightarrow\)SAE.

Our continuation-based experiments (§4.1.2, §4.1.3, and §4.2.1) all require splitting each AAE–SAE pair into a context and a continuation; we describe the alignment-based segmentation procedure in Appendix 9.1.

4.1.3 Multiple-Choice Evaluation↩︎

We assess explicit preference by presenting each model with a context (in either SAE or AAE) and two candidate continuations (one SAE, one AAE), asking which better continues the context. Option order is randomized to control for positional bias. A dialect-invariant model should prefer the continuation matching the context dialect. The prompt template is in Appendix 9.2.

4.2 Feature-Level Bias Localization↩︎

4.2.1 Context-specific log-probability bias↩︎

For each of the \(N\) AAE–SAE pairs, let \(x_i^{\mathrm{SAE}}, x_i^{\mathrm{AAE}}\) and \(y_i^{\mathrm{SAE}}, y_i^{\mathrm{AAE}}\) denote the matched SAE/AAE contexts and continuations, and let \(P(y \mid x)\) denote the model’s likelihood of continuation \(y\) given context \(x\). We define context-specific SAE and AAE log-probability (\(\mathrm{LP}\)) bias scores as \[\begin{align} b^{\mathrm{SAE}}_i &= \log P(y_i^{\mathrm{SAE}} \mid x_i^{\mathrm{SAE}}) \nonumber\\ &\quad - \log P(y_i^{\mathrm{AAE}} \mid x_i^{\mathrm{SAE}}), \tag{1} \\ b^{\mathrm{AAE}}_i &= \log P(y_i^{\mathrm{AAE}} \mid x_i^{\mathrm{AAE}}) \nonumber\\ &\quad - \log P(y_i^{\mathrm{SAE}} \mid x_i^{\mathrm{AAE}}). \tag{2} \end{align}\] Positive values indicate preference for dialect-matched continuations; negative values indicate preference for mismatched ones. The aggregate \(\mathrm{LP}\) bias is \[\mathrm{LP} = \frac{1}{N}\sum_{i=1}^{N} \big(b^{\mathrm{SAE}}_i - b^{\mathrm{AAE}}_i\big), \label{eq:LP}\tag{3}\] where larger positive values indicate overall preference for SAE across both contexts, and values near zero indicate dialect-matched behavior.

4.2.2 Localizing bias by feature category↩︎

To identify which linguistic markers most strongly trigger dialect preference, we partition AAE sentences by the presence of each feature category and compare mean \(\mathrm{LP}\) (Eq. 3 ) across categories. Using an AAE grammatical lexicon adapted from [26], we group features into four classes: Auxiliary (non-standard auxiliary forms, e.g., “we was”, “finna”), Aspectual (habitual and completive markers, e.g., habitual “be” as in “he be working”), Preverbal (pre-verbal lexical items, e.g., “she done finished”), and Syntactic (clause-level constructions, especially negative concord as in “ain’t nobody”). The full lexicon is in Appendix 9.3.

4.3 Debiasing with Activation Steering↩︎

Our mitigation method rests on a simple insight: if a model treats AAE and SAE inputs differently, then their internal representations must differ in some direction of activation space. Adding a vector along that direction at inference time should shift the model’s behaviour, without retraining. We operationalize this idea in two phases (pictorial overview in Appendix 9.4, Fig. 9). Phase 1 (one-time, offline) extracts a per-layer dialect direction from paired AAE–SAE inputs and uses causal tracing to identify which layers carry the dialect-preference computation. Phase 2 (test-time) injects the dialect direction into those layers, weighted by their causal importance, with no parameter updates. Reproducibility details are in Appendix 9.4.

4.3.0.1 Phase 1a: Extracting the dialect direction.

Given \(N\) paired inputs \(\{(x_i^{\mathrm{AAE}}, x_i^{\mathrm{SAE}})\}_{i=1}^{N}\), let \(h_l(x)\) denote the layer-\(l\) hidden state at the last context token. We compute the per-example contrast \[\Delta_{i,l} = h_l(x_i^{\mathrm{AAE}}) - h_l(x_i^{\mathrm{SAE}}), \label{eq:contrast}\tag{4}\] average across examples, and normalize to obtain a unit-norm dialect direction at each layer \(l\): \[\bar{\Delta}_l = \frac{1}{N}\sum_{i=1}^{N}\Delta_{i,l}, \qquad \hat{v}_l = \frac{\bar{\Delta}_l}{\lVert \bar{\Delta}_l \rVert_2}. \label{eq:direction}\tag{5}\] By construction, \(\hat{v}_l\) points from SAE-conditioned activations toward AAE-conditioned activations at layer \(l\).

4.3.0.2 Phase 1b: Identifying causally important layers.

Not every layer contributes equally to dialect preference. To isolate the layers where the bias is computed, we use a corruption-and-restore causal tracing analysis [27]. For each example, we corrupt the input embeddings at AAE feature-token positions with Gaussian noise: \[e_p^{\mathrm{corr}} = e_p + \epsilon_p, \quad \epsilon_p \sim \mathcal{N}(0, \sigma^2 I),\] where \(\sigma = \alpha \cdot \mathrm{std}(E_{\mathrm{feat}})\), \(E_{\mathrm{feat}}\) is the set of feature-token embeddings in the example, and \(\alpha = 3.0\) is fixed across all models to produce a clear perturbation signal without overly destabilizing the representation. We then run the model on the corrupted input and, at a single site \(z = (l, t)\), restore the clean layer-\(l\) hidden state at token position \(t\). Letting \(g^{\mathrm{corr}}\) denote the fully corrupted run and \(g_z^{\mathrm{rest}}\) the run with restoration at site \(z\), we measure the effect of restoration on the AAE-context preference score (Eq. 2 ): \[R_i(z) = b_i^{\mathrm{AAE}}(g^{\mathrm{corr}}) - b_i^{\mathrm{AAE}}(g_z^{\mathrm{rest}}). \label{eq:restoration}\tag{6}\] Because \(b_i^{\mathrm{AAE}} > 0\) corresponds to the model preferring the AAE-matched continuation, \(R_i(z) < 0\) means restoration at site \(z\) shifts the model toward dialect-matched behavior, while \(R_i(z) > 0\) means restoration amplifies SAE preference. We average over examples and the relevant token positions \(T\) to obtain a per-layer score: \[R_l = \frac{1}{N \cdot \lvert T \rvert} \sum_{i=1}^{N} \sum_{t \in T} R_i\big((l, t)\big), \label{eq:Rl}\tag{7}\] and select the four layers with the most negative \(R_l\) values as the steering set \(K\), i.e., the layers whose restoration most strongly shifts the model toward AAE-matched behavior.

4.3.0.3 Phase 2: Inference-time steering.

At inference, we attach forward hooks at each layer \(l \in K\) and modify the hidden state at every token position \(t\) as \[h_{l,t}' = h_{l,t} + \beta\, w_l\, \hat{v}_l, \qquad l \in K,\; \beta \geq 0, \label{eq:steering}\tag{8}\] where \(\beta\) controls the overall steering strength and \(w_l\) is a normalized per-layer weight proportional to causal importance: \[w_l = \frac{-R_l}{\sum_{j \in K}(-R_j)}, \quad l \in K. \label{eq:weight}\tag{9}\] Since \(\hat{v}_l\) points from SAE-conditioned activations toward AAE-conditioned activations, adding \(\beta w_l \hat{v}_l\) applies stronger steering at layers with larger beneficial restoration effects. Setting \(\beta = 0\) recovers the unmodified model; larger \(\beta\) yields stronger steering. The method modifies activations only on the forward pass and does not update any parameters.

5 Experimental Setup↩︎

5.0.0.1 Models.

We evaluate six instruction-tuned LLMs spanning families and scales (14B to 70B): Gemma-3-27B [28], Mistral-Small-3.1-24B [29], Phi-3-Medium-14B [30], Phi-4-14B [31], DeepSeek-R1-Distill-Qwen-32B [32], and Llama-3.1-70B [33].

5.0.0.2 Dataset.

All evaluation uses Real-AAE (§3). For sentiment classification, models predict positive, negative, or neutral. For continuation-based experiments, we apply the SAE-guided alignment-based segmentation described in Appendix 9.1 to construct context–continuation pairs.

5.0.0.3 Activation steering splits.

For steering experiments, we use three non-overlapping splits of Real-AAE: 1,200 AAE-feature sentences for causal-tracing layer selection (such examples provide clearer corruption-and-restore targets), 7,000 AAE–SAE pairs for dialect-direction extraction, and a held-out 3,000-pair evaluation set covering all four context\(\rightarrow\)continuation combinations (AAE\(\rightarrow\)AAE, AAE\(\rightarrow\)SAE, SAE\(\rightarrow\)AAE, SAE\(\rightarrow\)SAE). We sweep the steering strength \(\beta \in \{0.1, 0.2, 0.4, 0.6, 0.8, 1.0\}\) on the evaluation set to study the trade-off between bias reduction and language modeling performance.

5.0.0.4 Baselines.

We compare activation steering against two baselines: the unsteered base model, and a prompting baseline that explicitly instructs the model to treat AAE and SAE as semantically equivalent (full prompt in Appendix 10.1).

5.0.0.5 Implementation.

Experiments use PyTorch with vLLM [34] for inference and Hugging Face Transformers [35] for steering (which requires direct access to residual activations). All runs on NVIDIA H100 80GB GPUs.

6 Results↩︎

6.1 Discriminative Consistency Bias↩︎

We measure discriminative bias via sentiment classification on Real-AAE. Table 2 reports DGI (AAE–SAE label agreement), two-way DGI (agreement across AAE, SAE, and \(\text{AAE}_{\text{back}}\)), and cDGI (AAE–SAE agreement restricted to translation-stable examples) for each model.

A perfectly dialect-invariant model would score \(\mathrm{DGI} = 1.0\); no model reaches this. Gemma-3 comes closest (\(\mathrm{DGI} = 0.849\)), followed by Phi-4 (\(0.814\)). At the other end, DeepSeek-R1 (\(0.491\)) and Llama-3.1-70B (\(0.468\)) disagree with themselves on more than half of AAE–SAE pairs.

Comparing cDGI to DGI isolates the model’s dialect bias from translator artifacts. Even on translation-stable examples, cDGI remains below \(1.0\) for every model; for Llama-3.1-70B and DeepSeek-R1, more than a third of translation-stable AAE–SAE pairs still receive different sentiment labels. Translation drift therefore accounts for only part of the observed gap: dialect alone changes model predictions, independent of translation artifacts.

2pt

6.2 Generative Preference Bias↩︎

We next ask whether dialect preference appears at the level of language modeling itself, independent of any classification task. We measure this through model likelihoods over candidate AAE and SAE continuations, using two complementary metrics here: perplexity and forced-choice continuation. A log-probability bias score follows in §6.3.

2.0pt

6.2.0.1 Perplexity Analysis.

Fig. 3 reports continuation perplexity for all four context\(\rightarrow\)continuation combinations. Under SAE context, all six models assign lower perplexity to SAE continuations than to AAE: the expected pattern when continuation matches context. Under AAE context, however, every model still assigns lower perplexity to SAE continuations. Models adapt to SAE context but fail to adapt to AAE context, defaulting to SAE regardless of input dialect.

6.2.0.2 Multiple-Choice Evaluation.

Fig. 4 confirms this asymmetry under explicit forced choice. A dialect-invariant model should prefer AAE continuations under AAE context and SAE continuations under SAE context. Instead, most models continue to prefer SAE even when the context is AAE: Phi-4 selects SAE \(59.9\%\) of the time under AAE context (vs.\(40.1\%\) for AAE). Two models deviate. Gemma-3 reverses the pattern, preferring AAE under AAE context (\(63.4\%\)), while DeepSeek-R1 stays near \(50\%\) in both contexts, suggesting weak context-sensitivity rather than active dialect-matching. Table 4 illustrates the typical failure mode: given an AAE context, Mistral-Small-3.1 assigns higher probability to the SAE continuation, effectively “correcting” the dialect.

6.3 Feature-Level Bias Localization↩︎

Having established that dialect preference bias is pervasive, we next ask which AAE features most strongly trigger it. Fig. 5 shows the average \(\mathrm{LP}\) bias for each of the four AAE feature classes across models, where higher positive \(\mathrm{LP}\) indicates stronger SAE preference.

All four feature classes yield positive \(\mathrm{LP}\) across all six models: SAE preference is not tied to any single marker but spans a broad range of AAE forms. Syntactic constructions, especially negative concord (e.g., “ain’t nobody”, “can’t nobody”), yield the highest \(\mathrm{LP}\) for every model tested, making them a universal trigger of dialect preference bias. Preverbal markers also produce relatively strong effects, while auxiliary and aspectual markers show the same direction but smaller magnitudes.

6.4 Bias Mitigation↩︎

We evaluate activation steering as a test-time bias mitigation method against the unsteered model (Base) and a prompt-based baseline (Prompt) instructing the model to treat AAE and SAE equivalently. Table 3 reports DGI, cDGI, PPL, MC, and LP across all six models, with steering strength \(\beta\) selected per model to balance bias reduction and language modeling capability; the full sweep and bias–utility frontier appear in Appendix 10.4.³

Activation steering reduces dialect bias 5 to 20 times more effectively than prompting on LP, while preserving SAE behavior across all metrics. We report results per metric below.

6.4.0.1 DGI and cDGI:

Both metrics improve under steering for every model (e.g., Phi-4: DGI \(0.74 \rightarrow 0.85\), cDGI \(0.77 \rightarrow 0.88\)), indicating more consistent predictions across AAE and SAE inputs. Prompting yields only marginal gains (\(\leq 0.03\) across models). cDGI here is computed on the 368 annotator-judged-natural sentences not used for layer selection or dialect-direction extraction.

6.4.0.2 Perplexity (PPL):

Under steering, AAE perplexity decreases across all models (e.g., Phi-4: \(123.9 \rightarrow 111.4\)), indicating higher probability for AAE-matched continuations, while SAE perplexity remains close to baseline (within \(\sim\)​1 to 5 points; e.g., Phi-4: \(26.7 \rightarrow 28.7\)), preserving SAE fluency.

6.4.0.3 Multiple-choice preference (MC):

Steering boosts AAE Match under AAE context for all models (e.g., Phi-4: \(39.3\% \rightarrow 58.8\%\), \(+19.5\) points), while SAE Match under SAE context stays near baseline (e.g., Phi-4: \(89.1\% \rightarrow 88.3\%\)). The prompt baseline produces much smaller shifts.

6.4.0.4 Log-probability bias (LP):

LP decreases for every model under steering (e.g., Phi-4: \(60.6 \rightarrow 47.3\)), whereas prompting yields negligible changes (\(\leq 2.1\) across models, slightly worse for Llama-3.1-70B at \(-0.1\)). Activation steering achieves roughly 5 to 20 times the bias reduction of prompting on this metric.

Much weaker prompting performance on PPL and LP shows dialect bias is embedded in internal representations, not removable via surface instructions. Three ablations, prompt wording, single- vs.multi-layer steering, and steering strength \(\beta\), confirm robustness (Appendix 10.2, 10.3, 10.4).

7 Conclusion↩︎

We showed that all six state-of-the-art LLMs we tested systematically favor Standard American English over African American English (AAE), even when the user is writing in AAE. Our contributions are: Real-AAE, the largest real-AAE parallel corpus to date; cDGI and a feature-level analysis identifying syntactic constructions as a universal trigger of dialect preference; and the first application of activation steering to dialect bias, which mitigates 5 to 20 times more effectively than prompting without retraining. The framework is dialect-agnostic and can extends to other marginalized varieties. As LLMs increasingly enter high-stakes settings like hiring and healthcare, handling dialect variation fairly is essential for equitable deployment.

Limitations↩︎

7.0.0.1 Source data and register.

Real-AAE draws from the TwitterAAE corpus [25], which identifies AAE-speaker tweets via demographic inference. This provides naturally-occurring AAE at scale but captures the written social-media register, which may not fully generalize to spoken, formal, or other written registers. Extending the framework to additional registers is straightforward and a natural direction for future work.

7.0.0.2 LLM-generated SAE translations.

Our SAE translations are produced by Gemini-3-flash and validated both automatically (BERTScore F1 = 0.95) and by three native AAE speakers. Our cDGI metric is specifically designed to isolate model bias from residual translation drift in the conditioning examples.

7.0.0.3 Annotator regional coverage.

Our human validation involves three native AAE speakers from the West Coast and Northeast/Mid-Atlantic. Because annotators validate corpus quality (translation faithfulness and AAE naturalness) rather than the bias findings themselves, their regional coverage does not affect the LLM-behavior results we report; it does, however, shape the naturalness judgments used to construct the cDGI conditioning set. Expanding the annotator pool to include Southern AAE speakers is an important next step.

Ethical Considerations↩︎

This work aims to reduce the performance disparities and dialect erasure that AAE speakers may experience when interacting with language technologies. By quantifying bias and providing mitigation tools, we hope to support more equitable NLP systems.

The underlying tweets originate from the TwitterAAE corpus [25], which is publicly distributed for non-commercial academic use. Our release shares tweet identifiers along with the SAE translations and AAE back-translations rather than raw tweet text, in line with Twitter’s terms of service.

Three native AAE speakers validated the 1,000-sample subset. Their role was scoped to corpus validation: they were not involved in the design of the LLM audit, the aim of the study, the choice of bias metrics, or the steering experiments. Items were presented in randomized order, and they did not see model outputs, BERTScore values, or any automatic-validation signal during annotation. They judged each item independently along four dimensions: sentiment preservation, semantic equivalence, original AAE naturalness, and \(\text{AAE}_{\text{back}}\) naturalness. There was no discussion among annotators during the task. These three annotators are recognized as co-investigators, both to acknowledge the substantial validation effort they contributed and to keep the study within our institution’s exemption from human-subjects review: research conducted by investigators on their own annotations does not constitute human-subjects research and does not require IRB approval.

Activation steering is intended to mitigate bias in classification and generation, not to enable non-consensual dialect imitation. AAE is not monolithic, and our methods capture statistical patterns rather than the full sociolinguistic diversity of AAE speakers. We encourage practitioners to deploy these tools in consultation with affected communities and to prioritize user consent and transparency.

8 Additional Quality Validation Results↩︎

This appendix provides additional results for the automatic validation described in Section 3. We include the sentiment consistency confusion matrix between AAE and SAE sentences, as well as the distribution of BERTScore F1 between original AAE sentences and their back-translated AAE versions.

6pt

9 Additional Methodological Details↩︎

9.1 SAE-Guided Semantic Segmentation↩︎

Fig. 7 illustrates our alignment-based segmentation procedure. The goal is to split each paired SAE-AAE sentence into matched context and continuation segments.

We first identify the split point in the SAE sentence. Given the SAE sentence “I was going to call you later, but I had to finish my homework first,” we consider several candidate boundaries near the middle of the sentence, such as \(j_1\), \(j_2\), and \(j_3\). For each candidate \(j\), we compute the semantic shift between the left segment \(L_j\) and the right segment \(R_j\): \[d_j = 1-\cos(\phi(L_j),\phi(R_j)).\] We then select the boundary with a large semantic shift while penalizing highly imbalanced splits: \[j^* = \arg\max_j \left[d_j - \gamma \left| \frac{j}{n} - \rho \right| \right] \label{eq:split95obj}\tag{10}\] In Fig. 7, this selects the boundary after “later,” in the SAE sentence.

In Eq. 10 , we set \(\rho\) = 0.5 to encourage roughly balanced context-continuation splits. We use \(\gamma\) = 0.3 as a fixed balance penalty to trade off semantic shift against split length imbalance. This avoids boundaries that are too close to either end of the sentence while still favoring semantically meaningful splits. We then project this SAE boundary to the paired AAE sentence using word-level alignment. Instead of mapping the boundary by the same relative position, we align SAE and AAE words using aneuraz/awesome-align-with-co. The aligned words around \(j^*\) are used to choose the corresponding AAE boundary \(k^*\). In the example, the SAE context “I was going to call you later,” is aligned with the AAE context “I was finna call you later,” while the SAE continuation “but I had to finish my homework first” is aligned with the AAE continuation “but I hadda finish my homework first.”

This procedure avoids assuming that SAE and AAE have the same length or word order. It instead projects the split through word-level correspondence, which gives cleaner paired context-continuation segments for the continuation preference experiments.

9.2 Prompt Template↩︎

Fig. 8 shows the prompt template for multiple-choice evaluation. We use a minimal structure that avoids instructions about dialect, tone, or style, isolating the model’s intrinsic continuation preference. Option ordering (SAE vs.AAE as option A or B) is randomized to control for positional bias.

9.3 AAE Grammatical Feature Lexicon↩︎

Table 6 presents the AAE grammatical markers used in our feature-level analysis, adapted from [26]. Features are categorized into four classes: Auxiliary (non-standard auxiliary constructions), Aspectual (habitual and completive markers), Preverbal (pre-verbal lexical items), and Syntactic (clause-level constructions such as negative concord).

9.4 Reproducibility Details↩︎

All experiments are in PyTorch. We use vLLM [34] for inference and Hugging Face Transformers [35] for steering, which requires direct access to residual activations. Perplexity is computed via autoregressive likelihood; multiple-choice evaluation extracts log-probabilities and selects the higher-probability continuation. Computations were performed on NVIDIA H100 80GB GPUs.

All causal-tracing and steering experiments were implemented in PyTorch with Hugging Face transformers. For each model, we used 1,200 AAE-feature sentences for layer selection, 7,000 AAE–SAE pairs for dialect-direction extraction, and a separate 3,000-pair set for evaluation. Causal tracing corrupted the AAE feature-token embeddings with Gaussian noise \(\epsilon \sim \mathcal{N}(0,\sigma^2 I)\), where \(\sigma = 3.0 \times \mathrm{std}(E_{\mathrm{feat}})\) for the selected feature embeddings in that example, and restored clean activations layer-by-layer at the same token positions. For steering, we extracted one unit dialect direction per selected layer from the last context-token hidden-state contrast, ranked layers by mean restoration effect, and used the top \(4\) layers in the multi-layer setting with weights proportional to the absolute restoration effect. We evaluated \(\beta \in \{0, 0.1, 0.2, 0.4, 0.6, 0.8, 1.0\}\) and report the corresponding selected rows, traced-layer summaries, chosen layers, train/eval splits, and steering outputs for each model.

10 Supplementary Steering Analyses↩︎

10.1 Prompting Baseline↩︎

For the prompting baseline in Table 3, we use the original multiple-choice continuation prompt and add the following instruction:

Treat African American English (AAE) and Standard American English (SAE) as semantically equivalent dialects, and base your judgment only on the underlying meaning rather than dialect-specific wording.

10.2 Prompt Wording Ablation↩︎

A possible concern is that the word best in our prompt may favor SAE by implying correctness or safety. We test this by replacing it with most likely continuation and most stylistically consistent continuation, and rerun the continuation preference evaluation for all models.

Table 7 shows the main pattern remains unchanged. The stylistic prompt slightly reduces the asymmetry, but models still rarely switch to AAE under SAE context, while often switching to SAE under AAE context. In our results, AAE continuation rates under SAE context remain around 20–30%, whereas SAE continuation rates under AAE context often remain around 50–70%. This suggests that the observed bias is not driven only by the word best, but reflects a more general preference for SAE continuations.

4pt

10.3 Single-Layer vs.Multi-Layer Steering↩︎

Table 8 compares single-layer steering with multi-layer weighted steering. For the single-layer setting, we select the most causally important layer based on the recovery analysis in Section 4.4 and apply the steering direction only at that layer. By contrast, the multi-layer setting applies weighted steering across the top traced layers.

For each strategy, we report the best setting for each model. The overall pattern is clear. Multi-layer steering gives lower LP than single-layer steering for all models in this comparison. It also produces larger bias reduction in every case. At the same time, multi-layer steering keeps AAE perplexity lower across all models, while the SAE perplexity values remain close between the two settings. This means that distributing the intervention across several traced layers is not only more effective at reducing bias, but also does not introduce a larger utility cost. These results suggest that the debiasing signal benefits from coordinated steering across the top traced layers rather than being captured fully by only one layer. In this setting, combining the strongest traced layers is usually better than restricting the intervention to a single layer.

2.5pt

10.4 Full Steering Results↩︎

4pt

Table 9 reports the full multi-layer steering sweep, and the trends across \(\beta\). In the table, LP measures log-probability bias, where lower values indicate less preference for SAE continuations. AAE Match and SAE Match are multiple-choice consistency scores under AAE and SAE contexts, respectively, where higher values are better. AAE PPL and SAE PPL are the perplexities of the AAE-matched and SAE-matched continuations, respectively, where lower values indicate better support for the matched continuation. DGI and cDGI measure prediction consistency across dialectal variants, with higher values indicating better invariance.

(a) Discriminative consistency (DGI, cDGI): Both metrics increase monotonically with \(\beta\) for all models (e.g., Phi-4: DGI \(0.74 \rightarrow 0.88\), cDGI \(0.77 \rightarrow 0.90\); Llama-3.1-70B: DGI \(0.51 \rightarrow 0.63\), cDGI \(0.57 \rightarrow 0.66\)), showing that steering also improves AAE–SAE prediction agreement on the discriminative task, not just continuation preference.

(b) Perplexity (PPL): AAE PPL decreases with \(\beta\) at small to moderate values (e.g., Phi-4: \(123.85 \rightarrow 110.16\) at \(\beta = 0.4\); Mistral-Small-3.1-24B: \(82.75 \rightarrow 75.9\) at \(\beta = 0.8\)), while SAE PPL remains close to baseline across the same range (within \(\sim 1\)–\(2\) points for Mistral-Small-3.1-24B, DeepSeek-R1, and Llama-3.1-70B, and within Gemma-3’s tokenizer-scaled noise floor). At high \(\beta\) (\(\geq 0.8\)), AAE PPL rebounds and SAE PPL grows sharply for Phi-4 (\(26.73 \rightarrow 45.17\)) and Phi-3 (\(20.59 \rightarrow 34.53\)), reflecting the same over-steering effect visible in SAE Match.

(c) Log-probability bias (LP): LP decreases approximately monotonically with \(\beta\) for all six models, from \(60.64 \rightarrow 45.16\) on Phi-4, \(70.79 \rightarrow 62.07\) on Phi-3, and \(34.86 \rightarrow 29.76\) on Llama-3.1-70B, corresponding to relative bias reductions of \(10\)–\(26\%\) at \(\beta = 1.0\). The largest LP gains come in the \(\beta \in [0.4, 0.8]\) range, beyond which returns flatten.

(d) Multiple-choice preference (MC). AAE Match under AAE context rises monotonically with \(\beta\) for every model (e.g., Phi-4: \(39.28 \rightarrow 70.36\); DeepSeek-R1: \(43.79 \rightarrow 64.43\); Llama-3.1-70B: \(41.29 \rightarrow 64.75\)), confirming that steering shifts dialect-matched preferences in the intended direction. SAE Match under SAE context is largely preserved for most models (within \(1\)–\(5\) points of baseline for Mistral-Small-3.1-24B, DeepSeek-R1, Gemma-3, and Llama-3.1-70B across all \(\beta\)), but degrades at high \(\beta\) for Phi-4 (\(89.14 \rightarrow 68.09\) at \(\beta = 1.0\)) and Phi-3 (\(79.65 \rightarrow 66.29\)), indicating that over-steering can erode SAE behaviour on these models.

Overall, across all six models, moderate steering (\(\beta \in [0.4, 0.8]\)) achieves most of the available LP and MC improvements while keeping SAE PPL and SAE Match close to baseline; the bolded selections in Table 9 reflect this trade-off. Pushing to \(\beta = 1.0\) yields small additional bias-reduction gains but begins to hurt SAE-side fluency on the Phi family, supporting the choice of a medium \(\beta\) rather than the most aggressive setting.

10.5 Qualitative Example of a Cross-Model Preference Flip↩︎

We show one qualitative example in which the full source sentence is recognizably AAE and the evaluated AAE context itself contains AAE markers. We report the segmented context–continuation pair used in evaluation, together with the preference shift induced by steering. This example is also representative at the cross-model level: before debiasing, five of the six models prefer the SAE continuation under the AAE context, whereas after debiasing, five of the six steered models prefer the AAE continuation.

5pt

10.6 Confidence Intervals for Bias Mitigation Results↩︎

To quantify uncertainty in the mitigation results reported in Table 3, we compute 95% paired bootstrap confidence intervals over the 3,000 evaluation pairs. For each bootstrap sample, we resample evaluation pairs with replacement and recompute all metrics for Base, Prompt, and Ours on the same resampled set. This preserves the paired structure across methods and yields confidence intervals that reflect example-level variation rather than run-to-run variance.

Table 11 reports the corresponding confidence intervals for all metrics in Table 3. Overall, the uncertainty estimates support the same qualitative conclusion as in the main text: activation steering consistently improves dialect-matched behavior under AAE context while largely preserving SAE-side behavior.

3.2pt

10.7 Bias–Utility trade-off↩︎

In Figure 10, we show how the bias and utility change by sweeping over \(\beta\). Each point corresponds to one \(\beta\) setting for one model. The x-axis shows the LP bias score, where lower values indicate less preference for SAE continuations, and the y-axis shows the SAE perplexity ratio relative to the unsteered baseline \(\beta=0\), where values close to 1.0 indicate better preservation of SAE-side fluency.

References↩︎

---

1. Corresponding author.↩︎

2. cardiffnlp/twitter-roberta-base-2022-154m.↩︎

3. Steering uses separate splits (§5); absolute values are not directly comparable to §§6.1–6.3.↩︎