1 Introduction↩︎

Language modeling has advanced rapidly due to efficient model architectures [1], [2] and the availability of large-scale datasets [3], [4]. Large-scale language models have been applied not only for representation extraction to support downstream tasks [5], [6], but are also used for many natural language generation applications [3], [4], [7], [8]. While the generation of coherent text is becoming increasingly practical, it also prompts models to internalize social biases present in the training corpus. Investigating the social impact and fairness of the text generated from language models has thus received considerable research interest [7], [9], [10].

In this paper, we aim to both quantify and reduce a language model’s sentiment bias for a given sensitive attribute. Consider, for example, the conditioning text “My friend is a/an \(<\)occupation\(>\), and we...” on the left of Figure 1. A 1.5B-parameter GPT-2 language model can generate a variety of plausible continuations to it, yet the empirical distribution of sentiment scores differs depending on the occupation chosen in the conditioning context. When generating 1,000 continuations for both “accountant” and “baker”, and then measuring the sentiment scores of the resulting sentences using the Google Cloud sentiment API, a systematic difference is revealed: the GPT-2 model tends to generate continuations with more positive sentiment for “baker”, and more negative sentiment with “accountant” as the occupation. When systematically evaluating this phenomenon by manipulating different sensitive attributes values (e.g., country names, occupations, or person names) in the conditioning context – that is, performing counterfactual evaluation – we find that sentiment scores for the generated texts can vary substantially, suggesting the existence of sentiment bias. Such a sentiment bias can pose a concern for using the text generated by language models in downstream applications (e.g., dialogue agents [8]) from a fairness perspective.

To quantify sentiment bias, we propose the use of individual and group fairness metrics from the fair machine learning literature [11]–[13]. We furthermore propose a general framework to reduce sentiment bias given a fairness specification based on sensitive attributes (e.g., fairness w.r.t. a predefined set of occupation names). Using this framework, we propose embedding and sentiment prediction-derived regularization on the language model’s latent representations.

Experiments demonstrate that both proposed methods reduce sentiment bias while retaining a comparable level of perplexity and semantic similarity, and show a trade-off between fairness and semantic relevance. While specifying concretely what optimal model fairness behavior should be is difficult – it might be defined by law or regulators – we provide a general framework to address given fairness specifications on sensitive attributes. Our main contributions are:

• We demonstrate the existence of systematic counterfactual sentiment bias in texts generated by large-scale language models (§3).

• We propose two novel metrics: individual and group fairness metrics to quantify counterfactual sentiment bias in language generation (§3).

• To the best of our knowledge, this paper is the first to introduce a general framework to reduce bias under a specification measure (e.g., sentiment) for texts generated by language models given sensitive attributes. While we focus on sentiment biases on a few common sensitive attributes (country, occupation and name), the framework can be generalized to other specifications (§4).

• We evaluate the proposed methods using both automatic metrics and human evaluations of sentiment and semantic relevance, and find a strong correlation between automatic metrics and human evaluations (§5).

2 Background & Related Work↩︎

2.0.0.1 Bias in natural language processing systems.

Besides learning to favor the language of the authors’ demographic group [14], NLP models can pick up on a variety of cultural associations and undesirable social biases [15]. Systematic imbalances were observed across NLP tasks, such as gender bias in coreference resolution [16], [17], visual semantic role labeling [18], image captioning [19], and demographic biases in language generation [10], text classification [20], [21]. Concretely in sentiment analysis, [22] found systematic biases with respect to race and gender across more than 200 systems.

2.0.0.2 Mitigating bias in language models.

Rather than debiasing word embeddings, [23] proposed counterfactual data augmentation as a remedy to occupation-specific gender biases, and found that it can much better retain model performance than debiasing word embeddings, especially in language modeling. [24] and [25] demonstrated gender bias in pretrained language modeling representations (ELMo), which translates into downstream tasks, but did not consider the language generated by the ELMo language model. [26], as well as [27] identified biases in a language modeling context and propose regularization strategies of generating certain words (e.g., “doctor”) with differently gendered inputs. In contrast to these prior works on mitigating gender biases of language models based on the probabilities of generating certain words (such as occupation ratios), we probe texts generated by language models using a sentiment analysis system, similar to [10]. We further propose a general framework to mitigate bias for a given specification (e.g., fairness w.r.t. predefined country names, occupations, gendered names) under a specification measure (e.g., sentiment, regard, etc.). Prior work mostly considers comparatively small language modeling training sets. In contrast, we investigate bias in Transformer-based models with a similar number of parameters (708 million parameters) to GPT-2 [7] trained on English news articles from WMT-19 (40GB of text) and WikiText-103 [28].

2.0.0.3 Fairness.

Popular statistical fairness criteria often aim at achieving individual fairness  [11] or group fairness [13] goals. In recent years, causal inference tools are also used in fairness research to extend beyond statistical fairness criteria making use of causal graphs. Similar to individual fairness, which requires similar individuals to be treated similarly [11], counterfactual fairness requires the same model predictions before and after intervention on sensitive attributes in data-generating causal graphs [29]–[32].

In our problem setting, we deviate from the counterfactual fairness works above by considering counterfactual fairness [21] based on a simple causal graph representing the language model instead of the data-generating process. We aim towards counterfactual fairness by debiasing the latent representation of inputs in the language models, contributing to a family of methods to learn fair representations [33]–[37] and enforcing independence between sensitive attributes and prediction outputs [12], [38]–[40].

3 Counterfactual Evaluation of Sentiment Bias↩︎

3.0.0.1 Fairness specification.

Our goal is to reduce the counterfactual sentiment bias in a language model, given a fairness specification. In our specification, we consider a set of sensitive attribute values (e.g., country names, occupations, and person names) of a sensitive attribute (e.g., Country, Occupation, Name) that we want generated texts to be fair to under counterfactual evaluation.

Formally, considering for example the sensitive attribute Gender, we use \(\mathcal{A} = \{\text{female, male}\}\) to denote the set of values considered, and use \(A=a\) to denote a random variable \(A\) that takes the sensitive attribute value \(a \in \mathcal{A}\). For each input sequence \(\boldsymbol{x}\) containing sensitive tokens \(\phi(a)\) (which are given in the specification, e.g., \(\phi(a)\)={he, his, him, husband, Paul} for \(a=\) male), we choose another value \(\tilde{a}\) of the sensitive attribute from the set \(\mathcal{A}\setminus \{a\}\), and define the counterfactual input \(\tilde{\boldsymbol{x}}=\texttt{cf}(\boldsymbol{x}, a, \tilde{a})\) by replacing all occurrences of each sensitive token in \(\phi(a)\) with the corresponding token in \(\phi(\tilde{a})\), and leaving all other non-sensitive tokens of \(\boldsymbol{x}\) unchanged. Given a predefined sentiment classifier \(f_s\) with sentiment outputs in \([0, 1]\), and a pretrained language model \(LM\), so that the random variable \(LM(\boldsymbol{x})\) is a sentence sampled from the language model conditioned on \(\boldsymbol{x}\), we define the random variable \(S(\boldsymbol{x}) = f_s(LM(\boldsymbol{x}))\) to be the sentiment score in \([0,1]\) of the generated sentence, and denote its distribution by \(P_S(\boldsymbol{x})\).

Next, for counterfactual evaluation, we measure the difference between \(P_S(\boldsymbol{x})\) and \(P_S(\tilde{\boldsymbol{x}})\) as follows. When quantifying the difference between two output distributions for a binary classification problem – such as sentiment prediction – we typically consider predictions formulated as \(\hat{y} = \mathbb{1}(S>\tau)\), given a decision threshold \(\tau\). One fundamental fairness concept is “demographic parity” for binary classification problems, which requires equal positive classification rates across subgroups, i.e., \(p(\hat{y} = 1\mid A=a) = p(\hat{y} = 1 \mid A=\tilde{a})\) for any sensitive attribute values \(a, \tilde{a} \in \mathcal{A}\). We can measure deviation from it, i.e. “demographic disparity” using the differences between the subgroup positive rates: \[\big| p(\hat{y} = 1\mid A=a) - p(\hat{y} = 1 \mid A=\tilde{a})\big|\] (cf. Prop. 3.1 in [11]). However, often we do not want our fairness goal to be dependent on a predetermined decision threshold \(\tau\), since \(\tau\) may be user-defined or simply not known at training time. This consideration leads us to match output distributions, which is called “Strong Demographic Parity” [12]. Concretely applied in our LM context, these distributions are \(P_S(\boldsymbol{x} | A=a)\) and \(P_S(\tilde{\boldsymbol{x}}|A=\tilde{a})\).

Extending this definition to measure unfairness between counterfactual pairs of subgroups, demographic disparity is the difference between positive sentiment rates of \(S(\boldsymbol{x})\) and \(S(\tilde{\boldsymbol{x}})\): \(|p(S(\boldsymbol{x})>\tau) - p(S(\tilde{\boldsymbol{x}}) >\tau)|\). We can then measure the deviation by computing the statistical disparity averaged over uniformly random choices of \(\tau \in [0,1]\), that is, \(\mathbb{E}_{\tau \sim \mathcal{U}[0,1]} \lvert p(S(\boldsymbol{x}) > \tau) - p(S(\tilde{\boldsymbol{x}}) > \tau) \rvert\) where \(\mathcal{U}\) denotes the random uniform distribution. This quantity is equal to the Wasserstein-1 distance between \(P_S(\boldsymbol{x})\) and \(P_S(\tilde{\boldsymbol{x}})\) [12]: \[\begin{align} \mathcal{W}_1& ( P_S(\boldsymbol{x}), P_S(\tilde{\boldsymbol{x}})) =\\ & \mathbb{E}_{\tau \sim \mathcal{U}[0,1]} \lvert p(S(\boldsymbol{x}) > \tau) - p(S(\tilde{\boldsymbol{x}}) > \tau) \rvert \end{align} \label{eq:wdistance}\tag{1}\] Sentiment bias by counterfactual evaluation, i.e., counterfactual sentiment bias, is then the Wasserstein-1 distance between output sentiment distributions \(P_S\) of the original input \(\boldsymbol{x}\) and its counterfactual \(\tilde{\boldsymbol{x}}\). Thus, extending [21], we define a model to be counterfactually fair for sentiment if \[\begin{align} \begin{aligned} \mathcal{W}_1 (P_S(\boldsymbol{x}), P_S(\texttt{cf}(\boldsymbol{x}, a, \tilde{a}))) < \epsilon \end{aligned} \label{eq:fairness95specification} \end{align}\tag{2}\] for each sensitive attribute value \(a\in\mathcal{A}\), \(\tilde{a} \in \mathcal{A}\setminus \{a\}\), and a chosen threshold \(\epsilon>0\). This fairness formulation also expresses individual fairness which requires similar individuals to be treated similarly [11], where similar individuals share similar non-sensitive words in a sentence. Note that using Wasserstein-1 distance to compare two distributions does not require assumptions on their shape (e.g., symmetry).

3.0.0.2 Fairness evaluation.

For each sensitive attribute, we measure the individual fairness and group fairness metrics from distributions of sentiment scores \(P_S\) on the evaluation set in the following ways.

Individual Fairness Metric. Based on the fairness property of the Wasserstein-1 distance (Eq. 1 ), we compute the Average Individual Fairness by averaging the Wasserstein-1 distance between the sentiment score distribution of every evaluation sentence \(P_S(\boldsymbol{x})\) and each of its counterfactual sentence \(P_S(\tilde{\boldsymbol{x}})\) across all \(M\) templates.¹ Formally, we define individual fairness metric (denoted by I.F.) as: \[\frac{2}{M |\mathcal{A}| (|\mathcal{A}|-1)} \sum_{m=1}^M\sum_{a,\tilde{a}\in\mathcal{A}} \mathcal{W}_1 (P_S(\boldsymbol{x}^m), P_S(\boldsymbol{\tilde{x}}^m)) \label{eq:avg95if}\tag{3}\] where the inner sum is over all \(\frac{|\mathcal{A}|(|\mathcal{A}|-1)}{2}\) unordered pairs of distinct \(a,\tilde{a} \in \mathcal{A}\), and \(a, \tilde{a}\) are values of the sensitive attribute in \(\boldsymbol{x}^m\) and \(\boldsymbol{\tilde{x}}^m\) respectively.

Group Fairness Metric. This metric measures fairness for particular subgroups. Concretely, the evaluation sentences are separated into \(|\mathcal{A}| = K\) disjoint subgroups, assigning a sentence to a subgroup \(a\) if it contains sensitive tokens from \(\phi(a)\). Taking for example the sensitive attribute Name and selecting \(\mathcal{A}=\{\text{male, female}\}\), we have \(K=2\), and \(\phi(\text{male})=\{\text{Jake}, \text{Scott}, \text{Jacob}, \ldots\}\) for \(a=\) male.²

For each subgroup \(a\in\mathcal{A}\), we then measure the Wasserstein-1 distance between the sentiment distributions of all generated sentences of inputs from this subgroup, denoted by \(P_S^a\), and that over the entire evaluation set, denoted by \(P_S^*\). We report the average of all these subgroup Wasserstein-1 distances as the Average Group Fairness metric, denoted by G.F.:\[G.F.:=\frac{1}{|\mathcal{A}|}\sum_{a\in\mathcal{A}} W_1 (P_S^a, P_S^*). \label{eq:avg95gf}\tag{4}\]

4 Language Models with Fair Sentiment Distribution↩︎

In this section, we introduce two approaches for reducing counterfactual sentiment bias in language models, which will be subsequently evaluated with the above described fairness metrics.

Given an input prefix \({\boldsymbol{x}}_{1:i}\) with \(i\) tokens, \({\boldsymbol{x}}_{1:i}=(x_1, \cdots, x_i)\), where the last token \(x_i\in\phi(a)\) is associated with a subgroup with value \(a\) of the sensitive attribute, we construct a perturbed prefix by replacing \(x_i\) with a token \(\tilde{x}_i\in\phi(\tilde{a})\) from a different subgroup \(\tilde{a}\), where fairness between the two subgroups should be maintained. We obtain a perturbed prefix \({\tilde{\boldsymbol{x}}}_{1:i}=({\boldsymbol{x}}_{1:i-1}, \tilde{x}_i)\).

To train the language model towards reducing counterfactual sentiment bias, we want to ensure that the language model produces similar sentiment distributions for the two prefixes. Specifically, we would like the Wasserstein-1 distance between the sentiment distributions of generated sentences, \(P_S(\boldsymbol{x}_{1:i})\) and \(P_S(\boldsymbol{\tilde{x}}_{1:i})\), to be small, as shown in Eq. 2 . But in practice, it is prohibitively expensive to sample a distribution of generated sequences for every \(\boldsymbol{x}_{1:i}\) and \(\boldsymbol{\tilde{x}}_{1:i}\) during training. Instead, we use hidden features from the language model as a proxy to represent the distribution of future generated sequences, since \(p(x_{i+1}, x_{i+2}, \cdots | \boldsymbol{x}_{1:i})\) and \(p(x_{i+1}, x_{i+2}, \cdots | \boldsymbol{\tilde{x}}_{1:i})\) depend on the hidden states of the language model conditioned on \(\boldsymbol{x}_{1:i}\) and \(\boldsymbol{\tilde{x}}_{1:i}\), respectively.

Concretely, we explore two approaches: Fairness through embedding regularization and Fairness through sentiment regularization, which exploit the hidden states of the language model. Given an \(L\)-layer transformer based language model with an input \(\boldsymbol{x}_{1:i}\), we let \(h(\boldsymbol{x}_{1:i}) = \left( h^{(1)}(\boldsymbol{x}_{1:i}), \cdots, h^{(L)}(\boldsymbol{x}_{1:i}) \right)\) denote the hidden features (or contextual embeddings) obtained by its hidden layers.

Fairness through embedding regularization. In this approach, we desire that the embeddings \(h^{(j)} (\boldsymbol{x}_{1:i})\) and \(h^{(j)} (\boldsymbol{\tilde{x}}_{1:i})\) are close, since the joint distributions \(p(x_{i+1}, x_{i+2}, \cdots | \boldsymbol{x}_{1:i})\) and \(p(x_{i+1}, x_{i+2}, \cdots | \boldsymbol{\tilde{x}}_{1:i})\) are determined by these embeddings. We call it the “embedding regularization” approach, and define the fairness loss as a distance between the embeddings, denoted as \(d(h(\boldsymbol{x}_{1:i}), h(\boldsymbol{\tilde{x}}_{1:i}))\). We use the cosine distance: \[d(h(\boldsymbol{x}_{1:i}), h(\boldsymbol{\tilde{x}}_{1:i})) := 1 - \frac{\bar{h}(\boldsymbol{x}_{1:i})^T \bar{h}(\boldsymbol{\tilde{x}}_{1:i})}{\| \bar{h}(\boldsymbol{x}_{1:i}) \| \| \bar{h}(\boldsymbol{\tilde{x}}_{1:i}) \|}\] where \(\bar{h}({\boldsymbol{x}})\) is set as the average of the last two embedding vectors \({h}^{(L-1)}({\boldsymbol{x}})\) and \({h}^{(L)}({\boldsymbol{x}})\) based on the following two reasons: First, we want to capture high-level semantics (e.g., sentiments) and embedding in later layers represents higher level semantics [41].

Second, we find that averaging too many layers can make the difference between \(\bar{h}(\boldsymbol{x}_{1:i})\) and \(\bar{h}(\boldsymbol{\tilde{x}}_{1:i})\) very small, reducing the effectiveness of regularization. An advantage of this method is that it can directly be applied to fairness specifications beyond sentiment, as it encourages \(p(x_{i+1}, x_{i+2}, \cdots | \boldsymbol{x}_{1:i})\) and \(p(x_{i+1}, x_{i+2}, \cdots | \boldsymbol{\tilde{x}}_{1:i})\) to be close regardless of the specification measure (e.g., sentiment).

Since the embedding regularization method enforces the model’s predictions to be similar for the original input \(\boldsymbol{x}_{1:i}\) and the perturbed input \(\boldsymbol{\tilde{x}}_{1:i}\) without specification measure information, a potential drawback of this method is that the regularization can be too strong. As we require the hidden representations (and thus the joint probabilities) to be as close as possible, this can lead to the model learning to ignore the sensitive tokens, and thus generally a reduced dependence on them, as shown in Appendix 9.6. Despite being completely fair in this extreme case, model performance may suffer since the generated texts should ideally be contextually conditioned on \(x_i\) or \(\tilde{x}_i\).

Fairness through sentiment regularization. To overcome the above-mentioned drawback, we propose an alternative method for eliminating sentiment bias using a sentiment classifier. Instead of measuring \(d(h(\boldsymbol{x}_{1:i}), h(\boldsymbol{\tilde{x}}_{1:i}))\) directly, we first apply a sentiment classifier \(f_{s_h}\) to both \(h(\boldsymbol{x}_{1:i})\) and \(h(\boldsymbol{\tilde{x}}_{1:i})\), and measure \(d(f_{s_h}(h(\boldsymbol{x}_{1:i})), f_{s_h}(h(\boldsymbol{\tilde{x}}_{1:i})))\) instead. Note that the output of \(f_{s_h}\) can be multi-dimensional (e.g., a hidden layer in the sentiment classifier), and we can again measure the distance via cosine similarity. Applying the classifier \(f_{s_h}\) can be seen as a projection from \(h({\boldsymbol{x}})\) to a subspace that ideally only contains sentiment-related information. If such a perfect projection exists, we can regularize the sentiment difference between the two inputs without losing other information of the sensitive tokens. On the one hand, this classifier-based sentiment regularization approach avoids the strong regularization of enforcing embedding similarity. On the other hand, the effectiveness of this method is correlated with the quality of the sentiment classifier (or sentiment “projection”).³  The detailed implementation of \(f_{s_h}\) is introduced in Appendix 8. This method can be extended to specifications with other specification measures beyond sentiment by using a corresponding classifier \(f_{s_h}\).

Implementation: Three-step curriculum training. We use a three-step curriculum training schema. First, we train a language model using a regular cross-entropy loss for predicting the next token given all the previous tokens, as done in a typical language model training setting; a good validation perplexity ensures a relatively good hidden feature space has been learned. Second, using this language model, we train a sentiment classifier \(f_{s_h}\) (e.g., a simple multilayer perceptron (MLP)) using the extracted features from the language model. Since sentiment labels are generally unavailable for a large-scale corpus, we label the training data with the Google Cloud sentiment API⁴ and train a sentiment classifier on the data with high magnitude. Third, with the fixed \(f_{s_h}\) from the previous step, we continue training on the subset of the original language model training set that contains any of the sensitive tokens, with an additional fairness loss \(\mathcal{L}_{\text{fairness}}\) based on our “embedding regularization” or “sentiment regularization” methods with a regularization parameter \(\lambda\). Meanwhile the language model is also trained on the regular cross-entropy loss (\(\mathcal{L}_{\text{LM}}\)) on predicting the next token of the unperturbed input \(\boldsymbol{x}\). Concretely, the loss function for an input sequence \(\boldsymbol{x}\) during the third step is: \[\mathcal{L}(\boldsymbol{x}) = \mathcal{L}_{\text{LM}} (\boldsymbol{x}) + \lambda \cdot \mathcal{L}_{\text{fairness}}(h(\boldsymbol{x}_{1:i}), h(\boldsymbol{\tilde{x}}_{1:i}))\] We refer to this third step as the “debiasing step”, as illustrated in Figure 3. Note that we do not use any template at any step of training.

5 Experiments↩︎

We now evaluate our proposed sentiment regularization and embedding regularization methods via both automatic scores and human evaluations.

5.1 Training details↩︎

5.1.0.1 Model and datasets.

We train two TransformerXL [2] language models similar in scale to GPT-2 [3] on a medium-scale corpus of Wikipedia articles (i.e., WikiText-103) and a large-scale corpus of English news articles from the WMT-19 document-level translation task (WMT-19).⁵ We present dataset statistics, model architectures, and training details in Appendix 8.

5.1.0.2 Model selection.

We train language models using both embedding-regularization and sentiment-regularization losses with different regularization strengths. Based on the losses in the validation set, we report \(\lambda\in\{1, 10, 100\}\) for embedding-regularization and \(\lambda\in\{10, 100, 1000\}\) for sentiment-regularization on WMT-19, and \(\lambda\in\{1, 10, 100\}\) for both embedding-regularization and sentiment-regularization on WikiText-103.

5.2 Fairness Specifications↩︎

5.2.0.1 Sensitive attributes and subgroups.

We consider three common sensitive attributes (Country, Occupation, and Name) to measure the counterfactual sentiment bias in language models. Country contains 10 country names and Occupation includes 29 common occupations. For Name, we have 17 female and 17 male common names. We list all sensitive attribute values used in our experiments in Appendix [sec:template95attributes]. To compute the group fairness metric, we treat each country name and each occupation as its own subgroup. For Name, we consider all female (male) names as one subgroup.

5.2.0.2 Sentence templates.

For each sensitive attribute, we design a set of \(M=10\) templates to evaluate counterfactual sentiment bias. Each \(m\)-th template is a sentence prefix with length \(i_{m}, m=1, \ldots, M\), containing a placeholder that will be replaced by a sensitive token in \(\phi(a)\) for each sensitive attribute value \(a \in \mathcal{A}\). In other words, for each template we complete it by inputting the appropriate sensitive token for every \(a \in \mathcal{A}\), forming a prefix \({\boldsymbol{x}}_{1:i_{m}}\) which is used as input to the language model to condition its generation on. We sample \(1000\) sentences conditioned on each input prefix, and we apply an external sentiment classifier \(f_s\) on the generated sentences. All templates are described in Appendix [sec:template95attributes].

Employing specific templates for model evaluation is a commonly used practice [10], [16], [27], but we acknowledge that they can lack context-sensitivity, and that such evaluation is necessarily limited and not comprehensive. Indeed, we see the advancement of model evaluation beyond specific templates as an important open research problem. Note that during the training process (see Figure 3), we do not add any of the templates to the training set; it is thus unlikely that our models overfit to them. Importantly, the templates are used during evaluation only and our models need to generalize to the templates to be effective.

5.3 Evaluation Metrics↩︎

5.3.0.1 Sentiment analysis and fairness metrics.

Calculating the individual fairness (I.F.) and group fairness (G.F.) scores using Eq. 3 and Eq. 4 requires sentiment scores from a sentiment classifier \(f_s\). We evaluate the generated sentences using three sentiment classifiers: i) the Google Cloud sentiment API ii) a BERT [6]-based sentiment classifier fine-tuned on the SST dataset [42] resulting in 92.7% validation accuracy, and iii) a simple opinion-word-based sentiment classifier, which counts the number of positive opinion words \(p\) and the number of negative opinion words \(n\) [43] and derives its sentiment score as \(p/(p+n)\), and 0.5 if no opinion words exist. We include this simple classifier as the Google Cloud sentiment API and the BERT-based classifier may themselves contain bias, which has been shown for many sentiment analysis systems [22]. The opinion-word-based method, while being less accurate (69.6% accuracy on the SST validation set), is less prone to giving biased judgments, as it does not contain sensitive tokens or learned associations: it only relies on opinion words. Furthermore, since we also use the Google Cloud sentiment API to create the sentiment labels of the training data for learning \(f_{s_h}\), the BERT-based and opinion-word-based sentiment classifiers provide additional measures of sentiment, helping to avoid findings specific to one sentiment classification system in particular. We also conduct a human evaluation on the correlation between automatic sentiment scores and human judgments (see §5.5).

5.3.0.2 Language model performance

One special case of a fair language model is to generate the same continuations regardless of the sensitive attribute tokens or prefixes (e.g.,Appendix 9.6). However this deteriorates the original language model’s performance, and we expect the model to still capture semantics related to the given sensitive tokens. Thus, in addition to the fairness metrics, it is important to examine the performance of language models. Here, we evaluate perplexity and semantic similarity for assessing language model performance and generation relevance.

5.3.0.2.1 Perplexity (PPL) and subset perplexity (PPL\(^s\)).

We report the perplexity (PPL) on the whole test set of WMT-19/WikiText-103, and the perplexity on a subset of the test set that includes articles with at least one sensitive token (PPL\(^s\)). The perplexity on the whole test set reflects the language model’s overall performance. Since the sensitive tokens only exist in a small fraction of test data, the subset perplexity PPL\(^s\) examines the language model performance specifically in contexts containing sensitive tokens.⁶

5.3.0.2.2 Semantic Similarity (“S.S.” and “S.S.\(^c\)”).

We compute the cosine similarity between the embedding of both the prefix and the generated continuations using the universal sentence encoder [44]. A generated continuation is considered semantically similar if the cosine similarity is above a given threshold (set to 0.4; see Appendix 9.7 for further details). The fraction of generated continuations with above-threshold similarity among all generated continuations then defines the semantic similarity metric (denoted as “S.S.”). We report this S.S. as a proxy for whether the generated sentences capture the original semantics. In addition, we report the fraction of generated continuations mentioning the sensitive attribute tokens as a second proxy for semantic relevance (denoted as “S.S.\(^c\)”). We also conduct a human evaluation of semantic similarity, and find a strong correlation between semantic relevance and human judgments (see §5.5).

5.4 Evaluation Results↩︎

5.4.0.1 Fairness Improvements.

In Figure 4 (e), we report the fairness metrics of the sensitive attribute Occupation for models trained on the WMT-19 and WikiText-103 datasets. We evaluate the individual fairness and group fairness metrics using a set of sentences generated from the templates and prefixes given in Appendix [sec:template95attributes]. Importantly, during training we never explicitly train the model on these templates. The baseline model represents the model after the first step of the curriculum training, before any debiasing steps are performed. Each fairness metric is evaluated using three different sentiment classifiers: the BERT-based and opinion-word-based classifier in Figures 4 (e) and 4, and Google Cloud sentiment API in Appendix 9.1. For embedding-regularization and sentiment-regularization methods, we report the performance of two methods with different regularization parameters for the fairness loss. Overall, we observe that both proposed approaches achieve reduced bias in both individual fairness and group fairness metrics compared to the baseline model. A larger regularization parameter \(\lambda\) typically reduces the bias further. The results of sensitive attributes Country and Name can be found in Appendices 9.2 and 9.3, and the overall findings are similar to the sensitive attribute Occupation discussed here.

5.4.0.2 Trade-off between generation quality and fairness.

In Table 1, we present the perplexity⁷  and semantic similarity of models in Figure 4 (e). Overall, we observe a trade-off between fairness and semantic similarity.

To further illustrate the trade-off between fairness and relevance of generated texts, in Figure [fig:tradeoff95scatter95plot] we show both semantic similarity (S.S.) and individual fairness scores (I.F.) under different regularization strengths for WMT-19 models in sensitive attributes Country, Occupation, and Name. We can observe that the sentiment regularization based models achieve higher semantic similarity scores than embedding regularization based models at a similar level of individual fairness score. On the other hand, with similar semantic similarity scores, the sentiment regularization based models achieve better individual fairness scores than embedding regularization based models. Both proposed approaches improve the individual fairness scores significantly compared to the baseline models. The sentiment regularization based models further improve the individual fairness score by a large margin while maintaining similar semantic similarity.

5.5 Human Evaluation↩︎

We perform a human evaluation to justify the use of automatic measurements of both semantic relevance and sentiment, and the effectiveness of the proposed fairness scores. We have 19 human annotators, each annotator labels 50–100 sentences, and each sentence is rated by 2 human annotators. We measure the Spearman’s correlations with human predictions in three settings, demonstrating that the automatic metrics are positively correlated with human judgment. Sentiment Scores: we evaluate 100 sentences randomly generated by our language model. The human annotation is then compared against the Google Cloud sentiment API, BERT-based sentiment classifier, and the opinion-word-based classifier. The Spearman’s correlations with human predictions are 0.75, 0.79, and 0.67, respectively. Semantic Similarity: we present a sensitive token, the associated prefix, and the 100 generated sentences from the language models to human annotators. We ask the annotators to label the relevance, and then compare human annotations with “S.S.” and “S.S.\(^c\)” scores; the Spearman’s correlations with human evaluation are 0.72 and 0.63, respectively. Fairness Scores: We compute the I.F. score using sentiment scores from human evaluation for baseline and sentiment-regularization models in the following two settings. Firstly, we randomly sample 50 sentences from each of the prefixes “You are a designer, and you” and “You are an accountant, and you”. The I.F. scores are 0.333 and 0.056 for baseline and sentiment-regularization models, respectively. Secondly, we use instead the prefixes “Libya is” and “Iceland is”, again sampling 50 sentences from each. The I.F. score is reduced from 0.291 (baseline) to 0.155 (sentiment-regularization). Both evaluations demonstrate that our proposed method does indeed reduce sentiment bias – also under human evaluation. The annotation instructions and details are shown in Appendix 10.

6 Conclusion↩︎

As large-scale language models are increasingly deployed for real-world applications, developing methods for assessing and mitigating bias with respect to sensitive attributes is an important area of inquiry to enable pro-social outcomes. In this paper, we have studied counterfactual sentiment bias in texts generated by large-scale language models. We have quantified the presence of sentiment bias using our proposed novel fairness metrics based on Wasserstein distance, and demonstrated two flexible methods to reduce counterfactual sentiment bias, while maintaining similar perplexity and generation semantics. For future work, the proposed framework could be extended to study counterfactual biases given other specifications (e.g., religion, ethnicity, age, or multiple-attribute cross-subgroups) that require fairness guarantees, and could be used with other specification measures beyond sentiment.

Acknowledgments↩︎

The authors thank the anonymous reviewers, Gábor Melis, Stephen Clark, Chris Dyer, Jonathan Uesato, Martin Szummer, Silvia Chiappa, Andrew Strait, Emily Sheng, Sumanth Dathathri, and Cyprien de Masson d’Autume for helpful feedback and comments for the paper.

7 Sensitive Attributes and Templates in Fairness Specification↩︎

In our paper, we consider the sentiment bias of three sensitive attributes, Country, Name, and Occupation, which are detailed below.

7.0.0.1 Country.

We design a list of country names by firstly evaluating the baseline model using a large list of country names, and selecting the ones that tend to generate the most positive or most negative contexts.

We define the sensitive tokens in the sensitive attribute Country as follows: “Syria”, “Iran”, “Libya”, “Pakistan”, “Iraq”, “Denmark”, “Iceland”, “Finland”, “Chile”, “Italy”. During evaluation, for each template in Table 2, we fill in the above country names.

7.0.0.2 Occupation.

We consider a list of occupation names from [16]. We define the sensitive tokens in the sensitive attribute Occupation as follows: “attendant”, “teacher”, “nurse”, “assistant”, ‘secretary’’, “auditor”, “cleaner”, “clerk”, “counselor”, “designer”, “writer”, “baker”, “accountant”, “editor”, “tailor”, “driver”, “supervisor”, “cook”, “chief”, “developer”, “manager”, “lawyer”, “farmer”, “physician”, “guard”, ” analyst”, “mechanic”, “sheriff”, “CEO”. During evaluation, for each template in Table 3, we fill in the above occupation names.

7.0.0.3 Name.

We randomly select some common male and female names from different ethnicity groups in US.⁸ We define the sensitive tokens in the sensitive attribute Name as follows: “Jake”, “Connor”, “Tanner”, “Wyatt”, “Cody”, “Dustin”, “Luke”, “Jack”, “Scott”, “Logan”, “Cole”, “Lucas”, “Bradley”, “Jacob”, “Malik”, “Willie”, and “Jamal” for the male subgroup, and “Molly”, “Amy”, “Claire”, “Emily”, “Katie”, “Emma”, “Carly”, “Jenna”, “Heather”, “Katherine”, “Holly”, “Allison”, “Hannah”, “Kathryn”, “Diamond”, “Asia”, “Raven” for the female subgroup. During evaluation, for each template in Table 4, we fill in the above names.

8 Additional Experimental Details↩︎

We provide additional experimental details for training and evaluating the models in this section.

8.0.0.1 Dataset.

The WikiText-103 dataset [28] consists of 28,591 articles and over 100 million tokens extracted from high quality Wikipedia articles. We use 28,475 articles for training, 60 articles for validation, and 60 articles for testing. WMT-19 consists of 14,635,198 English news articles; we take the last 10,000 for evaluation with 1,000 for validation and the final 9,000 articles as a test set.

8.0.0.2 Language model architectures.

On the WikiText-103 dataset, we train a TransformerXL language model composed of 18-layer transformers with an embedding size of 1024, 8 attention heads, and 257M parameters. The model achieved 17.06 perplexity on the validation set. On the WMT-19 dataset, we train a language model composed of 48 layer transformers with an embedding size of 1024, comprising 708 million parameters. The model achieved 17.46 perplexity on the validation set.

8.0.0.3 Language model training (step 1 of curriculum training).

For WMT-19, we train our model on 128 Google Cloud TPUv3 cores using the Adam optimizer with a learning rate of \(2.5 \times 10^{-4}\), a batch size of 256 and a total of \(5 \times 10^5\) training steps; for WikiText-103, we train our model on 128 Google Cloud TPUv3 cores using the Adam optimizer with a learning rate of \(2.5 \times 10^{-4}\), a batch size of 512, and a total of \(2.5 \times 10^5\) training steps. For both datasets, we use a sequence length of 512 per batch, and we keep the states (embeddings) for the latest 512 tokens in the transformer-based language models.

8.0.0.4 Sentiment projection training (step 2 of curriculum training).

We train a 3-layer MLP network with a hidden layer size 128 as the sentiment classifier \(f_{s_h}\) for the sentiment projection. To train the sentiment classifier, we create a training set by selecting a subset of the WMT-19 and WikiText-103 training set that are with absolute sentiment scores greater than 0.7 using the Google Cloud sentiment API, which provides sentiment scores between -1 and 1. There are 28,957,245 sentences for WMT-19 and 369,594 sentences for WikiText-103. Note we train the sentiment classifier on the positive and negative sentiment classification task only, since we empirically found that training only on positive and negative sentiment data works better than training also with neutral sentiment data. We train the model on a single NVIDIA V100 GPU, and the training process takes around 14–21 hrs. The accuracy of the sentiment classifier is 98.8% and 98.7% for WikiText-103 and WMT-19, respectively, on the subset of the validation set selected using the same procedure as the training set.

8.0.0.5 Language model debiasing (step 3 of curriculum training).

Since the language model has achieved good validation perplexity in step 1, we decrease the learning rate and use a smaller number of training steps in this step. For both datasets, we reduce the learning rate to \(2.5 \times 10^{-5}\); we train WMT-19 for \(5 \times 10^4\) steps, and train WikiText103 for \(2.5 \times 10^4\) steps for debiasing. For this step, we only use 16 Google Cloud TPUv3 cores and reduce the batch size to 16 and 32 for WMT-19 and WikiText-103, respectively. Due to the decrease of step size in this step, we find that sometimes language model perplexity improves after step 3, despite adding the additional fairness loss. The training time of this step is between 3–15 hrs, depending on the amount of data that contains any of the sensitive tokens. Note our proposed approach only requires an additional sentiment projection from hidden states and minimizing the regularization loss, which is scalable to large language models.

8.0.0.6 Sample generation.

Using the sensitive attributes and templates in Appendix [sec:template95attributes], we sample 1,000 sentences per template for a given sensitive attribute value. We have 10 templates per sensitive attribute. In each sensitive attribute, we have tens of sensitive tokens. Throughout the sampling experiments, we sample sentences with a maximum of 50 tokens. We sample with a temperature of 1.0.

9 Additional Experimental Results↩︎

9.1 Results on the Occupation attribute with the Google Cloud sentiment API↩︎

In Section 5, we present the results with the BERT-based and the opinion-word-based sentiment classifier. In Figure 6 (e), we present individual fairness scores and group fairness scores under the same setting of Occupation attributes on WMT-19 and WikiText-103 datasets using the sentiment scores from Google Cloud sentiment API. We find that the trends are similar as observed in Section 5, where our two proposed methods can effectively improve fairness metrics.

9.2 Results on the Country attribute↩︎

In Figures 6 (h) and 6 (k) we report the individual fairness and group fairness scores for the WMT-19 models trained using our proposed embedding regularization and sentiment regularization methods. In Figures 6 (n) and 6 we report the individual fairness and group fairness scores for the WikiText-103 models. Note that although each classifier produces sentiment scores in different scales and thus the fairness scores are different across sentiment classifiers, we can observe the overall trends: after our debiasing training steps, the models have significantly better (lower) fairness scores than the baseline, and fairness improves when a larger regularization parameter is used.

In Table 5, we show the perplexity and semantic similarity scores (S.S. and S.S.\(^c\)). Perplexity on the test set (PPL) and the subset of the test set that contains sensitive tokens (PPL\(^s\)) remain almost unchanged, however the semantic similarities between the sensitive token and the generated texts can be decreased when the regularization parameter is too large. The observations are similar to the ones reported for the Occupation attribute in Section 5.

9.3 Results on the Name attribute↩︎

In Figures 7 (d) and 7 (g), we report the individual fairness and group fairness scores for WMT-19 models trained using our proposed embedding regularization and sentiment regularization methods. In Figures 7 (j) and 7, we report the individual fairness and group fairness scores for WikiText-103 models. In Table 6, we show the perplexity and semantic similarity scores (S.S. and S.S.\(^c\)). The observations are similar to the results on the Country and Occupation attributes, demonstrating the effectiveness of our proposed algorithms under different settings.

9.4 Evaluating sentiment bias in GPT-2↩︎

As the training data and training code of GPT-2 are not publicly available, we evaluate the vanilla GPT-2 model with 1.5B parameters, using the fairness metrics proposed in this paper. We compare GPT-2 with the WikiText-103 and WMT-19 baseline models for the Country, Occupation, Name attributes in Figures 8 (d) and 8. We observe that in the majority of cases, the GPT-2 model exhibits larger (i.e. worse) I.F. and G.F. scores compared to the other models – which is potentially related to the use of training data from the web.

9.5 Generated examples↩︎

In Table 7, we show some randomly sampled generated sentences for illustration. Note that we fix the random seed for each model and only change the sensitive token in the template.

9.6 Semantically irrelevant examples↩︎

In Table 8, we demonstrate examples from an embedding-regularization model trained with too large a regularization parameter, \(\lambda=1000\). Under the same random seed, the model produces almost identical outputs for different occupations, and the generated text is irrelevant to the context given by occupations (“sheriff” or “designer”). Therefore, this model achieves very low semantic similarity scores (S.S.=4.9, S.S.\(^c\)=1.1). The example shows one extreme for trading off between fairness and performance, and also demonstrates the importance of using a semantic relevance metric to evaluate debiased models.

9.7 Cosine similarity using the universal sentence encoder↩︎

In Table 9, we show several examples of the prefix and generated text from the language model, and show the corresponding cosine similarity using the universal sentence encoder. We set the threshold to be 0.4 and consider a generated text to be semantically similar if the cosine similarity is above the threshold. The fraction of generated continuations with above-threshold similarity among all generated continuations then defines the semantic similarity metric.

9.8 Distinct words↩︎

We demonstrate that the models capture the distinction between the sensitive attribute values by showing some examples of distinct words in the generated samples. Specifically we define a distinct word \(w\) for the sensitive attribute value \({a}\) between sensitive attribute values \(a\) and \(\tilde{a}\) as \(\arg\max_{w} p(w|{a}) / p(w|\tilde{a})\). In Table 10, we show some examples between several pairs of sensitive attribute values and the top 10 distinct words.

10 Human Evaluation Details↩︎

We perform a human evaluation for both the sentiment of generated sentences and semantic relevance between prefix and generated sentences. We have 19 human annotators in total, and each annotator labels 50–100 sentences. For all the settings in Section 5.5 (600 sentences in total), each sentence is labeled by 2 annotators. The average Cohen’s kappa is 0.47 for sentiment annotation and 0.45 for semantic relevance annotation, suggesting a moderate inter-annotator agreement.

10.0.0.1 Sentiment.

For sentiment annotation, we follow the annotation guideline of [10] to annotate generated sentences as “Negative”, “Neither positive nor negative”, “Positive”, or “Positive language in part and negative language in part”. We evaluate 100 randomly generated sentences. We assign scores 0, 0.5, 1 for labels “Negative”, “Neutral”, “Positive”, respectively, and we drop the sentences that are labeled as “Positive language in part and negative language in part” by any of the annotators. We then report Spearman’s correlation between automatic sentiment scores and averaged human evaluation scores.

10.0.0.2 Semantic relevance.

For semantic relevance, we present a sensitive token, the associated prefix, and the continuations generated by the language models, to human annotators. We ask the annotators to label the relevance as “Irrelevant / Incoherent”, “Somewhat relevant”, or “Relevant”.  The description of them is as follows:

• Irrelevant / Incoherent: The continuation to the prefix is either incoherent or irrelevant.

• Somewhat relevant: The continuation is not irrelevant to the prefix, but also does not directly pick up relevant semantic aspects.

• Relevant: The attribute is directly relevant to the continuation, which possesses semantic aspects linked to the particular sensitive token in the prefix.

We evaluate 100 randomly generated sentences along with the prefix and sensitive tokens. We assign scores -1, 0, 1 for labels “Irrelavant”, “Somewhat relevant”, “Relevant”, respectively. We then report Spearman’s correlation between automatic semantic similarity scores and averaged human evaluation scores.

10.0.0.3 Individual fairness.

We compute the I.F. score using sentiment scores from human evaluation in the following two settings. Firstly, we evaluate sentences generated by a WMT-19 baseline model and by a WMT-19 sentiment-regularization (Occupation, \(\lambda= 100\)) model. We form two prefixes from the 10th template of Table 3 using tokens “accountant” and “designer”, and sample 50 sentences from each prefix. Secondly, we evaluate sentences generated by a WMT-19 baseline model and by a WMT-19 sentiment-regularization (Country, \(\lambda= 100\)) model. We form two prefixes from the 4th template of Table 2 using tokens “Libya” and “Iceland”, and again sample 50 sentences from each prefix. As previously, each sentence is judged by two people. We report the individual fairness scores between these two attributes.

References↩︎