1 Introduction↩︎

An explanation or rationale², typically consists of a subset of the input that contributes more to the prediction. Extracting faithful explanations is important for studying model behavior [1] and assisting in tasks requiring human decision making, such as clinical text classification [2], misinformation detection [3], [4] and legal text classification [5], [6]. A faithful explanation is one which accurately represents the reasoning behind a model’s prediction [7]

Two popular methods for extracting explanations are through feature attribution approaches (i.e. post-hoc explanation methods) or via inherently faithful classifiers (i.e. select-then-predict models). The first computes the contribution of different parts of the input with respect to a model’s prediction [8]–[10]. The latter consists of using a rationale extractor to identify the most important parts of the input and a rationale classifier, a model trained using as input only the extractor’s rationales [11]–[13].³ Figure 1 illustrates the two approaches with an example.

Currently, these explanation methods have been mostly evaluated on in-domain settings (i.e. the train and test data come from the same distribution). However, when deploying models in real-world applications, inference might be performed on data from a different distribution, i.e. out-of-domain [15], [16]. This can create implications when extracted explanations (either using post-hoc methods or through select-then-predict models) are used for assisting human decision making. Whilst we are aware of the limitations of current state-of-the-art models in out-of-domain predictive performance [17], to the best of our knowledge, how faithful out-of-domain post-hoc explanations are has yet to be explored. Similarly, we are not aware how inherently faithful select-then-predict models generalize in out-of-domain settings.

Inspired by this, we conduct an extensive empirical study to examine the faithfulness of five feature attribution approaches and the generalizability of two select-then-predict models in out-of-domain settings across six dataset pairs. We hypothesize that similar to model predictive performance, post-hoc explanation faithfulness reduces in out-of-domain settings and that select-then-predict performance degrades. Our contributions are as follows:

• To the best of our knowledge, we are the first to assess the faithfulness of post-hoc explanations and performance of select-then-predict models in out-of-domain settings.

• We show that post-hoc explanation sufficiency and comprehensiveness show misleading increases in out-of-domain settings. We argue that they should be evaluated alongside a random baseline as yardstick out-of-domain.

• We demonstrate that select-then-predict classifiers can be used in out-of-domain settings. They lead to comparable predictive performance to models trained on full-text, whilst offering inherent faithfulness.

2 Related Work↩︎

2.1 Rationale Extraction↩︎

Given a model \(\mathcal{M}\), we are interested in explaining why \(\mathcal{M}\) predicted \(\hat{y}\) for a particular instance \(\mathbf{x} \in \mathbf{X}\). An extracted rationale \(\mathcal{R}\), should therefore represent as accurately as possible the most important subset of the input (\(\mathcal{R} \in \mathbf{x}\)) which contributed mostly towards the model’s prediction \(\hat{y}\).

Currently, there are two popular approaches for extracting rationales. The first consists of using feature attribution methods that attribute to the input tokens an importance score (i.e. how important an input token is to a model’s \(\mathcal{M}\) prediction \(\hat{y}\)). We can then form a rationale \(\mathcal{R}\), by selecting the \(K\) most important tokens (independent or contiguous) as indicated by the feature attribution method. The second select-then-predict approach focuses on training inherently faithful classifiers by jointly training two modules, a rationale extractor and a rationale classifier, trained only on rationales produced by the extractor [11]–[13], [18], [19]. Recent studies have used feature attribution approaches as part of the rationale extractor [12], [19], showing improved classifier predictive performance.

2.2 Evaluating Rationale Faithfulness↩︎

Having extracted \(\mathcal{R}\), we need to evaluate the quality of the explanation (i.e. how faithful that explanation is for a model’s prediction). Typically, post-hoc explanations from feature attribution approaches are evaluated using input erasure [20]–[22]. This approach masks segments of the input to observe if the model’s prediction changed. [23] proposed measuring the comprehensiveness and sufficiency of rationales as faithfulness metrics. A comprehensive rationale is one which is influential to a model’s prediction, while a sufficient rationale that which is adequate for a model’s prediction [23]. The term fidelity is also used for jointly referring to comprehensiveness and sufficiency [24]. [24] suggested normalizing these metrics using the predictions of the model with a baseline input (i.e. an all zero embedding vector), to account for baseline model behavior. Select-then-predict models are inherently faithful, as their classification component is trained only on extracted rationales [12]. A good measure for measuring rationale quality is by evaluating the predictive performance of the classifier trained only on the rationales [12], [19]. A higher score entails that the extracted rationales are better when compared to those of a classifier with lower predictive performance.

2.3 Explainability in Out-of-Domain Settings↩︎

Given model \(\mathcal{M}\) trained on an end-task, we typically evaluate its out-of-domain predictive performance on a test-set that does not belong to the same distribution as the data it was trained on [17]. Similarly, the model can also extract explanations \(\mathcal{R}\) for its out-of-domain predictions.

[25] studied whether generating explanations for language inference match human annotations (i.e. plausible explanations). They showed that this is challenging in-domain and becomes more challenging in out-of-domain settings. In a similar direction, [26] and [27] examined model generated explanations in out-of-domain settings and find that explanation plausibility degrades compared to in-domain. [28] proposed a method for detecting model bias towards group identity terms using a post-hoc feature attribution approach. Then, they use them for regularizing models to improve out-of-domain predictive performance. [1] have studied feature attribution approaches for identifying out-of-distribution images. They find that importance allocation in out-of-domain settings is similar to that of an in-domain model and thus cannot be used to detect such images. [29] finally argued that explanations can lead to errors in out-of-distribution settings, as they may latch onto spurious features from the training distribution.

These studies indicate that there is an increasing need for evaluating post-hoc explanation faithfulness and select-then-predict performance in out-of-domain settings. To the best of our knowledge, we are the first to examine these.

3 Extracting Rationales↩︎

3.1 Post-hoc Explanations↩︎

We employ a pre-trained BERT-base and fine-tune it on in-domain training data. We then extract post-hoc rationales for both the in-domain test-set and two out-of-domain test-sets. We compute input importance using five feature scoring methods and a random baseline:

• Random (Rand): Random allocation of token importance.

• Attention (\(\boldsymbol{\alpha}\)): Token importance corresponding to normalized attention scores [12].

• Scaled Attention (\(\boldsymbol{\alpha} \nabla \boldsymbol{\alpha}\)): Attention scores \(\alpha_i\) scaled by their corresponding gradients \(\nabla \alpha_i= \frac{\partial \hat{y}}{\partial \alpha_i}\) [20].

• InputXGrad (\(\mathbf{x} \nabla \mathbf{x}\)): Attributes input importance by multiplying the input with its gradient computed with respect to the predicted class, where \(\nabla x_i = \frac{\partial \hat{y}}{\partial x_i}\) [21], [30].

• ****Integrated Gradients (\(\mathbf{IG}\)): Ranking words by computing the integral of the gradients taken along a straight path from a baseline input (zero embedding vector) to the original input [8].

• ****DeepLift: Ranking words according to the difference between the activation of each neuron and a reference activation (zero embedding vector) [10].

3.2 Select-then-Predict Models↩︎

We use two select-then-predict models:

• HardKuma: An end-to-end trained model, where the rationale extractor uses Hard Kumaraswamy variables to produce a rationale mask z, which the classifier uses to mask the input [11]. Model training takes advantage of reparameterized gradients compared to REINFORCE style training employed by [18] and has shown improved performance [13].

• FRESH: We compute the predictive performance of a classifier trained on rationales extracted with feature attribution metrics (see \(\S\)3.1) using FRESH, following a similar approach to [12]. We extract rationales from an extractor by (1) selecting the top-\(k\) most important tokens (TopK) and (2) selecting the span of length \(k\) with the highest overall importance (Contiguous).

We use BERT-base for the extraction and classification components of FRESH similar to [12]. However, for HardKuma we opt using a bi-LSTM [31] as it provides comparable or improved performance over BERT variants [13], even after hyperparameter tuning.⁴

4 Experimental Setup↩︎

4.1 Datasets↩︎

For evaluating out-of-domain model explanation, we consider the following datasets (see Table 1 and Appendix 8 for details):

4.1.0.1 SST:

Stanford Sentiment Treebank (SST) consists of sentences tagged with sentiment on a 5-point-scale from negative to positive [32]. We remove sentences with neutral sentiment and label the remaining sentences as negative or positive if they have a score lower or higher than 3 respectively [33].

4.1.0.2 IMDB:

The Large Movie Reviews Corpus consists of movie reviews labeled either as positive or negative [33], [34].

4.1.0.3 Yelp:

Yelp polarity review texts. Similar to [35] we construct a binary classification task to predict a polarity label by considering one and two stars as negative, and three and four stars as positive.

4.1.0.4 Amazon Reviews:

We form 3-way classification tasks by predicting the sentiment (negative, neutral, positive) of Amazon product reviews across 3 item categories: (1) Digital Music (AmazDigiMu); (2) Pantry (AmazPantry); and (3) Musical Instruments (AmazInstr) [36].

4.2 Evaluating Out-of-Domain Explanations↩︎

4.2.0.1 Post-hoc Explanations:

We evaluate post-hoc explanations using:

2.2pt

• Normalized Sufficiency (NormSuff) measures the degree to which the extracted rationales are adequate for a model to make a prediction [23]. Following [24], we bind sufficiency between 0 and 1 and use the reverse difference so that higher is better: \[\small \begin{align} \text{Suff}(\mathbf{x}, \hat{y}, \mathcal{R}) = 1 - max(0, p(\hat{y}| \mathbf{x})- p (\hat{y}|\mathcal{R}))\\ \text{NormSuff}(\mathbf{x}, \hat{y}, \mathcal{R}) = \frac{\text{Suff}(\mathbf{x}, \hat{y}, \mathcal{R}) - \text{Suff}(\mathbf{x}, \hat{y}, 0)}{1 - \text{Suff}(\mathbf{x}, \hat{y}, 0)} \end{align}\] where \(\text{Suff}(\mathbf{x}, \hat{y}, 0)\) is the sufficiency of a baseline input (zeroed out sequence) and \(\hat{y}\) the model predicted class using the full text \(\mathbf{x}\) as input.

• Normalized Comprehensiveness (NormComp) measures the influence of a rationale to a prediction [23]. For an explanation to be highly comprehensive, the model’s prediction after masking the rationale should have a large difference compared to the model’s prediction using the full text. Similarly to [24], we bind this metric between 0 and 1 and normalize it: \[\small \begin{align} \text{Comp}(\mathbf{x}, \hat{y}, \mathcal{R}) = max(0, p(\hat{y}| \mathbf{x})- p (\hat{y}|\mathbf{x}_{\backslash\mathcal{R}}))\\ \text{NormComp}(\mathbf{x}, \hat{y}, \mathcal{R}) = \frac{\text{Comp}(\mathbf{x}, \hat{y}, \mathcal{R})}{1 - \text{Suff}(\mathbf{x}, \hat{y}, 0)} \end{align}\]

To measure sufficiency and comprehensiveness across different explanation lengths we compute the “Area Over the Perturbation Curve" (AOPC) following [23]. We therefore compute and report the average normalized sufficiency and comprehensiveness scores when keeping (for sufficiency) or masking (for comprehensiveness) the top 2%, 10%, 20% and 50% of tokens extracted by an importance attribution function.⁵

We omit from our evaluation the Remove-and-Retrain method [22] as it requires model retraining. Whilst this could be applicable for in-domain experiments where retraining is important, in this work we evaluate explanation faithfulness in zero-shot out-of-domain settings.

4.2.0.2 Select-then-Predict Models:

We first train select-then-predict models in-domain and then measure their predictive performance on the in-domain test-set and on two out-of-domain test-sets [12], [13]. Our out-of-domain evaluation is performed without re-training (zero-shot). Similar to full-text trained models, we expect that predictive performance deteriorates out-of-domain. However, we assume that explanations from a select-then-predict model should generalize better in out-of-domain settings when the predictive performance approaches that of the full-text trained model.

We do not conduct human experiments to evaluate explanation faithfulness, since that is only relevant to explanation plausibility (i.e. how intuitive to humans a rationale is [7]) and in practice faithfulness and plausibility do not correlate [21].

5 Results↩︎

5.1 Post-hoc Explanation Faithfulness↩︎

Table 2 presents the normalized comprehensiveness and sufficiency scores for post-hoc explanations on in-domain and out-of-domain test-sets, using five feature attribution methods and a random baseline. For reference, we include the averaged F1 performance across 5 random seeds, of a BERT-base model finetuned on the full text and evaluated in- and out-of-domain (Full-text F1).⁶

In-domain results show that feature attribution performance varies largely across datasets. This is in line with the findings of [21] and [22] when masking rationales (i.e. comprehensiveness). We find the only exception to be \(\alpha \nabla \alpha\), which consistently achieves the highest comprehensiveness and sufficiency scores across all in-domain datasets. For example \(\alpha\nabla\alpha\) evaluated on in-domain AmazDigiMu, results in sufficiency of 0.56 compared to the second best of 0.39 with IG.

Contrary to our expectations, results show that post-hoc explanation sufficiency and comprehensiveness are in many cases higher in out-of-domain test-sets compared to in-domain. For example using DeepLift, comprehensiveness for the in-domain test-set in Yelp (0.16) is lower compared to the out-of-domain test-sets (0.21 for SST and 0.23 for IMDB). This is also observed when measuring sufficiency with \(\alpha\nabla\alpha\), scoring 0.32 when tested in-domain on Yelp and 0.45 for the out-of-domain SST test-set.

Apart from increased sufficiency and comprehensiveness scores in out-of-domain post-hoc explanations, we also observe increased scores obtained by our random baseline. In fact, the random baseline outperforms several feature attribution approaches in certain cases in out-of-domain settings . As an example, consider the case where the model has been trained on AmazInstr and tested on AmazPantry. Our random baseline achieves a comprehensiveness score of 0.27 while \(\alpha\), DeepLift, \(x\nabla x\) perform similarly or lower (0.22, 0.25 and 0.27 respectively). Similarly, using a model trained on Yelp and tested on SST, the random baseline produces equally sufficient rationales to \(x \nabla x\) and IG, with all of them achieving 0.41 normalized sufficiency. A glaring exception to this pattern is \(\alpha \nabla \alpha\), which consistently outperforms both the random baseline and all other feature attribution approaches in in- and out-of-domain settings, suggesting that it produces the more faithful explanations. For example with out-of-domain AmazPantry test data, using a model trained on AmazInstr results in sufficiency scores of 0.39 with \(\alpha\nabla\alpha\). This is a 0.15 point increase compared to the second best (\(x\nabla x\) with 0.24).

We recommend considering a feature attribution for producing faithful explanations out-of-domain, if it only scores above a baseline random attribution. We suggest that the higher the deviation from the random baseline, the more faithful an explanation is.

5.2 Select-then-predict Model Performance↩︎

5.2.0.1 HardKuma:

Table 3 presents the F1-macro performance of HardKuma models [11] and the average rationale lengths (the ratio of the selected tokens compared to the length of the entire sequence) selected by the model. For reference, we also include the predictive performance of a full-text trained bi-LSTM. Results are averaged across 5 runs including standard deviations in brackets.

As expected, predictive performance of HardKuma models degrades when evaluated on out-of-domain data. Surprisingly, though, we find that their performance is not significantly different (t-test; p-value \(>\) 0.05) to that of the full-text LSTM in 9 out of the 12 out-of-domain dataset pairs. For example, by evaluating the out-of-domain performance of a HardKuma model trained on AmazDigiMu on the AmazPantry test-set, we record on average a score of 54.3 F1 compared to 55.3 with an LSTM classifier trained on full text. We also observe that HardKuma models trained on SST and IMDB generalize comparably to models trained on full-text when evaluated on Yelp, however the opposite does not apply. Our assumption is that HardKuma models trained on Yelp, learn more domain-specific information due to the large training corpus (when compared to training on IMDB and SST) so they fail to generalize well out-of-domain.

Results also show, that the length of rationales selected by HardKuma models depend on the source domain, i.e. training HardKuma on a dataset which favors shorter rationales, leads to also selecting shorter rationales out-of-domain. For example, in-domain test-set explanation lengths are on average 56.8% of the full-text input length for SST. In comparison, training a model on Yelp and evaluating on SST results in rationale lengths of 14.1%. We observe that in certain cases, HardKuma models maintain the number of words, not the ratio to the sequence in out-of-domain settings. For example, in-domain Yelp test-set rationales are about 11 tokens long that is the similar to the length selected when evaluating on IMDB using a model trained on Yelp. This is also observed where in-domain AmazInstr test-set rationales are on average 5 tokens long, which is the same rationale length when evaluating on AmazDigiMu using a model trained on AmazInstr.

In general, our findings show that in the majority of cases, using HardKuma in out-of-domain data results to comparable performance with their full-text model counterparts. This suggests that HardKuma models can be used in out-of-domain settings, without significant sacrifices in predictive performance whilst also offering faithful rationales.

5.2.0.2 FRESH:

Table 4 shows the averaged F1-macro performance across 5 random seeds for FRESH classifiers on in- and out-of-domain using TopK rationales.⁷. We also include the a priori defined rationale length in parentheses and the predictive performance of the Full-Text model for reference.⁸ We first observe that in-domain predictive performance varies across feature attribution approaches with attention-based metrics (\(\alpha\nabla\alpha\), \(\alpha\)) outperforming the gradient-based ones (\(x\nabla x\), IG), largely agreeing with [12]. We also find that \(\alpha \nabla \alpha\) and DeepLift are the feature attribution approaches that lead to the highest predictive performance across all datasets.

As we initially hypothesized, performance of FRESH generally degrades when testing on out-of-domain data similarly to the behavior of models trained using the full text. The only exceptions are when using \(x \nabla x\) and IG in IMDB. We argue that this is due to these feature attribution methods not being able to identify the appropriate tokens relevant to the task using a rationale length 2% of the original input. Increasing the rationale length to 20% (SST) and 10% (Yelp) also increases the performance. Results also suggest that \(\alpha\nabla\alpha\) and DeepLift outperform the rest of the feature attributions, with \(\alpha \nabla \alpha\) being the best performing one in the majority of cases. In fact when using \(\alpha \nabla \alpha\) or DeepLift, the out-of-domain performance of FRESH is not significantly different to that of models trained on full text (t-test; p-value \(>\) 0.05) in 5 cases. For example, a FRESH model trained on AmazPantry and evaluated on AmazInstr records 63.6 F1 macro (using DeepLift) compared to 64.5 obtained by a full-text model. However, this does not apply to the other feature attribution methods (\(\alpha\); \(x\nabla x\); IG).

To better understand this behavior, we conduct a correlation analysis between the importance rankings using any single feature attribution from (1) a model trained on the same domain with the evaluation data; and (2) a model trained on a different domain (out-of-domain trained model). High correlations suggest that if a feature attribution from an out-of-domain trained model produces similar importance distributions with that of an in-domain model, it will also lead to high predictive performance out-of-domain. Contrary to our initial assumption we found that the lower the correlation, the higher the predictive performance with FRESH. Results show low correlations when using \(\alpha\nabla\alpha\) and DeepLift (highest FRESH performance). Surprisingly, IG and \(x\nabla x\) (lowest FRESH performance) showed consistently strong correlations across all dataset pairs. Thus, we conclude that lower correlation scores indicate lower attachment to spurious correlations learned during training. We expand our discussion and show results for the correlation analysis in Appendix 12.

Our findings therefore suggest that using FRESH in out-of-domain settings, can result to comparable performance with a model trained on full-text. However this highly depends on the choice of the feature attribution method.

5.2.0.3 HardKuma vs. FRESH:

We observe that HardKuma models are not significantly different compared to models trained on the full text in out-of-domain settings in more cases, when compared to FRESH (9 out of 12 and 5 out of 12 respectively). However, FRESH with \(\alpha\nabla\alpha\) or DeepLift records higher predictive performance compared to HardKuma models (both in- and out-of-domain) in all cases. We attribute this to the underlying model architectures, as FRESH uses BERT and HardKuma a bi-LSTM. As we discussed in §3.2, we attempted using BERT for HardKuma models in the extractor and classifier similar to [12]. However, the performance of HardKuma with BERT is at most comparable to when using a bi-LSTM similar to findings of [13].

5.3 Correlation between Post-hoc Explanation Faithfulness and FRESH Performance↩︎

We hypothesize that a feature attribution with high scores for sufficiency and comprehensiveness, should extract rationales that result in high FRESH predictive performance. We expect that if our hypothesis is valid, faithfulness scores can serve as early indicators of FRESH performance, both on in-domain and out-of-domain settings.

Table 5 shows the Spearman’s ranking correlation (\(\rho\)) between FRESH F1 performance (see Table 4) and comprehensiveness and sufficiency (see Table 2). Correlation is computed using all feature scoring methods for each dataset pair. Results show that only 4 cases achieve statistically significant correlations (p-value \(<\) 0.05) with only 3 out-of-domain and mostly between sufficiency and FRESH performance. We do not observe high correlations with comprehensiveness which is expected, as comprehensiveness evaluated the rationale’s influence towards a model’s prediction. Our findings refute our initial hypothesis and suggest that there is no clear correlation across all cases, between post-hoc explanation faithfulness and FRESH predictive performance. Therefore, sufficiency and comprehensiveness scores cannot be used as early indicators of FRESH predictive performance.

6 Qualitative Analysis↩︎

Table 6 presents examples from a qualitative analysis we performed, aimed at better understanding out-of-domain post-hoc explanations. Rows with highlighted text in blue are from a model trained in the same domain as the presented example (ID), whilst those with highlighted text in red are from models trained on a different domain. Importance scores are computed using scaled attention (\(\nabla \alpha \nabla\)).

In Example (1), we observe that models trained on two closely related tasks (AmazInstr and AmazDigiMu) place more importance to the phrase “sound good”. On the contrary, the model trained on AmazPantry which has not encountered such phrases during training, mostly focuses on “Work great”. This is expected as the term “sound” is not typical of pantry reviews. Similarly, in Example (2) from the AmazPantry dataset, the in-domain model focuses on a domain-specific word “delicious”. On the contrary, the two models trained on music-related tasks focus on more generic terms such as “good” and “would recommend”. In Example (3) the model trained on Yelp focuses mostly on the word “behavior”, a term we consider more relevant to restaurant reviews rather than movie reviews. In comparison, the other models which are both trained on movie reviews focus both on the term “funny”. In Example (4), again the two movie-review models focus on more generic terms (i.e. “amazing”) compared to “must taste” that the model trained in-domain (i.e. Yelp) identifies as important.

Overall, results show that rationales from models applied to a different domain (other than that they were trained for), comprise of terms that are mostly present within the domain they were trained for. This can partly explain the performance of out-of-domain FRESH classifiers. Similar to [1], our assumption is that a model’s inability to generalize to other domains, is based on the model latching on to specific features from the training dataset.

7 Conclusion↩︎

We conducted an extensive empirical study to assess the faithfulness of post-hoc explanations (i.e. using feature attribution approaches) and performance of select-then-predict (i.e. inherently faithful) models in out-of-domain settings. Our findings highlight, that using sufficiency and comprehensiveness to evaluate post-hoc explanation faithfulness out-of-domain can be misleading. To address this issue, we suggest comparing faithfulness of post-hoc explanations to a random attribution baseline for a more robust evaluation. We also show that select-then-predict models, which are inherently faithful, perform surprisingly well in out-of-domain settings. Despite performance degradation, in many cases their performance is comparable to those of full-text trained models. In future work, we aim to explore methods for improving the evaluation of faithfulness for out-of-domain post-hoc explanations.

Acknowledgements↩︎

We would like to thank Katerina Margatina and Tulika Bose for their insightful feedback in a preliminary version of this paper. GC and NA are supported by EPSRC grant EP/V055712/1, part of the European Commission CHIST-ERA programme, call 2019 XAI: Explainable Machine Learning-based Artificial Intelligence.

8 Dataset Characteristics↩︎

Table 7 presents extended data characteristics for all datasets. We present information across the three data splits, including: (1) The average sequence length; (2) The number of documents in each split and (3) the number of documents under each label.

Our dataset selection was highly motivated for also examining the differences when we have gradual shifts in-domain. For example for the triplet SST - IMDB - YELP, two datasets are closely associated (SST, IMDB) as they are movie reviews, whilst Yelp is a task for classifying restaurant reviews. Similarly, AmazDigiMu and AmazInstr share similar characteristics, as they are reviews about items related to music. On the contrary, AmazPantry consists of reviews about pantry items. This is also the primary reason why we focused on text classification tasks, as it is easier to control for the output and other parameters, whilst allowing for control over the task it-self.

9 Models and Hyper-parameters↩︎

9.0.0.1 For feature attributions:

We use BERT-base with pre-trained weights from the Huggingface library [37]. We use the AdamW optimizer [38] with an initial learning rate of \(1e-5\) for fine-tuning BERT and \(1e-4\) for the fully-connected classification layer. We train our models for 3 epochs using a linear scheduler with 10% of the data in the first epoch as warm-up. We also use a grad-norm of 1 and select the model with the lowest loss on the development set. All models are trained across 5 random seeds and we report the average and standard deviation. We present their test-set performance in Table 8 and their development set performance in Table 9.

9.0.0.2 For FRESH:

For the rationale extractor, we use the same model for extracting rationales from feature attributions. For the classifier (trained only on the extracted rationales), we also use BERT-base with the same optimizer configuration and scheduler warm-up steps. We also use a grad-norm of 1 and select the model with the lowest loss on the development set. We train across 5 random seeds for 5 epochs.

In Table 8 we present full-text BERT-base F1-macro scores averaged across 5 random seeds with standard deviations included in the brackets. Additionally, we present the mean Expected Calibration Error (ECE) scores. Finally, in Table 9 we present the in-domain F1-macro performance and loss on the development set.

9.0.0.3 For HardKuma:

We use the 300-dimensional pre-trained GloVe embeddings from the 840B release [39] as word representations and keep them frozen during training. The rationale extractor (which generates the rationale mask \(z\)) is a 200-d bi-directional LSTM layer (bi-LSTM) [31] similar to [11], [13]. We use the Adam optimizer [40] for all models with a learning rate between \(1e-5\) and \(1e-4\) and a weight decay of \(1e-5\). We also enforce a grad-norm of 5 and train for 20 epochs across 5 random seeds. Similar to [13] we select the model with the highest F1-macro score on the development set and find that tuning the Lagrangian relaxation algorithm parameters beneficial to model predictive performance. We also attempted training HardKuma models with BERT-base, similar to [12], however we found performance to be at best comparable with our LSTM variant, as in [13], even after hyperparameter tuning.

All experiments are run on a single NVIDIA Tesla V100 GPU.

10 HardKuma - Extended↩︎

In Table 10 we present for reference the performance of a 200-dimensional bi-LSTM classifier trained on full-text. We train the full-text LSTM for 20 epochs across 5 random seeds and select the model with the highest F1-macro performance on the development set. We use the Adam optimizer with a learning rate of \(1e-3\) and \(1e-5\) weight decay. We report predictive performance and ECE scores on the test-set. In Table 11 we include HardKuma performance with standard deviations, and expected calibration error (ECE), across five runs.

11 FRESH - Extended↩︎

Tables 12 and 13 presents FRESH F1 macro performance and Expected Calibration Error (ECE) for classifiers trained on TopK and Contiguous rationales respectively, with standard deviation in brackets. We include the a priori defined rationale length in the brackets (.%) and for reference, the ID performance of the Full-Text model (as also seen in Table 8).

Comparing with FRESH performance with Contiguous rationales rather than TopK (see Table 12), we first observe that performance degrades for most feature attribution methods. These findings are largely in agreement with those of [12]. However, \(x\nabla x\) and IG, which perform poorly with TopK, record surprisingly better scores with Contiguous type rationales. For example, in-domain performance with IG becomes comparable with \(\alpha\nabla\alpha\) in in-domain IMDB (83.2 with \(\alpha\nabla\alpha\) and 82.5 with IG). This is in sharp contrast with TopK, where IG recorded an F1 score of only 59.7, compared to 87.9 of \(\alpha\nabla\alpha\).

These findings also hold in out-of-domain settings, where \(\alpha\nabla\alpha\), \(\alpha\) and DeepLift result in poorer FRESH performance with Contiguous type rationales, compared to TopK. However, IG and in many cases \(x\nabla x\) improves. For example with TopK rationales, evaluating on Yelp using IG from a model trained on IMDB, results on an F1-score of 69.1. On the contrary, with Contiguous rationales and the same set-up, IG results in FRESH performance of 87.0.

Our findings lead us to assume that, the rationale type has a large impact on FRESH performance, both in-domain and on out-of-domain settings. Certain feature attribution methods benefit from one type of rationales (e.g. DeepLift with TopK), whilst others from another (e.g. IG with Contiguous).

12 Extended Analysis↩︎

12.1 Correlation of Rankings↩︎

We examine why \(x \nabla x\) and IG, do not perform as well as DeepLift and \(\alpha\nabla\alpha\) when using FRESH. We therefore conduct a study to gain better understand this. We first fix the domain of the data we evaluate on and then compute the correlation between importance rankings using any single feature attribution from: (1) a model trained on the same domain with the evaluation data and (2) a model from trained on a different distribution (out-of-domain trained model). High correlations suggest that a feature attribution from an out-of-domain trained model, produce similar importance distributions with that of an in-domain model (i.e. both attend to similar tokens to make a prediction). Therefore, we assume that this will lead to high predictive performance out-of-domain. In Figure 2 we show Spearman’s ranking correlation across dataset pairs, between a model trained on the same distribution as the evaluation data (ID) and an out-of-domain trained model (OOD), such that (ID <-> OOD).

As expected, the random baseline produced almost no correlation between models. An interesting observation is that two of the gradient-based methods (\(x \nabla x\) and IG) produce strongly correlated rankings. This suggests that these two metrics produce generalizable rankings irrespective of the domain shift, when comparing to the remainder of the feature attribution approaches. Surprisingly, Deeplift importance rankings exhibit almost low to no correlation betweenen them, despite being also gradient-based. We hypothesize that this happens because DeepLift considers a baseline input to compute its importance distribution, which highly depends on the model and as such is de-facto normalized and perhaps generalizes better.

12.1.0.1 \(\boldsymbol{\alpha}\) for out-of-domain detection?:

An interesting case is that of \(\alpha\), where we observe moderate to strong correlations across all test-cases. What is more evident, is that in the OOD tuples we considered, it appears that stronger correlations appear where the OOD task and the ID task are closer together. For example in the case of SST and IMDB (both sentiment analysis tasks for movie reviews), \(\alpha\) produces a strong correlation (0.68). This contrasts the moderate correlation of 0.58 between SST and Yelp, which is for restaurant reviews. This is also evident in the case of AmazDigiMu and AmazInstr, where both tasks are for review classification, but for musical related purchases. They both score strong correlations between them and moderate correlations with reviews for pantry purchases (AmazPantry). This observation might suggest, that using these correlation metrics with \(\alpha\) might be an indicator of the degree of task-domain-shift. Our observation is also supported by the findings of [1], who show that feature attributions are good indicators of detecting spurious correlation signals in computer vision tasks.Considering \(\alpha \nabla \alpha\) we observe a wide range of correlations, ranging from low in the AmazInstr-AmazDigiMu pair to strong in the AmazPantry-AmazInstr pair, which we cannot interpret as something meaningful.

12.1.0.2 Correlation values and FRESH:

We first observe that the lowest correlated feature attributions \(\alpha \nabla \alpha\) and DeepLift perform the better on FRESH, followed by \(\alpha\) which displays moderate correlations and at the end of the spectrum the two gradient-based methods which display high correlations. Contrary to our initial assumption, this suggests that the attributions which generalize better (i.e. return rationales that result in higher FRESH performance) are those which exhibit low to no correlations.

12.1.0.3 Agreement at different rationale lengths:

As the correlation analysis considers the entire length of the sequence, we now examine a scenario where we have a priori defined rationale lengths. Similarly to the correlation analysis, we now compute the agreement in tokens between ID feature attribution rankings to those of an OOD trained model. In Tables 14, 15 and 16 we therefore show the token agreement between in-domain and out-of-domain post-hoc explanations (on the same data) for 2%, 10% and 20% rationale lengths.

Our findings show that across all rationale lengths, results largely agree with the correlation analysis. The two gradient-based methods exhibit higher agreement than the remainder, with \(\alpha\nabla\alpha\) and DeepLift recording the lowest agreements. Surprisingly, the poorest performers on out-of-domain FRESH record the highest agreement in tokens with in-domain models. Whilst this suggests that they generalize better, we believe that the inhibiting factor to their performance is their limited in-domain capabilities (i.e. they record the lowest in-domain FRESH performance with TopK).

13 Post-hoc Explanation Faithfulness - Extended↩︎

In Tables 17, 18 and 19, we present post-hoc explanation sufficiency and comprehensiveness scores at 2%, 10% and 20% rationale lengths.

References↩︎