0.0.1 Introduction↩︎

Recent advancements in Natural Language Processing (NLP), particularly with the introduction of Large Language Models (LLMs), have dramatically enhanced model performance. These models are now capable of executing a wide array of tasks and have been adopted across various domains and research fields [1]–[3]. Their applications extend beyond the NLP community, and they are widely used by the general public [4]–[6]. However, these black-box models are complex and opaque [7]–[9]. While performance has advanced, this comes at the cost of understanding their underlying mechanisms [10]–[12].

The ability to explain decisions is particularly crucial, given that NLP models, especially LLMs, significantly influence individual decision-making [13], [14], society [15], [16], the job market [17], and academic research [18], [19]. Moreover, model interpretability and analysis are utilized for scientific insights and discoveries [20]–[24].

Unsurprisingly, research on model interpretability and analysis has become one of the most prolific areas within the NLP community and beyond, yielding thousands of publications in recent years, as illustrated in Figure 1. Consequently, many technical NLP model interpretability and analysis surveys have emerged, reviewing hundreds of methods [9], [25]–[31]. In this paper, we aim to bridge a gap in the existing literature and discuss model interpretability from the stakeholders’ perspective. Our goals are to broaden the NLP community’s point of view on the application of interpretability methods in various fields and to promote the design and development of methods that align with the objectives, expectations, and requirements of various stakeholders.

We will explore three key questions: why do we need interpretability (§0.0.2), what are we interpreting (§0.0.3), and how are we interpreting (§0.0.4)? This approach allows us to examine common interpretability paradigms (Table ¿tbl:tab:categorization?), their properties and their applications by different stakeholders.

We start by presenting four perspectives on interpretability and their relevant stakeholders in §0.0.2. Next, in §0.0.3, we address a pressing issue in the literature: the lack of consensus on the definition of interpretability. We examine various definitions within and outside the NLP community and propose a broad definition: Extracting insights into a mechanism of the NLP system and communicating them to the stakeholders in understandable terms.

In §0.0.4, we propose six properties of interpretability methods and discuss the relevance of each property to different stakeholders. For example, the scope property distinguishes between local and global explanations. If the stakeholder is a physician, a local explanation that clarifies the prediction for an individual patient is preferred. Conversely, a global explanation is more suitable for a scientist, as it facilitates understanding broader phenomena.

We survey in §0.0.5 seven prevalent interpretability paradigms, explain which properties characterize each (see Table ¿tbl:tab:categorization?), and discuss their applications by different stakeholders. Throughout the survey, we review over 200 works.

Following that, in §0.0.6 we aim to understand how the paradigms and their properties are reflected in practice by analyzing trends over the years and across different research fields. To this end, we retrieved over 14,000 papers using the Semantic Scholar API and employed an LLM to select only relevant ones, resulting in 2,000 papers. Furthermore, we utilized the LLM to annotate papers with their interpretability paradigm and properties.¹ Importantly, the LLM annotation is in strong agreement (over 90%) with human expert annotation. To the best of our knowledge, this is the first successful application of an LLM for such a task.

Below, we summarise our main findings:

1. Within the NLP community, interpretability paradigm trends have remained stable over the decade. However, the introduction of LLMs in the past two years has prompted a notable shift.

2. Outside the NLP community, our main claim gains support: different stakeholders have varying needs, reflected in significant differences between research fields in terms of both the paradigms and their properties.

3. Comparing NLP developers to non-developers reveals that the latter group is less interested in understanding internal model components.

4. Non-developers opt for popular methods not originally developed within the NLP community, such as SHAP and LIME, likely due to their user-friendly and easy-to-apply software.

5. LLMs have shifted the trends in interpretability research: not only has the number of published papers doubled, but there has also been a substantial increase in the use of natural language explanations. These explanations are utilized in nearly half of the papers outside the NLP field.

We hope this first-of-its-kind paper, which reviews NLP interpretability through the stakeholders’ perspective and rigorously analyzes trends within and outside the NLP field, will pave the way for improved design, development, and application of these essential methods. To further this aim, we outline in §0.0.7 practical steps that NLP researchers can undertake to promote the adoption of interpretability methods in other disciplines.

0.0.2 Why Do We Need Interpretability?↩︎

Understanding why interpretability is necessary provides a solid framework for discussing, assessing, and enhancing interpretability methods, ensuring they meet practical objectives and expectations. So, when and why do we need interpretability? We gather ideas from other surveys [32]–[34] and propose a decomposition of the need for interpretability into four perspectives (see Figure 2): algorithmic, business, scientific, and social.

The four perspectives define the objective and use case of the interpretability method. Clearly, there can be overlaps between the different perspectives, particularly with the algorithmic one. For example, using interpretability to build a better model (algorithmic perspective) might coincide with making it fairer (social perspective) or one that promotes more informed business decisions (business perspective). Similarly, promoting social values through interpretability (social perspective) can build customer trust (business perspective).

Besides the objectives of the interpretability method, another key consideration is the stakeholders–the audience to whom the explanation is aimed and communicated. Accordingly, when designing the interpretability method, we should consider not only the objective (and the usage) of the explanation but also the stakeholders, their level of expertise, and their familiarity with NLP models. By identifying different stakeholders’ specific requirements and concerns, we can foster practical interpretability methods that align with their expectations [35]. We next discuss the four perspectives and the main stakeholders (in bold):

1. The Algorithmic Perspective: emphasizes using interpretability for building better models. Thus, the stakeholders are developers. Interpretability allows for an open-ended, more rigorous evaluation beyond standard metrics [36]–[38]. It helps uncover why the model fails, offering insights into identifying and rectifying mistakes [39] and improving its generalization. For instance, [40] use saliency maps, and [41] use counterfactual explanations for modifying the training objective and improving model robustness. [42] study the source of hallucinations in LLMs by curating a diagnostic set that utilizes the model’s pre-existing knowledge. Moreover, by understanding how the model works, we can intervene and modify it or design better models from the start (e.g., reverse engineering) [43], [44]. For example, [45] use attributions to locate knowledge neurons, modify them, and edit factual knowledge of the NLP model. The algorithmic perspective underscores interpretability for debugging, refining model deployment, and forecasting progress.

2. The Business Perspective: focuses on leveraging interpretability across various sectors to enhance informed decision-making, legal compliance, and user trust. Models often aid decision-making at the business level (e.g., sentiment analysis for market research [46]) and at the user level (e.g., LLMs assisting physicians in patient diagnostics [47]). In both cases, interpretability aids ensure well-grounded and trustworthy decisions [48].

Legal compliance includes cases where interpretability is explicitly the regulator’s requirement, such as the GDPR’s “right to explanation” [49] and the Algorithmic Accountability Act proposed in the US [50], or cases where interpretability is instrumental in ensuring the business adheres to legal standards, thereby reducing the risk of legal penalties. For instance, when NLP models are used to process credit applications [51]–[53], they must comply with the Equal Credit Opportunity Act (ECOA), which prohibits discrimination.

Finally, interpretability enhances transparency, fostering trust and goodwill. When end-users understand how decisions are made, they are more likely to trust AI systems, improving business reputation. For example, Facebook’s ‘Why am I seeing this ad’ tool is designed specifically to provide more transparency and build trust [54]. The link between interpretability and trust is well-documented [55]–[57].

3. The Scientific Perspective: Language is tightly connected to human behavior, cognition, and communication. Researchers and scientists from various disciplines, such as social science [58], [59], psychology [60], psychiatry [61], psycho-linguistics [62], health [63], [64], neuroscience [65], [66], finance [67], behavioral economics [68], [69], political science [70], and beyond, are now turning to NLP to model scientific phenomena, decode complex patterns and derive meaningful insights about humanity. Science is all about gaining knowledge, and interpretability enables us to understand the underlying mechanisms and patterns the NLP model identifies, facilitating deeper comprehension and advancing scientific discoveries [20]. For example, by interpreting the representations of Facebook posts extracted by an NLP model, [24] identify a new risk factor for suicide ideation: boredom.

4. The Social Perspective: addresses the broader impact of NLP systems on society, fairness, the ethical implications of its use and AI safety. Since NLP models are optimized using data that may contain human biases and prejudices [71], [72], interpretability is crucial for understanding the rationale behind the models, ensuring they serve what they are designed for rather than reflecting their training data [73]. Interpretability can confirm the model predictions are just and equitable [74]–[76], foster public trust, promote ethical practices, and prevent misuse or other harmful consequences [29], [77]. Accordingly, interpretability helps society embrace the model or reject it, depending on how well it aligns with expected social values.

0.0.3 Definitions↩︎

width=0.975

0.0.3.1 What is an Interpretability Method?

In the AI literature, the terms interpretability and explainability are often subjects of debate, and there is no clear consensus on their definitions [78]–[80]. While these terms are used interchangeably in much of the NLP literature [10], [30], [81], many papers in the XAI literature distinguish between the two [82], [83], see our note in §0.0.4.2.1. Moreover, within this broad umbrella of model interpretability, the NLP literature also discusses model analysis [25], [31].

For the purposes of this paper, we embrace a broad perspective and define both interpretability and explainability methods as:

We justify this broad definition, which explicitly encompasses model analysis, because our paper focuses on the perspective of stakeholders for whom, to some extent, analysis alone may suffice to achieve their objectives. For instance, a regulator might only need to ensure that model performance does not significantly differ between two subpopulations. This does not necessarily demand that the interpretation elucidate the precise cause of each decision. Moreover, our broad definition does not restrict the interpretability method to explain the full system, but rather, only a mechanism within it. For example, developers might want to gain insights about specific components of the system to improve or modify their functionality.

0.0.3.2 What is an Explanation?

[84] and [79] rightfully emphasize that interpretability should not be confused with an explanation. [84] distinguishes between (causal) attributions and (causal) explanations. Attribution involves extracting relationships and causes, but it is not necessarily an explanation, even if a person could use attributions to derive their own explanation. Explanation also involves selecting, contextualizing, and presenting causes and relationships to the stakeholders. Thus, explanations are about communicating insights in a way that aligns with human cognitive biases and social expectations. In some sense, the output of interpretability methods is an attribution.

Most existing work in the NLP literature is on how we extract insights and not about communicating them. Since this paper is directed at this community rather than the HCI or XAI communities, we mostly focus on interpretability methods. However, to begin the discussion about the what and how parts (see the paragraph below the following definition), we must first define an explanation. This is because the what and how are derived from the why – the stakeholders, and clearly, they are part of an explanation. To this end, we have gathered common (though not formal) definitions from seminal works in the literature [10], [33], [78], [79], [83], [85], and propose the following definition:

We define and elaborate on the mechanism and understandable terms aspects of the above definition in Appendix §1.0.2. These two aspects are related to the what part: what mechanism are we interpreting, what terms are we using to describe its states, and what is the scope of the explanation?

Conversely, the extracting and communicating aspects are related to the how part: how are we interpreting and extracting insights and how are we presenting and communicating insights? Note that extracting is essentially the interpretability method defined in §0.0.3.1.

To summarize, an interpretability (or explainability or analysis) method extracts insights from a model, whereas an explanation involves communicating these insights to stakeholders. This process includes filtering and selecting relevant insights, processing them, and presenting them in an understandable terms. For example, computing SHAP values is an interpretability method, while visualizing these values using the SHAP Python package² and providing guidance on interpreting these visualizations constitute an explanation.

0.0.4 Properties and Categorization↩︎

In this section, we propose and describe properties that answer the what and how questions derived from our interpretability definitions. We aim to provide the stakeholders’ perspective, deepening our understanding of how these properties align with their objectives and requirements. We begin by discussing the what aspect properties in §0.0.4.1, followed by the how aspect properties in §0.0.4.2.

In Table ¿tbl:tab:categorization?, we present a categorization of interpretability paradigms based on the properties. For the reader’s convenience, we briefly describe each property in Appendix §1.0.1.

0.0.4.1 What Properties

0.0.4.1.1 The Explained Mechanism

In Appendix §1.0.2.1, we formally define what a mechanism is. Broadly, a mechanism can refer to the entire NLP system or a specific process or component within it. To better categorize interpretability methods, we distinguish four types of mechanisms. While most methods explain the whole system (an input-output mechanism), other methods explain input representations or hidden states (an input-internal mechanism). Another mechanism type focuses on explaining the functionality of internal components such as neurons, attention heads, circuits, and more (an internal-internal mechanism).

In addition, the mechanism property covers any abstraction of the mechanism states (see §1.0.2.2), for example, explaining the impact of concepts conveyed in the text input instead of explaining long and complex raw input. In this case, which is thoroughly discussed in the next subsection §0.0.4.1.2, the explained mechanism is concept-output.

The choice of which mechanism to explain depends on the why: the objective of the explanation and the stakeholder’s needs. Stakeholders mostly utilize methods that explain the full system (an input-output mechanism). However, many are interested in other mechanisms. For example, developers aim to understand the functionality of internal components such as neurons or layers to modify and edit factual knowledge encoded by them [86]. Scientists might explore the representational space, for example, neuroscientists examine the brain by aligning model representations with brain activity [87], and social scientists cluster representations to monitor opinions, such as attitudes towards COVID-19 vaccines [88].

0.0.4.1.2 Raw Input or Abstracted Input

A common interpretability paradigm is feature attributions, where each input feature is assigned an importance score reflecting its relevance to the model prediction. In computer vision, the raw inputs consist of pixels, and feature attributions effectively highlight relevant areas that can be immediately and intuitively grasped [89], [90]. In contrast, explaining the raw input in NLP, often a lengthy and complex text, presents distinct challenges. For end-users, assigning scores to each token can be overwhelming as the cognitive load increases with the text length.

Instead, simplifying the system by abstracting the input to concepts or a summary, thus reducing the number of features explained, could lead to a better mental model of the system [91]. For example, concept counterfactual methods (see §0.0.5.5, [92] and [93]) change a specific concept conveyed in the text. By contrasting the counterfactual predictions with the original prediction, we can gain digestible insights into how the concept impacts the prediction (a concept-output mechanism). Moreover, due to the vast space of textual data, providing global explanations by explaining the raw input is challenging. In contrast, concept-level explanations naturally support global explanations.

0.0.4.1.3 Scope: Local or Global

This categorization is based on the scope of the explanation: local or global. A local explanation describes the mechanism for an individual instance. For example, feature attributions and attention visualizations (§0.0.5.1). Conversely, global explanations describe the mechanism for the entire data distribution, for example, probing (§0.0.5.2) and mechanistic interpretability (§0.0.5.3). Many local explanations can be generalized into global ones. For example, concept counterfactuals (§0.0.5.5) measure the causal effect of a concept on the prediction of an individual instance. A global average causal effect estimation can be derived by iterating the entire dataset and applying adjustments [93].

The choice of scope, local or global, depends on the objectives of the explanation and its stakeholders. For instance, developers debugging edge cases may prefer local explanations. Conversely, when aiming to improve the functionality of model components, developers might lean towards global explanations offered by mechanistic interpretability. End-users, such as clients and customers, require local explanations since they are concerned with decisions directly affecting them; this local need is also reinforced by the “right to explanation” [49]. Similarly, physicians using NLP systems must rely on local explanations. On the other hand, business decision-makers and scientists generally favor global explanations, which help identify broader trends and underlying patterns. From a social perspective, global explanations hold more significance. However, accumulating local evidence can progressively provide insights into global tendencies.

0.0.4.2 How Properties

0.0.4.2.1 Time: Post-hoc or Intrinsic

This property distinguishes between methods based on the time the explanation is formed. Post-hoc methods produce explanations after the prediction and are typically external to the explained model. Conversely, intrinsic methods are built-in; the explanation is generated during the prediction, and the model relies on it. Intrinsic methods include, for example, natural language explanations (§0.0.5.6) or self-explaining models (§0.0.4.1.2) such as concept bottleneck models, which train a deep neural network to extract human-interpretable features, which are then used in a classic transparent model (e.g., logistic regression).

In the XAI literature, this distinction also defines the difference between explainable AI (post-hoc) and interpretable AI (intrinsic) [82], [83]. However, interpretable AI generally refers to transparent models (see [79]), while in our categorization, intrinsic models can be opaque to some extent: in self-explaining methods, an opaque neural network extracts human-interpretable features; similarly, in natural language explanations, the explanation is generated by an opaque neural network. Intrinsic methods aim to produce more faithful and understandable insights and could better serve all stakeholders. However, they may also limit model architecture and thus could potentially degrade system performance, although this is not always the case (see [22] for an example).

0.0.4.2.2 Access: Model Specific or Agnostic

This property distinguishes interpretability methods based on their access to the explained model. Model-agnostic methods do not assume any specific knowledge about the model and can only access its inputs and outputs. For example, diagnostic sets (§0.0.5.4), perturbation-based attributions (§0.0.5.1), or some counterfactual methods (§0.0.5.5). The latter two modify only the input and measure its impact on model prediction. On the other hand, model-specific methods require access to the explained model during the training time of the interpretability method. They can also access its internal components and representations. Hence, while a model-specific method can be applied only to one explained model, the same model-agnostic method can be applied to any model simultaneously.

Unlike model-specific methods, model-agnostic methods can not explain internal mechanisms. However, they can still be extremely valuable for some stakeholders. From an algorithmic perspective, they are useful during model selection and deployment. For example, developers juggling multiple models can easily rank them based on their vulnerability to confounding biases, such as gender bias. Moreover, regulators would prefer model-agnostic methods, utilizing a dedicated diagnostic set or a pool of counterfactuals to verify whether the model meets the required standards.

0.0.4.2.3 Presenting Insights

The presentation of insights extracted by the interpretability method falls under the communicating aspect of the explanation definition in §0.0.3.2. There is extensive research in the XAI field that explores this aspect and examines how the presentation affects different stakeholders [94]–[98]. Even though we do not delve into the stakeholder perspective, we still discuss this property since not all methods support every form of presentation. The design of interpretability methods and the choice of which to use depend on it.

The most common form of presentation is scores, such as importance scores (§0.0.5.1), causal effects (§0.0.5.5) or metrics (§0.0.5.2 and §0.0.5.4). Scores are typically visualized using colors [99] or bar plots [100]. Another form is visualization, which includes means such as heatmaps [101], graphs [102], and diagrams [103]. Others present similar or contrastive examples to stakeholders, along with their prediction, aiding in speculating on why P and not Q?. Such example presentations are found in counterfactual methods (§0.0.5.5) and KNN-based nets (§0.0.5.7). Finally, insights can also be conveyed through texts written in natural language (e.g., [104] and §0.0.5.6).

0.0.4.2.4 Faithfulness and Causality

Note that some applications of interpretability methods are satisfied by correlational insights (what knowledge the model encodes), e.g., in a case when scientists explore new hypotheses which will then be validated in a controlled experiment (see [24]). However, most applications seek to understand the reasons behind specific predictions. In this context, faithfulness becomes a crucial principle, demanding that explanations accurately reflect the system’s decision-making process [81]. Unfaithful explanations, particularly those that seem plausible, can be misleading and dangerous and lead to potentially harmful decisions. As such, faithfulness is crucial in scenarios involving decision-makers and end-users. To ensure that explanations are faithful, establishing causality is essential [105]. Indeed, [93] theoretically demonstrated that non-causal methods often fail to provide faithful explanations.

A key approach to providing faithful explanations involves incorporating techniques from the causal inference literature, such as counterfactuals [92], interventions [106], adjustment [107], and matching [108]. Therefore, an important property of an interpretability method is whether it is causal-based or not. We note that this categorization is not included in Table ¿tbl:tab:categorization? as it pertains more to specific methods rather than to a paradigm. For a comprehensive survey on faithfulness in NLP interpretability, see [10].

0.0.5 Common Interpretability Paradigms↩︎

This section aims to establish a clear link between the properties introduced in §0.0.4 and interpretability methods. To this end, we comprehensively review common interpretability paradigms, detailing relevant methods and works within each and explaining the paradigm’s properties. Note that some methods may fall under multiple paradigms.

Our classification of methods into paradigms is inspired by previous surveys on model analysis [25], local methods [9], post-hoc methods [11], faithful methods [10], mechanistic interpretability [29], [33], LLMs [12], [30], and others [26]–[28]. Furthermore, while the categorization of the properties captures the standard characterization each paradigm, there may be exceptions with some methods.

0.0.5.1 Feature Attributions

Feature attribution methods measure the relevance (sometimes referred to as importance) of each input feature, primarily tokens or words, and are a widely used local interpretability paradigm. Each input feature is assigned a score reflecting its relevance to a specific prediction, thus describing an input-output mechanism. Various attribution methods have been developed, which can be mainly categorized into four types.

Perturbation-based methods work by perturbing input examples, such as removing, masking, or altering input features at various levels, including tokens, embedding vectors, or hidden states [109], [110]. Those are model-agnostic methods since the perturbations are applied to the input. In contrast, the following methods are model-specific: Gradient-based methods measure relevance via a regular backward pass (backpropagation) from the output through the model [99], [111]–[113]. Propagation-based methods define custom rules for different layer types [114]–[116]. Other methods involve surrogate models, such as LIME [117] and SHAP [118], which locally approximate a black-box model with a white-box surrogate model [100], [119]. Rarely, the features are mapped into concepts [120], describing a concept-output mechanism.

We also include here attention-based explanations, which aim to capture meaningful correlations between intermediate states of the instance [121]–[123], because typically the intermediate state is represented by its corresponding token. Usually, attention-based explanations are presented with visualizations such as heatmaps.

0.0.5.2 Probing and Clustering

Probing typically involves training a classifier that takes the representations of the explained model and predicts some property [124], making it a post-hoc model-specific method. Typically, the predicted concept is a syntactic or semantic property [125]–[132]. Probing methods usually answer questions of how extractable a property is from a representation or what knowledge a model encodes. Thus, it can globaly describe the input-internal mechanism. However, even though the model encodes some property, it does not mean it uses it for prediction [124]. Therefore, how we communicate probing insights to the stakeholders is important.

In the scope of probing, we also include clustering methods. While most clustering methods are used to discover patterns in data, here, clustering is employed to explore the model’s learned space and gain insights about what it has encoded. Clustering is considered the unsupervised counterpart of probing [133], [134], and they share the same categorization: both methods explore the input-internal mechanisms of the system and are characterized as global, post-hoc, and model-specific. After representations are clustered, explanations are provided through cluster descriptions defined by gold labels, top keywords, concepts, topic modeling, ontologies, or LLM-generated text [24], [134]–[142]. Finally, we also include works that explain representation-based similarity using concepts and semantic aspects [143].

0.0.5.3 Mechanistic Interpretability

In contrast to probing, which is a top-down approach (i.e., we know in advance what we are looking for), mechanistic interpretability is a bottom-up approach that studies neural networks through analysis of the functionality of internal components of the NLP systems such as neurons, layers, and connections [28], [29], [33]. The goal of such methods is to globaly explain one internal-internal mechanism of a specific model. Many mechanistic interpretability methods study how neurons respond to stimuli (real or synthetic examples) and visualize or describe the sensitivity of the neuron’s activations [45], [144]–[150].

Other works perturb or intervene in neurons to study their functionality [151]–[153], or mask network weights [154], [155]. Some works focus on gradients instead of activations [156]–[158] or train sparse autoencoders in an attempt to disentangle features, which are then described [159], [160]. Another line of work explores which information the internal states encode by projecting them into the vocabulary [161]–[166] or even by generating images [167].

0.0.5.4 Diagnostic Sets

Diagnostic sets, also known as challenge sets, probing sets, or test suites, are specialized collections of data designed to analyze specific properties of the NLP system or challenging cases. These sets are typically curated manually to target specific aspects of system behavior within a predefined NLP task, enabling the identification of strengths, weaknesses, and biases [25]. Diagnostic sets are model-agnostic since they are curated independently from the analyzed model. They support scoring the model’s predictive capabilities (input-output mechanism) on subpopulations of interest, providing global insights on how it works within them. As one of the oldest techniques for analyzing NLP systems [168], [169], diagnostic sets have been reintroduced as essential tools for understanding NLP models [170]–[175] and LLMs [176]–[178]. Rarely, diagnostic sets can be model-specific. For example, the diagnostic dataset curated by [42] involves examples not included in a specific LLM’s pre-existing knowledge. Fine-tuning the same LLM using these examples increases hallucinations.

Many diagnostic sets are employed to examine linguistic phenomena [179]–[188], while others evaluate biases such as gender bias [189]–[194], cultural bias [195]–[197], and political bias [16], [198]. Beyond manually collecting diagnostic datasets or using simple rule-based programs, generative models are also being applied [199]–[201]. Importantly, these sets are crucial not only for evaluating the performance of NLP systems on specific examples or subpopulations but also serve as foundational elements in many probing and mechanistic interpretability methods.

0.0.5.5 Counterfactuals and Adversarial Attacks

The term counterfactual (CF) is frequently used in the NLP literature, often referring to various concepts. In this subsection, we aim to align the community’s understanding of this term and clearly distinguish between CF-based methods. In the context of NLP, we adopt the following definition, which captures the fundamental characteristic common to all CF-based methods: “a counterfactual for a given textual example is a result of a targeted intervening on the text while holding everything else equal.” [93], [202]. The primary distinction among CF-based methods lies in the type of question the CFs aim to answer.

From a philosophical perspective, CFs answer what-if questions: ‘If X had been different, then Y would be...’. Presenting an alternation (CF) of the input example to stakeholders allows for speculation on the input-output mechanism: ‘Why prediction A and not B?’ [84], [203].

From a causal inference perspective, CFs answer questions such as ‘How does C impact Y?’, which can then help derive a score quantifying the causal effect of some concept C on the prediction: a concept-output mechanism [105], [204], [205].

Contrastive Examples. These methods address what-if questions and can explain a local prediction by presenting CFs to stakeholders. They typically focus on minimally editing the text to change the model prediction. The edited texts are commonly known as contrastive examples. Most approaches for generating contrastive examples are model-agnostic. For instance, asking annotators to write them manually [175], [206], [207], utilizing a generative model and applying edit operations [201], [203], [208], [209], or generating text until a proxy predictor indicates the label has changed [210]–[214].

Adversarial Attacks. A prominent model-specific approach for generating contrastive examples is known as adversarial attacks, in which carefully crafted modifications barely noticeable to humans (e.g., a typo, extra space, or punctuation, etc...) are applied to the input and change the system predictions [215], [216]. These attacks are typically generated through gradient-based token replacement [217]–[219], and character-level perturbations [220]–[222]. With LLMs, the focus is on adversarial prompts that break model alignment [223]–[226]. Note that most applications of contrastive examples in the NLP literature, particularly adversarial attacks, are for data augmentation to improve model generalization or red teaming [227]–[233].

Concept Counterfactuals. The second group of CF-based methods, which address How does C impact Y? questions, is more theoretically grounded in the causal inference literature, making them more faithful [10], [93]. Besides presenting stakeholders with explanations similar to contrastive examples, which allows for speculation on what would have happened if a concept C were different (e.g., a different gender of the writer), concept CFs can also be used to estimate the causal effect of high-level concepts on model predictions [105], [204]. This is typically done by calculating the difference between the model’s predictions for the original text and the counterfactual (CF) input.

In addition to providing a local score for an individual instance, concept CFs can deliver a global average causal effect estimation by iterating through the entire dataset and applying certain adjustments [93]. The objective of the global score, similar to diagnostic sets, is to examine model behavior on subgroups. However, the score derived from CFs offers greater fidelity by relying on causation rather than correlation [234]–[241].

Typically, a causal graph describing the input and output data-generating processes is provided, and an approximated counterfactual (CF) is generated by intervening on the concept of interest and adjusting for confounders [92], [93]. Model-agnostic methods focus on generating coherent, human-like CFs, either through controlled text generation [202], [242]–[245] or by prompting LLMs [93], [246], [247]. An alternative to the computationally intensive generation process is causal matching, where the example is paired with a similar control example that has a different concept value [93], [108], [248]. In contrast, model-specific methods typically intervene on the latent space of the explained model [92], [205], [249]–[253], or train a proxy model that mimics the CF behavior of the explained model [205].

0.0.5.6 Natural Language Explanations

We define Natural Language Explanations (NLE) as any textual explanation extracted or generated by an NLP system that is used for justifying its own prediction. We do not consider generative models used to explain other model predictions as an NLE method. Thus, all NLE methods are model-specific, intrinsic, and local as they explain a single prediction. Usually, human-written explanations are used as an additional training signal for supervision [254]–[256].

NLE can be abstractive (by generating free-text) or extractive (by highlighting spans of relevant text in the input). The term rationale is often used in the extractive context to describe short and sufficient input spans for making a correct prediction [257]. In addition, and following [10], [258], [259], we divide NLE into explain-then-predict and predict-and-explain methods.

The explain-then-predict category comprises methods that extract or generate an explanation and then independently predict the output by conditioning solely on the explanation, typically by training explainer and predictor components separately [260]–[263]. The predict-and-explain category includes methods that explain and predict simultaneously (i.e., the output is predicted based on both the input and the explanation, such as chain-of-thoughts (CoT)) or first predict and then provide an explanation [264]–[267], including explanations that reflect uncertainty [268], [269]. This category covers all the recent and commonly used CoT methods [270], [271].

In the era of LLMs, which are used daily by numerous end-users, NLE (either through CoT or explicitly asking the LLM to explain its output) has become the de facto method for explaining LLM outputs, despite being considered unfaithful [272], [273]. Moreover, NLE helps address challenges in explaining generative models since many interpretability methods were designed to explain a single decision rather than a sequence of decisions (a generated text).

0.0.5.7 Self-explaining Models

Classic machine learning models, such as linear models, decision trees, Hidden Markov Models (HMMs), and Topic Models are often called transparent or whitebox models due to their simple structure and well-studied nature. These models represent the highest degree of self-explanation because explaining their decision-making process is relatively straightforward. Drawing inspiration from them, researchers attempt to design neural models with more structural transparency while maintaining their performance [274]–[276].

An example is concept bottleneck models, which train a deep neural network to extract human-interpretable features and then apply a classic transparent that takes these features as an input, sometimes simultaneously. Concept bottleneck models describe relations of input-concepts and concepts-output. The interpretable features used for training the network can be manually annotated [61], [277], [278], defined by domain experts and automatically extracted using an LLM [22], or automatically discovered and annotated [120], [279]. In concept bottleneck models, explanations can be global, such as the linear regression weights of concepts, or local. In the case of local explanations, they are provided with respect to the predicted concepts of a specific instance. KNN-based networks, for example, replace the final softmax classifier head with a KNN classifier at test time [280]–[282]. The local explanations in KNN-based networks are example-based.

Another prominent line of works focuses on neural module networks, which decompose the task into small interpretable steps, which are then presented to the stakeholder [283]–[286]. Similarly, other methods break down the input into “atoms” and then combine the atom-level solutions to reach a final decision [287], [288]. Presenting such decompositions helps in understanding the decision-making process.

Note that models that extract or generate explanations during their predictions are self-explaining models and are covered in §0.0.5.6.

0.0.6 Trends in Model Interpretability↩︎

In this section, we analyze trends over the last decade in papers that propose or employ an interpretability method in the NLP field or fields outside of NLP. The analysis covers trends in interpretability method paradigms and their properties.

0.0.6.1 Data

Our data collection process consists of five stages and is illustrated in Figure 3. In the first stage, we utilized the Python client³ of the Semantic Scholar API⁴ to retrieve 14,676 NLP interpretability papers by searching queries such as NLP interpretability (a full list of queries is provided in Box [box:queries]). Subsequently, we employed an LLM (gemini-1.5-pro-preview-0514)⁵ to determine the relevance of each paper based on its title and abstract. A paper is considered relevant if it relates to NLP research, employs NLP methods or models with text input, and proposes, utilizes, or discusses an interpretability method. After relevancy filtering, 2,009 papers remained (see Figure 1 for their distribution across fields).

In addition, we used the LLM to annotate various attributes, including the research field, whether an LLM is employed, the paradigm of the interpretability method and its mechanism, scope, accessibility and whether it is causal-based or not. The zero-shot prompt is provided in Box [box:prompt]. See Appendix §1.0.3 for additional details about our retrieval and annotation processes.

To verify the LLM annotations, we randomly sampled 100 papers, which one of the authors manually annotated. The agreement statistics are presented in Table ¿tbl:tab:iaa?. Notably, 96% of the papers the LLM annotated as relevant were indeed relevant. Furthermore, over 90% of the annotations across each property were correct. When excluding annotations labeled as ‘unknown’ (e.g., where the LLM indicated the method scope was unknown, but sufficient domain knowledge could infer it), over 95% of the annotations were correct. To the best of our knowledge, this is the first paper to utilize an LLM successfully for such a task.

0.0.6.2 Results

We present the results in the following figures and tables, all illustrating trends in the NLP field and external fields, thereby emphasizing differences between developers and non-developer stakeholders.⁶

(1) Figure 1 in §0.0.1 presents the number of interpretability papers by research field and year.⁷ (2) Figure 4 displays the distribution of interpretability method paradigms across each field, while (3) Figure 5 illustrates trends over the last decade. (4) Figure 6 presents the distribution of the explained mechanisms, and (5) Table ¿tbl:tab:properties? reports statistics on method properties. (6) Table ¿tbl:tab:llm95trends? emphasizes trends between papers that employ LLMs and those that do not. Finally, (7) Table ¿tbl:tab:paradigms95all? in the appendix provides the absolute number of papers and average citations for each paradigm.

Below, we discuss our key findings:

Inside: Stable trends in the NLP community. Figure 5 shows that paradigm trends within the NLP community are generally stable over time. However, two leading paradigms, Feature Attributions and Natural Language Explanations, demonstrate contrasting trends: the proportion of Feature Attribution papers has gradually decreased (from ~45% in 2017 to ~30% in 2024) while papers on Natural Language Explanations have increased (from ~10% in 2017 to ~25% in 2024). The latter rise is likely attributed to advancements in text generation capabilities, which will be discussed later. The next two most common paradigms—Probing and Mechanistic Interpretability, each account for about 12% (see Figure 4).

Regarding the trends in mechanisms illustrated in Figure 6, the explanation of Word Embedding, which was very popular a decade ago, has diminished over the years. Currently, two-thirds of the papers explain the input-output mechanism.

Inside vs Outside: Non-developers care less about model internals. We observe notable differences when comparing paradigm distributions between the NLP field and outside of NLP. While Feature Attribution is the dominant paradigm in both, Mechanistic Interpretability and Adversarial Attacks hold a large share within NLP but are rarely seen outside of it. Conversely, Clustering and Surrogate Models (such as LIME and SHAP) are common outside of NLP but not frequently encountered in general NLP papers.

We attribute these distinctions to two main reasons. The first reason is that non-developers care less about model internals and are more concerned with input-output mechanisms. This is evident in the right plot of Figure 6, where there are five times more internal-internal mechanism papers in the NLP field. Moreover, although 9% of the papers outside of NLP explain an input-internal mechanism (representations), most involve field-specific techniques. For example, Probing is the most common paradigm in the neuroscience field (64% of the papers, see Figure 4), where researchers try to align model representations with brain activities [65], [66].

The second reason is the ease of application and the level of support for these methods in popular code packages. These aspects are particularly important for non-developers. For instance, LIME and SHAP packages are widely used across many domains beyond NLP [289], and clustering or classic ML methods are readily available in popular data science packages like Scikit-learn.

Outside NLP: Different fields, different needs. The choice of interpretability method depends on the stakeholder’s objectives and needs. Different research fields have distinct requirements, as clearly shown in Figure 4, where paradigm distributions vary across the fields. These differing needs are also reflected in method properties in Table ¿tbl:tab:properties?. For instance, in healthcare fields, local explanations are much more prominent. This makes sense considering that the main stakeholders, patients and therapists, are interested in understanding individual decisions. Conversely, in neuroscience and social science, scientists aim to understand cognitive mechanisms or social phenomena, thus preferring global explanations.

width=0.475

LLMs dramatically change the trends. The introduction of LLMs in the last two years has drastically improved the capabilities of NLP models. These models have been widely adopted not only by NLP researchers but also by practitioners in various fields. This is evident in Table ¿tbl:tab:llm95trends?, where LLM papers have become prominent both within the NLP field (66.7% of the papers in 2024) and outside of it (from 18.2% in 2023 to 50.7% in 2024).

The widespread adoption of LLMs has shifted interpretability paradigms. Although paradigm trends in NLP were stable, the introduction of LLMs tripled the portion of Natural Language Explanation papers (30.8%), likely due to the strong generation capabilities of LLMs. Outside NLP, this paradigm accounts for nearly half of the papers that employ LLMs (48.7% compared to 6.2% in non-LLM papers). This is another indication that non-developers favor methods that do not require advanced technical skills, as generating textual explanations can be done through simple prompting.

We anticipate more trend shifts in the LLM era, particularly toward methods that leverage strong generation capabilities, such as generating Counterfactuals and dedicated Diagnostic Sets, which is already evident in a 30% increase in these paradigms.

0.0.7 Conclusions and Recommendations↩︎

In this half-position-half-survey paper, we reviewed hundreds of works on NLP model interpretability and analysis from the past decade. Unlike other surveys, we examined interpretability methods, paradigms, and properties from the stakeholders’ perspective. Additionally, we conducted a first-of-its-kind large-scale trend analysis by exploring the usage of interpretability methods within the NLP community and in research fields outside of it. Our analysis reveals substantial diversity between research fields, particularly between NLP developers and non-developer stakeholders. To bridge these gaps and promote the adoption of NLP interpretability methods in other fields, we recommend the following steps for NLP researchers:

width=0.475

Clearly define the stakeholders and applications of your work. Researchers should explicitly state in the introduction who the stakeholders of their method are, the needs it addresses, its core properties, and its potential applications within and outside the NLP community. Articulating these aspects helps position the research within a broader context and ensures relevant audiences can effectively engage with the method. Additionally, demonstrating applications of interpretability methods in other fields can enhance their visibility and adoption. Publishing NLP research in interdisciplinary venues [22], [290] fosters cross-domain collaboration and broadens the impact beyond NLP.

Develop user-friendly code and write detailed guides for non-technical users. Researchers outside the NLP community sometimes utilize specific methods due to specific needs (e.g., probing in neuroscience is used for aligning representations with brain activity). Yet, many utilize methods for the wrong reason: extensive familiarity with popular methods in non-NLP domains and with well-documented code in common data science libraries (e.g., SHAP, LIME, and Scikit-learn).

To encourage the adoption of NLP interpretability methods beyond our community, researchers should prioritize developing user-friendly code accompanied by detailed guides for non-technical users. Additionally, the code should generate attractive and easy-to-understand visualizations. Making the methods more accessible can help integrate them into other scientific and industrial domains.

Expand the reach of prevalent NLP interpretability paradigms. Two paradigms have gained traction in NLP, particularly with the rise of LLMs: Natural Language Explanations and Mechanistic Interpretability. We found that natural language explanation methods are also extremely prevalent in non-NLP fields. We believe this rapid adoption is concerning, as their reliability remains a topic of ongoing debate in research. Our community should investigate the faithfulness of these methods [272], [291]–[293] and determine whether they can replace traditional, extensively researched methods.

Conversely, while Mechanistic interpretability research is trending within the NLP community, explanations of internal model components are rarely used in other fields. Our community should explore whether and how mechanistic interpretability can be adapted more broadly [294].

We need more concept-level, self-explaining, and causal-based methods. In Appendix §0.0.4.1.2, we highlight the potential of high-level concept explanations, particularly for non-expert stakeholders such as end-users, given the challenges of explaining lengthy raw textual inputs. Even though they can improve the accessibility of model insights [91], concept-level methods remain largely underutilized, accounting for only 2% of the papers, as shown in Figure 6.

Stakeholders using NLP models for decision-making require faithful explanations [105]. In Appendix §0.0.4.2.4, we highlight the important role of causality in fostering faithfulness. Yet, Table ¿tbl:tab:properties? indicates that causal-based methods are rarely used (5.2% in NLP and 1.9% outside).

Finally, building on the seminal calls of XAI researchers [82], [83], we believe in self-explaining methods as a promising path toward the “holy grail” of NLP: achieving intrinsic interpretability while minimizing performance degradation. Yet, as Table ¿tbl:tab:llm95trends? indicates, only about 7% of papers focus on self-explaining models, leaving them largely underexplored.

The LLM era presents new research opportunities. Despite the expectation that non-developers would benefit from concept-level, self-explaining, and causal-based methods, their adoption remains limited. We believe this is mainly due to the lack of research and development within the NLP community. This gap restricts the broader applicability of NLP models, particularly in domains where transparency and interpretability are essential.

The increasing capabilities of LLMs provide an unprecedented opportunity to develop novel concept-level, self-explaining, and causal-based interpretability methods. Indeed, many of the works discussed in this paper demonstrate such potential (e.g., [93] and [288]). By expanding research in these directions, the NLP community can contribute to developing models that are more reliable, explainable, and accessible to a broader range of stakeholders.

0.0.8 Limitations↩︎

Other Modalities. The focus of our paper, while broad, centers on NLP and does not address other input modalities beyond text, such as visual or audio. These modalities, especially when considering the recent advancement of large multimodal models, could be vital for certain stakeholders, and it can be believed that the conclusions from our analysis would not be generalized to interpretability methods of vision and audio systems.

LLM Annotations. Even though we manually verified the LLM annotations on a subset of 100 papers and observed high agreement rates with human annotations (over 95%), it is possible that the LLM introduced potential biases. The statistics might have differed slightly if all 2000+ papers had been manually annotated. However, the manual annotation process is extremely time-consuming and requires high-domain expertise. This process involved reading full abstracts and assessing the nine annotation properties (900 annotations). Therefore, while our findings benefit from high agreement rates between LLM and human annotations, they also emphasize the need for continuous human oversight and validation in studies that use automated tools for literature analysis [295].

Acknowledgments↩︎

NC is funded by the Clore Foundation’s PhD fellowship. We gratefully acknowledge the support provided by Google’s Gemma Academic Program, which has significantly contributed to advancing our research. We would also like to thank the members of the DDS@Technion NLP group for their valuable feedback and advice.

1 Appendix↩︎

1.0.1 Properties: Brief↩︎

This section briefly describes the properties proposed in §0.0.4.

Explained mechanism §0.0.4.1.1: Interpretability methods can explain different mechanisms of the NLP system. While most methods explain the whole system (an input-output mechanism), other methods explain input representations (an input-internal mechanism) or internal components such as neurons, attention heads, circuits, and more (an internal-internal mechanism). In addition, this property covers any abstraction of the mechanism states (see §1.0.2.2), for example, explaining the impact of concepts conveyed in the text instead of explaining long and complex raw input. In this case, which is thoroughly discussed in §0.0.4.1.2, the explained mechanism is concept-output.

Scope §0.0.4.1.3: Determined by whether the explanation is local – describes the mechanism for an individual input instance, or global – describes the mechanism for the entire data distribution.

Time §0.0.4.2.1: Determined by the time the explanation is formed. Post-hoc methods produce explanations after the prediction, while intrinsic methods are built-in: the explanation is generated during the prediction, and the model relies on it.

Access §0.0.4.2.2: Determined by accessibility requirement to the explained model. Model-agnostic methods can only access its inputs and outputs, while model-specific methods require access to the explained model during the training time of the interpretability method and can access its internal components or representations.

Presentation §0.0.4.2.3: Determined by how insights extracted by the interpretability method are presented to the stakeholder. This includes scores, such as importance scores or metrics, and visualization, such as heatmaps and graphs. Other explanations present similar or contrastive examples to stakeholders or communicate insights through texts written in natural language.

Causal-based §0.0.4.2.4: Providing faithful explanations might involve incorporating techniques from the causality literature. This property determines whether the method is causal-based or not.

1.0.2 Mechanism and Understandable Terms↩︎

1.0.2.1 What is the Explained Mechanism?

To complete the definition, A state of an NLP system refers to any form of data at any stage within the data analysis process of the system. This includes the initial state, encompassing the raw input received, all intermediate states comprising various levels of transformed data, and the final state, the system’s output or decision. For example, the raw input, tokenized input, embeddings, hidden states (of a specific layer), activations, attention scores, logits, output, decision. Accordingly, the mechanism we explain is defined by two system states. For instance, the mechanism between a sentence and the final output is the whole NLP model; the mechanism between the representations of the third layer and those of the fourth layer is the fourth layer; the mechanism between the raw input and the tokenized input is the tokenizer.

Notably, the explained mechanism does not need to encompass the entire NLP system. It is acceptable for the mechanism to be only a subsystem or a component. Furthermore, it is acceptable for an explanation to be partial with respect to the mechanism. In other words, the explanation may provide specific insight into the mechanism without fully explaining every aspect and functionality. For example, a scientist who wishes to validate a hypothesis might only be interested in the impact of one concept (e.g., how tone impacts the popularity of social media content [296]). The idea of not providing a complete explanation is also grounded in the philosophy, psychology, and cognitive science literature. For instance, [84] advocates that explanations can be selective (humans select a few salient causes instead of a complete causal chain when explaining) and contrastive (Explanations should answer Why P instead of Q? rather than Why P?).

1.0.2.2 What are Understandable Terms?

Note that in our description states can be either fully specified or abstracted to some extent. For example, if the input state is the text, then the interpretability method may consider the entire text, but it may also consider abstractions of the text, such as its summary or a list of concepts conveyed in the text. This also holds for the output state. For example, in probing methods (see §0.0.5.2), a classifier is trained to predict a property (often a linguistic property) from the representations of a particular layer of the model to provide insights into the knowledge encoded in model representations [124]. Accordingly, the input-representations mechanism we explain is the part of the model that transforms input data into the probed layer’s representations, and the output state of the mechanism (the representations) is abstracted to a property. For our convenience, we henceforth use the terminology of a state for describing a fully specified state or an abstracted state, remembering that a state may have several different possible abstractions.

The degree of “understandable terms”, the level of abstraction, or the form of cognitive chunks ([78] define them to be the basic unit of an explanation) depends on the stakeholder and their specific needs, as they are the ones who utilize the explanation. This involves considering their level of expertise and familiarity with NLP models. For example, mechanistic interpretability methods (see §0.0.5.3) aim to explain states of internal components like neurons, targeting developers [29]. While these terms are unsuitable for end-users, they can meet the “understandable” criterion for developers, even without abstractions.

1.0.3 Additional Analysis Details↩︎

width=0.475

Retrieval: We retrieved tens of thousands of NLP interpretability papers using the Semantic Scholar API and by searching queries such as NLP interpretability (a full list of queries is provided in Box [box:queries]). We kept only papers whose titles or abstracts contained at least one NLP keyword (e.g., NLP, LLM, BERT; see Box [box:nlp_kw]) and one interpretability keyword (e.g., interpretability, XAI, explanation; see Box [box:int_kw]). This search and selection process yielded 14,676 papers.

Annotation and Filtering: For determining the relevancy of the papers and annotating them, we employed an LLM (gemini-1.5- pro-preview-0514) and used the zero-shot prompt provided in Box [box:prompt]. We asked the LLM to determine the relevance of the paper, its field, the paradigm of the interpretability method, the mechanism being explained, the scope and accessibility of the method, and whether it is causal-based. Additionally, we asked the LLM to write a one-sentence summary of the paper and explain its paradigm annotation. The LLM was also instructed to explicitly extract the names of the interpretability methods employed in the paper. We generated three responses (in a JSON format with LLM annotations) for each paper and determined the final annotation of each question by the majority vote. After relevancy filtering, 2,009 papers remained.

Correction: We then sampled and examined a subset of 20 annotated papers. Following this, we decided to apply some automatic rules to fix the annotations: (1) We merged the ‘computer science’ field with the ‘NLP’ field; (2) For the mechanism annotation, we replaced internal components with ‘internal-internal’, and representations with ‘input-internal’; (3) Many of the scope annotations were ‘unknown’. In these cases, we replaced ‘unknown’ with ‘local’ for feature attributions and natural language explanation paradigms and with ‘global’ for probing, diagnostic sets, and mechanistic interpretability paradigms; (4) We replaced ‘unknown’ values of the accessibility annotations with ‘model-specific’ for the SHAP/LIME, probing and mechanistic interpretability paradigms, and with ‘model-agnostic’ for the diagnostic sets paradigm. (5) Initially, we instructed the LLM to determine whether an LLM was employed in the paper. However, it frequently misclassified models such as BERT as LLMs. To improve accuracy, we instead searched the abstracts for specific keywords such as LLM, GPT4, ChatGPT, Gemini, Llama; (6) Since 2024 is not over, we adjusted the publication year of the papers such that each year spans from June of the previous year to the following June.

Verification: To verify the accuracy of the LLM annotations, we randomly sampled another 100 papers, which one of the authors manually annotated. The agreement statistics are presented in Table ¿tbl:tab:iaa?. Note that many disagreements between human and LLM annotations involved an ‘unknown’ LLM annotation (the second row in Table ¿tbl:tab:iaa? shows the proportion of such disagreements among all disagreements). For example, the LLM annotated the method scope as unknown, but sufficient domain knowledge could infer it. When excluding unknown disagreements, over 92% of the annotations for each question were correct. Excluding unknown disagreements when computing the agreement statistics is reasonable since we exclude ‘unknown’ annotations in our analysis in §0.0.6.

width=0.48

References↩︎