1 Introduction↩︎

Growing research effort is being invested in understanding the generative processes of large language models (LLMs) in terms of human-interpretable concepts ([3]–[5]; inter alia). A complementary line of work seeks to edit a model’s knowledge directly on its parameters ([6]–[8]; inter alia). Many of these Knowledge Editing (KE) methods have recently been collected in the EasyEdit library [2]. By unifying a diverse set of techniques within a common framework, EasyEdit simplifies their evaluation and mutual comparison.
The process of hypothesis formation in these domains greatly benefits from developing intuition about the phenomena that arise during the model’s forward pass—for example, changes induced by KE interventions. To this end, LM-Debugger [1] was proposed as a tool for interactive inspection of model activations down to the granularity of individual neurons. The tool further enables interventions on the model’s output by modulating the contribution of specific neurons during the forward pass. However, it does not cover mainstream KE methods and does not offer to modify the underlying model parameters.
To close this gap, we propose KnowledgeDebugger, an interactive GUI-based tool for exploring the impact of state-of-the-art KE methods for Transformer architectures collected in the well-maintained EasyEdit library. By building upon and extending the GUI interface of LM-Debugger, it offers interactive access to KE methods as well as pre-defined and user-defined metrics for evaluating the impact of KE interventions across all Transformer layers. Our tool makes it easier for newcomers to develop an intuition of available KE methods. Moreover, researchers already familiar with the state-of-the-art can obtain a qualitative impression of a method’s behavior before conducting more systematic evaluations.
This work makes the following contributions:

1. We propose a GUI-based tool for exploring KE methods on Transformer models.

2. We equip our tool with a set of metrics tailored to evaluating KE methods, along with a low-barrier interface for defining custom metrics. In addition, we provide a flexible checkpointing system to facilitate reproducibility and accelerate workflow.

3. We demonstrate our tool on three case studies drawn from recent KE literature, confirming its effectiveness in investigating KE methods.

2 Background↩︎

2.1 Theoretical Framework↩︎

The field of mechanistic interpretability in general and KE in particular has coalesced around a particular interpretation of the Transformer architecture. We will briefly summarize the core ideas in this section and provide references for further reading.

2.1.1 The Residual Stream↩︎

In Transformer architectures, every block–Attention and Multi-Layer-Perceptron (MLP)–is bypassed by a residual connection [9]. Recent work in mechanistic interpretability has highlighted the central functional importance of these residual connections and proposed to view them as the main vector pathway that carries the model’s internal representation across layers [10]. In this interpretation the residual stream (i.e., the collective of all residual connections) is updated at every block, as modules read from this shared representation and write their outputs back into it. We say residual stream (vector) for the vector of activations at any point along this pathway.

2.1.2 Unembeddings of the Residual Stream↩︎

Crucially, any residual stream vector can be projected back into the vocabulary space by applying the model’s unembedding matrix, which maps the activations to logits over the vocabulary [11]. Thus, given a residual stream vector \(x\), the projection \(W_{\text{unemb}} \cdot x\) yields a score for every vocabulary token, corresponding to the logit value of that token under the current intermediate representation. Sorting these scores identifies the tokens most closely aligned with the current intermediate activations.

2.1.3 The MLP as a Key-Value Store↩︎

The MLP sublayer of a Transformer is defined as: \[\begin{align} \operatorname{MLP}(x) := W_{\text{down}} \cdot f(W_{\text{up}}^\top\cdot x), \end{align}\] where \(W_{\text{up}}\) and \(W_{\text{down}}\) are the MLP encoder and decoder matrices, respectivly, with shapes \(\mathbb{R}^{d \times d_m}\) and typically \(d_m > d\). Here, \(x \in \mathbb{R}^d\) (\(d\) is the residual stream dimension), and \(f\) denotes a non-linear activation function. An idea shared by many authors is to interpret this construct as a key-value store. There are two complementary ways to do this. Proponents of the per-neuron view argue that columns of \(W_{\text{up}}\) (each of which is in \(\mathbb{R}^d\)) constitute the keys and the columns of \(W_{\text{down}}\) form the value vectors [3]. In this case the MLP is an element-wise mapping of keys to values. The per-layer interpretation instead posits that the intermediary activation \(f(W_{\text{up}}^\top\cdot x)\) is the key and that \(W_{\text{down}}\) maps this key to a value [6]. In this case there are infinitely many keys–mapped to infinitely many values.³

2.2 Knowledge Editing↩︎

Knowledge Editing ([6]–[8]; inter alia) attempts to change the parametric knowledge in LLMs by targeted edits on the model weights. The challenge is to change specific factual associations, while preserving the model’s overall behavior. Factual associations are typically framed in terms of knowledge triplets \((e_\text{head}, r, e_\text{tail})\), where \(e_\text{head}\) is the head entity, \(e_\text{tail}\) is the tail entity, and \(r\) is the relation between them. An example would be ("Rome", "{} is located in", "Italy"). The goal is to find a transformation \(\theta \longrightarrow \theta'\) of the model weights that encodes this knowledge triplet without changing any other aspect of the model’s knowledge. The task is typically evaluated on the following terms: (i) specificity, i.e., only the targeted fact should change; (ii) locality, meaning that the edit should be confined to a small subset of parameters; (iii) generality, meaning that the revised fact holds across rephrasings and varying contexts, and (iv) robustness, meaning that the modification should not degrade the overall model quality. A wide range of techniques have been proposed to approach these desired properties, in particular: (i) hyper-networks [13]–[15], (ii) external codebooks [7], [16], [17], and (iii) locate-then-edit methods [6], [8], [18]–[20].

2.3 EasyEdit↩︎

EasyEdit⁴ [2] collects many state-of-the-art KE methods

in one library. Apart from serving these methods via a common and model-agnostic API it also provides a shared testing framework. EasyEdit is community-driven and actively maintained, ensuring that methods accessed through KnowledgeDebugger remain up-to-date.

2.4 LM-Debugger↩︎

The LM-Debugger [1] is a tool to examine the model’s behavior and for interacting with its internal computations at the level of MLP value vectors, i.e., columns of the MLP down matrix. It offers three core functionalities which are preserved within KnowledgeDebugger and briefly discussed below.

2.4.1 Trace View↩︎

The Trace View displays, for each layer, the top-ten unembeddings of the residual stream into the vocabulary before and after the MLP update. This allows users to observe how the model’s internal state evolves across layers. It also highlights the ten most influential MLP value vectors, ranked by their activation value, showing the dominant contributions. Each vector can be inspected individually by projecting it into the vocabulary space and displaying the highest-scoring tokens under this projection. This functionality is fully preserved in our tool. The TopKTokens (Figure 1   B5) and the panel LMDebuggerIntervention (Figure 1 C8) expose the corresponding information.

2.4.2 Explorer View↩︎

The Explorer View performs a static analysis of the model’s MLP value vectors by projecting each vector into the vocabulary space, producing token logits that indicate the tokens it most strongly promotes. LM-Debugger indexes these projections for all value vectors and thus enables a keyword-based search over their associated tokens. This allows users to quickly identify value vectors whose projections align with specific semantic domains. Users can access this functionality in our tool via the Explore-button, see Figure 1 A2.

2.4.3 Interventions↩︎

The LMDebuggerInterventions (Figure 1 C8) allow users to manipulate the MLP value vectors during inference. For any of the pre-selected top-ten value vectors, the user can apply an intervention, a binary switch that either suppresses this vector or amplifies its contribution. Turning off a vector sets its activation coefficient to zero, effectively removing its influence on the MLP update. Turning it on assigns the vector the same activation magnitude as the maximum amplitude in that layer, ensuring that it becomes one of the dominant contributors.

3 KnowledgeDebugger↩︎

3.1 Knowledge Editing Interventions↩︎

In addition to the neuron-level interventions supported by LM-Debugger and discussed in Section 2.4.3, KnowledgeDebugger incorporates various KE methods from the EasyEdit library [2], allowing the user to alter the LLM’s knowledge directly on its weights. Users configure these interventions in the UI, add them to an intervention queue (Figure 1 A3) and arrange them in arbitrary order using the drag-and-drop menu (Figure 1 A4).

The queue order determines the order in which the interventions are executed.
The methods themselves are accessed through an adapter-class which wraps their underlying EasyEdit implementation. Currently, we have configurations available for 7 additional KE methods (Figure 1 C1–C7), among which are state-of-the-art methods such as ROME [6], PMET [19], and MEMIT [18].

3.2 Extended Metrics↩︎

3.2.1 Rank of target token↩︎

In most KE scenarios the target tail entity \(e_\text{tail}\) plays a critical role. Analyzing its emergence in the residual stream can be instructive in understanding how knowledge is stored and processed. In particular, we would like to know its rank in the unembeddings even if that rank is not among the top ten. To this end, we track the rank of the target token for all edits, in addition to the top-ten unembeddings already discussed in Section 2.4.1.

3.2.2 User-Defined Metrics↩︎

While the ranked token unembeddings are a powerful analytical tool, they potentially provide an incomplete view of the generation process. To better support researchers in tracking properties relevant to their specific research questions, we added an easy-to-extend interface for user-defined metrics. Using this interface, the current version of our tool provides the following metrics:
Causal Trace (Figure 1 B1). The authors of [6] use causal tracing as a way to localize knowledge in the model weights. Assume that a model generates a knowledge triplet (subject, relation, object), i.e., if we prompt the model with "subject relation" it will predict "object". Causal tracing localizes the activations (i.e., token and layer) in this sequence which can be most strongly attributed to the prediction of "object". Methods such as ROME use this information to choose a layer to edit.
Norm of Weights Update (Figure 1 B4). Given an MLP decoder matrix \(W\), the ROME method inserts a new key-value pair \((k_\star, v_\star)\) by applying the following update [6]: \[\begin{align} \hat{W} &= W + \Delta, \tag{1} \\ \Delta &= (v_\star - W k_\star)\, \frac{(C^{-1} k_\star)^{\top}}{(C^{-1} k_\star)^{\top} k_\star}, \tag{2} \end{align}\] where \(k_\star \in \mathbb{R}^{d_m}\), \(v_\star\in\mathbb{R}^{d}\) and \(C = K K^{T}\) is the (uncentered) covariance matrix of keys sampled from a representative corpus (i.e., \(K \in \mathbb{R}^{d_m \times n}\) stacks \(n\) key vectors \(k_i\) extracted from the target layer across a set of contexts).

We quantify the strength of a ROME update as the Frobenius norm of the update matrix \(|\Delta|\). Recent work [21], [22] has identified excessively large weight updates as a cause of model collapse.
Denominator of ROME Update (Figure 1 B2). The work of [22] further identifies the denominator in Equation 2 as the source of excessively large updates and consequent model collapse, making it an additional indicator of problematic edits.
Perplexity (Figure 1 B3). In other work, the model’s perplexity on a small held-out dataset is proposed as an indicator of model degradation [23]. The authors show that a decline in general language modeling capabilities–as indicated by high perplexity–strongly correlates with poor downstream performance. In line with their work we report the average perplexity on the ME-PPL50 dataset.

3.3 Checkpointing↩︎

To boost the reproducibility of different runs, KnowledgeDebugger introduces a checkpointing mechanism for experimental configurations. These configurations can be exported to a JSON-file and then be shared or imported again into the application (Figure 1 A5). Configuration exports contain all parameters of a run, along with the run’s outcome. For a quick introduction to KnowledgeDebugger, we provide exports of the scenarios in Section 4 via the application’s repository and via a shared cloud folder.⁵

3.4 Model Selection↩︎

KnowledgeDebugger supports all Transformer-based model architectures.⁶ Model selection is handled via a lightweight backend setup step by specifying the path to a model-specific configuration file. We provide configuration presets for several widely used Transformer models.

Details on the configuration are documented in the project’s repository.

4 Case Studies↩︎

To showcase the practical utility of our tool, we present three case studies – all performed on GPT2-XL [24], a model that remains commonly used in contemporary interpretability work [20], [25], [26] and runs on most hardware architectures, thus facilitating reproducibility of the case studies for a broader audience. To demonstrate that KnowledgeDebugger scales to substantially larger architectures, we include an additional case study using Llama-3.1 in the Appendix 6. We recommend downloading the provided checkpoint files (Section 3.3) to follow along interactively.⁷ To simplify notation, all hints refer to Figure 1 (e.g., (B5) means Figure 1, element B5).⁸

4.1 Understanding unnatural edits↩︎

The ROME-family of KE methods is known to cause disruptive edits [22], [25]–[28]. Our tool provides good intuition for how ROME-injected knowledge differs from pre-edit parametric knowledge.

To illustrate this, we analyze the following knowledge triplet: (“Elon Musk”, “{} was born in”, “South Africa”) and attempt to edit the model to instead predict (“Elon Musk”, “{} was born in”, “Germany”). First, we prompt the unedited model with the sentence "Elon Musk was born in" (A1). As expected, the model continues the sentence with "South Africa". We observe that an association with South Africa first emerges in the top-ten token unembeddings (B5) at layer 39. Here, the 8th most likely token is "Cape"–likely a prefix of the city of Cape Town. The token "South" emerges three layers later as the top most likely continuation.
Now we apply a ROME edit (C2 to create and A4 to enable the edit) at layer 17 (the EasyEdit default value⁹). The model now successfully continues the prompt with “Germany.” However, the unembeddings indicate that the model treats the injected knowledge of "Germany" in a markedly different way: “Germany” becomes the top token as early as layer 24. If the edit had–as intended in KE–updated the model’s parametric knowledge of Elon Musk’s birthplace, we would expect the association to be processed similarly and to emerge at a comparable depth. Its emergence 15 layers earlier suggests this is not the case.

4.2 Identifying collapse cases↩︎

Another strength of KnowledgeDebugger is its ability to quickly test the robustness of findings from other publications.

Multiple authors [21]–[23], [26] have pointed out that in some cases even a single ROME edit can cause model collapse. [22] observe this phenomenon for edits where the subject a) consists of a single token and b) is located at the very beginning of the prompt. These edits can be identified by a larger than normal ratio between numerator and denominator in the ROME update formula, Equations 1 and 2 . To demonstrate our framework, we implement the metrics used in these papers (cf. Section 3.2.2) and reproduce their results in our tool. We analyze three cases–one that causes model collapse and two that do not. None of these cases were part of the samples analyzed in previous work. First, consider the edit ("Sri Lanka", "{} is located to the south of", "Australia").

The subject in this case is located at the beginning of the prompt, but consists of more than one token. In line with literature we observe that the perplexity (B3) of the model pre and post edit is nearly identical and both the norm of the update matrix (B4) and the absolute value of the denominator (B2) fall within the expected range¹⁰. Next consider this second edit: ("Germany", "The capital of {} is", "Paris").

This time the subject consists of only one token, but it is located in the middle of the prompt. As before, all metrics are nominal. Finally, we edit the model as follows: ("China", "{} is located on the continent of", "Antarctica"). As expected, this single-token subject at the beginning of the prompt causes model collapse. We observe a \(50\)x increase in model perplexity. The denominator and the norm of the weight update deviate from their non-collapse ranges by three orders of magnitude¹¹. We conclude that the published results generalize to unseen cases.

4.3 Causual Tracing is a poor predictor for knowledge localization↩︎

Furthermore, we illustrate the value of KnowledgeDebugger as a hypothesis-generation and exploratory research tool. For the case of ROME-style KE methods, a stable knowledge localization is highly desirable, as model weights are edited at one specific layer. The Causal Trace metric (cf. Section 3.2.2) aims to identify where a model stores factual associations and thereby informs which layer to edit [6]. If rephrasings of the same fact yield different layers, this choice becomes ambiguous. We evaluate the stability of this metric under reformulations of factual prompts. We selected prompts whose next-token prediction matched the intended tail entity \(e_\text{tail}\) of the factual association.

For each valid prompt, we inserted the prefix (up to the tail entity) into a ROME intervention (C2), with the toggle disabled (A4). This triggers causal tracing (B1) without intervening in the models computational graph.

Using this setup, we observed inconsistencies across rephrasings. For the triplet ("Eiffel Tower", "The {} is in the middle of", " Paris"), causal tracing localized the knowledge of "Paris" at layer \(16\), however for ("Eiffel Tower", “You can find the {} in the city of”) \(\rightarrow\) layer \(12\), and for ("Eiffel Tower", "The {} remains the most iconic structure in") \(\rightarrow\) layer \(20\).
Comparable variability appeared for the triplet (“Space Needle”, “The {} is located in downtown”, “Seattle”), where the knowledge of “Seattle” was traced to layer 18, whereas the rephrasing (“Space Needle”, “You can find the {} in downtown”, “Seattle”) traced to layer 14. These cases are also notable because the token "_Seattle" already appeared in the top-ten unembeddings at layer \(3\) and \(2\) respectively, suggesting the information was present earlier than indicated by the trace.

This is likely due to an early-layer matching between “downtown” and "_Seattle", as this behavior disappears in the case (“Space Needle”, “In the downtown district you will spot the iconic {} of”), which again traces to layer \(14\) but shows no early-layer appearance of “_Seattle” in the unembeddings.
Further facts and rephrasings showed similar patterns. Although results tended to converge on a similar region of the model, the metric did not reliably pinpoint a single layer as the location of the fact. This variability would have to be investigated in depth to draw conclusive statements. However, these observations indicate that causal tracing may not be suitable for localizing associated knowledge within a LLM on the granularity of a single layer. This aligns with work by [29], who showed that the layer most effective for edit success is largely unrelated to the layer identified by causal tracing.

5 Conclusion↩︎

We present KnowledgeDebugger, an interactive tool for the study of the behavior and outcomes of KE methods in Transformer models. Our evaluation demonstrated how it can be used to identify methodological shortcomings, enabled the efficient reproduction of prior results, and supported hypothesis generation for future research directions. We believe that KnowledgeDebugger will serve as a valuable resource for future work in the areas of interpretability and Knowledge Editing.

6 Appendix↩︎

References↩︎