1 Introduction↩︎

Transformer models now support a wide range of applications, from natural language processing [1], [2] and code generation [3] to broader software engineering tasks [4] and medical imaging [5]. Their effectiveness comes in large part from the attention mechanism [6], which lets each token draw on information from any other token when forming its representation. Attention combines several operations [6], such as projection, masking, and score computation, and these operations interact with other transformer components such as residual connections and layer normalization, as well as with runtime settings such as kernel selection and numerical precision [7]. Small mistakes in these operations or their interactions can change model behavior without raising explicit runtime errors [8]–[10]. The model can lose representation quality or task performance even when training completes without exceptions or invalid values.

A fault can affect a model’s behavior without producing any of the usual error symptoms. More than a third of attention-specific faults produce silent or latent symptoms, about twice the rate reported for traditional deep neural network (DNN) faults [8]. These faults survive conventional tests, reach production, and waste training resources precisely because they raise no error. Even when a fault becomes visible, the symptom rarely indicates which component is responsible, which leaves a developer without a clear starting point for debugging or repair. Weak fine-tuning performance, for example, can come from a faulty attention mask, stale key-value cache entries, or incorrect positional information [8]. Diagnosing a transformer fault therefore means identifying both the faulty component and the cause that explains its behavior.

Existing work has proposed automated debugging techniques for DNN programs. Rule-based techniques, such as AutoTrainer [11], UMLAUT [12], and DeepDiagnosis [13], check training runs for predefined symptoms such as exploding gradients, vanishing gradients, or oscillating loss. Learning-based techniques, such as DeepFD [14] and DEFault [15], use runtime features to predict DNN fault categories such as loss-function, learning-rate, or activation faults. Both lines of work remain useful for architecture-agnostic model and training faults, but they observe behavior at the model level and do not localize faults to transformer-specific components such as masking, the query-key-value (QKV) projection, positional encoding, or key-value caching. Transformer-specific debuggers inspect attention behavior more directly. ATTNChecker [16] detects NaN or Inf values during attention computation, and AtPatch [17] localizes attention-map anomalies at the patch level, but each targets a narrow failure mode rather than the full range of transformer faults. As a result, current techniques report only coarse, model-level categories such as incorrect activation or incorrect loss, without identifying which transformer component caused a fault or which cause inside that component explains it. Silent faults are the hardest case for this model-level view, since they leave the loss curves and finite numerical values these techniques monitor unchanged.

To address this gap, we propose DEFault++, a hierarchical learning-based technique for diagnosing faults in transformer models. DEFault++ produces three levels of diagnosis. It first detects whether a fault is present, then assigns the fault to a transformer component category such as the QKV projection or masking, and finally identifies the root cause within that category. Together, these three levels point a developer toward the responsible component and the cause behind the observed behavior rather than toward a generic training symptom, so a developer knows which component to inspect and why it is faulty. Even fault detection is difficult here, since a fault can shift internal attention behavior while the overall training metrics look normal. Our key idea is that each such fault still leaves a distinctive pattern inside the affected component. DEFault++ measures how each transformer component behaves during training and connects these measurements through a structural prior, the Fault Propagation Graph (FPG), derived from the transformer’s forward and backward equations. This prior organizes the runtime measurements so the diagnostic model can link an observed deviation to a fault category and report the measurements behind each diagnosis. Training and evaluating DEFault++ requires transformer faults labeled at the component level, which no existing benchmark provides. We therefore construct DEFault-bench, a benchmark of 5,556 labeled instances from encoder and decoder architectures, generated with a transformer-specific mutation technique. On DEFault-bench, DEFault++ improves fault detection over four prior DNN debugging techniques, reaching an F1 of up to 0.91 for fault detection. To confirm that these findings extend to faults found in practice, we further evaluate DEFault++ on real-world faults collected from open-source repositories. In a developer study with 21 participants, repair-action accuracy rises from 57.1% without assistance to 83.3% with DEFault++.

In summary, we make the following contributions:

1. We propose DEFault++, a hierarchical technique that detects a fault in a transformer model, categorizes the affected component, and identifies the root cause within that component.

2. We construct DEFault-bench, a benchmark of 5,556 labeled transformer instances across encoder and decoder architectures, and release the transformer-specific mutation and fault-injection framework behind it, which others can reuse to generate further transformer fault data [18].

3. We define a Fault Propagation Graph (FPG), a structural prior that encodes the dependency paths among transformer components and gives the diagnostic model the transformer’s architectural structure.

4. We design a training objective for root-cause diagnosis that combines supervised contrastive learning with prototype matching, together with an FPG-based explanation that reports which feature groups support each diagnosis.

5. We evaluate DEFault++ on DEFault-bench, against four prior techniques, on real-world faults, and in a developer study with 21 participants.

2 Motivating Example↩︎

To show the kind of fault DEFault++ targets, consider the real-world fault in [lst:qkv95fusion95bug], taken from the Hugging Face Diffusers library (Issue #11903¹). The method fuse_qkv_projections merges the separate query, key, and value projections into one fused layer and routes the forward pass through that fused layer. The original projection modules stay in the model so that LoRA [19] can still attach adapters to them. The forward pass, however, no longer uses those modules, so LoRA updates parameters that no longer affect the model output. The run completes without exceptions, the loss decreases, and no invalid values appear. The only visible sign of the fault is that fine-tuning does not change model behavior.

Diagnosing this fault is hard for existing techniques. A static analysis that inspects the model structure or parameter list still finds valid q_proj, k_proj, and v_proj modules, so it cannot tell that their updates no longer reach the forward pass. Rule-based dynamic analyses that monitor training, such as AutoTrainer [11], UMLAUT [12], and DeepDiagnosis [13], do not flag the run because it shows none of their predefined symptoms, such as NaN or Inf values, exploding gradients, or oscillating loss. Learned classifiers, such as DeepFD [14] and DEFault [15], recognize predefined DNN fault categories such as activation-function faults, so they offer little help for a fault in a transformer QKV projection path. Transformer debuggers such as ATTNChecker [16] and AtPatch [17] inspect attention behavior more directly, but they target narrower symptoms such as invalid attention scores, which this example does not produce. In contrast, DEFault++ measures runtime traces such as QKV alignment, attention entropy, and projection update activity during training. These measurements characterize the faulty QKV projection path, so DEFault++ detects the fault, assigns it to the QKV category, and identifies the root cause as a stale projection update path.

3 Background and Related Work↩︎

Fault Taxonomies. Empirical studies of deep learning (DL) programs classify faults by root cause, covering faults in training procedures, hyperparameters, and layer configurations across feed-forward, convolutional, and recurrent architectures [20], [21]. These taxonomies supply the fault categories used by several DL diagnosis techniques, but they predate attention-based architectures and do not describe faults that arise from attention computation, masking, projection paths, positional encodings, key-value caching, or transformer-specific kernel behavior. In our prior work, we addressed part of this gap with a taxonomy of attention faults that classifies 555 attention faults from open-source projects into seven categories and 25 root causes [8], [22]. The present work builds its fault labels on these attention-specific categories together with five non-attention categories from the DL fault taxonomies [20], [21], which cover the surrounding model and training faults such as loss, learning-rate, and activation faults (6.1).
Mutation Testing for Deep Learning. Mutation testing assesses how well a test suite detects faults by introducing small, controlled changes into a program, called mutants, and checking whether the tests reveal them [23]. The deep learning community has adapted this idea to neural models. DeepMutation [24] and DeepMutation++ [25] introduce mutation operators that modify model structure, training code, or learned parameters, and report mutation scores to quantify test adequacy. DeepCrime [26] injects faults into the source code at training time, reruns training several times, and applies statistical tests with effect-size thresholds to decide whether a mutation changes behavior. This repetition matters because the same injected change can look harmful under one random seed and negligible under another [27]. PyTorchFI [28] injects faults at the tensor and bit levels to study the fault tolerance of PyTorch models. Mutation testing can also supply labeled training data for fault diagnosis. DeepFD [14] injects five common training faults, such as an incorrect learning rate, to create labeled samples for supervised diagnosis, and later work extends this idea to larger and more diverse sets of faulty DNN programs [15], [23], [29]. These datasets mostly cover architecture-agnostic DNN faults, such as loss misconfiguration, incorrect activation functions, or incorrect layer counts. They do not target the transformer-specific mechanisms or attention-related behavior needed to diagnose transformer faults (¿tbl:tab:related-comparison?).

4pt

Debugging Techniques for Deep Learning Programs. Existing DL debugging techniques fall into three broad groups. The first group learns fault patterns from training behavior. DeepFD [14] extracts four feature types, namely loss, gradient, weight, and activation features, and classifies five common training faults. Our prior work DEFault [15] extends this idea into a hierarchical classifier that first detects whether a program is faulty and then assigns it to one of seven generic DNN fault categories. These two methods are the closest dynamic baselines to our work, since they infer fault labels from training-time measurements, and DEFault++ extends our hierarchical design from generic DNN fault categories to transformer-specific components. They observe behavior at the model level and report model-level fault categories. As a result, they cannot tell whether a symptom comes from the QKV projection, masking, positional encoding, residual connections, normalization, key-value caching, or another transformer component. DEFault++ targets exactly this finer, component-level granularity.

10pt

Rule-based and heuristic techniques monitor predefined failure symptoms. AutoTrainer [11], UMLAUT [12], and DeepDiagnosis [13] check training-time symptoms such as exploding gradients, vanishing gradients, and oscillating loss. DeepLocalize [30] and GRIST [32] focus on numerical instability, while Tensfa [33] and TFCheck [34] check tensor shapes and training-process invariants. Other methods work at finer granularity, including MODE [35], DeepFault [36], Apricot [37], and DeepSeer [38]. These techniques work well when a fault produces one of the symptoms or structural patterns they monitor. They have limited reach, however, for transformer faults that keep loss curves normal, tensor shapes valid, and numerical values finite while changing internal attention behavior.

A third group of techniques analyzes a model statically, before any training run, so it operates at a different level of abstraction from ours. NeuraLint [39], DEBAR [40], and NerdBug [41] check meta-model graphs, tensor abstractions, and API usage, while DeepCheck [42] and CRADLE [43] analyze trained networks or compare backend implementations. FL4Deep [44] sits between the static and dynamic views, combining both through a knowledge graph to rank candidate root causes. These techniques find structural, symbolic, API-level, or backend-level faults, but they do not diagnose transformer-component faults from training traces. We therefore treat them as complementary to DEFault++ rather than as direct baselines.
Transformer-Specific Fault Analysis. A smaller body of work analyzes transformer-internal behavior, but each technique targets a narrower failure mode than the full transformer fault taxonomy (3). ATTNChecker [16] detects NaN or Inf values during attention computation, AtPatch [17] localizes attention-map anomalies at the patch level, FT-Transformer [31] studies hardware soft-error tolerance for vision transformers, and Bug Attention Probe [45] applies attention probing to code-understanding tasks. None of these techniques produces a combined hierarchy of fault detection, component category, and root-cause diagnosis. A related direction perturbs inputs to study model behavior [46]–[50]. Their goal differs from ours, since these methods expose behavioral failures through modified inputs, whereas we diagnose implementation faults that arise during ordinary training. Debuggers that use large language models (LLMs), such as SoapFL [51], LLM4FL [52], ChatDBG [53], and BugReAct [54], reason over source code, execution logs, or bug reports to localize faults in general software. They are also complementary, since they operate after a developer provides software artifacts or failure reports, while DEFault++ reads training-time traces from transformer components to infer a transformer-specific fault category and root cause.
Explainability for Fault Diagnosis. Diagnostic output is more useful when it gives developers evidence for a prediction rather than only a label, because the evidence shows where to look and gives a reason to trust the diagnosis. Existing explanation methods fall into two families. Post-hoc methods compute explanations after a trained model makes a decision. SHAP [55], DiCE [56], and Anchors [57] return feature attributions, counterfactual examples, or rule-based explanations, and our prior method DEFault adopts this post-hoc approach [15]. Inherently interpretable methods instead build explanation into the prediction mechanism. Prototype networks [58] explain a prediction through similarity to class prototypes, and concept-bottleneck models [59] route predictions through human-readable intermediate concepts. DEFault++ follows this interpretable-by-design approach and reports the training-time evidence behind each transformer fault diagnosis.

4 Study Design↩︎

Our study comprises three parts. We first construct DEFault-bench, a transformer fault benchmark built through mutation testing that provides labeled examples of correct and faulty fine-tuning behavior. We then design DEFault++, a hierarchical diagnostic technique that detects a fault, categorizes the affected component, and identifies its root cause, organized around how faults propagate across transformer components. Finally, we evaluate DEFault++ on DEFault-bench, on real-world faults, and through a developer study. We organize the evaluation around four research questions that move from whether the technique works, to how it compares with prior work, to which of its design choices matter, to whether the evidence it reports can be trusted.

5 Experimental Setup↩︎

This section describes the transformer models and tasks we study and the cross-validation protocol we use to evaluate generalization. The benchmark construction (6) and the diagnostic technique (7) both build on this setup.

5.1 Subject Models and Tasks↩︎

We select the transformer models for our study by three criteria. A model must be widely used, openly available through a standard Hugging Face Transformers implementation [60], and small enough, at most 125M parameters, to fine-tune repeatedly under matched seeds within our compute budget. Applying these criteria gives four encoder-only and three decoder-only models that together cover both architecture families, two depths, and distilled and full-size variants of the same family. The encoder models are BERT-base (110M parameters, 12 layers) [61], RoBERTa-base (125M, 12) [62], DistilBERT (66M, 6) [63], and DistilRoBERTa (82M, 6) [63]. The decoder models are GPT-2 (124M, 12) [64], GPT-Neo-125M (125M, 12) [65], and DistilGPT-2 (82M, 6) [63].

We select downstream tasks that are commonly adopted in prior transformer and deep learning studies and that come with openly released datasets. Each task is a publicly available benchmark with an established evaluation metric, and the set spans both classification and language modeling so that the injected faults are exercised under different training objectives. For the encoder models, we use five GLUE classification tasks [66], namely SST-2, QNLI, RTE, MRPC, and QQP, with accuracy as the task metric. For the decoder models, we use four language-modeling corpora, namely LAMBADA [67], PTB [68], WikiText-2 [69], and OpenWebText [70], with log-perplexity as the task metric. All models use their default tokenizers and the standard transformer-block structure that our mutation operators target.

We fine-tune every model with a single shared setup across tasks, namely AdamW [71] with linear warmup and mixed precision [27], [61], so that observed differences trace to the injected fault rather than to hyperparameter variation. We repeat each fine-tuning run under five fixed seeds \(\mathcal{S} = \{42, 123, 456, 789, 101112\}\) [27] and vary the affected layers and the fault severity uniformly, so that no single layer or magnitude dominates the benchmark.

5.2 Cross-Validation Protocol↩︎

We evaluate generalization to unseen model–task pairs using nested grouped cross-validation with \(k=5\) outer folds. The grouping unit is the model–task pair, so all runs from the same pair remain in the same fold. This prevents correlated traces from the same model and task from appearing in both training and test data. The encoder subset contains 20 model–task pairs (4 models \(\times\) 5 GLUE tasks), and the decoder subset contains 12 model–task pairs (3 models \(\times\) 4 language-modeling tasks). Because we repeat each run across seeds, we average those per-seed measurements into one example, so the random seed is not used as a grouping variable. The outer loop evaluates held-out model–task pairs, and the inner loop uses StratifiedGroupKFold for model selection [72]. We fit preprocessing steps and tune hyperparameters only inside the inner training fold, and all methods use the same outer-fold assignments. Because each outer fold holds out whole model–task pairs, a fold-level standard deviation would mostly reflect which pairs are held out rather than training noise, so we report results at the model–task level.

6 DEFault-bench: Benchmark Construction↩︎

Training and evaluating a component-level diagnostic technique requires transformer faults labeled by component, which no existing benchmark provides. We therefore construct DEFault-bench, a benchmark of clean and faulty fine-tuning runs for the seven transformer models and nine tasks introduced in 5. We create the faulty runs with a transformer-specific mutation technique that injects documented attention and DNN faults into the components of the transformer architecture, either by changing stored parameters or by changing computations during the forward pass. We organize these faults with a fault taxonomy (6.1) and realize each one as a mutation operator (6.2). Each injected fault is one configuration \(\mathcal{C} = (m, t, u, f, v, \ell, \sigma)\) that fixes the model, task, target component, fault category, mutation variant, target layer, and severity. We validate each mutant in two steps. We first check that the intended structural change was applied. We then test whether the faulty run behaves differently from its matched clean run under the same seeds, following prior mutation-testing work [24], [26] (1).

6.1 Fault Taxonomy↩︎

We base the fault taxonomy on the main components of a transformer architecture (2). A transformer model contains input embeddings, a stack of repeated transformer blocks, and an output projection head [6]. Each block contains multi-head self-attention (MHA), a position-wise feed-forward network (FFN), residual connections, and layer normalization (LayerNorm). For an input sequence \(\mathbf{x} \in \mathbb{R}^{n \times d}\), a post-norm block adds each sublayer’s output back to its input and then normalizes the sum, computing \(\mathbf{h} = \mathrm{LayerNorm}(\mathbf{x} + \mathrm{MHA}(\mathbf{x}))\) for the attention sublayer and \(\mathbf{y} = \mathrm{LayerNorm}(\mathbf{h} + \mathrm{FFN}(\mathbf{h}))\) for the feed-forward sublayer. We group faults by six components, namely embeddings, MHA, FFN, layer normalization, residual connections, and the output projection head. The same component set applies to pre-norm variants, where only the position of LayerNorm changes.

We then map these transformer components to documented fault categories and root causes (3). The attention-related categories come from our prior fault taxonomy for attention-based neural networks (ABNNs), which we derived from 555 real-world attention faults [8], [22]. The non-attention categories come from two existing DNN fault taxonomies based on 970 faults in total [20], [21].

The seven attention categories map to different parts of the attention computation [6], [8], illustrated in the attention-internal view of 2, \[\mathrm{Attention}(Q, K, V) = \mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}} + M\right)V \label{eq:dpp-attention}\tag{1}\] where \(Q\), \(K\), and \(V\) are linear projections of the input, \(d_k\) is the key dimension, and \(M\) is an additive mask. Masking faults corrupt the mask \(M\) through padding masks, causal masks, or mask reshaping across batch and head dimensions. QKV, Score, and Positional faults affect the projections, the attention-score computation, and the positional information, respectively. Kernel faults involve the configured attention backend or numerical execution path. Variant faults use the wrong form of attention, such as single-head attention or causal masking in an encoder. KV Cache faults apply to autoregressive decoders and affect cache-state management [73]. The remaining components each contribute one category, following prior DNN fault taxonomies [20], [21]. Embedding faults involve token, segment, or feature-dimension embeddings. FFN faults cover feed-forward weights, neurons, and activation functions. LayerNorm faults involve the learned scale \(\gamma\), bias \(\beta\), numerical-stability term \(\epsilon\), or regularization of LayerNorm parameters [27], [74]. Residual faults cover skip connections and residual dropout, and Output faults involve the prediction head or logits.

Overall, the taxonomy contains 12 fault categories and 45 root causes, including five KV Cache root causes that apply only to autoregressive decoders. Transformer models without autoregressive cache behavior, such as encoder-only models, use 11 of these categories and 40 root causes (3).

3pt

6.2 Mutation Operators↩︎

To generate faults systematically, we realize each root cause as one or more mutation operators, where a mutation operator is a specific code change that injects that root cause into a transformer model. We read the root causes from the taxonomy in 3 and define an operator for each distinct way a root cause can be realized as a code change, which yields 52 operators across the 12 fault categories. We present the operators in two tables that follow the two taxonomy sources. The 29 attention-specific operators come from our attention fault taxonomy and target masking, the QKV projection, score computation, positional information, the attention kernel, attention variants, and the decoder KV cache (2). The 23 architecture-level operators come from the DNN fault taxonomies and target embeddings, the FFN, layer normalization, residual connections, and the output head (3). This grouping reflects where a fault originates in the model, which is separate from the run-time mechanism used to inject it (6.3).

We give each operator a three-letter identifier, following the naming style of prior DL mutation work [24], [26]. The first letter identifies the transformer component, and the remaining two letters form a short mnemonic for the change. For example, QZQ uses Q for the QKV component, Z for zeroing, and Q for the query projection, while SPD uses S for the Score component, P for pre-softmax, and D for dropout. A fault category can hold more than one operator because one root cause can be triggered through more than one separate change. The QKV parameter-initialization root cause, for example, has three operators that zero the query, key, or value projection (QZQ, QZK, QZV), since each projection is a separate entry point for the same root cause. This is why the 52 operators map to 45 root causes rather than one operator each.

The Search Type (ST) column records how an operator selects the parameter listed in the Executable Mutation Parameters column. A B (binary) operator takes no parameter and applies a single fixed change. A EU operator has a numeric parameter swept over a predefined list of values, such as a scaling factor or a dropout probability. A EL operator has a categorical parameter chosen from a fixed set, such as FCA, which replaces the activation function with each function in \(\{\mathrm{ReLU}, \mathrm{GELU}, \mathrm{Tanh}, \mathrm{Sigmoid}\}\). For parameterized operators, our mutation technique evaluates low, medium, and high settings, so the benchmark spans a range of fault strengths rather than only extreme mutants. Extreme mutants carry little diagnostic information, since almost any test exposes them [26].

3pt

6.3 Fault Injection Mechanism↩︎

We inject every fault programmatically, so each configuration is applied, evaluated, and reverted without manual edits to the model source, and we release this injection framework in our replication package [18]. We denote the clean model parameters by \(\theta\) and the injected mutant by \(\theta'\), and we evaluate the two under matched random seeds and otherwise identical fine-tuning conditions, so that any measured difference reflects the injected configuration \(\mathcal{C}\). Recall that \(\mathcal{C} = (m, t, u, f, v, \ell, \sigma)\) fixes the model architecture, the downstream task, the target unit, the fault category, the fault variant, the affected layer indices \(\ell \subseteq \{1,\dots,L\}\) for a model with \(L\) transformer blocks, and the severity level \(\sigma \in \{\mathrm{low}, \mathrm{medium}, \mathrm{high}\}\). The severity sets the magnitude of the injected change for numeric variants, such as scaling factors and dropout rates, and maps to a discrete intensity for non-numeric variants, such as the visibility ratio in causal-mask breaking. We use only single-fault configurations, with one target unit, one category, and one variant per run, so that each observed change traces to one injected fault. Transformer faults can affect either stored parameters or forward-pass computations, so injection uses two mechanisms [16], [28].

Static faults change parameter tensors at rest, before forward execution. An FFN weight-scaling fault, for example, multiplies the targeted weight matrices by a scalar, \[\theta'_i = \begin{cases} \alpha \cdot \theta_i & \text{if } i \in \mathrm{params}(u,\ell),\\[0.2ex] \theta_i & \text{otherwise} \end{cases} \label{eq:dpp-static-fault}\tag{2}\] where \(\mathrm{params}(u,\ell)\) selects the parameters of unit \(u\) in layers \(\ell\). Static faults include embedding corruption, parameter scaling, permanent neuron dropout, and LayerNorm parameter changes. Dynamic faults change the forward computation path through wrappers or hooks at selected module call sites. A mask-zeroing fault, for example, intercepts the attention call and replaces the mask \(M\) with zeros, \[\texttt{forward}'(x, M) = \texttt{forward}(x, \mathbf{0}_{|M|}) \label{eq:dpp-dynamic-fault}\tag{3}\] where \(\mathbf{0}_{|M|}\) is a zero tensor of the same shape as \(M\). Dynamic faults include mask manipulation, \(Q\)/\(K\) swapping, removal of score scaling, and forced kernel selection. In both mechanisms the transformer architecture stays unchanged, and only the targeted computation or parameter values change. 4 shows one representative example of each mechanism.

Masking faults need special handling for decoder-only models, where self-attention is causal by construction and a token at position \(i\) cannot attend to a future position \(j>i\). The causal mask enforces this by placing \(-\infty\) in the future positions of the additive mask \(M\), so that softmax assigns those positions zero weight. To inject a causal violation, we weaken this constraint rather than add a new mask. We replace a configurable fraction of those \(-\infty\) entries in the upper-triangular region of \(M\) with \(0\), controlled by the severity \(\sigma\), so each query can attend to a proportion of keys beyond its own position. We treat causal violations as decoder instances of attention masking faults.

To keep clean and faulty runs comparable, we hold the main experimental settings fixed between the \(\theta\) and \(\theta'\) runs, such as the dataset, optimizer, and number of epochs, and we pair each faulty run with a clean run under the same seed. A context manager carries out the injection. It stores the original parameter tensors and forward methods, applies the fault specified by \(\mathcal{C}\), and restores the original state after evaluation. Restoring the state at the end of each configuration prevents one injected fault from carrying over into the next run.

Finally, we apply a structural verification check to each injected fault. For static faults, we verify that the parameter difference is restricted to \(\mathrm{params}(u,\ell)\) and that its magnitude matches \(\sigma\) within a relative tolerance of \(10^{-6}\) to account for floating-point rounding. For dynamic faults, we verify that hooks or wrappers attach only to the intended module call sites and that the original forward function is restored after the context manager exits. We also verify execution completeness by requiring each configuration to produce training logs, and we exclude configurations that crash or raise runtime errors.

6.4 Mutant Validation↩︎

Injecting a fault does not guarantee that it changes model behavior, so we decide which mutants to keep through a statistical mutation-killing test, following prior work [26]. For each configuration, 5 injects the fault, verifies the structural change, fine-tunes the clean and mutant models under the shared seeds, and applies the killing test to the paired results.

We fine-tune the clean model \(N\) (parameters \(\theta\)) and the mutant \(M\) (parameters \(\theta'\)) each \(n\) times under matched random seeds, and we compare their performance distributions \(A_N(\mathit{TestS}) = \langle A_{N_1}, \ldots, A_{N_n}\rangle\) and \(A_M(\mathit{TestS}) = \langle A_{M_1}, \ldots, A_{M_n}\rangle\) on a held-out test set \(\mathit{TestS}\). The predicate \(\mathit{isKilled}\) decides whether the mutant is killed: \[\mathit{isKilled}(N, M, \mathit{TestS}) = \begin{cases} \mathit{true} & \text{if } p\text{-value}\bigl(A_N(\mathit{TestS}),\, A_M(\mathit{TestS})\bigr) < \alpha, \\ \mathit{false} & \text{otherwise.} \end{cases} \label{eq:dpp-iskilled}\tag{4}\]

We use accuracy as the test-set metric for encoder classification tasks and log-perplexity for decoder language-modeling tasks [8], and we evaluate each clean and faulty configuration with the same \(n=5\) seeds at \(\alpha=0.05\). The matched seeds let us compare each faulty run directly with its clean counterpart. For each mutation, we compute the five paired differences and test whether they move consistently in the expected direction with a one-sided paired sign-flip permutation test [75]. This test asks whether the observed direction of the paired differences is unlikely under the null hypothesis that the fault has no effect. With five paired comparisons, the smallest possible one-sided exact \(p\)-value is \(1/2^5 \approx 0.031\), which lies below \(\alpha=0.05\), so five matched seeds suffice for this exact test. To confirm that the test does not flag ordinary training noise as a fault, we also apply \(\mathit{isKilled}\) to pairs of clean runs that differ only in random seed, and the rate at which these clean-versus-clean pairs are flagged estimates the false-positive rate. We summarize each operator’s effectiveness with the mutation score [26]:

\[\mathit{MS}(\mathit{MO}) = \frac{|\{c \in \mathit{MO} \,:\, \mathit{isKilled}(N, M_c, \mathit{TestS})\}|}{|\mathit{MO}|} \label{eq:dpp-ms}\tag{5}\]

where \(\mathit{MO}\) is the set of injected configurations of an operator, and we report the overall mutation score as the average of \(\mathit{MS}(\mathit{MO})\) across operators. For each killed mutant we record a label \(y = (u, f, v, \ell, \sigma)\) with the target unit, fault category, fault variant, affected layer set, and severity. We omit \(m\) and \(t\) because each instance comes from one model–task context. A surviving mutant, for which \(\mathit{isKilled}\) returns false, is an injected fault whose measured effect does not pass the killing test, so its label cannot be inferred reliably. We discard surviving mutants before building the final benchmark instances.

6.5 Benchmark Statistics↩︎

After structural verification, we obtain 4,096 single-fault configurations (1,955 for encoders and 2,141 for decoders), with every category represented. Validating these mutants requires 40,960 paired clean and faulty fine-tuning runs, and constructing the balanced correct class adds 13,890 clean-variant runs, for a total of 54,850 fine-tuning runs and about 27,000 GPU-hours on NVIDIA A100 and H100 GPUs. We apply \(\mathit{isKilled}\) to all structurally valid configurations and compute the mutation score by category and architecture (4, 7). Among encoder configurations, 1,598 of 1,955 mutants are killed (81.7%). Among decoder configurations, 1,180 of 2,141 mutants are killed (55.1%). Encoder mutation scores are high across most categories. Variant reaches 100%, and Score, QKV, Positional, FFN, Residual, LayerNorm, and Embedding all exceed 80%, while Kernel, Output, and Masking are the lowest. Decoder mutation scores vary more across categories and model–task pairs (6). Masking and Kernel reach the highest decoder scores, and Score the lowest. The low Score rate is consistent with how softmax dampens score-level changes. Softmax maps the pre-softmax logits to a normalized distribution, so a moderate score change can be partly absorbed before it reaches the attention weights [6], [10], [76]. For decoder language modeling, this can keep the perplexity change below the threshold the killing test uses.

6pt

After discarding surviving mutants, we build the correct class from clean runs and label-preserving clean variants. Using a single clean run per base model could let a classifier learn model-specific artifacts rather than fault-related behavior. We therefore generate clean variants by changing factors that should not affect task behavior, such as the random seed, the training-data order where applicable, and hyperparameters within ranges that stay below the significance threshold. We evaluate each clean variant against its base model with the same significance test used for killed mutants, retain variants that remain statistically indistinguishable from the base model, and discard variants that meet the killed criterion. For each base model \(B\) that produces \(k\) killed mutants, we keep \(k\) clean variants from the same base model, which holds the faulty-to-correct ratio balanced within each base-model stratum. DEFault-bench contains 5,556 instances, with 3,196 encoder and 2,360 decoder instances (4).

Each instance carries labels at three levels. The Level 1 label for fault detection is binary, either faulty or correct. Faulty instances also carry Level 2 and Level 3 labels. The Level 2 label is the injected fault category. Encoders use 11 categories, namely Score, Positional, FFN, QKV, LayerNorm, Residual, Masking, Embedding, Kernel, Output, and Variant, and decoders use the same 11 plus KV Cache. The Level 3 label is the injected root cause within the fault category. Each category contains 2–5 root causes, and each root cause belongs to exactly one category.

7 DEFault++ : Hierarchical Diagnostic Technique↩︎

DEFault++ takes the 5,556 labeled instances from DEFault-bench as input, each with the runtime measurements collected during training, and diagnoses them through the three-level hierarchy in 8. Level 1 uses the full feature representation to detect whether an instance is faulty. Only instances predicted as faulty pass to Level 2, which assigns a fault category. Level 3 then predicts the root cause within that category by comparing the instance to learned class prototypes, in an embedding space organized by the Fault Propagation Graph (FPG), the structural prior over transformer components defined in 7.1. The same embedding space produces the feature-group importance scores that explain each diagnosis.

7.1 Fault Propagation Graph (FPG)↩︎

The fault categories and root causes in our benchmark identify where a fault originates. They do not, however, explain how the effects of a fault can propagate to other transformer components. For example, a QKV projection fault can change the projection output and then affect attention scores, attention weights, and the residual stream. To represent these dependency paths, we define the Fault Propagation Graph (FPG), a directed graph whose nodes are transformer components and whose edges represent data-flow dependencies (9). The FPG is deterministic because it follows the transformer computation graph. Each edge marks a possible path through which a fault can affect another component. The magnitude and representation of that effect can still change as it passes through learned transformations and operations such as softmax, LayerNorm, and activation functions.

We define the FPG as \(\mathcal{G} = (\mathcal{V}, \mathcal{E})\). Each node \(v \in \mathcal{V}\) represents a transformer component. Residual connections and LayerNorm appear separately for the attention and FFN sublayers. Decoder architectures also include a KV cache node. Each directed edge \((v_i, v_j) \in \mathcal{E}\) marks a dependency derived from the forward equations, residual structure, or autoregressive state reuse.

We derive the edges by analyzing the transformer’s forward and backward equations, which yield seven dependency mechanisms (9). We write \(\delta x\) for a small change to a variable \(x\), and each mechanism below describes how such a change moves from one component to another.

1. M1: Forward sequential propagation. If \(B = f(A)\), then a change in \(A\) can change \(B\) according to \(\delta B = (\partial f / \partial A)\, \delta A\). This mechanism captures the main forward path through embeddings, attention computation, residual connections, LayerNorm, FFN, and the output head.

2. M2: Simultaneous propagation. QKV projections feed both score computation (via \(Q, K\)) and attention output (via \(V\)). A single projection fault can therefore affect scores and values at the same time. In decoders, \(K\) and \(V\) also enter the cache.

3. M3: Residual bypass. In \(h = x + \text{MHA}(x)\), the skip path carries changes in \(x\) directly to the residual output. Later normalization steps can still adjust the magnitude or representation of the propagated change.

4. M4: Cross-layer propagation. The residual stream provides repeated identity paths across stacked layers [77]. A change at layer \(\ell\) can therefore reach later representations, although intermediate normalization and nonlinear operations may reduce or reshape it.

5. M5: Backward pass propagation. A fault that changes the loss \(\mathcal{L}\) can change \(\partial \mathcal{L} / \partial \theta_i\) for parameters whose computation contributes to that loss, which can affect weight updates across multiple components during training.

6. M6: Architecture-wide intervention. Some fault types affect multiple components at once. A variant fault (e.g., single-head instead of multi-head attention) changes the attention computation pattern, parameter shapes, and kernel dispatch. This mechanism is a joint structural effect rather than a propagation relation.

7. M7: Cache propagation across time steps. In decoder architectures, the KV cache stores \(K\) and \(V\) across generation steps. A cache fault can affect the current token and future tokens.

Three nonlinear operations limit propagation along specific edges. Softmax maps changes into a normalized distribution and reduces large shifts. LayerNorm [78] rescales activations by their mean and variance. Activation functions such as ReLU and GELU suppress or pass changes depending on their operating region [79], [80]. We annotate these limiting operations on the corresponding FPG edges. The seven mechanisms fall into three classes by how the diagnostic model uses each one, namely propagation relations that become graph edges, recursive composition that repeats across stacked layers, and a structural intervention that the model reads from the fault labels rather than from edges (5). The propagation relations (M1, M2, M3, M5, M7) describe paths through which a change reaches another component, and the message-passing graph (7.3) uses the forward relations (M1, M2, M3, M7) as edges while representing the backward relation (M5) through gradient features. Recursive composition (M4) repeats M1 and M3 across stacked layers, and the structural intervention (M6) is a joint effect in which one architectural change alters several components at once.

4pt

7.2 Feature Representation↩︎

We represent each training run with features that capture component-level behavior. The FPG identifies the component paths along which a fault can propagate, and we use this structure to organize the collected metrics. Architecture-agnostic DNN metrics, such as loss trajectories, accuracy, and gradient statistics, are often too broad to separate transformer-specific fault categories. A fault in a QKV projection, a mask, or a positional encoding may barely move accuracy or perplexity while clearly changing attention-level and component-level measurements. We therefore collect metrics for attention distributions, projection alignment, residual-stream behavior, optimization behavior, training dynamics, and validation behavior, and we organize them into feature groups, where each feature group holds the metrics for one transformer component or one model-wide aspect of training (10, 6).

By how they are measured, these feature groups fall into four families, and 6 lists the metric types in each group, where \(n_g\) is the number of metric types in group \(g\) before aggregation. Layer-internal metrics capture attention and component behavior inside transformer layers, since attention distributions [48], [49], representation geometry [81], [82], normalization parameters [74], [78], and residual propagation [77] can each vary across layers. They total \(C_{int}=15\) metric types for encoders and \(C_{int}=16\) for decoders, where the decoder adds future attention mass. Gradient metrics total \(C_{opt}=21\) types and record gradient norm, update ratio, and update activity across parameterized components and global gradients [11], [14], [15], [30]. Behavioral metrics total \(C_{train}=10\) types for encoders and \(12\) for decoders and capture output-head behavior, positional sensitivity, step time, peak memory allocation, and decoder cache behavior. Validation metrics total \(C_{eval}=2\) types and capture task accuracy or perplexity and calibration error at validation checkpoints [83].

4pt

Training runs can differ in model depth, number of steps, and number of epochs. We therefore aggregate the raw measurements at the layer, epoch, and training-phase levels, as shown in 10. Layer aggregation reduces each per-layer metric to \(A_\ell=5\) statistics, preserving early, middle, and final-layer behavior while removing dependence on model depth. Epoch-level aggregation reduces the step-level sequence within each epoch to \(A_e=3\) statistics. Training-phase aggregation reduces the epoch sequence to \(A_p=5\) statistics. This aggregation produces one feature vector per training run while retaining information about the affected component and the training phase in which the effect appears. Combining the per-family metric-type counts with these aggregation factors gives the length of the final feature vector,

\[d_{\text{final}} = A_p\!\left(A_e\!\left(A_\ell \, C_{int} + C_{opt} + C_{train}\right) + C_{eval}\right), \label{eq:feature95vector95length}\tag{6}\]

6 gives \(1{,}600\) features for encoders (\(C_{int} = 15\), \(C_{opt} = 21\), \(C_{train} = 10\), \(C_{eval} = 2\)) and \(1{,}705\) features for decoders, where \(C_{int} = 16\) adds future attention mass and \(C_{train} = 12\) adds two cache metrics. We remove near-constant columns using coefficient-of-variation filtering (\(\mathrm{CV} < 0.01\)). DEFault++ uses the filtered fixed-length feature vector as input to the diagnostic model.

7.3 Diagnostic Model↩︎

DEFault++ performs hierarchical diagnosis for one input instance at a time. Level 1 detects whether the instance is faulty. If Level 1 predicts a fault, Level 2 assigns the instance to a fault category. Level 3 then predicts the root cause within that category and computes feature-group importance scores for the diagnosis.
Feature-Group Encoding. DEFault++ first encodes each feature group (6) with a dedicated multilayer perceptron (MLP), which keeps the component-level organization of the measurements. Encoders and decoders have different numbers of feature groups, \(G = 12\) for encoders and \(G = 13\) for decoders, since the KV cache group applies only to decoders, so we train a separate diagnostic model for each architecture.

Given a feature vector \(\mathbf{z} \in \mathbb{R}^d\) partitioned into \(G\) groups \(\{\mathbf{z}_{g_1}, \ldots, \mathbf{z}_{g_G}\}\) (see 6), each group \(g\) is encoded as \[\mathbf{h}_g = \text{MLP}_g(\mathbf{z}_g) \in \mathbb{R}^{h}, \quad g = 1, \ldots, G, \label{eq:group95encoding}\tag{7}\] where \(h = 32\) is the hidden dimension per group, selected by grid search on the inner validation fold. The group embeddings form the matrix \(\mathbf{H} = [\mathbf{h}_1; \ldots; \mathbf{h}_G] \in \mathbb{R}^{G \times h}\).

A learnable projection matrix \(W_{\text{proj}}\) then maps the concatenated group embeddings to a single shared representation, \[\mathbf{z}_{\text{proj}} = W_{\text{proj}} \, \text{vec}(\mathbf{H}) \in \mathbb{R}^{e}, \label{eq:projection}\tag{8}\] where \(e = 64\) is the embedding dimension and \(\text{vec}(\mathbf{H}) \in \mathbb{R}^{Gh}\) denotes the row-major vectorization of \(\mathbf{H}\). Levels 1 and 2 use this shared vector \(\mathbf{z}_{\text{proj}}\) for fault detection and fault categorization, while Level 3 uses the group embedding matrix \(\mathbf{H}\) to preserve group-level structure for root-cause diagnosis and explanation.
Message Passing in the Fault Propagation Graph (FPG). We use the FPG to exchange information between related feature groups. Each round updates a feature group’s embedding using its current value and the embeddings of the feature groups connected to it by FPG edges. After several rounds, a group embedding includes evidence from nearby groups in the FPG. For example, the QKV alignment group can aggregate evidence from the score, attention, and residual-stream groups, which lie on the propagation paths of QKV projection faults.

The group-level graph uses the forward propagation mechanisms defined in the FPG. Backward effects are captured by gradient-based feature groups instead of graph edges. We construct the adjacency matrix \(\mathbf{A} \in \mathbb{R}^{G \times G}\) by mapping FPG components to their corresponding feature groups (7). A group-level edge exists when any component in the source group has an FPG edge to any component in the target group. Components mapped to the same feature group are merged, and groups that cover multiple components inherit the edges of those components. Representation drift, Training dynamics, and Validation performance do not correspond to individual transformer components, so they use self-loops only.

7pt

For the decoder diagnostic model (\(G = 13\)), 11 shows the row-normalized adjacency matrix with self-loops, \(\hat{\mathbf{A}}\). Nonzero values near the main diagonal follow the sequential data flow through the transformer block. Off-diagonal values capture branching paths such as QKV\(\to\)Cache, Cache\(\to\)Attention, and FFN\(\to\)Residual. Representation drift, Training dynamics, and Validation performance appear as isolated self-loops, since they provide model-wide context rather than lying on a component-level propagation path.

Each round mixes every group embedding with those of its FPG neighbors and then applies a learned transformation. Following graph convolutional networks [84], one round of message passing is \[\mathbf{H}' = \operatorname{ReLU}\left(\hat{\mathbf{A}}\mathbf{H}W_{\text{msg}}\right), \label{eq:message95passing}\tag{9}\] where \(\hat{\mathbf{A}}\) is the row-normalized adjacency matrix with self-loops and \(W_{\text{msg}} \in \mathbb{R}^{h \times h}\) is a learnable weight matrix. We use row normalization because feature groups have different numbers of FPG neighbors, and it keeps the message-passing updates on a comparable scale across groups.

We apply three rounds of message passing, with a separate \(W_{\text{msg}}\) for each round. We select the number of rounds by grid search over \(\{1,\dots,5\}\) on the inner validation fold. Three rounds cover short FPG paths, such as QKV \(\to\) Score \(\to\) Attention \(\to\) Residual, while avoiding deeper aggregation that can oversmooth node representations [85]. DEFault++ uses the updated group embeddings in the following diagnosis stages.

Hierarchical Classification. DEFault++ uses a shared encoder across the three diagnosis levels. The encoder combines feature-group encoding, FPG message passing, and projection into a common representation. Each diagnosis level then applies its own prediction head to this representation. Sharing the encoder avoids separate representations for closely related tasks and keeps the model small, since message passing operates over only 12 feature groups for encoders and 13 for decoders.
\(\scalebox{0.7}{\bullet}\) Level 1: Fault detection. The first level performs binary detection. A two-layer MLP maps \(\mathbf{z}_{\text{proj}}\) to logits for faulty and correct instances.
\(\scalebox{0.7}{\bullet}\) Level 2: Fault categorization. Instances predicted as faulty are passed to the second level, which assigns them to a fault category. A two-layer MLP maps \(\mathbf{z}_{\text{proj}}\) to \(C\) category logits, where \(C = 11\) for encoders and \(C = 12\) for decoders since the KV Cache category applies only to decoders.
\(\scalebox{0.7}{\bullet}\) Level 3: Root-cause diagnosis. The third level predicts the root cause within the fault category selected by Level 2. Since each category contains only 2–5 root causes and per-root-cause support is limited, DEFault++ uses a prototypical classifier [58], [86], which represents each root cause by a single point in the embedding space, its prototype, and classifies an instance by its distance to those points. A prototype is the mean group embedding of the training examples for a root cause, and at inference DEFault++ assigns the instance to the nearest root-cause prototype within the predicted category.

Let \(\mathcal{D}_{c,r}\) denote the training samples with fault category \(c\) and root cause \(r\). For each root cause \(r\) within category \(c\), we compute the prototype \(\boldsymbol{\pi}_{c,r}\) as the mean group embedding: \[\boldsymbol{\pi}_{c,r} = \frac{1}{|\mathcal{D}_{c,r}|} \sum_{i \in \mathcal{D}_{c,r}} \mathbf{H}_i \in \mathbb{R}^{G \times h}. \label{eq:prototype}\tag{10}\]

DEFault++ compares an input instance with the prototypes from the predicted category. The squared distance to a prototype decomposes over feature groups: \[d(\mathbf{H}, \boldsymbol{\pi}_{c,r}) = \sum_{g=1}^{G} \| \mathbf{h}_g - \boldsymbol{\pi}_{c,r,g} \|^2 = \sum_{g=1}^{G} d_g, \label{eq:proto95distance}\tag{11}\] where \(d_g = \| \mathbf{h}_g - \boldsymbol{\pi}_{c,r,g} \|^2\) is the contribution of feature group \(g\). DEFault++ predicts the nearest root-cause prototype: \[\hat{r} = \arg\min_{r \in \mathcal{R}_{\hat{c}}} d(\mathbf{H}, \boldsymbol{\pi}_{\hat{c},r}).\] The group-wise decomposition of the prototype distance allows DEFault++ to estimate each feature group’s contribution to the root-cause decision.

7.4 Training Objective↩︎

DEFault++ trains the three diagnosis levels jointly with a shared encoder. During training, the detection loss, categorization loss, and root-cause diagnosis losses all update the shared encoder.

The training objective has four components. Level 1 uses binary cross-entropy, \(\mathcal{L}_{\text{detect}}\), over all samples in the batch. Level 2 uses class-weighted cross-entropy, \(\mathcal{L}_{\text{cat}}\), over fault categories and applies it only to faulty samples. Level 3 uses two losses, a per-category cross-entropy loss \(\mathcal{L}_{\text{rc}}\) for root-cause prediction and a separation loss \(\mathcal{L}_{\text{sep}}\) for prototype-based diagnosis.

For root-cause training, DEFault++ learns root-cause distinctions separately within each fault category. For each fault category, DEFault++ uses a separate root-cause head. The head takes \(\mathbf{z}_{\text{proj}}\) as input and predicts among the root causes in that category. For example, QKV root causes are compared with other QKV root causes, not with LayerNorm or FFN root causes. At inference, the model first predicts a fault category and then diagnoses the root cause within that category. We compute \(\mathcal{L}_{\text{rc}}\) only over faulty samples within their ground-truth fault category.

The cross-entropy loss used for root-cause diagnosis trains the per-category heads, but prototype-based diagnosis also depends on the group embedding space \(\mathbf{H}\). Root causes from the same fault category may still have similar group embeddings, which can make nearest-prototype diagnosis less reliable. DEFault++ therefore uses \(\mathcal{L}_{\text{sep}}\) to separate root causes in the group embedding space. The separation loss combines supervised contrastive learning [87] with prototype matching [58]. The contrastive term separates embeddings of different root causes within the same category, while the prototype-matching term moves each sample closer to its own root-cause prototype.

The contrastive term uses one vector per training sample, computed by flattening the group embedding matrix: \(\mathbf{u}_i = \text{vec}(\mathbf{H}_i)\). For a training sample \(i\), the comparison set is restricted to samples from the same fault category. Within this set, positives are samples with the same root cause as \(i\), and negatives are samples with a different root cause. The contrastive term therefore pulls together embeddings from the same root cause and separates embeddings from different root causes within the same category. Following supervised contrastive learning [87], the contrastive term for category \(c\) is \[\mathcal{L}_{\text{ctr}}^{(c)} = -\frac{1}{|\mathcal{P}_c|} \sum_{(i,j) \in \mathcal{P}_c} \log \frac{\exp(\text{sim}(\mathbf{u}_i, \mathbf{u}_j) / \tau)}{\sum_{\substack{k \ne i \\ c_k = c_i}} \exp(\text{sim}(\mathbf{u}_i, \mathbf{u}_k) / \tau)} \label{eq:contrastive}\tag{12}\] where \(\mathcal{P}_c\) is the set of positive pairs within category \(c\), \(\text{sim}(\cdot, \cdot)\) is cosine similarity, and \(\tau = 0.1\) is the temperature. The denominator excludes sample \(i\) itself and sums over the other samples in the batch that share the same fault category. We compute \(\mathcal{L}_{\text{ctr}}\) by averaging \(\mathcal{L}_{\text{ctr}}^{(c)}\) over categories with at least one positive pair in the batch.

The prototype-matching term uses the grouped squared distance from 11 . It encourages each sample to lie closer to its own root-cause prototype than to other prototypes in the same category: \[\mathcal{L}_{\text{pm}} = -\frac{1}{|\mathcal{B}|} \sum_{i \in \mathcal{B}} \log \frac{\exp(-d(\mathbf{H}_i, \boldsymbol{\pi}_{c_i, r_i}) / \tau_p)}{\sum_{r' \in \mathcal{R}_{c_i}} \exp(-d(\mathbf{H}_i, \boldsymbol{\pi}_{c_i, r'}) / \tau_p)} \label{eq:proto95matching}\tag{13}\] where \(\mathcal{B}\) is the training batch, \(\mathcal{R}_{c_i}\) is the set of root causes within category \(c_i\), and \(\tau_p\) is a temperature parameter.

The separation loss combines these two terms: \[\mathcal{L}_{\text{sep}} = \beta\,\mathcal{L}_{\text{ctr}} + \gamma\,\mathcal{L}_{\text{pm}}. \label{eq:separation95loss}\tag{14}\]

Finally, the full DEFault++ objective is \[\mathcal{L}_{\text{DEFault++}} = \mathcal{L}_{\text{detect}} + \alpha\,\mathcal{L}_{\text{cat}} + \lambda\,\mathcal{L}_{\text{rc}} + \mathcal{L}_{\text{sep}}. \label{eq:total95loss}\tag{15}\] We apply \(\mathcal{L}_{\text{sep}}\) only to faulty samples within their ground-truth category. We set \(\alpha = 1.0\), \(\lambda = 1.0\), \(\beta = 0.5\), and \(\gamma = 0.3\) through grid search.

7.5 FPG-based Explanation↩︎

Level 3 predicts the root cause by selecting the nearest prototype in the grouped embedding space. To explain this prediction, DEFault++ compares the predicted prototype \(\boldsymbol{\pi}_{\hat{c},\hat{r}}\) with the nearest alternative prototype \(\boldsymbol{\pi}_{\hat{c},\tilde{r}}\) in the predicted fault category: \[\tilde{r} = \arg\min_{r \in \mathcal{R}_{\hat{c}} \setminus \{\hat{r}\}} d(\mathbf{H}, \boldsymbol{\pi}_{\hat{c},r}).\] For each feature group \(g\), DEFault++ computes how much closer the group embedding is to the predicted prototype than to the nearest competing prototype: \[\Delta_g = d_g(\boldsymbol{\pi}_{\hat{c},\tilde{r}}) - d_g(\boldsymbol{\pi}_{\hat{c},\hat{r}}), \quad g = 1, \ldots, G, \label{eq:group95contribution}\tag{16}\] where \(d_g(\boldsymbol{\pi}_{\hat{c},r}) = \| \mathbf{h}_g - \boldsymbol{\pi}_{\hat{c},r,g} \|^2\). A positive \(\Delta_g\) means that feature group \(g\) supports the predicted root cause over the nearest competing root cause. A negative \(\Delta_g\) means that the group is closer to the competing prototype. Since the full prototype distance is the sum of group-level distances, \(\sum_g \Delta_g\) equals the distance gap between the predicted prototype and the nearest competing prototype.

DEFault++ normalizes the positive group contributions into importance scores: \[w_g = \frac{\max(\Delta_g, 0)}{\sum_{g'=1}^{G} \max(\Delta_{g'}, 0)}, \quad g = 1, \ldots, G. \label{eq:explanation}\tag{17}\] Groups with negative contributions receive \(w_g = 0\). The importance scores sum to one and identify the feature groups that most support the predicted root cause.

For structural groups, which correspond to transformer components (e.g., attention, QKV, FFN), each \(w_g\) maps to a transformer subsystem. A high score therefore identifies the subsystem whose message-passed embedding most supports the diagnosis [88]. For the cross-layer observation group (e.g., representation drift), a high score indicates that cross-layer behavior contributes to the diagnosis. For model-wide context groups (e.g., training dynamics, validation performance), a high score means that training or validation behavior helped distinguish the predicted root cause. These non-structural groups provide supporting evidence. They should not be interpreted as the faulty component.

8 Evaluation↩︎

We evaluate DEFault++ on the benchmark (i.e., DEFault-bench) to assess its effectiveness across fault detection, fault categorization, root-cause diagnosis, and feature-group evidence. We report encoder and decoder results separately because the two architectures use different feature groups and label spaces.

8.1 Evaluation Metrics↩︎

We evaluate DEFault++ using precision, recall, and F1. At Level 1, the faulty and correct classes are balanced by construction, so we report binary precision, recall, and F1. At Levels 2 and 3, fault categories and root causes are imbalanced, so we report macro-averaged precision, recall, and F1 across classes [89]. We do not report accuracy because it can be misleading under class imbalance [89], [90]. We define true positives according to the hierarchy. Level 1 requires correct fault detection. Level 2 requires the correct fault category. Level 3 counts a prediction as correct only when both the predicted category and the predicted root cause are correct.

8.2 Baselines↩︎

We compare DEFault++ with four prior fault-detection techniques for DNNs, namely AutoTrainer [11], DeepDiagnosis [13], DeepFD [14], and DEFault [15]. These baselines distinguish faulty runs from correct runs. They do not define fault labels that match our transformer fault categories or root causes, so we limit the comparison to the fault-detection level. For the rule-based baselines, AutoTrainer and DeepDiagnosis, we use their published rules and thresholds and map the required observables to the closest available trace features in DEFault-bench. AutoTrainer’s gradient-threshold rules map to the Training dynamics group, and DeepDiagnosis includes rules that partially overlap with the Attention and Score groups. For the learning-based baselines, DeepFD and DEFault, we train the original model types from their replication packages on the fault-detection task [14], [15].

8.3 Implementation↩︎

We implement DEFault++ in PyTorch [91]. Each feature group is encoded by a separate MLP with hidden dimension \(h = 32\), the shared projection maps the concatenated group embeddings to \(e = 64\) dimensions, and DEFault++ applies three rounds of FPG message passing over the group-level adjacency matrix. Training uses Adam [92] with learning rate \(10^{-3}\), weight decay \(10^{-4}\), and batch size 256. We train for up to 150 epochs with early stopping on the validation split. We set the contrastive temperature to \(\tau = 0.1\) and tune the loss weights by grid search. We address fault-category and root-cause imbalance using inverse-frequency weighting in the corresponding classification losses. We use a fixed set of random seeds for reproducibility. The replication package contains the implementation, hyperparameter lists, and trained models [18]. By approximately epoch 100, fault-detection F1 on the validation split shows little further improvement, while fault-categorization F1 continues to improve modestly until early stopping.

8.4 Answering RQ\(\mathbf{_1}\): Effectiveness of Diagnosis↩︎

To answer RQ\(_1\), we evaluate DEFault++ at its three levels, namely fault detection, fault categorization, and root-cause diagnosis.

Fault Detection. [tab:perf95level195overall] shows that DEFault++ achieves F1 of 0.8259 for encoders and 0.9094 for decoders. Recall is also high on both architectures, indicating that most faulty instances are detected before the later diagnosis levels.

Fault Categorization. [tab:perf95level295overall] shows strong fault-categorization performance for both encoders (F1 of 0.8497) and decoders (F1 of 0.8679). Precision and recall stay close in value, which indicates that the result does not come from a strong precision-recall tradeoff. The per-category results in [tbl:tab:perf95level295perclass] show that Residual and Positional faults are among the strongest categories in both settings.

For encoder models, DEFault++ scores lowest on Variant (0.741) and Kernel (0.771). Variant differs sharply across the two families, reaching 0.948 on decoders despite a similar number of instances (25 for encoders and 23 for decoders, 4), so the small Variant set does not by itself explain the encoder result. One possible reason is that decoder Variant faults produce cache-related differences that the encoder-only models cannot show. Kernel faults are difficult to categorize because they affect low-level execution behavior, such as numerical precision and backend selection, rather than a specific transformer component [93].

For decoder models, DEFault++ scores lowest on FFN and KV Cache faults. FFN faults are harder to categorize because their effects can appear in downstream components, such as the residual stream and LayerNorm [6]. KV Cache faults are challenging because the same cache fault can produce different measured behavior across token-generation steps [73], [94].

Root-cause Diagnosis. [tab:perf95level395overall] shows that DEFault++ achieves F1 of 0.851 on encoders and 0.867 on decoders for overall root-cause diagnosis. We observe higher precision than recall on both architectures, indicating that incorrect root-cause predictions are less common than missed root-cause cases. The per-category results ([tbl:tab:perf95level395category]) show that DEFault++ diagnoses root causes most accurately for Positional and Residual faults. DEFault++ performs lowest when diagnosing root causes within the QKV fault category. Several QKV root causes can change projection alignment, attention scores, and update behavior in similar ways [8]. This overlap may explain why QKV root causes are harder to distinguish. The Level 3 per-category results also indicate that the number of root causes within a fault category alone does not explain the performance. The Residual fault category includes five root causes and remains one of the strongest categories for root-cause diagnosis. QKV includes four root causes but remains the weakest. This pattern suggests that separability within each category plays a larger role than the number of root causes alone.

8.5 Answering RQ\(\mathbf{_2}\): Comparison with Existing Techniques↩︎

To answer RQ\(_2\), we compare DEFault++ with four existing DNN fault-detection techniques. AutoTrainer [11] and DeepDiagnosis [13] report training-symptom outputs. DeepFD [14] and DEFault [15] use DNN-level fault categories, which do not correspond to the transformer component categories or root causes used in our evaluation. We therefore restrict the comparison to fault detection (Level 1).

The baseline results (¿tbl:tab:rq295detection95comparison?) highlight the difference between general DNN fault detection and transformer-component fault detection. AutoTrainer and DeepDiagnosis use fixed training symptoms (e.g., gradient failures, loss oscillation, overfitting). Many DEFault-bench faults do not necessarily produce these training-level symptoms. A masking, QKV, positional, or cache fault can change internal transformer behavior while the loss curve and numerical values remain within ordinary ranges. As a result, rule-based methods can miss faults that do not match their predefined symptoms. In contrast, DeepFD and DEFault perform better than the rule-based methods because they learn from runtime features rather than relying on fixed rules. However, they still focus on general DNN training behavior and do not directly capture transformer-specific behavior (e.g., attention distributions, QKV alignment, cache behavior, or component-level propagation). These results suggest that transformer fault detection is not only a model-level prediction task. It also requires component-level measurements and the dependencies among transformer components.

max width=

8.6 Answering RQ\(\mathbf{_3}\): Ablation↩︎

To answer RQ\(_3\), we ablate two major parts of DEFault++ and assess their impact on performance. In particular, we focus on (a) message passing in the Fault Propagation Graph (FPG) and (b) the separation loss \(\mathcal{L}_{\text{sep}}\). FPG message passing applies to all three diagnosis levels, while \(\mathcal{L}_{\text{sep}}\) applies only to root-cause diagnosis.

We evaluate four model variants. The full DEFault++ model uses both FPG message passing and separation loss. The \(-\)FPG variant removes message passing and keeps the remaining model unchanged. The \(-\)Sep variant removes the separation loss by setting \(\beta = 0\) and \(\gamma = 0\). The \(-\)FPG \(-\)Sep variant removes both components and keeps only the feature-group encoders, shared projection, and prediction heads.

We also evaluate two graph-topology variants to separate the effect of the FPG’s transformer-specific structure from the influence of graph connectivity alone. The Rewired variant keeps the same number of nodes and edges as the FPG but randomly reassigns edge endpoints while preserving the degree distribution [95]. The Random variant replaces the FPG with an Erdős–Rényi graph [96] with approximately the same edge density.

Fault Detection & Categorization. For Level 1 fault detection, removing FPG message passing shows a slight performance decline on both architectures (¿tbl:tab:abl95l1?). This result suggests that the feature groups already contain enough information for binary fault detection, while FPG message passing adds a slight advantage by combining evidence from related components.

For Level 2 fault categorization, removing FPG message passing leads to a larger performance drop ([tbl:tab:abl95l2]). Categorization requires the model to distinguish fault types that can affect related parts of the transformer. For example, QKV and Score faults both affect attention behavior, but they reach downstream components through different propagation paths. FPG message passing allows the model to combine evidence from related feature groups before predicting the fault category.

Root-cause Diagnosis. At Level 3, removing the separation loss causes a larger performance drop than removing FPG message passing (8). This pattern suggests that the main challenge at this level is distinguishing root causes within the same fault category. Root causes in the same category often affect the same transformer component, which can make their runtime measurements similar. The separation loss directly targets this problem by encouraging embeddings from different root causes in the same category to move apart.

FPG message passing also contributes to Level 3, but its effect is smaller than that of the separation loss. Removing both FPG message passing and the separation loss leads to the lowest Level 3 performance, suggesting that they provide complementary benefits.

Topology Ablation. 9 shows that the original FPG gives the best performance at every diagnosis level for both encoders and decoders. The Rewired and Random variants degrade performance, and removing FPG message passing gives the lowest performance. The drop from the original FPG to the Rewired and Random variants shows that transformer-specific propagation links matter beyond graph connectivity alone. We observe that the topology effect is largest at the fault-categorization level. The FPG-removal ablation shows the same Level 2 pattern, with the largest performance drop appearing in fault categorization ([tbl:tab:abl95l2]). Since fault categorization depends more on propagation structure, it must differentiate fault categories that can affect related transformer components. The topology differences are smaller at Level 1 and Level 3. Level 1 is a binary detection task, while Level 3 depends more on within-category root-cause separation through \(\mathcal{L}_{\text{sep}}\). These ablations show that the FPG structure helps categorization, but they do not establish that its edges match the trained model’s actual fault-propagation paths.

8.7 Answering RQ\(\mathbf{_4}\): Explanation Faithfulness↩︎

DEFault++ uses the prototype classifier for root-cause diagnosis. The prototype distance decomposes over feature groups, so DEFault++ can measure how much each group contributes to the selected root-cause prototype. To answer RQ\(_4\), we assess the faithfulness of these feature-group explanations to DEFault++’s prototype-based root-cause decisions.

We first evaluate the reported feature groups through ablation. For each prediction, we remove the two feature groups that DEFault++ ranks highest. We then recompute the prototype-distance gap between the predicted root-cause prototype and the nearest alternative prototype. The gap is computed as the distance to the nearest alternative prototype minus the distance to the predicted prototype. Larger gaps indicate stronger support for the predicted root cause. As a baseline, we repeat the same test after removing two randomly selected feature groups. 14 shows that removing the top-ranked groups reduces the prototype-distance gap more than removing random groups. A similar pattern appears for both encoder and decoder models, and it applies to all fault categories.

We next evaluate the prototype classifier used for the explanation. DEFault++ also trains a supervised root-cause classifier, which predicts root causes directly from the training labels. We compare these two classifiers by measuring how often they predict the same root cause for the same sample. 15 shows that prototype accuracy increases during training, and its agreement with the supervised classifier also increases for both encoder and decoder models. Since DEFault++ computes explanations from the prototype classifier, the increasing agreement supports the use of prototype-based explanations for the learned root-cause decisions.

DEFault-bench provides the root-cause annotation for each faulty run, but it does not specify which feature groups should explain that fault. We therefore assess faithfulness by testing whether the reported groups affect DEFault++’s prototype-based decision. This evaluation supports internal faithfulness, but it does not establish the reported groups as the true causal explanation for the fault.

9 Real-world Fault Evaluation↩︎

We use real-world GitHub issues to evaluate whether DEFault++ generalizes beyond the synthetic faults in DEFault-bench. Following prior work [8], [30], [44], we collect 20 transformer-related bug-fix issues from open-source repositories and reproduce 11 of them. For each reproduced issue, we compare the faulty version with the fixed version using the same feature extraction procedure as in the main evaluation. DEFault++ detects 8 of the 11 faults, assigns the correct fault category for 8, and identifies the correct root cause for 4 (¿tbl:tab:realworld95eval?). Fault detection (FD) checks whether DEFault++ marks the faulty version as faulty. Fault categorization (FC) checks whether it assigns the fault to the correct transformer category. Root-cause diagnosis (RC) checks whether the predicted cause matches the cause confirmed by the corresponding bug fix.

3.5pt

For the real-world faults that DEFault++ detects and categorizes correctly, the top-ranked feature groups generally align with the reported fault category. For the masking faults, DEFault++ assigns high importance to attention-mass features, which measure attention given to invalid tokens. For the QKV faults, the top-ranked groups include score and projection-update features, which correspond to QKV projection behavior and attention-score computation. For positional and variant faults, DEFault++ assigns high importance to position-sensitive and attention-behavior features, although these cases are not always diagnosed at the root-cause level. DEFault++ detects and categorizes several real-world faults correctly but does not always identify the correct root cause, as in the positional, KV Cache, and variant cases. This matches our benchmark results, which show that root-cause diagnosis is the most difficult level.

DEFault++ does not identify three real-world faults. In transformers#37574, the fault appears during autoregressive generation, where the model stores KV states from earlier decoding steps and reuses them when generating later tokens. DEFault++ extracts cache-related behavior during fine-tuning, but it does not trace how the cache changes at each generation step, which can limit its ability to diagnose faults that appear only through generation-time cache updates. In pytorch#116333, the fault depends on backend stride validation rather than transformer-layer behavior, so diagnosing it would require system-level information beyond the model trace DEFault++ uses. Finally, transformers#36096 falls outside DEFault++’s scope because the fault crashes before a valid trace can be collected.

We use diffusers#11903 as a real-world case study for DEFault++. The fault redirects the forward pass to a fused QKV projection while LoRA updates the original projection modules. In this case, training completes without an explicit failure, but the updates do not affect the projection path used during execution. 16 shows the DEFault++ output for this case. DEFault++ detects the run as faulty, assigns it to the QKV category, and diagnoses the root cause as dynamic parameter registration. The explanation assigns most of the normalized feature importance to QKV Alignment and Optimization, which together account for 73% of the score. These groups point to a mismatch between the projection modules updated by LoRA and the fused QKV projection used in the forward pass. The GitHub bug fix addresses the same mismatch by ensuring that LoRA updates the projection module executed by the model. This case also shows that DEFault++ generalizes within the dynamic parameter registration root cause rather than memorizing a single operator. The training operator QFG freezes QKV gradients while the parameters stay in the forward path, whereas diffusers#11903 is the opposite case, where gradients flow but the parameters leave the forward path. DEFault++ recognizes the second case through the same near-zero QKV update-activity signature, which suggests the root-cause label spans sub-mechanisms within a category rather than fitting one operator.

10 Developer Study↩︎

We evaluate DEFault++ through a developer study using real-world transformer debugging scenarios derived from reproduced GitHub issues. Participants review four scenarios and select the most appropriate repair action for each one. For two scenarios, participants receive only the debugging artifacts available from the reproduced issue (e.g., logs, model outputs). For the other two scenarios, participants receive the same artifacts together with the diagnosis generated by DEFault++. We use the study to measure how DEFault++ affects repair correctness, confidence, and self-reported completion time, and how participants perceive the usefulness of its diagnoses.

Developer Study Design. We use a within-subjects, counterbalanced design with two conditions [97]. In the baseline condition, participants receive the scenario description, observed behavior, logs, and metrics. In the DEFault++ assisted condition, they receive the same materials plus the DEFault++ diagnosis, including fault status, fault category, predicted root cause, and supporting feature group. Each participant completes all four scenarios, with two scenarios assigned to each condition. We counterbalance condition assignment across two survey forms, following prior scenario-based developer studies of DL debugging techniques [12], [98].

The four scenarios cover Masking, Positional, KV Cache, and Variant faults (10). For S1, DEFault++ provides a correct diagnosis at all three levels. For S2, S3, and S4, DEFault++ correctly detects the fault and predicts the correct fault category, but predicts an incorrect root cause (11). We include these imperfect-diagnosis scenarios to evaluate whether category-level diagnosis and supporting feature groups can still guide repair selection when the root-cause prediction is incorrect.

The primary outcome is repair correctness, scored against the ground-truth fix from the source issue. We also collect self-reported confidence on a 5-point scale and self-reported time as a perceived-effort measure. After completing the scenarios, participants rate the clarity, usefulness, and practical value of the DEFault++ diagnosis on 5-point Likert scales and answer preference questions comparing the two conditions.

3pt

5pt

Developer Study Participants. The Dalhousie University Research Ethics Board approved the study.² Participants are eligible if they have experience with transformer architectures in training, fine-tuning, inference, debugging, or evaluation [12], [98]. We recruit 21 participants through graduate-student mailing lists, academic networks, and direct outreach ([tab:devstudy95demographics]). Participation is anonymous, voluntary, and uncompensated, and runs through Microsoft Forms.

Developer Study Results. We report descriptive statistics, following prior vignette-based debugging studies such as UMLAUT [12] and KUnit [98]. Each participant sees only two scenarios per condition, so we treat the difference between conditions as an effect-size estimate rather than as a hypothesis test.

Participants select the correct repair more often with DEFault++ assistance than in the baseline condition. Repair accuracy increases from 57.1% in the baseline condition to 83.3% in the assisted condition ([tbl:tab:devstudy95results] 17). Confidence also increases from 3.38 to 4.00. The largest gains appear in S2 and S3, where DEFault++ predicts the correct fault category but an incorrect root cause. This suggests that category-level diagnosis and supporting feature groups can still help developers choose the correct repair direction when the root-cause prediction is imperfect.

Participants also rate the DEFault++ output positively. Most participants agree or strongly agree that the diagnosis is easy to understand, helps them decide on a repair, and would be useful in practice (17).

11 Threats to Validity↩︎

Threats to Internal Validity. In our evaluation, information leakage could occur if runs from the same model–task pair were split across training and test folds. We address this risk with nested grouped cross-validation, which keeps each model–task pair within a single outer fold and limits preprocessing and hyperparameter selection to the inner training data. The joint training objective could also affect the evaluation because DEFault++ shares an encoder across diagnostic levels. As a result, the loss from one level may influence the representation used by another level. We mitigate the risk by restricting the separation loss to faulty samples within their ground-truth category and by evaluating its contribution through ablation.

Threats to Construct Validity. Mutation-generated faults may not fully represent faults that developers encounter in real-world transformer models [22], [26]. We reduce this risk by deriving each mutation operator from documented root causes in attention and DNN fault taxonomies [20]–[22], which are based on real-world examples. Class imbalance may also affect the evaluation results. We mitigate this risk by using macro-averaged F1 and inverse-frequency class weighting [90]. The mutation-killing test might fail to identify weak or noisy mutants, even when the intended change is successfully applied through fault injection. This may create a conservative fault set, but it improves label reliability because each retained faulty run differs statistically from its matched clean run. Extreme mutants may also inflate diagnostic performance when their effects are easy to detect [26]. We reduce this risk by varying fault severity and target layer, and by reporting mutation scores separately for each category. A final construct concern is the causal reading of the outputs. The training labels are causally grounded by construction, since each mutant differs from its clean counterpart through one targeted change under matched seeds. The Fault Propagation Graph and the diagnostic model, however, capture statistical associations rather than causal mechanisms, because the graph follows the computation graph and the model predicts mutation-generated labels. The RQ\(_4\) group ablation and the real-world trace-to-code checks show that the reported feature groups are consistent with the predicted faults, but they do not establish a causal explanation, so we treat FPG edges as plausible propagation paths rather than as verified causal links.

Threats to External Validity. Generalization beyond the evaluated model–task pairs remains an external validity concern. We reduce this risk by using grouped outer folds, so each method is tested on pairs not seen during training or hyperparameter selection. Comparisons with prior techniques may also be affected by differences in prediction targets, since those techniques do not support DEFault++’s multi-level diagnoses (1). We therefore restrict direct baseline comparison to fault detection. In the developer study, differences between participants could affect repair choices. We reduce this risk with a counterbalanced within-subjects design, where each participant completes scenarios with and without DEFault++ support.

12 Limitations and Future Work↩︎

DEFault++ currently diagnoses one injected fault per training run. This setting isolates each fault for categorization and root-cause diagnosis, but it does not cover programs with several co-occurring faults. Extending DEFault-bench to multi-fault configurations and studying how diagnostic behavior changes when several faults appear in the same run is a natural next step.

The Fault Propagation Graph (FPG) currently acts as a structural prior. Its edges mark possible propagation paths between transformer components, but they do not encode dependency strength or propagation mechanism. Calibrating these edges with fault-injection measurements and annotating them with dependency types would let DEFault++ distinguish stronger, weaker, and mechanism-specific propagation paths, and would let the model learn edge weights rather than treat every edge alike.

Our evaluation covers four encoder-only and three decoder-only models with 66M–125M parameters and 6–12 layers. It does not cover encoder-decoder models, vision models, multimodal models, mixture-of-experts architectures, or larger transformers. Since each mutation operator targets a component type, DEFault-bench extends to another model in the same architecture family by mapping its modules to those component types, although DEFault++’s trained classifier weights are model-specific and need retraining.

Our explanation evaluation measures faithfulness to DEFault++’s own decision process. DEFault-bench provides root-cause labels but not ground-truth feature-group explanations, so future work can strengthen the evaluation by adding independently validated feature-group annotations or by designing controlled faults with known feature-level effects.

Finally, our developer study is modest (N = 21) and vignette-based rather than live, so we treat self-reported time as estimated effort rather than as a direct measure of task completion time. A larger live study that varies the type and severity of diagnostic errors would better characterize how practitioners use DEFault++ during debugging.

13 Conclusion↩︎

We present DEFault++, a transformer-specific hierarchical technique for fault detection, fault categorization, and root-cause diagnosis. DEFault++ extends our prior hierarchical DNN fault diagnosis to transformer architectures and reports which feature groups support each diagnosis. It organizes component-level measurements with a Fault Propagation Graph, a structural prior derived from the dependency paths in transformer computation. To train and evaluate it, we construct DEFault-bench, a benchmark of 5,556 labeled transformer runs generated with transformer-specific mutation testing. On DEFault-bench, DEFault++ reaches F1 of 0.826–0.909 for detection, 0.850–0.868 for categorization, and 0.851–0.867 for hierarchical root-cause diagnosis. In a developer study with 21 participants, repair accuracy rises from 57.1% without DEFault++ to 83.3% with it. The ablation and the developer study together point to one design lesson, that component-level measurements organized along the transformer’s propagation structure separate faults that model-level training signals leave indistinguishable. We release DEFault-bench and the DEFault++ implementation in our replication package [18].

References↩︎