1 Introduction↩︎

An ECG assesses the electrophysiology of the heart in a quick and non-invasive manner. The x-axis comprises time, usually expressed as milliseconds, and voltage in millivolts on the y-axis. As seen in 1, it comprises several prominent features which map to specific phases of the cardiac cycle. For instance, the P-wave indicates atrial systole, the QRS complex ventricular systole and the T-wave repolarization. Hence, it is possible to diagnose multiple pathologies from an ECG.

The development of wearable devices has made possible the prospect of remote ECG monitoring. This could help facilitate effective triage and the appropriate allocation of health resources. However, this requires the AI model to be small enough to fit onto such a device. If an anomaly is detected, this would need to be verified by an expert to reduce the risk of a false positive and the inappropriate allocation of health resources. Alternatively, false negatives indicate an undetected arrhythmia. This could also lead to the inappropriate allocation of finite health resources. XAI facilitates quick and effective verification by drawing an expert’s attention to the features which contributed the most to the model’s classification, as long as the XAI itself does not introduce too much “noise” by highlighting irrelevant features not actually used by the model.

Multiple XAI tools have been used to explain the outputs of AI arrhythmia detectors, with Shap [2] and Grad-CAM [3] being particularly common choices [4]. However, nearly half of all published XAI applications do not evaluate the explanations themselves [4]. This omission is even more common in the medical domain [5], where expert knowledge is at a premium and frequently not available for large scale studies. Recent guidelines, called FUTURE-AI, for the development and deployment of AI tools in a medical context recommend utilizing XAI [6]. This includes, among other recommendations, quantitative evaluations and measuring sensitivity to noise of the XAI tool itself (as distinct from the AI model). The difficulty of scaling the assessment of XAI tools together with the development of clinical guidelines, such as FUTURE-AI, has given rise to a growing interest in establishing proxy-measures for quantitatively assessing the outputs of XAI.

In this paper, we treat the output of the XAI tool itself as a signal. This signal is a combination of model signal (the actual features used by the model for classification) and XAI-introduced noise. All post hoc explainability tools are heuristic based: Shap, ReX [7], [8] and Lime [9] for example, rely on some form of sampling, and Lime also uses segmentation information for images. These heuristics inevitably introduce some noise into the underlying model signal, which we will call self-noise. A good XAI tool should correctly identify model signal, but should ideally introduce the smallest amount of self-noise, or – failing that – predictable self-noise. We propose using spectral entropy as a means to quantify how much self-noise XAI tools add to the signal. To the best of our knowledge, no one has proposed a means to systematically examine self-noise before. We evaluate spectral entropy to quantify self-noise on ECG data.

XAI output is not time-series; we are using spectral entropy in an unusual way. This paper demonstrates that this slight abuse is justified by practical use. Spectral entropy can augment other XAI assessment methods (see 5), but does not answer all questions. Entropy can measure the self-noise of the XAI tool but cannot be used directly to assess the “accuracy” of an explanation. Indeed, it is an open question whether a model’s criteria for classification should align with a human expert’s criteria [10], although in the short-term, clinical applications will need to align with prior medical knowledge [11]. However, spectral entropy can be used to measure the “precision” of an explanation. An explanation with low accuracy, i.e. in the wrong place, may still have high precision due to its concentration or lack of diffusion. This high precision can be captured by spectral entropy.

Qualities of a good explanation

 [11] specify that explanations should be understandable to end users, clinically relevant, agree with prior medical (informative plausibility) knowledge and truthfulness. The former requires that XAI outputs are concise and presented coherently. Indeed, one of the principal complaints from medical users of XAI is the amount of superfluous signal  [12], [13]. Spectral entropy quantifies this intuition. The latter requirement is common to all similar recommendations and necessitates that the XAI tool is assessed for the degree to which it captures what the AI model is actually doing. This would include assessing whether the XAI tool adds self-noise due to its heuristics.

2 Spectral Entropy↩︎

Shannon entropy is a measure of the average surprisal in a system [14]. A system in which every outcome is equally surprising is a maximum entropy system. In XAI output, maximum entropy would be achieved by uniform-at-random allocation of importance to the different input features ([fig:pvc95ks]). This would be akin to white noise in signal processing. Any XAI tool which outputs white noise would be maximally ‘bad’, on the assumption that the explanation itself if not uniformly distributed over the feature space (see 5). We use the scipy implementation of the periodogram function to compute the power spectral density of the explanation (1 ).

\[\label{eq:1} P(\omega_i) = \frac{1}{N}\mid X(\omega_i)\mid^2\tag{1}\]

This is then normalized (2 ) and passed directly to the standard formula for Shannon entropy, to give us the power spectral entropy \(PSE = - \sum_{i=1}^{n} p_i \log p_i\). Such a calculation is inexpensive to perform in real time. All results are in information theoretic bits, i.e. the \(\log\) is base \(2\).

\[\label{eq:2} p_i = \frac{P(\omega_i)}{\sum_i P(\omega_i)}\tag{2}\]

XAI explanations obviously do not have a sample rate. The scipy implementation of the periodogram function assumes a default sample rate of \(1\). We find in our evaluation that changing the sample rate has no effect on the calculation of spectral entropy.

3 Experiments↩︎

To demonstrate the usefulness of this measure we utilized a model trained to detect cardiac arrhythmias from ECG data. [15] specifically developed a model small enough to be contained within a wearable device. Their 17 layer, 1D-convolutional neural network was trained on the MIT-BIH dataset [16] to classify 17 arrhythmias to an accuracy of \(93.5\)%. As the model from [15] was not available, we trained a model ourselves from their instructions.

We used all \(1000\) ECG from this dataset to classify and then explain the model predictions using Grad-CAM, Lime, DeepLIFT [17] and ReX [7], specifically its version for spectral data [18]. We also used two variations of Shap: GradientShap and KernelShap. We utilized a Lime variant designed for time series data [19], rather than standard Lime. Apart from this variant and ReX¹, all XAI tools were obtained from the captum.ai² library. Grad-CAM directly accesses the final convolutional layer of a model, therefore its output is the same size as this final layer, which in this model results in an activation map of 6 features. This is then upsampled via linear interpolation into the original \(3600\) features.

All tools were used with their default settings. Such is the parameter space of each tool that it would be unfeasible to explore these spaces adequately. The techniques run the gamut from white box, i.e.Grad-CAM with its direct access to convolutional layers, to purely black box, i.e.ReX, which relies only on the model classification. All the XAI techniques provide post hoc explanations of a particular instance of an ECG. The model that we use is not inherently explainable, so we were unable to estimate the entropy of any self-explanatory method.

The output of each tool is not directly comparable. ReX and Lime, for example, do not compute values below \(0\), whereas tools which estimate Shapley values produce both positive and negative values. Negative values indicate features which either “distract” from the actual classification, or point towards another classification. For the sake of fair comparison, we ignore sub-zero values when calculating the spectral entropy of a given explanation as we are only interested in features which positively contribute to the classification. We additionally took ECGs from the PVC (Premature Ventricular Contraction) class and annotated a ground truth explanation based on domain expertise. Such manual annotation is notoriously expensive: we annotated 20 ECGs from this class. From this, we calculate a ground truth entropy for this class.

4 Results↩︎

1 shows the average spectral entropy for each tool across all \(17\) classes. The most obvious feature is that, for most tools, the entropy does not vary much over all \(17\) classes. This is a good indicator that, for this model and data, most tools produce a near constant amount of self-noise. The exception to this is ReX, which has the highest standard deviation. Unlike the other tools, ReX is capable of identifying multiple explanations [20] in images. It may be the case that the feature attribution map output by ReX contains multiple explanations for certain classes, resulting in the wide spread of entropy values. This needs further investigation.

We focus in on one class to show the utility of the method. 3 shows an ECG with a PVC (premature ventricular contraction) highlighted, representing the only region required for a human domain expert to classify it. This particular example has only one explanation: the region between the dotted red lines. 4 shows the output of the different XAI tools on this ECG instance. The appropriate region has again been highlighted in red.

The most obvious difference is between tools which estimate Shapley values, and those which do not. [fig:pvc_gs,fig:pvc_dl,fig:pvc_ks] all estimate Shapley values. KernelShap has a signal which is close to pure noise ([fig:psd95ks]). One would have to assume that KernelShap is a poor choice of tool for ECG explainability. Both GradientShap and DeepLIFT highlight the correct region in that their highest peaks are within the red zone. However, they have other, clearly separate, peaks which are spurious. Indeed, GradientShap has a low level of noise which is fairly constant ([fig:psd95gs]). ReX output is different, showing a gradual rise towards the model signal and then a fall away. Grad-CAM shows a similar pattern, though with almost no self-noise. It does not, however, highlight the correct part of the signal. This may be due to the necessity to upsample Grad-CAM’s output. In lieu of more detailed information, this upsampling is uniform, assigning each of the \(6\) neurons to equal width bins in the output space. It may be that equal binning is not the most appropriate method, but it is unclear what one might do in its place. Lime output is curious, showing a relatively small amount of noise ([fig:psd95lime]) while still failing to highlight the correct part of the signal.

5 Discussion↩︎

The tools fall broadly into two classes: high noise and low noise. This does not necessarily align with accuracy of explanation. For instance, Lime has relatively low entropy across all classes, but on PVC at least does not accurately select any of the appropriate peaks. Conversely, although GradientShap has high entropy on the same signal, the largest peaks are clearly aligned with the relevant parts of the ECG for PVC. While this level of accuracy does not generalize over all classes, the noise does: spectral entropy is fairly consistent over all classes for all tools, with the exception of ReX. This suggests that it is accurately capturing the baseline heuristic noise of each tool.

An important assumption when discussing model signal and self-noise is that the explanation (or rather, the data that the model is using to make its classification) is local, i.e. confined to just a few points in the signal. A non-local explanation, where the explanation is distributed across the entire input, might indeed look close to white noise. A low spectral entropy will be unfortunate in the event that an explanation is non-local, and therefore the signal really is evenly distributed over the input space. This is not the case with the ECG data that we use, however.

Various metrics have been suggested to quantify the quality of XAI output. [21] introduce deletion and insertion curves, which track model confidence as features are ablated or introduced. [22] propose the concepts of sensitivity and fidelity, which measure the changes in an explanation due to perturbations on the original input. All of these methods test the resilience or portability of an explanation in some way, by removing or introducing features, or perturbing feature values. All of these methods require multiple calls to a model and are computationally expensive.

None of these metrics attempt to quantity how much noise the XAI tool itself adds into the explanation. [7] use the number of pixels in an explanation of an image as a quality measure. They find that the Shapley value based tools require more pixels in general than either ReX or Lime. This is likely analogous to self-noise, whereby unnecessary pixels are added to an explanation which do not actually contribute to the class. They do not, however, attempt to quantity how much spurious information is added by the tools they investigate.

[23] argues that an explanation in the medical context is inherently social. Any quantitative performance measure is a proxy measure of its usefulness. This meta-analysis problem is well documented in the literature [24]. Spectral entropy is an elegant technique which measures the degree of spurious information the XAI method adds, aligning with the qualitative feeling of an output containing “too much” to be useful.

6 Conclusion↩︎

We have proposed the use of spectral entropy as an XAI evaluation metric for medical, tabular-style, data. It quantifies the intuition that XAI outputs with spurious information obscures domain expert interpretation. Spectral entropy can be used alongside other quantitative measures when assessing the XAI tool which seeks to explain the outputs of an ECG classifier. Spectral entropy is computationally inexpensive and can be used in real time. This would not be instead of domain expert assessment, but may be helpful when scaling such assessments to a quantity beyond their resource limitations.

Acknowledgments↩︎

David Kelly and Nathan Blake are funded by CHAI: EPSRC - Causality in Healthcare AI Hub (grant number EP/Y028856/1).

References↩︎