1 Introduction↩︎

Machine learning algorithms are built on the assumption that data encountered at test-time comes from the same distribution as seen during training. When this assumption is violated, test data is said to be out-of-distribution (OOD) and prediction accuracy rapidly degrades [1], particularly for deep neural networks (DNNs) which are prone to overconfident predictions [2]. However, because DNNs are effectively “black boxes”, it is difficult to assess when their predictions can be trusted [3].

In this work, we address the problem of OOD detection [4]: classifying test-time inputs as either in-distribution (ID) or OOD. Effective OOD detection is critical for real-world deployment, as models are frequently exposed to inputs that deviate from their training distribution for which they should avoid making predictions. Yet, current OOD detection methods exhibit two significant limitations. First, these methods typically rely almost exclusively on the final-layer representation of the network (e.g., logits), disregarding the rich, hierarchical representations [5] present throughout the intermediate layers [6]–[10]. As we illustrate in Fig. 1, this can overlook discriminative structure that is visible earlier in the network but becomes less separable in later representations. Second, even when OOD detectors perform well, their scores typically provide limited insight into the internal representations that distinguish ID from OOD inputs. This makes it difficult to diagnose where distributional differences emerge within a network, and whether they are captured by low-level, mid-level, or high-level features. Such a layer-wise view can inform which representations should be monitored for OOD evidence and whether a shift reflects low-level visual corruption, higher-level semantic mismatch, or both.

To address these challenges, we propose Sparse Autoencoders for Interpretable Detection (SAID), a post-hoc OOD detection method that leverages sparse autoencoders (SAEs) trained on intermediate network activations. SAEs have recently been used to extract sparse, concept-like features from otherwise opaque neural representations [13]. We adapt this idea to vision-based OOD detection by training SAEs on selected layers of a frozen image classifier, including both convolutional and Transformer-based vision models. At test time, SAID compares the SAE encoding of an input to the mean SAE encoding of the predicted ID class, producing a similarity-based OOD score. We show that these sparse intermediate representations improve detection performance across both semantic and covariate distribution shifts.

Beyond detection performance, the sparse structure of SAE representations provides a diagnostic lens for analyzing how ID and OOD inputs activate the model’s learned concept space. We automatically annotate SAE latents with human-readable concept labels using a vision-language model, then evaluate whether the concepts activated by a test input are semantically consistent with the image. Our results show that ID samples tend to exhibit higher concept agreement than both semantic- and covariate-shift OOD samples, and that this agreement gap varies across layers.

Our contributions are threefold. First, we introduce SAID, a post-hoc OOD detection method that uses SAE encodings of intermediate-layer representations to improve detection performance. Second, we show that intermediate layers contain discriminative information that is often lost or compressed in final-layer representations, improving performance across semantic and covariate shift benchmarks. Third, we analyze SAE-derived concept labels as a diagnostic tool for understanding how distribution shift affects learned representations, showing that ID inputs activate concepts that are more semantically consistent with the image than OOD inputs.

2 Related Work↩︎

Out-of-Distribution Detection. One prominent category of OOD detection methods leverages the uncertainty of model predictions or the logits (outputs of the final layer). For instance, softmax confidence scores have been used as a baseline for OOD detection [6]. Building upon this, energy-based models have been proposed, where the energy score derived from the model’s output logits is used to distinguish between in-distribution and OOD samples [7]. While these methods offer a starting point, they primarily rely on information from the final layer, potentially overlooking valuable insights embedded within the intermediate representations of the neural network [14].

To overcome the limitations of solely relying on final layer outputs, some research has explored using features from earlier layers. The Mahalanobis distance applied to intermediate feature representations has shown promise in improving near-OOD detection [9], [15]. Similarly, methods utilizing deep nearest neighbors in feature space have been developed for OOD detection [8]. More recent advancements include activation shaping techniques, which aim to directly influence the activations of neural networks to improve OOD discrimination [10]. Despite these advancements, a challenge remains in effectively utilizing the vast amount of information contained in all intermediate activations, as it can be redundant, noisy, or difficult to interpret.

Interpretable Machine Learning. The increasing interest in interpretable machine learning has led to methods that explain model predictions using human-understandable concepts [16], [17]. This is especially relevant for OOD detection, where concept-level representations can help analyze how ID and OOD inputs differ. Recent work has also begun to explore the role of feature sparsity in OOD detection [18].

Sparse Autoencoders (SAEs) provide a natural mechanism for learning sparse, interpretable features from neural network activations [19]–[22]. By decomposing dense activations into localized, concept-like features [23], [24], SAEs offer a way to analyze otherwise opaque representations and have connections to anomaly detection [25], [26]. Universal SAEs further suggest the possibility of aligning sparse concepts across models [27].

Building on this, we use SAEs to extract sparse concept representations from intermediate layers for OOD detection. Rather than relying only on final-layer features, our approach leverages the distinct activation patterns induced by ID and OOD inputs throughout the network, providing both strong OOD detection and a concept-level view of distribution shift.

3 Preliminaries↩︎

3.1 Out-of-Distribution Detection↩︎

Let a machine learning model be trained on an ID dataset, \(\mathcal{D}_{in} = \{ (\mathbf{x}_i, y_i) \}_{i=1}^N\), where \(\mathbf{x}_i \in \mathbf{R}^d\) are input data points and \(y_i\) are their corresponding labels. We formulate the task of OOD detection as a binary classification problem: given a test sample \(\mathbf{x}\), the goal is to determine whether it is drawn from the same distribution as the training set (ID) or from a different distribution (OOD). In this work, we focus on two causes of distributional shift that can result in data being OOD [4]. The first is semantic shift, where the label distribution \(p(y|x)\) of the test distribution differs from training. The second is the problem of covariate shift, where the distribution over the inputs, \(p(x)\), differs.

3.2 Sparse Autoencoders↩︎

An autoencoder is a neural network designed to learn a compressed representation (encoding) of input data. It is comprised of an encoder, \(E(\cdot)\), which maps an input \(\mathbf{x}\) to a latent representation (or code) \(\mathbf{z}\), and a decoder, \(D(\cdot)\), which reconstructs the input from this latent representation, aiming for \(D(\mathbf{z}) \approx \mathbf{x}\). The primary objective during training is to minimize the mean squared error between the original input and its reconstructed version plus an \(\mathcal{L}_1\) term to induce sparsity.

A sparse autoencoder introduces a sparsity constraint on the latent representation \(\mathbf{z}\). This encourages most of the units in the hidden (latent) layer to be inactive (i.e., have outputs very close to zero) for any given input, ensuring that only a small, specific subset of features are “active."

In this work, we specifically utilize a top-\(k\) sparse autoencoder. For a given input \(\mathbf{a}\) (representing an intermediate activation from a pre-trained neural network), the encoder \(E(\cdot)\) first computes an intermediate latent vector. This vector is then subjected to a top-\(k\) activation function, which explicitly forces all but the \(k\) most active latent dimensions to zero. Top-\(k\) SAEs do not have the \(\mathcal{L}_1\) sparsity term in the objective function.

This direct method of enforcing sparsity encourages the learned features to be monosemantic, meaning each feature in the sparse representation ideally corresponds to a single, distinct concept. This property is suited for interpretability of the learned features, providing insight into what specific concepts the network is activating for a given input.

4 Sparse Autoencoders for Interpretable Detection↩︎

We introduce a post-hoc method for OOD detection which leverages the latent encodings from SAEs. Our approach, Sparse Autoencoders for Interpretable Detection (SAID), trains SAEs on the intermediate feature activations of a pre-trained vision-based classification model, as shown in Fig. 2.

4.1 Training Algorithm↩︎

Given a pre-defined set of layers \(L\) we train one SAE for each layer \(l\in L\) of a pre-trained neural network \(f(\cdot)\). For each layer, the corresponding SAE receives as input the intermediate activations \(\mathbf{a}=f_l(\mathbf{x})\), where \(f_l\) is the \(l\)-th layer of \(f\). All SAEs use identical hyperparameters, excluding their sparsity coefficient \(k\). To construct the SAE training set, we pass the entire original dataset through the pre-trained network \(f\) and store the resulting activations from each intermediate layer \(l \in L\). Each SAE is trained to minimize a mean squared error (MSE) reconstruction term. Formally, for a batch of intermediate activations \(\mathbf{a}\), the MSE reconstruction loss is defined as follows:

\[\mathcal{L}_{\mathrm{MSE}} = \frac{1}{N} \sum_{i=1}^{N} \|\mathbf{a}^{(i)} - D(E(\mathbf{a}^{(i)}))\|_2^2 .\] where \(E\) and \(D\) denote the SAE encoder and decoder respectively, and \(N\) the number of samples in the batch. After training, we freeze the SAE and only use its encoder \(E_l\) for OOD scoring.

4.2 Similarity-Based Scoring↩︎

After training each SAE for a given layer \(l \in L\), we encode all training samples from each ID class into the corresponding sparse latent space. We then compute and store the mean latent encoding \(\mu_{{c},l}\) for every ID class \(c\). Thus, the total number of stored class means equals the number of ID classes multiplied by the number of SAEs (one per layer).

For a test input \(\mathbf{x}\), let \(\hat{c} = \arg\max_c f_c(\mathbf{x})\) denote the class predicted by the pre-trained classifier, where \(f_c(\mathbf{x})\) denotes the predicted probability assigned to class \(c\). During inference, each SAE produces a latent encoding \(\mathbf{z}_l =E_l(f_l(\mathbf{x}))\). We compute a layer-specific OOD score \(s_l(\mathbf{x})\) as the cosine similarity between this latent encoding and the mean encoding of the predicted class \(\mu_{\hat{c},l}\): \[s_l(\mathbf{x}) = \frac{\mathbf{z}_l \cdot \mu_{\hat{c},l}}{\|\mathbf{z}_l\| \|\mu_{\hat{c},l}\|}.\]

These layer-wise scores are then aggregated into a single unified OOD confidence score \(S(\mathbf{x})\) by taking the average over all layer scores for a given sample. Thus, the overall OOD score \(S\) is computed as follows,

\[S(\mathbf{x}) = \frac{1}{|L|}\sum_{l \in L} s_l(\mathbf{x}).\]

At test-time, we apply a threshold \(\tau\) to classify samples: inputs for which \(S(\mathbf{x}) \geq \tau\) are classified as ID, while inputs with \(S(\mathbf{x}) < \tau\) are classified as OOD. Intuitively, higher scores indicate greater similarity between the sample and the known ID distribution.

4.3 VLM-Based Concept Annotation↩︎

A key benefit of using SAEs is their ability to learn interpretable features. To leverage this interpretability for concept-based explanations of OOD predictions, we introduce an automated labeling process guided by a vision-language model (VLM). Following strategies from natural language domains [28], we use a pre-trained VLM (GPT-5.5 [29]) to assign human-interpretable labels to latent features.

For each feature in the sparse latent space of each SAE, we identify the top 25 images from the ID training set which activate that latent most strongly, sorted by activation magnitude. These images represent the visual stimuli most closely associated with that latent concept. We then pass this set of images to a pretrained VLM along with a structured prompt (provided in the Appendix) that instructs the VLM to describe the shared visual patterns or semantic theme of these images, yielding a concise, human-readable concept label.

Fig. 5 shows examples of VLM-generated annotations, alongside their representative sets of images. Although each SAE theoretically possesses as many concepts as there are latent features, in practice, the presence of dead latents reduces the effective number of concepts. These labels provide an explanation of why specific inputs are considered out-of-distribution, highlighting precisely which concept-based features are or are not activated.

5 Experiments↩︎

We evaluate our proposed approach, SAID, with the goal of answering three questions: 1) Is SAID effective at detecting both semantic and covariate shifts in data across multiple architectures? 2) Does using the intermediate layers of a network provide added benefit over just using the final layer of a network? 3) Does SAID’s concept-based explanations reflect meaningful semantic differences between ID and OOD inputs?

Datasets. We evaluate our performance using ImageNet-1K [30] as the ID dataset and select OOD datasets containing both semantic and covariate distribution shifts. Semantic shifted datasets include: SSB-Hard [31], NINCO [32], iNaturalist [33], Textures [34], OpenImage-O [35], MNIST [36], FashionMNIST [37], and COVID [38]. To study covariate shifts, we use the following ImageNet variants: ImageNet-C [39], ImageNet-ES [40], and ImageNet-R [41].

We test our method on two different, publicly available architectures pretrained on ImageNet-1K. The first is a transformer-based architecture, ViT-B/16 [12], and the second is the CNN-based ResNet-50 [42]. Unlike methods which perform training-time regularization, our method does not modify the parameters of the backbone model. Thus, we only compare our method to other post-hoc detection methods.

Baselines. We compare our method to seven baseline methods: VIM [35], RMDS [15], MDS [9], KNN [8], EBO [7], MLS [43], and SHE [44]. These implementations were taken from OpenOOD [45], a library for benchmarking OOD performance.

Evaluation Metrics. We evaluate our method using two metrics. AUROC measures the method’s overall ability to distinguish ID from OOD samples across all thresholds, with 100 indicating perfect detection and 50 equivalent to random guessing. FPR95 measures the percentage of OOD samples incorrectly identified as ID when the true positive rate for ID samples is set to 95%; lower values indicate better performance.

All details regarding hyperparameters, training and evaluation details, and computational resources can be found in the Appendix.

3pt

3pt

5.1 OOD Detection Performance↩︎

The quantitative results for semantic and covariate shift are shown in Table 1 and Table 2, respectively. On both architectures, SAID achieves the best average detection performance on both shift types. The improvement is most pronounced for the ViT backbone, where SAID achieves the best average performance on both semantic-shift and covariate-shift benchmarks. These results suggest that sparse SAE representations provide a strong OOD signal when applied to intermediate representations of transformer-based vision models. SAID also remains competitive on ResNet-50, achieving the highest average AUROC across both shifts, although the margin over existing methods is smaller.

5.2 Impact of Intermediate Layers on OOD Detection↩︎

We first ask whether intermediate-layer information is useful at all. Table 2.b shows the AUROC gain from using SAEs trained on multiple intermediate layers rather than only the final layer. Across all datasets, adding intermediate-layer SAEs improves performance, indicating that intermediate representations contain discriminative information that complements the final-layer embedding.

The benefit is largest for datasets that are separable from ID at earlier layers. Figure 3.b shows that the most informative layer varies by dataset. Some datasets, such as NINCO and ImageNet-R, perform best at the final layer, while others, such as Textures and ImageNet-C, are better separated in earlier layers. This shows that the discriminative structure for these inputs lives in lower-level features that final-layer methods discard, and motivates aggregating across multiple layers.

A natural follow-up question is whether SAEs are the right way to extract that intermediate-layer information. Figure 3 shows that both SAID and KNN benefit from adding intermediate layer representations as both SAEs and raw activations, while RMDS performs worse in some cases. Although KNN obtains a larger relative gain from both types of representations, SAID achieves stronger absolute performance on both semantic- and covariate-shift benchmarks (Tables 1 and 2). SAID is also more compact at inference time: it stores one mean latent vector per class per layer, whereas KNN must store and search a large ID reference set. Together, these results show that intermediate layers provide useful OOD evidence, and that SAE features offer an effective and compact way to aggregate this information.

5.3 Sensitivity and Ablation Analysis↩︎

We next evaluate the sensitivity of SAID to its main hyperparameters and design choices (Fig. 4). Overall, SAID is relatively stable across a range of SAE latent dimensions. Performance improves when moving from very small latent spaces to moderate-sized ones, but changes only marginally once the latent dimension is sufficiently large (beyond 1024). This suggests that, beyond a certain capacity, additional latent dimensions do not substantially change the concept space used for OOD detection, likely because the SAE already has enough active features to represent the relevant ID structure. The sparsity level \(k\) has a more pronounced effect. Increasing \(k\) initially improves performance, indicating that too few active features may underrepresent the visual concepts needed to distinguish among ImageNet classes, though this too saturates beyond 448.

We also ablate how layer-wise scores are aggregated and how similarity is measured in the SAE latent space. For score aggregation (Fig. 4.c), we compare uniform averaging against several alternatives: linearly increasing or decreasing weights over layers, variance-based weights computed either globally over ID concept activations or with respect to the predicted class, and max/min aggregation using the largest or smallest layer-wise similarity score. Uniform averaging performs competitively across both semantic and covariate shifts, suggesting that a simple layer-agnostic aggregation is a robust default. Although some weighting schemes can improve performance for particular shift types, they implicitly assume that certain layers are more informative a priori, requiring either domain expertise or validation samples from the test distribution.

Among distance metrics (Fig. 4.d), cosine similarity performs best overall. We conjecture that this is the case due to the sparsity of the latent concept vectors. In this setting, the relative pattern of activated concepts may be more important than the overall activation magnitude. Cosine similarity captures this by assigning high similarity to inputs with aligned sparse activation patterns, even when their latent vectors differ in scale.

5.4 Analyzing Concept Activations Under Distribution Shift↩︎

We next evaluate whether SAE-derived concept labels reflect meaningful differences between ID and OOD inputs. We construct a test dataset of 400 images: 200 images uniformly sampled from among a subset of 40 classes in the ID dataset, and 200 images uniformly sampled across each OOD dataset in the corresponding shift setting (semantic or covariate). For each test image, we consider the top activated SAE concepts at each layer and use a VLM to judge whether each concept is semantically consistent with the image content. We report two metrics in Fig. 6: concept agreement, the fraction of activated concepts that agree with the image, and layer agreement, the fraction of layers for which at least one activated concept agrees with the image.

Qualitative examples are shown in Fig. 5. ID samples tend to activate concepts that align with the image class or with visually relevant attributes, while OOD samples more often activate concepts that are weakly related or semantically inconsistent. The quantitative results in Fig. 6 support this trend. Across layers, ID samples exhibit higher concept agreement than both semantic-shift and covariate-shift OOD samples. The ID–OOD gap is larger in later layers, suggesting that distribution shift is especially visible in higher-level concept representations. However, this layer-wise pattern should be interpreted diagnostically rather than causally: disagreement at a later layer may reflect either a mismatch at that layer or the downstream effect of differences that arise earlier in the network.

Overall, these results show that SAE concept labels provide a complementary view of OOD behavior. While SAID’s detection score measures similarity in sparse latent space, concept agreement helps characterize whether the activated latent features correspond to meaningful visual content in the input. This makes the learned concepts useful for analyzing how ID and OOD samples differ across the network’s representation hierarchy.

6 Discussion and Conclusion↩︎

We introduced Sparse Autoencoders for Interpretable Detection (SAID), a post-hoc OOD detection framework that applies sparse autoencoders to intermediate representations of frozen vision models. By comparing sparse latent encodings to class-conditional ID prototypes, SAID provides a simple similarity-based OOD score that leverages information distributed across the network rather than relying only on final-layer features. Empirically, SAID achieves strong performance across both semantic and covariate distribution shifts on ViT and ResNet-50 backbones. Our results show that intermediate representations contain discriminative structure that is often reduced or obscured in final-layer embeddings, and that SAE features provide an effective way to extract and aggregate this information for OOD detection.

Beyond detection performance, SAID provides a concept-level view of how ID and OOD inputs are represented across layers. Using VLM-labeled SAE features, we find that ID samples tend to activate concepts that are more semantically consistent with the input image than those activated by OOD samples. This suggests that SAE concepts can serve as a useful diagnostic tool for analyzing how distribution shift affects learned representations. Future work could extend this framework to larger models, language and multimodal domains, and more natural forms of distribution shift.

Limitations: Despite promising results, our approach has limitations, the most significant of which is that we have limited our current evaluation to vision domains. Methodologically, the use of a fixed-\(k\) sparsity constraint, where a constant number of features are active for any given input, may be a restrictive assumption. The inherent complexity of data varies; some ID inputs might be adequately described by very few concepts, while others may require more. A more flexible sparsity mechanism could potentially improve representation quality and, consequently, OOD detection performance.

Furthermore, the computational cost associated with the proposed method is non-trivial. Training a separate, high-capacity SAE for each of the intermediate layers is a resource-intensive process. This requirement for substantial computational power and time could pose a barrier for applications involving extremely large models or environments with limited hardware resources.

7 VLM-Based Concept Annotation Prompt↩︎

8 VLM Concept-Image Agreement Prompt↩︎

def get_gpt_verdict(image_b64: str, concepts: list[dict]) -> Union[dict, str]:
    """
    Sends the image and all concepts to GPT-5.5 in a single call.
    Asks GPT to return a JSON object mapping each concept string to true/false.

    Returns a dict like:
        {"giant pandas": true, "grid or mesh-like patterns": false, ...}
    or a string error tag on failure.
    """
    if not concepts:
        return "no_concepts_found"

    concept_strings = [c["concept"] for c in concepts]
    numbered_list = "\n".join(f"{i+1}. {c}" for i, c in enumerate(concept_strings))

    prompt_text = (
        "I will provide you an image and a set of visual concepts in natural language. "
        "Consider each concept and decide whether it could be applied to the image. "
        "These concept labels may be abstract in nature. "
        "Err on the side of caution — only mark a concept false if it clearly does not describe "
        "the image or references visual features that are not present at all.\n\n"
        f"Concepts:\n{numbered_list}\n\n"
        "Respond with a single JSON object where every key is the exact concept string "
        "(copied verbatim from the list above) and the value is true or false. "
        "Return only valid JSON — no markdown, no explanation, no extra text."
    )

    messages = [
        {
            "role": "system",
            "content": (
                "You are a judge for a machine learning experiment evaluating visual concept labels. "
                "You respond only with a valid JSON object mapping concept strings to boolean values."
            ),
        },
        {
            "role": "user",
            "content": [
                {"type": "text", "text": prompt_text},
                {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{image_b64}"}},
            ],
        },
    ]
    ...

9 SAE training hyperparameters↩︎

Table 3 and Table 4 lists the hyperparameters we used to train all SAEs for our method with ViT and ResNet-50 as the backbone architectures respectively. The k hyperparameter is the k value used in the top-k function. The For ViT, the first k value corresponds to the k used at Layers 4 and 7, whereas the second is the k used at Layers 10 and 13. For ResNet-50, the first k value corresponds to the k used at Layers 10 and 22, whereas the second is the kused at Layers 40 and 49. Tied weights refers to initializing each SAE decoder weight matrix to the transpose of the encoder weight matrix (but not coupled during training). The SAEs were trained on A100 GPUS using 40GB of memory. Software libraries we used include Pytorch 2.7.0 and OpenOOD 1.5.

References↩︎