1 1 Introduction↩︎

Vision foundation models (VFMs) have become a standard source of general-purpose image embeddings. Yet the embeddings they produce are difficult to interpret, in part because of the polysemanticity: a single coordinate of the representation may respond to many seemingly unrelated concepts. A leading explanation for this phenomenon is superposition, whereby a network represents more features than it has dimensions by encoding them as overlapping, non-orthogonal vectors [1]. This opacity is an obstacle to auditing and controlling systems built on top of these models.

Sparse autoencoders (SAEs) have recently emerged as a promising tool for interpreting such representations. An SAE encodes an input embedding into a higher-dimensional latent space under a sparsity constraint and reconstructs the input from this sparse code, with the aim of recovering latent units that are as monosemantic as possible, ideally, each responding to a single human-interpretable concept. Initially used to interpret the internal activations of large language models (LLMs) [2], [3], SAEs have since been applied to the embeddings of vision foundation models, where they are used to extract and study interpretable visual features [4]–[6].

SAEs differ in how they impose sparsity. The original formulation, which we refer to as the vanilla SAE, adds an explicit \(\ell_1\) penalty on the latent code to the reconstruction loss. Because this penalty grows with activation magnitude, it biases the encoder toward systematically reducing the magnitudes of active units–a phenomenon known as feature shrinkage [7], [8]–and vanilla SAE training is in general prone to producing dead latents, units that cease to activate entirely [7]. The Top-\(k\) SAE [7], building on the \(k\)-sparse autoencoder [9], was introduced specifically to dispense with the \(\ell_1\) penalty: it imposes sparsity architecturally, through an activation function that retains, for each input, only the \(k\) latent units with the largest activations and zeroes the rest. This enforces a hard per-sample sparsity budget while avoiding the magnitude shrinkage induced by the \(\ell_1\) term.

Perhaps because the Top-\(k\) SAE was conceived precisely to remove the \(\ell_1\) penalty, the literature has not revisited augmenting it with an additional sparsity constraint acting on the latent code. Yet the Top-\(k\) SAE retains its own well-documented limitations, several of which an additional sparsity term could plausibly address. First, the sparsity level \(k\) is uniform across samples, even though the number of latent factors needed to describe an input is likely to vary with its complexity; this rigidity has motivated a line of work that relaxes the per-sample budget, for example by enforcing it only on average across a batch [10], by allocating it adaptively across units [11], or by predicting an input-dependent \(k\) [12], [13]. The standard objective, however, is indifferent to how many of the \(k\) available units are actually used, provided the reconstruction is accurate; it therefore gives the model no incentive to encode an easy sample with fewer than \(k\) units. A sparsity penalty acting on the activations before selection would supply precisely this missing incentive, letting the effective number of active units adapt to each input rather than remaining fixed at \(k\). Second, the value of \(k\) itself is chosen arbitrarily, and a Top-\(k\) SAE tends to overfit to its training \(k\), in the sense that its reconstruction quality degrades when the number of units retained at inference departs from the value used during training [7]. Encouraging the model to concentrate its reconstruction onto fewer units would reduce its reliance on the exact value of \(k\), and so mitigate this sensitivity.

To address these limitations, we introduce two sparsity regularizers compatible with the Top-\(k\) architecture. Both act on the activations before the Top-\(k\) selection and are restricted to the batch-active units: the units selected by the Top-\(k\) operator at least once within the batch. The first is an \(\ell_1\) penalty on the off-support activations–those of units not selected by the Top-\(k\) operator for a given sample. In a standard Top-\(k\) SAE these activations receive no gradient from the reconstruction objective and are therefore left unconstrained, allowing a unit to produce moderate, sub-threshold responses to inputs unrelated to the concept it encodes when selected. Penalizing them drives these off-support responses toward zero, sharpening each unit’s selectivity: a unit is encouraged to activate strongly only on the inputs for which it is among the top contributors and to remain near-zero otherwise. We show that this yields more coherent activating image sets and higher monosemanticity. The second regularizer penalizes the ratio of the \(\ell_1\) to the \(\ell_2\) norm of the activations, a scale-invariant sparsity measure introduced by [14]. Minimizing it concentrates the code onto fewer effective units; we show that this concentration directly mitigates several of the limitations above.

We evaluate on two datasets, ImageNet-1K [15] and Open Images V7 [16], using the embeddings of three frozen vision foundation models–CLIP [17], SigLIP2[18], and a supervised ViT-L/16 [19]—across a range of \(k\). We assess interpretability with the Monosemanticity Score of [6], a label-grounded class-purity measure, and qualitative inspection, and we study the downstream effects of the induced concentration. Across all configurations, the use of both regularizers yield markedly more concentrated and more interpretable codes without hurting reconstruction quality.

Our contributions are as follows:

• We introduce two sparsity regularizers compatible with the Top-\(k\) architecture, both acting on the pre-selection activations: an off-support \(\ell_1\) penalty and an \(\ell_1/\ell_2\)-ratio penalty.

• Across two datasets, three vision foundation models, and a range of \(k\), we show that both regularizers consistently improve interpretability–measured by monosemanticity and class purity–while preserving reconstruction quality.

• We show that the concentration induced by the \(\ell_1/\ell_2\)-ratio regularizer addresses a known limitations of the Top-\(k\) SAE: it makes reconstruction substantially more robust to the inference-time choice of \(k\). Additionnally it concentrates discriminative information into fewer leading units, improving small-budget linear probing.

2 2 Related Work↩︎

2.0.0.1 Sparse autoencoders for vision.

Sparse autoencoders (SAEs) were originally introduced to interpret the internal activations of large language models (LLMs), decomposing them through dictionary learning into a larger set of sparse, more monosemantic latent units [2]. The same approach has since been transferred to the embeddings of vision foundation models (VFMs), where SAEs are used to extract and study human-interpretable visual features [4], [5]. We follow this line of work and train SAEs on the embeddings of frozen vision foundation models.

2.0.0.2 Monosemanticity.

A central goal of an SAE is to recover latent units that are as monosemantic as possible, that is, units each responding to a single human-interpretable concept. To quantify this property for vision SAEs, [6] introduce the Monosemanticity Score, which measures how similar the images that most strongly activate a given latent unit are to one another; higher values indicate a more concept-selective unit. We adopt the Monosemanticity Score as our primary measure of interpretability.

2.0.0.3 Top-\(k\) sparse autoencoders.

SAEs differ in how they impose sparsity. Early SAEs add an explicit \(\ell_1\) penalty on the latent code to the reconstruction loss which causes the suppression of weaker features [7], [8]. The Top-\(k\) sparse autoencoder [7] instead imposes sparsity architecturally, through its activation function: for each input it keeps only the \(k\) latent units with the largest activations and sets the remaining ones to zero. This enforces a hard per-sample sparsity budget and avoids the magnitude shrinkage caused by the \(\ell_1\) penalty. We build directly on this architecture.

2.0.0.4 Regularizing the Top-\(k\) sparse autoencoder.

In the Top-\(k\) sparse autoencoder, sparsity is controlled solely through the hard selection of the \(k\) largest activations. Prior attempts to add regularization in this setting act elsewhere than on the latent code: for instance, weight regularization penalizes the encoder and decoder weights, rather than the activations, to improve the stability of the learned features [20]. To the best of our knowledge, no prior work augments a Top-\(k\) sparse autoencoder with an explicit sparsity penalty applied directly to the activation vector seen by the Top-\(k\) operator. This is the gap addressed by the two regularizers we propose.

2.0.0.5 The \(\ell_1/\ell_2\) ratio as a sparsity measure.

One of our regularizers penalizes the ratio of the \(\ell_1\) to the \(\ell_2\) norm of the activations. This ratio originates from the sparseness measure of [14], defined as a normalized \(\ell_1/\ell_2\) ratio that is invariant to the scale of the vector. [21] subsequently proposed the Hoyer-Square regularizer, the squared \(\ell_1/\ell_2\) ratio, as a differentiable and scale-invariant surrogate for the \(\ell_0\) count, and used it to train sparser neural networks at comparable accuracy. Our second regularizer adapts this \(\ell_1/\ell_2\)-ratio family to the activations of a Top-\(k\) sparse autoencoder.

3 3 Method↩︎

Top-\(k\) sparse autoencoder enforces sparsity of its latent code only through its Top-\(k\) activation function[7].

We propose two regularizers compatible with the Top-\(k\) architecture, each imposing an additional sparsity constraint on the activations before the Top-\(k\) operator. Their effects are illustrated in Figure 1.

3.1 3.1 Notation and Architecture↩︎

We train a Top-\(k\) sparse autoencoder on embeddings \(x \in \mathbb{R}^d\) produced by a frozen vision foundation model. A batch of \(N\) samples is denoted \(\mathcal{B} = \{x^{(1)}, \dots, x^{(N)}\}\). The SAE has \(m \gg d\) latent units, with encoder weights \(W_e \in \mathbb{R}^{m \times d}\), decoder weights \(W_d \in \mathbb{R}^{d \times m}\), an encoder bias \(b_e \in \mathbb{R}^m\) over the latent space, and a bias \(b_d \in \mathbb{R}^d\) over the input space. As is standard for this architecture [2], [7], \(b_d\) serves as a centering bias: it is subtracted from the input before the encoder and added back after the decoder, so no separate decoder bias is used.

For a sample \(x^{(i)}\), the encoder first produces the pre-activations \[\pi^{(i)} = W_e\,\big(x^{(i)} - b_d\big) + b_e \in \mathbb{R}^m ,\] to which a ReLU is applied to obtain the activations \[a^{(i)} = \mathrm{ReLU}\!\left(\pi^{(i)}\right) \in \mathbb{R}^m_{\geq 0}.\] The Top-\(k\) operator then retains the \(k\) largest activations and zeros the rest. Let \(S^{(i)} \subseteq \{1, \dots, m\}\), with \(|S^{(i)}| = k\), denote the index set of the \(k\) largest coordinates of \(a^{(i)}\). The sparse code is \[z^{(i)} = \mathrm{TopK}_k\!\left(a^{(i)}\right), \qquad z^{(i)}_j = \begin{cases} a^{(i)}_j, & j \in S^{(i)}, \\ 0, & j \notin S^{(i)}. \end{cases}\] Finally, the decoder reconstructs the input from the sparse code, adding the centering bias back, \[\hat{x}^{(i)} = W_d\, z^{(i)} + b_d .\]

3.2 3.2 Active-Unit Mask↩︎

Both regularizers are restricted to the activations selected by the Top-\(k\) operator for at least one sample of the batch. We define this active set as \[\mathcal{A}(\mathcal{B}) = \bigcup_{i=1}^{N} S^{(i)} \subseteq \{1, \dots, m\},\] and encode it as a binary mask \(\mu \in \{0,1\}^m\), the per-unit indicator of the active set, \[\mu_j = \mathbf{1}\big[\, j \in \mathcal{A}(\mathcal{B}) \,\big] = \begin{cases} 1, & \text{if } \exists\, i \in \{1,\dots,N\}:\;j \in S^{(i)}, \\ 0, & \text{otherwise.} \end{cases}\] The mask is recomputed for each batch and treated as a constant during backpropagation; no gradient flows through \(\mu\). Masking restricts the penalty to units that contribute to the reconstruction of the batch. Units that are never selected receive no reconstruction gradient, so penalizing them would drive their activations toward zero with no counteracting signal, inflating the number of dead neurons.

3.3 3.3 Regularizers↩︎

Both regularizers act on the activations \(a^{(i)}\), restricted to the batch-active units by the mask \(\mu\). They share the notation above and differ only in their expression. We denote the regularizer evaluated on a batch by \(R(\mathcal{B})\).

3.3.0.1 Regularizer 1 (off-support \(\ell_1\)).

We penalize the \(\ell_1\) norm of the masked residual between the activations and the sparse code, \[R(\mathcal{B}) = \sum_{i=1}^{N} \big\| \mu \odot \big( a^{(i)} - z^{(i)} \big) \big\|_1 ,\] where \(\odot\) is the elementwise (Hadamard) product. Since \(a^{(i)}_j = z^{(i)}_j\) on the selected units, this residual is supported exactly on the units not selected by the Top-\(k\) operator (the off-support units); the mask further restricts it to the units active across the batch. Minimizing \(R\) thus drives the off-support activation mass of the batch-active units toward zero.

3.3.0.2 Regularizer 2 (\(\ell_1/\ell_2\) ratio).

We penalize the ratio of the \(\ell_1\) to the \(\ell_2\) norm of the masked activations, \[R(\mathcal{B}) = \sum_{i=1}^{N} \frac{\big\| \mu \odot a^{(i)} \big\|_1}{\big\| \mu \odot a^{(i)} \big\|_2 },\] The ratio \(\|\cdot\|_1 / \|\cdot\|_2\) is scale-invariant, so the penalty concentrates the code without competing with the reconstruction term over its overall magnitude. Its square is a standard proxy for the effective number of active units [14], so minimizing \(R\) pushes each code toward fewer effective active units.

3.4 3.4 Training Objective↩︎

With the reconstruction loss \[\mathcal{L}_{\text{recon}}(\mathcal{B}) = \frac{1}{N} \sum_{i=1}^{N} \big\| x^{(i)} - \hat{x}^{(i)} \big\|_2^2,\] and the auxiliary Top-\(k\) loss \(\mathcal{L}_{\text{aux}}(\mathcal{B})\) that reconstructs the residual from the top unselected (dead) units [7], the total objective is \[\mathcal{L}(\mathcal{B}) = \mathcal{L}_{\text{recon}}(\mathcal{B}) + \alpha\, \mathcal{L}_{\text{aux}}(\mathcal{B}) + \lambda\, R(\mathcal{B}),\] where \(\lambda \geq 0\) controls the strength of the sparsity penalty and \(\alpha \geq 0\) the auxiliary term. The baseline Top-\(k\) SAE corresponds to \(\lambda = 0\).

3pt

4 4 Experiments↩︎

We investigate whether the proposed regularizers improve the interpretability of Top-\(k\) SAEs. We train matched pairs of models, with and without a regularizer, and evaluate them quantitatively on (i) reconstruction and monosemanticity and (ii) class purity, followed by a qualitative inspection of neurons across their full activation range.

4.0.0.1 Setup.

We train and evaluate Top-\(k\) SAEs on the image embeddings of three frozen vision foundation models—(i) CLIP ViT-L/14 [17], (ii) SigLIP2 [18], and (iii) a supervised ViT-L/16 [19]—and on two datasets, ImageNet-1K [15] and Open Images V7[16]. The latent dimension is \(8192\) and the only hyperparameter varied across runs is \(k \in \{32, 64, 128\}\). For each (encoder, dataset, \(k\)) configuration, we compare the unregularized Top-\(k\) baseline (\(\lambda = 0\)) against our regularized model (for both regularizers), keeping all other settings identical so that the two differ only in the regularizer \(R\) and its coefficient \(\lambda\).

4.0.0.2 Reconstruction and interpretability.

The primary goal of an SAE is to expose interpretable features, which we quantify with the Monosemanticity Score. All metrics are computed on the test set. \(R^2\) measures reconstruction quality, and monosemanticity is computed per latent unit and reported as its mean (M\(_\mu\)) and median (M\(_m\)) across units; for all three, higher is better. We additionally report the number of dead neurons–latent units never activated on the test set–for which lower is better.

For both regularizers, increasing \(\lambda\) initially improves the reconstruction before degrading it, whereas monosemanticity increases monotonically over the range we consider. Since our objective is to maximize interpretability, for each configuration we select the largest \(\lambda\) whose \(R^2\) remains superior to the baseline, yielding the largest monosemanticity gain at no reconstruction cost. Results are reported in Table 1.

Under this protocol, both regularizers improve monosemanticity while preserving reconstruction quality across all configurations, with a single exception: SigLIP on ImageNet under the Regularizer 2 (\(\ell_1/\ell_2\) ratio), where mean monosemanticity decreases marginally (\(-0.002\)). Regularizer 1 (off-support \(\ell_1\)) yields the larger monosemanticity gain in every configuration. The effect is most pronounced for the ViT encoder, where adding a regularizer improves mean monosemanticity by up to \(0.13\) (Regularizer 1) and \(0.07\) (Regularizer 2), while SigLIP shows the most modest improvements.

Finally, the two regularizers have opposite effects on the number of dead neurons: Regularizer 2 (\(\ell_1/\ell_2\)) tends to increase it, whereas Regularizer 1 (off-support \(\ell_1\)) tends to decrease it, in some cases substantially (e.g.on SigLIP).

3pt

4.0.0.3 Class purity.

The Monosemanticity Score captures monosemanticity through the similarity of embeddings. To corroborate it with a label-grounded measure, we evaluate the class purity of the latent units on the ImageNet-1K validation set. For each active latent \(j\), we define its dominant class as the label that fires it most often, \[c_j^\star = \arg\max_{c} \sum_{n\,:\,y_n = c} \mathbf{1}\!\left[z^{(n)}_j > 0\right],\] where \(y_n\) is the label of sample \(n\) and \(z^{(n)}_j\) its activation on latent \(j\). We then summarize how concentrated each latent is on its dominant class with two metrics. Binary purity is the fraction of samples activating the latent whose label is \(c_j^\star\), counting every activation equally. Weighted purity is the same fraction but weighted by activation strength, so that strongly activating samples contribute more; it measures whether a latent’s strongest responses, in particular, are class-consistent. A latent that responds exclusively to a single class attains a purity of \(1\), whereas one that responds indiscriminately scores near the class prior. We average each metric over the active latents and report the mean per configuration.

As shown in Table 2, both regularizers increase binary and weighted purity across all encoders (except a single regression–a \(-0.00035\) change in binary purity for SigLIP at \(k{=}128\) under Regularizer 2). The gains are largest on the ViT encoder, where Regularizer 1 raises binary purity by up to \(0.14\) and weighted purity by up to \(0.13\). Consistent with the monosemanticity results, Regularizer 1 (off-support \(\ell_1\)) improves purity more than Regularizer 2.

4.0.0.4 Qualitative analysis.

The Monosemanticity Score is intended to quantify how visually and semantically coherent the images activating a latent are. We complement it with a direct qualitative inspection of these images. We focus on the ViT encoder with \(k=32\) and rank the latents by their test-set monosemanticity score, separately for the baseline and for Regularizer 1 (off-support \(\ell_1\)). For a given latent, we examine its top-10, middle-10, and bottom-10 activating images–that is, the images eliciting its strongest, median, and weakest activations—which probes coherence across the full activation range rather than at the peak alone.

We conduct two complementary comparisons. The first is rank-matched: we compare the latent at a given monosemanticity rank with the regularizer against the latent at the same rank without it (in general two distinct units), isolating how coherent a latent of comparable standing is in each model. The second is unit-matched: we compare the same latent (identical position in the dictionary) with and without the regularizer, isolating the effect of the penalty on a fixed unit.

The rank-matched comparison (Figure 2) shows that, with the regularizer, the dominant concept of a latent remains consistent down to its weakest (bottom-10) activations, whereas for the baseline coherence degrades away from the top activations. The unit-matched comparison (Figure 3) shows that a given latent continues to encode the same concept with and without the regularizer, but its set of activating images becomes markedly more coherent.

4.0.0.5 Sparsity effects of the regularizers.

While the regularizers were shown above to improve monosemanticity, their original motivation is to impose an additional sparsity constraint. To visualize their effect on the activation distribution, we sort each test sample’s \(m\) activations in decreasing order and average them rank-by-rank across the ImageNet-1K test set, obtaining the mean magnitude of the \(r\)-th largest activation as a function of rank \(r\). Figure 4 plots this averaged, rank-ordered profile for CLIP ViT-L/14 at \(k=64\), for the baseline and for each regularizer; the red line at rank \(k=64\) separates the activations kept by the Top-\(k\) operator (left) from those it zeroes out (right).

The two regularizers act on this profile in qualitatively different ways. Regularizer 1 (off-support \(\ell_1\)) leaves the kept activations almost unchanged relative to the baseline and only suppresses the off-support tail, consistent with its design. Regularizer 2 (\(\ell_1/\ell_2\) ratio) instead reshapes the entire profile, concentrating mass onto the very top ranks: its largest activation is substantially higher than the baseline’s while the remaining ranks decay much faster. The \(\ell_1/\ell_2\) ratio thus has the more pronounced effect on the activation distribution, whereas the off-support penalty acts only where activations are already small.

So far, our interpretability analysis considered both regularizers and showed that Regularizer 1 yields the larger monosemanticity gains. The activation profiles, however, reveal that Regularizer 2 reshapes the latent geometry far more aggressively. The remaining experiments focus on Regularizer 2, showing that its concentrated geometry brings substantial benefits beyond monosemanticity.

4.0.0.6 Robustness to the inference-time \(k\).

A direct consequence of the concentration induced by Regularizer 2 is that activation magnitudes decay quickly with rank (Figure 4), so that most of the reconstructed signal is carried by a small number of units. This suggests that the exact number of units retained at inference should matter less than for the baseline: keeping slightly fewer or more than the training \(k\) should leave reconstruction largely unchanged. Such a property would mitigate a known limitation of Top-\(k\) SAEs, namely their sensitivity to the arbitrary choice of \(k\) and the resulting overfitting to the training value [7]. To test this, we train Top-\(k\) SAEs on ImageNet-1K with CLIP ViT-L/14 at \(k \in \{32, 64, 128\}\), with and without Regularizer 2, and evaluate each model while varying the number of retained units \(k_{\text{inf}}\) at inference around the training value. We report the normalized reconstruction error (NMSE) as a function of \(k_{\text{inf}}\).

Figure 5 reveals two complementary effects, both more pronounced at larger training \(k\). For \(k_{\text{inf}}\) below the training value (i.e.under truncation), Regularizer 2 attains a consistently lower NMSE than the baseline, and the gap widens as the model is truncated more aggressively–most strikingly at training \(k=128\), where the regularized model degrades far more gracefully. For \(k_{\text{inf}}\) above the training value, the baseline NMSE rises back up, indicating that it has overfit to using exactly \(k\) units, whereas Regularizer 2 remains essentially flat. Together, these show that the concentrated geometry makes reconstruction substantially more robust to the inference-time \(k\).

4.0.0.7 Probing under activation truncation.

If Regularizer 2 concentrates activation magnitude onto fewer units, it should also concentrate the information carried by the code, in the sense that fewer units suffice to identify the class of a sample. We test this by probing the code under activation truncation. For each trained SAE, we form a truncated code by keeping only the \(k'\) largest activations of each sample and zeroing the rest, fit a linear probe on these truncated codes for ImageNet-1K classification (\(1000\) classes), training the probe on the training split and reporting top-1 accuracy on the test split. Sweeping the truncation level \(k'\) yields an accuracy-versus-\(k'\) curve; a curve that rises steeply at small \(k'\) indicates that the discriminative information is packed into few coordinates. We train Top-\(k\) SAEs on ImageNet-1K with CLIP ViT-L/14, SigLIP2, and a supervised ViT-L/16, and compare the baseline against Regularizer 2.

Figure 5 shows the curve for CLIP ViT-L/14 at \(k=64\). The two models reach the same asymptotic test accuracy (\({\approx}0.82\)) once enough units are retained, so the regularizer does not change the total information recoverable from the code; it reaches high accuracy with far fewer units, dominating the baseline across the entire small-budget regime.

To quantify this concentration of discriminative information, we report the normalized area under the accuracy-versus-\(k'\) curve, defined as the AUC over \(k' \in [1, K]\) divided by \(K\); this rescaling places it in \([0,1]\) and keeps it comparable across integration ranges. A higher value indicates that accuracy is attained at smaller truncation levels. We set \(K = 32\), integrating only up to this point because the accuracy curves of the two models converge once enough units are retained, so beyond it their difference adds no discriminative signal and would only attenuate the metric.

Table 3 reports the normalized AUC across all nine configurations, together with the top-1 accuracy at the full budget \(k'=32\). The AUC is higher for Regularizer 2 in every configuration, confirming that it consistently front-loads discriminative information into the leading activations. At the same time, the accuracy at \(k'=32\) is essentially unchanged relative to the baseline, showing that this gain in early-budget accuracy comes at no cost to overall performance.

3pt

4.0.0.8 Necessity of the active-unit mask.

Both regularizers are restricted to the batch-active units–those selected in the Top-\(k\) of at least one sample of the batch (Section 2.2). We motivated this design by noting that an unselected unit receives no gradient from the reconstruction term; penalizing it directly therefore applies a one-sided force that drives its activations to zero and eventually kills the unit. To test this, we repeat every configuration of Tables 1 and 2 with the regularization applied to all units (no mask) and report the resulting number of dead neurons.

Table 4 confirms the claim. Removing the mask increases the number of dead neurons in nearly every configuration, frequently by one to two orders of magnitude (e.g.from \(12\) to \(2426\) dead units for the supervised ViT-L/16 on Open Images V7 at \(k=32\) under Regularizer 1). The few exceptions are configurations in which no units die even without the mask, such as the supervised ViT-L/16 on ImageNet-1K at \(k \in \{64,128\}\). Beyond the increase in dead neurons, the unmasked variant also systematically lowers mean monosemanticity relative to the masked one. The mask is thus necessary both for stability and for the interpretability gains reported above.

3pt

5 5 Conclusion↩︎

We introduced two sparsity regularizers compatible with the Top-\(k\) architecture, both acting on the activations before the Top-\(k\) selection: an \(\ell_1\) penalty on the off-support units and an \(\ell_1/\ell_2\)-ratio penalty on the full activation vector. Across two datasets, three vision foundation models, and a range of \(k\), both regularizers consistently improve interpretability—measured by monosemanticity and class purity–at no cost to reconstruction quality. Beyond interpretability, we showed that the concentration induced by the \(\ell_1/\ell_2\)-ratio penalty makes reconstruction substantially more robust to the inference-time choice of \(k\) and front-loads discriminative information into the leading units, improving small-budget linear probing.

These results carry a simple message: although the Top-\(k\) SAE was introduced specifically to avoid the \(\ell_1\) penalty and its known drawbacks, it still benefits from an explicit sparsity constraint–provided the constraint acts on the pre-selection activations rather than the sparse code, and is restricted to the batch-active units so as not to create dead latents. Hard architectural sparsity and soft regularization are thus complementary rather than mutually exclusive.

Several directions follow naturally. The regularizers could be extended to other members of the Top-\(k\) family, such as BatchTopK [10] and Matryoshka SAEs [22], whose selection mechanisms differ from the per-sample Top-\(k\) studied here. More broadly, the \(\ell_1/\ell_2\) ratio is a scale-invariant sparsity measure that remains underused in representation learning, and our results suggest it is a promising and largely unexplored tool for shaping the geometry of sparse codes.

References↩︎