Inhibited Self-Attention:
Sharpening Focus in Vision Transformers
July 14, 2026
Abstract
Vision Transformers (ViTs) have demonstrated remarkable performance in computer vision tasks. However, their self-attention mechanism often diffuses focus across background regions, relying on spurious correlations rather than object-relevant cues. Inspired by inhibitory mechanisms observed in biological vision systems, we propose the Inhibited Self-Attention (ISA), a novel self-attention that integrates inhibitory signals to enhance feature selectivity and suppress spurious responses. In contrast to conventional self-attention, which relies solely on positive attention values due to softmax normalization, our approach retains and utilizes negative attention scores to suppress irrelevant features and sharpen focus on objects of interest. Experiments across multiple datasets, including ImageNet-1k and COCO, and several robustness benchmarks demonstrate that ISA enhances object-centric selectivity, reduces shortcut reliance, and improves out-of-distribution generalization. Our analysis of relevance maps confirms that ViTs with ISA exhibit sharper, more localized focus on object-relevant regions while reducing distractions from non-relevant (background) features, enabling more reliable models. We release our code at https://github.com/prdvanderwal/inhibited-self-attention.
1 Introduction↩︎
Vision Transformers (ViTs) [1] have sparked a paradigm shift in computer vision, challenging the decade-long dominance of Convolutional Neural Networks (CNNs) [2]–[4]. Unlike CNNs, which rely on local feature extraction through convolutional filters, ViTs process an image as a set of patches, using self-attention to model global dependencies [1], [5], [6]. This enables ViT architectures to drop the inductive bias of convolutional operators, and use self-attention to expand the range of the receptive field to long-range relations among image patches. Remarkably, ViTs achieved higher performance than CNNs when trained on very large datasets, leading to foundation models that can be adapted to various tasks [1].
Despite their strong performance, ViTs do not always concentrate attention on the most informative image regions. They frequently distribute attention across background elements that contain easily detected but non‑informative spurious cues. This can introduce bias by over-relying on background features and reducing the ability of ViTs to learn class-specific representations [7]. This limitation increases the likelihood of relying on incidental correlations rather than meaningful object attributes [8]. In 1, we show images correctly classified by a ViT with high confidence but with attention focused on the background or other shortcuts rather than on the objects of interest. Steering attention towards objects of interest, has been an active line of research in CNN based models, but comparatively fewer works have addressed this challenge in the context of ViTs.
Figure 1: (a) Examples of correctly classified images by a ViT-S/16 with very high confidence where the attention of the model is not directed at the object of interest. (b) The same images classified by ISA-ViT-S/16 using our proposed inhibited self-attention.. a — ViT-S/16, b — ISA-ViT-S/16
In this paper, we reshape the self-attention mechanism of Vision Transformers by incorporating an inhibitory process that enhances the selectivity of the model attention. Our method draws inspiration from biological visual processing, where inhibitory receptive fields in the visual system suppress responses to spurious stimuli, thus reinforcing the response to the patterns of interest [9]–[11]. We refer to this approach as Inhibited Self-Attention (ISA). Unlike prior methods that rely on auxiliary models [12], [13] or add extra architectural branches (e.g., the Differential Transformer [14]), we make use of the already available negative attention scores, normally discarded by the softmax in standard Self-Attention, to introduce contextual inhibition and refine the final attention distribution. This mitigates a key limitation of conventional ViTs, which often allocate attention to spurious background features while reducing focus on class-relevant regions. By suppressing the non-relevant feature information, our approach enhances the ability of ViT models to learn more discriminative and meaningful feature representations. We investigate the improvement of the selectivity and robustness of ViTs brought by our method in object classification and detection, and particularly in the presence of shortcuts and distribution shifts.
Our main contributions are: (1) We introduce a novel inhibited self-attention mechanism that incorporates negative attention values into ViTs, which contribute to sharpening the focus of self-attention on relevant image regions, without the overhead or architectural complexity of auxiliary models. (2) We introduce a new metric called ‘Attention-on-Objects’ to quantify object-centered attention in ViTs.
2 Related Work↩︎
2.0.0.1 ViTs and Softmax in Self-Attention.
Inspired by the transformer model for natural language processing applications [15], the Vision Transformer (ViT) [1] has caused a paradigm shift in the field of computer vision. In contrast to CNNs which work on the locality principle with filters that only observe a region of the input image, the receptive field of ViTs is effectively expanded to the entire image through the self-attention mechanism [5]. This allows ViTs to capture rich and highly flexible global patterns [13]. In self-attention, each image patch is mapped to three vectors: queries, keys, and values. Similarity scores between queries and keys are computed, which can be positive or negative depending on the relationship between the patches. These raw scores are then scaled and passed through the softmax activation function, which normalizes them into a probability distribution. The softmax function converts both positive and negative similarities into positive weights, thereby eliminating the inhibitory effect of negative correlations and limiting the model’s ability to capture relational structures involving both excitatory and inhibitory interactions.
In an effort to combine the global connectivity of ViTs with the locality of CNNs, several variants have been proposed, including Swin Transformer [16], ScopeViT [17], LocalViT [5], and ReViT [18]. For object detection, DETR [19] introduced an end-to-end transformer framework later refined by DINO [20] with denoising queries and improved feature alignment. Data-efficient Image Transformers (DeiT) [21] addressed the need for costly pre-training through distillation, using a teacher classifier and a dedicated distillation token that interacts with all embeddings. More recently, it was shown that competitive ImageNet-1k performance can also be achieved when training from scratch [22].
2.0.0.2 Biologically-Inspired Vision Models
Biological neural networks have often served as inspiration for artificial neural networks (ANNs), particularly in vision research. Similar to a CNN that learns a filter to only respond to particular stimuli, the majority of the neurons in the V1 area of the visual system of the brain display a similar behaviour called orientation selectivity [9]. Some neurons in area V1 exhibit a response inhibition mechanism, called push-pull inhibition [9], [23], that sharpens the selectivity by suppressing responses to non-preferred stimuli, such as spurious texture. A computational model of simple cells for image processing was proposed in [24], and subsequently extended by including the push-pull inhibition mechanism in [25], with improved performance in noisy images. Other key inhibition mechanisms include center-surround inhibition, which enhances edge detection by suppressing uniform backgrounds [10], [26], and lateral inhibition, which sharpens contrast by reducing redundant responses by neighboring neurons [11]. Lateral inhibition was exploited in AlexNet [27], using a Local Response Normalization, yielding improved generalization performance. Furthermore, in [28], [29] push-pull inhibition was deployed in a new type of CNN layer, improving the robustness of models to image corruptions.
The implementation of an inhibition mechanism in [30] that functions in parallel with components of the Swin-Transformer [16], obtained improved accuracy, sensitivity, and specificity in comparison with the regular Swin-Transformer on a medical image dataset.
In this work, we propose a novel self-attention mechanism that exploits negative attention scores to inhibit the attention of non-relevant features, thus sharpening the reliance on relevant areas of the objects of interest.
2.0.0.3 Enhancing Attention Selectivity
Despite many variations of the self-attention mechanism, ViTs are prone to attend to irrelevant parts of the image such as background features [14], [31], risking low selectivity of the models on the features that strictly characterize the patterns of interest. This phenomenon was empirically demonstrated in [13] and identified as a leading reason for the training difficulties of ViTs.
While object-centric mechanisms have been widely investigated for CNNs [32]–[34], their direct adaptation to attention in Vision Transformers has received comparatively less focus. In order to improve the selectivity of ViT models for relevant features of the objects of interest, in [13] the authors proposed a convolution-based token-selector block that can be inserted right after the attention mechanism. In [31] a CNN extractor was used to select relevant patches, such that only the attention over these patches is computed. This approach improves the selectivity and reduces the computational costs (as the self-attention is computed between a lower number of patches) of ViTs. However, its performance relies on the quality of the CNN-based patch descriptor. Similarly, in [12] the authors aim to rectify shortcut learning in ViTs but rely on a separate saliency model. Instead of using an auxiliary model for patch selection or saliency calculation, in [6] a position-aware attention scaling was introduced.
Recent work has likewise identified that softmax discards information from negative query–key interactions. PolaFormer [35] recovers these components within linear attention, where non-negative feature maps are required for a valid kernel decomposition, by routing same-signed and opposite-signed interactions through parallel streams, motivated by expressivity and efficiency rather than selectivity. Cog Attention [36] allows negative output weights by multiplying softmax applied to absolute QK scores with the sign of the raw QK products, framing negativity as a dynamic token-deletion mechanism targeting expressivity in language modelling and diffusion models, without addressing object-centric selectivity or robustness. The Differential Transformer [14] was proposed for language modeling to reduce attention noise through subtractive filtering between two softmax branches; this design requires two separate query–key projection pairs per layer, which increases parameter overhead, and is less suited to vision tasks with large token counts. Furthermore, unlike ISA, it operates primarily at the attention output level, which may be insufficient to produce the highly selective attention focus required to mitigate shortcut reliance. The Integral Transformer [37] instead averages multiple QK logit signals before softmax, trading subtraction for integration, but reduces the per-signal head dimension by a factor equal to the number of signals, lowering the expressiveness of projections.
ISA utilizes a competitive interaction between two normalizations of the same attention signal (softmax and softmin), leveraging the negative attention information discarded by standard self-attention into an explicit inhibitory signal that sharpens attention and reduces noise. Unlike existing methods that require auxiliary models or extra query-key pairs, ISA is a seamless drop-in replacement for standard attention, with no additional parameter or FLOPs.
3 Method: Inhibited Self-Attention↩︎
We propose a new type of self-attention mechanism for Vision Transformers that utilizes all available information in the attention, exploiting both positive and negative scores. We call it Inhibited Self-Attention (ISA). The output of ISA is computed through an inhibition process, inspired by the push-pull inhibition in the visual system of the brain, which effectively uses negative attention scores usually discarded by the standard self-attention due to the softmax activation function. An overview of our proposed method is illustrated in 2a.
3.0.0.1 (Raw) Self-Attention
The purpose of the attention mechanism in Vision Transformers is to allow the model to learn how to dynamically focus on specific parts of the input for making predictions. This is done by calculating attention scores for each token representing an image patch, effectively determining how important each part of the image is. For the proposed inhibited self-attention, we calculate the raw attention scores as for the standard attention \(\boldsymbol{A}_{raw} = \boldsymbol{QK}^T / \sqrt{d_\textit{k}}\), where Q is the query, K the key, and \(\textit{d}_k\) a scaling factor.
3.0.0.2 Inhibited Self-Attention
Whereas the normal attention mechanism would only calculate the softmax of the raw attention scores, we calculate the softmin distribution as well. Given the raw attention scores \(\mathbf{A}_{\text{raw}} \in \mathbb{R}^{n \times n}\), we compute the positive attention \(\mathbf{A}^+\) and the negative attention \(\mathbf{A}^-\) as the softmax and softmin distributions, respectively. Their formulation is as follows:
\[\mathbf{A}^+_{i,j} = \frac{e^{\mathbf{A}_{\text{raw}, i,j}}}{\sum_k e^{\mathbf{A}_{\text{raw}, i,k}}}, ~\quad~ \mathbf{A}^-_{i,j} = \frac{e^{-\mathbf{A}_{\text{raw}, i,j}}}{\sum_k e^{-\mathbf{A}_{\text{raw}, i,k}}}\]
The softmin attention scores enable the model to capture the negative attention between patches in a complementary manner to the positive attention, assigning higher weights to elements with lower scores and effectively highlighting those patches that potentially contribute negatively to model predictions. We combine the positive and negative attention scores through an inhibition mechanism to obtain the inhibited self-attention scores:
\[\tilde{\mathbf{A}}_{i,j} = \max \left(\mathbf{A}^+_{i,j} - \alpha \mathbf{A}^-_{i,j}, \epsilon \right)\]
where \(\alpha\) is a hyperparameter that indicates the inhibition strength and \(\epsilon = 10^{-10}\) is a small constant to ensure the function is strictly positive while maintaining numerical stability and preventing vanishing gradients.
Finally, we L1-normalize row-wise to ensure a valid probability distribution of the self-attention scores:
\[\mathbf{A}^{\text{inhib}}_{i,j} = \frac{\tilde{\mathbf{A}}_{i,j}}{\sum_k \tilde{\mathbf{A}}_{i,k}}\]
By using the negative raw attention scores, our formulation captures both excitatory and inhibitory relationships between patches, allowing the model to infer not only what to focus on but also what to ignore. We show the difference between the standard attention mechanism using only softmax and ISA in [fig:methodexamples]. The same input yields an attention map with more focused attention on the object of interest and reduced attention on the surroundings. Importantly, ISA does not merely suppress low-attention patches; it selectively increases attention on some regions while decreasing it on others, producing a reweighted map that cannot be reproduced by any pointwise (non)linear transformation of the standard attention map, since ISA relies on a competitive interaction between two distinct attention signals.
3.0.0.3 Deployment in ViTs
We replace the standard self-attention mechanism in each layer of the ViT with our proposed Inhibited Self-Attention. For object detection experiments, inhibition is applied exclusively within the ViT backbone. In all experiments, \(\alpha\) is set as a learnable parameter, initialized with a value of 1.
4 Evaluation↩︎
We demonstrate ISA’s enhanced attention selectivity through quantitative and qualitative analysis of attention maps. We compare against several baselines on image classification and object detection, evaluating performance on clean data, shortcut reliance, and out-of-distribution settings.
4.1 Experimental Setup↩︎
4.1.0.1 Datasets
We evaluate ISA selectivity on the ImageNet-1k [38] and ImageNet-Segmentation (ImageNet-S) [39] datasets. ImageNet-S, a pixel-annotated subset of ImageNet-1k, provides ground-truth segmentation maps that allow us to quantify the attention focused on objects versus background regions.
To examine shortcut vulnerability, we employ ImageNet-Watermark (ImageNet-W) [40], which introduces Chinese character watermarks into the ImageNet-1k validation set to test sensitivity to spurious correlations.
Generalization to out-of-distribution samples is evaluated using ImageNet-Renditions (ImageNet-R) [41], while background reliance is assessed with the Waterbirds dataset [42]. We further evaluate robustness to image corruptions on ImageNet-C [43] and ImageNet-\(\bar{\text{C}}\) [44].
Finally, to demonstrate the advantages of ISA beyond image classification, we conduct object detection experiments on the COCO dataset [45].
4.1.0.2 Evaluation Metrics
We analyze the focus of ViT attention on class-relevant regions, namely on the areas belonging to the objects of interest, using the Attention-on-Objects metric that we propose. We elaborate on this metric in 4.2.0.1.
The performance of the models on clean data is measured by computing the classification accuracy and cross-entropy loss on the test set. For evaluation on ImageNet-W, we report the IN-W Gap which is the accuracy degradation compared to ImageNet-1k, and the Carton Gap that corresponds to the increase in the carton class accuracy from ImageNet-1k to ImageNet-W [40]. For corrupted data we report the mean Corruption Error (mCE) and relative mean Corruption Error (rmCE) [43], and normalize the results of each model with inhibition with respect to those of its respective baseline without inhibition. For object detection we evaluate the models on the mean average precision (mAP) and Attention-on-Detected-Objects (see 4.4).
4.1.0.3 Baselines
Considering that our proposed Inhibited Self-Attention is a drop-in replacement for the self-attention, we train our ViTs from scratch. We take ViT-S/16 as our baseline, which established a strong ImageNet-1k benchmark without the need of computationally expensive pre-training [22], together with the Differential Transformer [14], which we adapt for ViTs (DIFF-ViT). To demonstrate the stability of the proposed mechanism, we also train a DeiT-S model [21], with and without ISA. For object detection, we use our pre-trained ViT-S as backbone with a DINO detection head [20]. Models incorporating ISA are denoted as ‘ISA-ViT’, ‘ISA-DeiT’, and ‘ISA-DINO’, corresponding to the ViT, DeiT, and DINO architectures, respectively.
4.1.0.4 Implementation
We trained all models on the ImageNet-1k dataset from scratch. For the ViT-S/16, DIFF-ViT-S/16, and ISA-ViT-S/16, we adopted the training settings outlined in [22], with 90 training epochs and RandAugment augmentation [46]. Consistent with [22], we employed global average pooling for final classification instead of a CLS token2. For DeiT and ISA-DeiT, we followed the official DeiT settings, training for 300 epochs with straightforward augmentations, including grayscale, solarization, Gaussian blur, and color jitter. For knowledge
distillation, we used the pre-trained RegNetY-16GF [47] image classification model as the teacher model. We opted for
knowledge distillation with soft targets to help the student ViT model better capture inter-class relationships. Fine-tuning for downstream tasks was performed without knowledge distillation. Instead, we applied linear probing on the concatenation of the
CLS and DIST tokens. For DINO, we trained for 12 epochs and followed the configuration settings of [20]. With a simple
convolutional feature pyramid network we extracted 3 feature maps from the output of the ViT backbone as input for the DINO head. The batch size was set to 256 for classification and 2 for object detection. All training was conducted on a single node with
4 NVIDIA A100 GPUs. The code and hyperparameters used for training were kept consistent between the proposed models and respective baselines to ensure a fair comparison. Seed training was enabled for reproducible results.
4.2 Enhanced Attention Selectivity↩︎
We visualize the model attention using CheferCam [48], [49], which produces more accurate visualizations compared to raw attention, roll-out [50] or GradCAM [51], and investigate how ISA influences the final attention.
4.2.0.1 Attention-on-Objects (AoO).
We develop the Attention-on-Object (AoO) metric to measure attention focus, defined as the ratio of attention on object regions to that on the background. Unlike the Pointing Game [32], which considers only the single highest attention point, AoO accounts for all attention mass, and uses segmentation masks from ImageNet-S to delineate object and background regions, rather than the coarser boxes employed by the Energy-based Pointing Game [52]. The AoO ratio (\(A_{\text{object}}\)) is calculated as:
\[A_{\text{object}} = \frac{\sum_{i,j} \mathcal{M}_{i,j}A_{i,j}}{\sum_{i,j} A_{i,j}},\]
with \(A_{i,j}\) denoting the attention value at pixel \((i,j)\) of the interpolated attention map, and \(\mathcal{M}_{i,j} \in \{0,1\}\) a binary mask indicating whether pixel \((i,j)\) belongs to the object of interest. As we are not only interested in the amount of attention on the object but the strength of the attention, we incrementally threshold the attention and obtain the thresholded AoO (\(A_{\text{object}}^{\text{thresh}}\)) as follows:
\[A^{thresh}_{\text{object}} = \frac{\sum_{i,j} M_{i,j} A^{t}_{i,j}}{\sum_{i,j} A^{t}_{i,j}}, \quad A^{t}_{i,j} = \begin{cases} A_{i,j}, & A_{i,j} \ge T_\text{val} \\ 0, & \text{otherwise} \end{cases}, \label{threshold95formula}\tag{1}\]
where \(T_{\text{val}} = \frac{T}{100} \times A_\text{max}\) is the threshold value, with \(T\) being the threshold percentage (in the range of [0, 99]), and \(A_{\text{max}}\) the normalized maximum value in the attention map. This thresholding further distinguishes AoO from the Pointing Game [32], which considers only the single strongest attention point, and the Energy-based Pointing Game [52], which does not account for attention strength, providing a more complete picture of how attention strength is spatially distributed across object and background regions.
Figure 4: (a) Ratio of attention on the object of interest (AoO) for threshold levels of the normalized peak attention in the range \([0,99]\) on ImageNet-S [39]. The ratio was calculated at each threshold for all ±12.5k validation images with available segmentation maps. (b) AoO precision-recall curve computed for the same validation images, with higher thresholds retaining only the most confidently attended tokens. (c) Raw attention score distributions of the last layer. The plain ViT’s wider distribution causes peaked, winner-takes-all attention, while ISA’s narrower distribution yields more balanced, object-focused attention..
An example attention map and the corresponding segmentation mask are illustrated in the left part of 3. The right part of 3 shows that at higher thresholds, ISA-ViT-S/16 retains significantly more attention on the object compared to the baseline without inhibition. We computed the AoO ratio for all validation images at each threshold in the range \([0,99]\) according to 1 and plot the average ratio at each threshold in 4 (a). The figure illustrates that our inhibited self-attention leads to a higher Attention-on-Object ratio across all threshold levels. Notably, at higher threshold levels, the gap in the ratio between ISA-ViT-S/16 and ViT-S/16 exceeds 25 percentage points, highlighting that our proposed mechanism directs significantly stronger attention to the objects of interest. DIFF-ViT-S/16 lies between the two, improving over the baseline but plateauing around 71% and trailing ISA-ViT-S/16 by roughly 5 percentage points at high thresholds, indicating less concentrated attention on objects.
We additionally analyze AoO and object coverage across the same 100 thresholds, and plot a precision-recall style curve in 4 (b). At threshold 0 all tokens receive attention and object coverage is maximal, corresponding to the rightmost point of the curve. As the threshold increases, only the most confidently attended tokens remain, moving the curve leftward toward lower coverage. ISA dominates across the full curve, maintaining high AoO even as coverage shrinks, meaning its highest-confidence tokens consistently fall on objects. ViT’s AoO peaks at intermediate coverage and degrades toward the high-confidence end, confirming that its most-attended tokens frequently fall outside object boundaries. DIFF-ViT improves over ViT but plateaus well below ISA across the full curve, with the gap widening toward the high-confidence end, showing that its output-level subtraction cannot match ISA’s selectivity precisely where selectivity matters most.
4.2.0.2 Attention scores per layer.
To identify the mechanistic source of ISA’s selectivity, we examine the raw attention score distribution of the last layer and visualize them in 4 (c). The plain ViT exhibits a wide score distribution, which causes softmax to produce highly peaked, winner-takes-all attention concentrated on only a few tokens. DIFF-ViT shows a similarly wide distribution, spreading attention broadly but not selectively. ISA acts as a regularizer that narrows this distribution, resulting in more balanced attention weights over a focused set of object-relevant tokens. This is further confirmed by an analysis of attention concentration metrics in Appendix A.2, showing that ISA avoids single-token fixation while concentrating its attention on object-relevant regions. Further visualizations and analysis at different layers are provided in Appendix A.1.
4.2.0.3 Attention Visualizations
In 5, we visualize the attention maps (computed according to [48], [49]) for some sample images from ImageNet-1k. ISA has a significant impact on the strength and focus of the attention on the objects of interest. Generally, the attention maps of ISA-ViT are more object focused compared to those of the baseline. After analyzing numerous maps, we found that the baseline ViT and DIFF-ViT frequently rely on shortcuts and spurious background features. This tendency is illustrated in 1 and in the example image of a cat in the fifth column of 5 where the baseline models make correct classifications without attending to the object. In many of these instances, ISA-ViTs did successfully attend to the objects of interest. Also for cases where the baseline ViT had reasonably good focus on the object (see examples in the first four columns of 5), our proposed method frequently reduces attention on background patches, with consequently higher attention values on the object. While ISA does not always achieve strictly higher object pixel coverage (example of the dog in the last column of 5), the baseline advantage in such cases often reflects diffuse spreading across object boundaries rather than meaningful focus on discriminative parts. Extra visualizations are provided in Appendix B.
4.3 Results: Image Classification↩︎
We evaluated ViT-S/16, DIFF-ViT-S/16, and ISA-ViT-S/16 on ImageNet-1k and assessed whether higher-quality attention maps lead to improved performance. Additionally, we tested shortcut reliance and model generalization to OOD samples by evaluating the models on ImageNet-W and ImageNet-R. Since ImageNet-R contains only 200 ImageNet classes, we fine-tuned our models on ImageNet-200 using linear probing, following the same training setup but with 10% of the training epochs. To highlight the stability of ISA in other architectures, we also tested DeiT-S and ISA-DeiT-S. We ran each experiment three times and report the average for all metrics.
4.3.0.1 Accuracy, Shortcuts and OOD Evaluation
In ¿tbl:tab:results?, we report the results on ImageNet-1K, ImageNet-W and ImageNet-R. All results are averaged over three independent runs. ISA-ViTs improve robustness and OOD performance while maintaining competitive accuracy on ImageNet-1K. For ImageNet-W, we report the IN-W Gap and Carton Gap, which quantify the top-1 accuracy degradation and the class-wise accuracy improvement for the carton class, respectively, relative to ImageNet-1k. Lower IN-W Gap indicates that the model is more robust to the presence of the watermark across all classes, while a smaller Carton Gap suggests decreased reliance on the watermark shortcut for the carton class. For all models with ISA, we improve both ImageNet-W metrics, demonstrating reduced shortcut reliance. Similarly, on ImageNet-R, models incorporating ISA consistently achieve higher accuracy on OOD samples. Unlike DIFF-ViT, ISA introduces negligible overhead in model size and FLOPs compared to the ViT-S/16 baseline; detailed comparisons are given in Appendix D.
0.95@ l c c c c c & & & ImageNet-R
& Acc. (%) \(\uparrow\) & CE Loss \(\downarrow\) & IN-W Gap \(\uparrow\) & Carton Gap \(\downarrow\) &
Acc. (%) \(\uparrow\)
ViT-S/16 & 78.9 ± 0.1 & 0.890 ± 0.0 & -10.0 ± 0.8 & 28.7 ± 6.4 & 39.4 ± 0.1
DIFF-ViT-S/16 & 79.1 ± 0.0 & 0.868 ± 0.0 & -10.0 ± 1.5 & 31.3 ± 1.2 & 39.2 ± 0.1
ISA-ViT-S/16 & 79.0 ± 0.1 & 0.889 ± 0.0 & -9.3 ± 0.4 & 27.0 ± 3.1 & 40.4 ± 0.1
DeiT-S & 79.9 ± 0.0 & 0.956 ± 0.0 & -8.25 ± 1.0 & 28.7 ± 3.1 & 42.2 ± 0.3
ISA-DeiT-S & 80.1 ± 0.1 & 0.943 ± 0.0 & -7.84 ± 0.3 & 23.0 ± 3.1 & 42.9 ± 0.4
4.3.0.2 Reliance on Background
Given the increased object-focused attention in ISA-ViT, we investigate whether this corresponds to a reduced reliance on background features for classification of objects of interest. We use the Waterbirds dataset [42], which consists of photographs of various landbird and waterbird species. To introduce spurious correlations, the authors altered the backgrounds in 5% of the training images, placing waterbirds on land backgrounds and landbirds on water backgrounds. This results in four distinct groups: landbirds on land (0), landbirds on water (1), waterbirds on land (2), and waterbirds on water (3). At test time, landbirds and waterbirds are equally distributed across land and water backgrounds. Despite high overall accuracy, models may still underperform on specific subgroups due to reliance on spurious correlations [42]. We therefore use worst-group accuracy (i.e., waterbirds on land) as a key metric to assess susceptibility to confounding factors.
We fine-tuned ImageNet-1k pre-trained models for 30 epochs using linear probing. Each experiment was repeated three times, and we report the means in 1.
ISA-ViT yields higher class-wise accuracy on both under-represented groups during training with non-natural backgrounds (groups 1 and 2), indicating reduced reliance on spurious background features. Our method improves the worst-group accuracy by more than 2 percentage points on average with respect to ViT-S/16. The reported average accuracy is the weighted accuracy on the test set with the weights corresponding to the proportion of each group in the skewed training dataset, for which our model also outperforms the baseline.
| Model | Acc. (%) | Group Acc. (%) | |||
|---|---|---|---|---|---|
| 3-6 | 0 | 1 | 2 | 3 | |
| 1-6 ViT-S/16 | 89.5 | 94.2 | 87.0 | 81.6 | 89.6 |
| DIFF-ViT-S/16 | 89.3 | 93.3 | 87.2 | 82.9 | 89.5 |
| ISA-ViT-S/16 | 90.5 | 94.6 | 88.8 | 83.8 | 89.2 |
| 2-6 DeiT-S | 91.0 | 95.9 | 88.6 | 81.5 | 91.8 |
| ISA-DeiT-S | 90.9 | 96.0 | 88.0 | 83.5 | 90.9 |
4.3.0.3 Robustness to Corruptions
We further assessed the robustness of Inhibited Self-Attention to image corruptions using the ImageNet-C and ImageNet-\(\bar{\text{C}}\) benchmarks. We focus on the relative mean Corruption Error (rmCE), which normalizes corruption performance by each model’s own clean error rate, thereby isolating robustness gains from differences in clean accuracy. ISA achieves the best rmCE across all model families and both benchmarks. Detailed results are provided in Appendix C.
4.4 Results: Object Detection↩︎
We demonstrate the benefits of ISA beyond image classification on the COCO object detection benchmark [45]. Using ViT-DINO and DIFF-DINO as baselines, we compare their performance to ISA-DINO.
4.4.0.1 Attention-on-Detected-Objects (AoDO)
Unlike the ViT backbone, which employs dense attention, the official DINO implementation adopts Deformable Attention [53] for improved efficiency, faster convergence, and more accurate localization [20], [54]. To analyze how inhibition affects the spatial focus of the decoder, we quantify the Attention-on-Detected-Objects (AoDO) for all 36,335 COCO objects. For each object, we consider the best predicted bounding box and its corresponding sampling locations, and compute AoDO as: \[AoDO = \frac{\sum_{i \in \mathcal{F}} w_i}{\sum_{j=1}^{N} w_j}, \quad \text{where} \quad \mathcal{F} = \{i \mid \mathbf{p}_i \in \mathcal{B}_{\text{GT}}\}\]
where \(\mathbf{p}_i = (x_i, y_i)\) denotes the \(i\)-th sampling point, \(w_i\) its attention weight, and \(N=H \times L \times K\) the number of sampling points, with \(H\) attention heads, \(L\) feature pyramid levels, and \(K\) points per head-level. \(\mathcal{B}_{\text{GT}}\) is the ground-truth bounding box. AoDO extends the Energy Based Pointing game [52], specifically for deformable attention. Given the availability of COCO segmentation maps, we additionally compute AoO using the corresponding masks, which more accurately capture attention focus on the actual object than the bounding boxes used in the Energy-Based Pointing Game and AoDO.
For objects detected by both the baseline and inhibited models (\(\sim\)87% of all objects), we compare their AoO and AoDO values in ¿tbl:tab:aoo?. ISA-DINO learns to sample more precisely within both the ground truth bounding box of the object of interest (AoDO), and the actual segmentation map (AoO). The Differential Attention mechanism in DIFF-DINO hurts both mAP and attention focus compared to the plain ViT backbone, which may be caused by its global subtraction distorting spatial relationships and undermining object localization. This highlights that strong ImageNet performance, potentially achieved via shortcuts such as texture or background bias, does not guarantee success on downstream tasks like object detection, which demands precise spatial localization and is inherently more resistant to such shortcuts.
0.95@ l c c c c c c & &
& mAP (%) & AoO (%) & AoDO (%) & mAP (%) & AoO (%) & AoDO (%)
DINO-S/16 & 38.2 & 42.5 & 62.3 & 37.5 & 42.7 & 62.4
DIFF-DINO-S/16 & 37.7 & 39.4 & 57.9 & 37.7 & 39.4 & 58.2
ISA-DINO-S/16 & 38.6 & 44.6 & 63.8 & 38.6 & 44.6 & 63.9
4.4.0.2 Background reliance on COCO-Background
To assess the reliance of object detectors on background information, we introduce COCO-Background3 (COCO-B), a variant of the COCO dataset where objects are placed against non-natural backgrounds sampled from the Places365 dataset. Conceptually similar to the Waterbirds dataset [42], COCO-B preserves object locations while replacing the original scene context, as explained below. This enables a direct evaluation of background and contextual dependence in object detection.
We selected five background classes, namely Volcano, Canyon, Glacier, Snowy Mountains, and Desert Sand, which are unlikely to produce false positives for the majority of COCO objects. Background images containing humans were manually removed to avoid introducing spurious detections. The dataset was generated by taking all non-empty validation images in COCO and compositing the objects onto randomly sampled backgrounds from each class, while keeping the use of each background class balanced. Backgrounds were resized and center-cropped to match the original image dimensions, ensuring full coverage without distortion or empty regions. Example images from COCO-B are shown in 6.
In ¿tbl:tab:aoo?, we report results on background reliance. ISA-DINO maintains mAP under background alterations, whereas the model without inhibition experiences a substantial drop. DIFF-DINO shows no degradation but underperforms throughout, suggesting a lower performance ceiling rather than robustness. This indicates that the proposed inhibition effectively increases object-centric focus and reduces background reliance in object detection.
4.5 Discussion↩︎
With our proposed inhibited self-attention we address a key architectural limitation of standard attention mechanisms, namely the insufficient ability to suppress irrelevant features that usually lead to shortcut learning and biased predictions. We demonstrated that the focus of ViT attention can be sharpened by including inhibition. The improved selectivity of ViT that we achieved, driven by the effective suppression of irrelevant cues in the computation of the self-attention of tokens, is a key factor for developing more reliable models. It indeed contributes to reducing the reliance on spurious shortcuts and background correlations, which prompt biased model predictions. By promoting selective attention to truly informative regions, inhibition enhances the model ability to make causally grounded and generalizable decisions rather than depending on coincidental contextual cues, as demonstrated by the consistently improved performance on ImageNet-R. Moreover, the resulting relevance maps exhibit clearer focus patterns, pointing towards more interpretable inference and allowing for a more transparent understanding of what drives the model predictions.
Our experiments focused on Vision Transformer models, while future work could extend this analysis to larger models to further investigate the effects of inhibition in higher-capacity architectures.
5 Conclusion↩︎
We introduced Inhibited Self-Attention, a novel drop-in attention mechanism that enhances selectivity in Vision Transformers. We redesigned the self-attention mechanism by integrating an inhibitory process that sharpens the focus of self-attention to objects of interest, thus increasing the reliability of vision models. By leveraging negative attention scores, typically discarded by the softmax function, our approach applies contextual inhibition to suppress irrelevant features, thus sharpening attention on objects of interest.
To quantify this improved object-centered focus, we introduced a new metric called Attention-on-Objects (AoO), which measures how effectively models concentrate attention on target regions. ViTs with Inhibited Self-Attention produce attention maps with more precise and concentrated focus on class-relevant regions while reducing the impact of spurious cues. Our models assign significantly more attention to foreground objects than conventional ViTs, for both classification and detection tasks. The improved selectivity enhances decision-making by reducing reliance on shortcuts and spurious correlations. Furthermore, the improved focus on object-relevant regions translates to better generalization to out-of-distribution data, as reflected by higher robustness on ImageNet-R. Extending beyond classification benchmarks such as ImageNet and Waterbirds, ISA models also achieved improved results on COCO and COCO-B, showing more accurate object detection, better focus on target objects, and reduced reliance on backgrounds.