1 Introduction↩︎

Recent studies indicate that deep neural networks and the human brain interpret the environment differently, with the human visual system (HVS) dynamically filtering task-irrelevant information to focus on potential objects of interest, a selective mechanism that contributes to perceptual stability and resilience against noisy or misleading inputs [1]–[4]. The retina contains photoreceptors, with the fovea–a high spatial resolution area–playing a key role in color perception and visual detail recognition [5]. The highest photoreceptor density at the fovea decreases with eccentricity, resulting in a variable-resolution image transmitted to the brain, a phenomenon known as foveation, highlighting HVS’s multi-resolution perception. Such mechanisms suggest that vision models could benefit from integrating spatially adaptive processing strategies inspired by the HVS. In addition to efficiency and robustness, such biologically inspired mechanisms enhance spatial awareness, allowing models to preserve fine details in salient regions while maintaining global scene coherence, similar to how the human visual system balances local and global perception. Studying HVS to enhance deep neural network design is therefore a promising research avenue for developing intelligent agents.

Approximately \(10^7\) to \(10^8\) bits of information enter the visual nerve every second in the HVS [6]. To manage this data efficiently, the HVS uses saccadic eye movements to direct the fovea to selected targets, creating a detailed scene map from varied resolutions, known as fixation points, and saving computational resources [6], [7]. Inspired by the HVS, several studies have incorporated foveation and fixation mechanisms into neural networks [8]–[10]. Existing methods sequentially process the input image, first locating the fixation points and then processing the features around the fixation points. However, these approaches are not optimal for two reasons. First, several inferences are required for each fixation point. Second, they require a fusion mechanism to exploit the collective information acquired from various fixation points.

We therefore propose a biologically inspired transformer architecture dubbed the Foveated Dynamic Transformer (FDT), comprising foveation and fixation modules that dynamically select multiscale tokens based on the input image. To simulate foveation in HVS, we process input tokens with the foveation module to generate multiscale queries, keys, and values. Inspired by the radial-polar pooling model of foveation proposed in [12], foveation module transforms the token into several scales with increasing window size. We employ a dynamic, input-dependent mechanism to simulate eye movement, with the dynamic fixation module producing a fixation map for each token in each transformer block. Tokens that are not at the fixation point are not processed. The multi-head attention processes only the remaining tokens. The processed tokens that are located at the fixation points are merged with the non-fixated tokens and are sent to the next block. Multiple blocks in transformer process information from multiple fixation points by combining the information transferred from the previous blocks, which enables to implement foveation and fixation mechanisms in single pass. While inspired by the biological process, our implementation diverges from the iterative fixation control of the HVS, and also from methods like FoveaTer [13] that require multiple sequential attention steps to simulate gaze shifts. Instead, we embed both foveation and fixation directly into each transformer block, enabling spatial attention shifts in a single feedforward pass. This approach avoids iterative computation, unlike FoveaTer, and does not rely on token pruning heuristics like DynamicViT [14], allowing for end-to-end differentiable and biologically grounded token selection.

Using a DeiT architecture as a baseline, we evaluate our model on an image classification task. We integrate the fixation and foveation modules into the same architecture and evaluate the effectiveness, efficiency, and robustness of our model on the ImageNet100 database. The FDT architecture enhances the robustness of the vision transformers against adversarial attacks, shortcut learning, and natural corruption by 27%, 6%, and 3%, respectively, without being directly trained for these specific challenges. Moreover, FDT achieves a 34% reduction in computational demand, measured in Multiply-Accumulate operations (MACs), demonstrating its efficiency and effectiveness in processing while maintaining a lean computational footprint.

2 Related Work↩︎

Several approaches inspired by the functioning of the human visual system have been devised to replicate foveation and eye movement within the domain of computer vision. [8] introduced a method that employs a sequence of image movements to aggregate information before classification. This method uses a hard-attention mechanism trained via reinforcement learning to predict fixation points. In a similar vein, [9] developed a foveated object detection system that harnesses varying resolutions to comprehensively analyze the entire image, aligning its fovea with regions of interest in the input data. This approach amalgamates data from multiple fixations and leverages peripheral information, similar to the way the human visual system employs contextual cues to guide gaze.

[15] developed an efficient model that incorporates space-variant sampling, mimicking the human retina, and the mechanisms to generate sequences of fixations. They proposed a CNN-based method that uses Foveal Cartesian Geometry (FCG) sampling, as outlined by [16], to compress visual signals. An attention mechanism is employed for “eye movements" to progressively gather detailed information from a scene. Activation within the feature maps of the final convolutional layer is harnessed to guide the generation of fixation sequences. Additionally, [17] investigate the role of foveation and saccadic eye movements as biologically inspired proxies for data augmentation in the context of self-supervised learning (SSL). They suggest that foveation through cortical magnification and saccade-like sampling of images can replace conventional SSL augmentations, offering insights into potential biological implementations of self-supervision and challenging spatially uniform processing assumptions in both human and machine vision.

More recently, [13] introduced foveation and fixation mechanisms directly into the vision transformer architecture. In this method, the next fixation location is iteratively determined based on the final layer’s attention weights. While this approach captured dynamic attention patterns akin to human visual processing, its iterative nature results in inefficiencies during inference. Similarly, [18] introduced a model that incorporates human gaze sequence using a dual-sequence encoder alongside a vision transformer to correct shortcut biases in visual classification. However, their method required separately modeling gaze data and training an additional gaze encoder module. In contrast, our approach integrates foveation and fixation mechanisms directly into the vision transformer without the need for a separate gaze sequence encoder or auxiliary training components.

Building upon prior biologically inspired studies of foveation and fixation in machine vision [8]–[10], [19], our work extends this line of inquiry to the transformer framework. A detailed overview of these earlier models is provided in Appendix B. In parallel, recent research has focused on improving the computational efficiency of vision transformers through token pruning and slimming strategies [14], [20]–[27]; we summarize these efficiency-oriented approaches in Appendix B.

Our approach integrates foveation and fixation directly within each transformer block, enabling multiple spatial attention shifts in a single feed-forward pass without the iterative computation required by methods such as FoveaTer. Unlike approaches that assign continuous importance scores for gradual pruning, our fixation mechanism produces discrete, binary decisions informed by multi-resolution foveated features. This design reflects the all-or-nothing nature of biological fixation, simplifies the fusion of processed and unprocessed tokens, and yields a biologically grounded, computationally efficient alternative to existing token-slimming strategies.

3 Foveated Dynamic Transformer (FDT)↩︎

3.1 Foveation Module↩︎

We introduce FDT, an HVS-inspired transformer with two additional modules: Foveation and Fixation. The Foveation module embeds input features at multiple resolutions, drawing motivation from multi-resolution processing in the human visual system. The Fixation module selectively processes tokens through gaze-inspired sampling, producing binary decisions for each foveated query. The MHSA module then processes a subset of the foveated queries, keys, and values based on the Fixation module’s decisions, while the remaining tokens are passed to the next block. The FDT architecture is shown in 2.

The Foveation module in the FDT architecture is designed as a computational analogue of foveated processing by incorporating multi-scale information from neighboring features to generate query, key, and value features (3). This mechanism enables each token to aggregate information from progressively larger receptive fields. The Foveation module consists of multiple depthwise separable convolutional layers, each of which progressively extracts information from an expanding receptive field. In order to process auxiliary tokens, such as classification tokens, in convolutional layers, we reorganize input tokens \(t\) into patch tokens in image form (\(t_p\)) and auxiliary tokens (\(t_a\)) using the function \(\mathcal{K}\), as follows: \[t_{a}, t_{p}=\mathcal{K}(t) \quad \text{where} \quad \mathcal{K}:\mathbb{R}^{C\times N} \mapsto \mathbb{R}^{C\times A}, \mathbb{R}^{C\times H\times W}.\] where \(C\) represents the embedding size, \(N\) is the number of tokens, \(A\) is the number of auxiliary tokens, and \(H\) and \(W\) are the height and width of the patch-token grid.

The Foveation module leverages multi-resolution information by applying successive depthwise separable convolutions (\(\mathcal{DSC}\)) to patch tokens \(T_{p}\) and pointwise convolutions (\(\mathcal{PC}\)) to auxiliary tokens \(T_{a}\), as the latter are two-dimensional features. This is mathematically represented as follows: \[\begin{align} t_p^l = \mathcal{DSC}^{l-1}(t_p^{l-1}) ~~~~~~~~ \text{and} ~~~~~~~~ t_a^l = \mathcal{PC}^{l-1}(t_a^{l-1}). \end{align}\] Here, \(l\) refers to the layer and \(\mathcal{DSC}\) divides a kernel into two independent kernels that perform depthwise and pointwise convolutions, \(\mathcal{DSC}(x) = \mathcal{PC}(\mathcal{DC}(x))\). As a result, each successive layer of the Foveation module employs progressively larger receptive fields. In all experiments, we use three foveation levels (\(L_{\mathrm{fov}}=3\)): the input features and two successive convolutional outputs. This fixed scale count matches the three-way channel split below, so each level contributes \(C/3\) channels and the concatenated query, key, and value streams each recover width \(C\).

To embed multi-scale information into the query, key, and value features, we split the auxiliary and patch-related features of each layer into three equal-sized splits in the channel dimension using function \(\mathcal{S}\): \[\begin{align} t_{pq}, t_{pk}, t_{pv} &= \mathcal{S}(t_{p}), \\ \mathcal{S}: \mathbb{R}^{C \times H \times W} \;&\mapsto\; \Big(\mathbb{R}^{C/3 \times H \times W}, \mathbb{R}^{C/3 \times H \times W}, \mathbb{R}^{C/3 \times H \times W}\Big), \\[2mm] t_{aq}, t_{ak}, t_{av} &= \mathcal{S}(t_{a}), \\ \mathcal{S}: \mathbb{R}^{C \times A} \;&\mapsto\; \Big(\mathbb{R}^{C/3 \times A}, \mathbb{R}^{C/3 \times A}, \mathbb{R}^{C/3 \times A}\Big). \end{align}\]

Finally, we merge and concatenate the auxiliary and patch splits from the three foveation levels in the channel dimension to form foveated-query (\(q_{\mathrm{fov}}\)), -key (\(k_{\mathrm{fov}}\)), and -value (\(v_{\mathrm{fov}}\)) features using the inverse of the operation that was applied for initial rearrangement:

\[\begin{align} q_{\mathrm{fov}} &= [ \mathcal{K}^{-1}(t_{aq}^{0}, t_{pq}^{0}) \mid \mathcal{K}^{-1}(t_{aq}^{1}, t_{pq}^{1}) \mid \mathcal{K}^{-1}(t_{aq}^{2}, t_{pq}^{2}) ],\\[1mm] k_{\mathrm{fov}} &= [ \mathcal{K}^{-1}(t_{ak}^{0}, t_{pk}^{0}) \mid \mathcal{K}^{-1}(t_{ak}^{1}, t_{pk}^{1}) \mid \mathcal{K}^{-1}(t_{ak}^{2}, t_{pk}^{2}) ],\\[1mm] v_{\mathrm{fov}} &= [ \mathcal{K}^{-1}(t_{av}^{0}, t_{pv}^{0}) \mid \mathcal{K}^{-1}(t_{av}^{1}, t_{pv}^{1}) \mid \mathcal{K}^{-1}(t_{av}^{2}, t_{pv}^{2}) ]. \end{align}\]

3.2 Fixation Module↩︎

Humans execute a sequence of eye movements to construct a detailed scene map, selecting fixation points based on multi-resolution information from foveated perception. Inspired by this principle, we introduce a fixation module that leverages multi-resolution features to determine which input tokens should be processed.

Our fixation module uses the multi-resolution information in the foveated-query token \(q_{\mathrm{fov}}\) to identify fixation points. This token’s extensive receptive field allows fixation decisions to be based on a single token. We implement the module as a single linear layer that maps each foveated query to two logits for binary fixation decisions.

\[\begin{align} \ell_{\mathrm{fix}}^{i,j} &= \operatorname{Linear}(q_{\mathrm{fov}}^{i,j}), \\ &\quad \forall i \in \{0, \dots, H-1\},\; \forall j \in \{0, \dots, W-1\}. \end{align}\]

To generate a fixation map, we feed all the foveated query features to the fixation module (FIX). The fixation map is generated based on the logits produced by the module. The position where the first logit value is higher than the second logit value is set to 1.

\[\label{eq:gate} g(i,j)=\begin{cases} 1, & \ell_{\mathrm{fix}}^{i,j}(0)>\ell_{\mathrm{fix}}^{i,j}(1),\\ 0, & otherwise. \end{cases}\tag{1}\]

The fixation map is used to filter the foveated features that are processed in the Multi-Head Self-Attention (MHSA) module. See 4.

3.3 Overall Architecture↩︎

The HVS uses foveation and fixation mechanisms to efficiently process information. Inspired by this, we developed the Foveated Dynamic Transformer (FDT), a variant of the standard vision transformer that integrates computational foveation and fixation modules into its architecture. FDT retains the use of MHSA and MLP modules but processes only fixated tokens. Data flow in FDT, denoted by layer norm (LN), is structured as:

\[\begin{align} q_{\mathrm{fov}}, k_{\mathrm{fov}}, v_{\mathrm{fov}} = \operatorname{FOV}(\operatorname{LN}(x)), ~~~~~ g = \operatorname{FIX}(q_{\mathrm{fov}}), \\ q_{\mathrm{fix}}, k_{\mathrm{fix}}, v_{\mathrm{fix}}, x_{\mathrm{fix}} = \left\{ q_{\mathrm{fov}}^{i,j}, k_{\mathrm{fov}}^{i,j}, v_{\mathrm{fov}}^{i,j}, x^{i,j}\mid g(i,j) = 1 \right\}. \end{align}\]

In this architecture, the MHSA module forms a global relationship among fixated tokens and produces an attention matrix for each input token, focusing on specific fixated values. This module adapts to varying input sizes by processing solely the fixated tokens (where \(d_{h}\) denotes the number of heads and \(\sigma\) represents the softmax function):

\[\begin{align} \mathrm{MHSA}(q_{\mathrm{fix}},k_{\mathrm{fix}},v_{\mathrm{fix}}) &=\sigma(q_{\mathrm{fix}}k_{\mathrm{fix}}^{\top} / \sqrt{d_{h}}) \cdot v_{\mathrm{fix}} \end{align}\]

FDT takes a full-size token map and produces foveated tokens, but since only fixated tokens are processed, their output shape (\(x_{\mathrm{fix}}\)) does not align with the expected input size for subsequent blocks. To resolve this, we blend processed and unprocessed tokens, ensuring proper input for the next stages.

\[out^{i,j}=\begin{cases} x_{\mathrm{fix}}^{i,j}, & \text{if} \quad g(i,j) = 1,\\ x^{i,j}, & \text{otherwise}. \end{cases}\]

3.4 Training Dynamic Network↩︎

In the FDT architecture, the fixation module acts as a gating network that applies a selection operation to the foveated tokens of the input image. Variations in input images result in differing numbers of selected tokens, which complicates training with mini-batches. To facilitate mini-batch training, all foveated tokens are fed into the MHSA module without applying fixation sampling. To enable end-to-end training with a differentiable fixation map, we apply Gumbel-Softmax with hard labeling to the output of the foveation module during training, replacing the logit comparison described in 1 . Taking \(g\) as the output of Gumbel-softmax;

\[\begin{align} x_{\mathrm{masked}} & =\operatorname{MHSA}_{\mathrm{masked}}(q_{\mathrm{fov}},k_{\mathrm{fov}},v_{\mathrm{fov}},g)+x,\\ x_{\mathrm{masked}} & =\operatorname{MLP}(\operatorname{LN}(x_{\mathrm{masked}}))+x_{\mathrm{masked}}. \end{align}\]

The fixation map is then utilized for the calculation of masked attention (where \(\mathcal{C}\) is a large constant):

\[\begin{align} \mathrm{MHSA}_{\mathrm{masked}}(q_{\mathrm{fov}},&k_{\mathrm{fov}},v_{\mathrm{fov}},g) \\ &= \sigma \Big( q_{\mathrm{fov}} k_{\mathrm{fov}}^{\top} / \sqrt{d_h} \;+\; \mathbb{M} \Big) \cdot v_{\mathrm{fov}}, \\[1mm] \mathbb{M} = -\mathcal{C} \;\times\; &\Big( 1 - flatten(g) \otimes flatten(g) \Big). \end{align}\]

3.4.0.1 Budget Constraint.

In order to achieve high performance, the fixation module tends to assign all tokens as fixation points when there are no budget constraints. However, the ideal behavior of the fixation module should be to focus on the minimum number of tokens necessary for accurate prediction. To achieve this, we introduce a fixation budget constraint that forces the network to allocate a certain percentage of all tokens. The fixation budget loss is defined as the blockwise \(\ell_2\) norm between the desired percentile and the mean of the fixation map to measure the deviation from the desired budget (\(g\) is the fixation map, \(\beta \in (0, 1]\) denotes the desired fixation budget, i.e., the target fraction of tokens to be selected as fixation points in each block, where \(L\) is the number of blocks):

\[\mathcal{L}_{\mathrm{Budget}}=\frac{1}{L}\sum_{\ell=0}^{L-1}\left\Vert \mathbb{E}(g)-\beta\right\Vert ^{2}.\]

Therefore, the total loss for training FDT on a classification task is the sum of cross-entropy loss as the task loss and the fixation budget loss (\(\lambda\) is the balancing factor):

\[\mathcal{L}=\mathcal{L}_{\mathrm{CE}} + \lambda \cdot \mathcal{L}_{\mathrm{Budget}}.\]

4 Empirical Analyses↩︎

We demonstrate that FDT outperforms DeiT in robustness against adversarial attacks, natural corruption, and shortcut learning. We investigate the impact of budget size on both robustness and computational efficiency and provide attention and gating visualizations to support our findings. For experimental details, see Appendix D.

4.1 Adversarial Robustness↩︎

Although adversarial attacks target neural networks specifically, the human visual system is naturally robust to noise, clutter, and distribution shift. Mechanisms like foveation and fixation help humans emphasize meaningful features while discounting irrelevant perturbations. Motivated by this, we evaluated FDT using 12 adversarial attacks and one Gaussian-noise baseline. The results, shown in 5, indicate that FDT outperformed the DeiT method in all types of attacks (for numerical results, see [tab:adversarial95attack95all95models]). We further provide a qualitative visualization of adversarial examples in 10, where FDT perturbations appear more concentrated on semantically relevant image regions than DeiT perturbations.

4.2 Natural Corruption Robustness↩︎

1.7pt

We evaluated models using the ImageNet100-C [28] dataset, which includes common corruptions. ImageNet100-C denotes ImageNet-C restricted to the same 100-class ImageNet100 subset used for clean training and validation. 1 shows that FDT consistently outperforms DeiT across all severity levels, particularly in weather-related corruptions. This robustness is consistent with the HVS-inspired motivation for adaptive, selective processing.

4.3 Effect of Budget↩︎

Our model incorporates the MHSA module, which exhibits quadratic growth in computational complexity as the number of tokens increases. The fixation module within each FDT block dynamically selects a subset of tokens for processing in the MHSA based on the input image’s complexity, significantly reducing the overall computational cost.

To evaluate our model’s efficiency, we quantified the Multiply-Accumulate operations (MACs) required for inference for both DeiT and FDT under various gating budgets. We calculated FDT’s computational complexity by averaging the computations needed per sample in the validation set, reflecting the dynamic nature of our approach. MACs are normalized to DeiT to highlight computational efficiency gains. 2 shows that FDT requires fewer expected MACs for inference than DeiT. At a 50% budget, FDT uses an average of 3.01 GMACs, achieving a 34.57% reduction compared to DeiT. Even at a full budget, where all tokens are utilized, the increase in complexity is only 0.7%, demonstrating the minimal overhead of the foveation and fixation modules. These results indicate that FDT can dynamically allocate computational resources to more informative regions of the image, following an HVS-inspired selective-processing principle. At the 50% budget, this allocation corresponds to higher clean accuracy and a 34.57% MAC reduction relative to DeiT-S. The robustness and shortcut-learning results are consistent with the hypothesis that targeted processing can reduce reliance on irrelevant or misleading features. These findings suggest that FDT can trade off clean accuracy, robustness metrics, and expected MACs across budgets, making it a candidate for future study in resource-constrained vision systems. Furthermore, 2 also serves as an ablation study: FDT\(_{1.0}\)(full budget) isolates the effect of the foveation module, while models with lower budgets highlight the contribution of the fixation mechanism to performance and robustness.

3.0pt

4.4 Robustness against Shortcut Learning↩︎

Shortcut learning occurs when ANNs form decision rules that excel on specific datasets by exploiting spurious correlations or statistical irregularities instead of learning the underlying task. These strategies, though effective on familiar data, do not generalize well across different data distributions [29]. To assess models, we trained both DeiT and the FDT under varying budget constraints using the Tinted-ImageNet100 dataset, where each training sample is modified with a class-specific tint, following the approach in the Tinted-STL10 dataset. Shortcut-learning performance is evaluated on the untinted standard ImageNet100 validation set, so the reported accuracy measures whether models generalize without the tint cue. 6 suggests that FDT is more robust to this synthetic shortcut than DeiT, especially under larger budgets.

4.5 Feature Inversion↩︎

We employ “feature inversion" to enhance our understanding and visualization of transformer-based representations. This technique reconstructs an input image from specific model features or activations, offering insights into how the model processes inputs and makes predictions. While widely used with CNNs [30], [31], its application to transformers remains less explored. Using the ‘Deep Image Prior’ image parameterization of [32] and the feature-inversion setup of [33], we optimize a CNN-based generator \(F_\theta(\cdot)\) to transform random noise input \(z\) into an image. The goal is to match the features of the output image, particularly the \(\mathrm{CLS}\) features, with those of a target image \(I\), defined by:

\[\underset{\theta}{\arg \min }\left\|\phi\left(F_\theta(z)\right)-\phi(I)\right\|_{\mathrm{F}} \label{eq:inverted95features}\tag{2}\]

where \(\phi(I)\) denotes the target features and \(\|\cdot\|_{\mathrm{F}}\) represents the Frobenius norm, focusing on the classification token (\(\mathrm{CLS}\)); thus, \(\phi(I)=\mathrm{CLS}(I)\).

We employ the same network architecture and parameters as [33] for the generative network \(F_\theta(z)\). The qualitative examples in 7 show clearer object structure in FDT reconstructions than in DeiT reconstructions, consistent with the hypothesis [33] that robustness-oriented models can yield more interpretable representations.

4.6 Effect of Model Size↩︎

The relationship between model size and performance is pivotal in neural network design, balancing capacity with computational efficiency. Generally, larger models can achieve higher accuracy but may overfit and require more resources. Smaller models might better generalize and suit practical needs but have limited learning capacity for certain functions. Understanding this trade-off is crucial for choosing the appropriate model size for specific tasks.

We explore the impact of model size on our FDT by evaluating three sizes: tiny, small, and base, following DeiT conventions. Importantly, to ensure a fair and controlled comparison, we maintained the architectural backbone identical to DeiT, with the only differences being the addition of the foveation and fixation modules. These additions are lightweight: for example, in the small model, FDT has 21.85 million parameters compared to DeiT-S’s 21.70 million, with the foveation block replacing the 5.32 million parameter QKV linear layer with 5.47 million parameters (a marginal 0.15 million increase) and the fixation block adding just 9k parameters. This minimal parameter increase ensures that any performance gains can be attributed to the introduced modules rather than increased capacity.

We compared FDT’s performance against DeiT across the same evaluation metrics, except for the learning rate of the base model set to 2e-4. Our results, presented in 3, cover accuracy on clean, adversarially attacked, and naturally corrupted images, and the relative accuracy gains over DeiT. These results suggest that FDT remains competitive across model sizes in the evaluated clean, adversarial, and corruption settings.

4.0pt

4.7 Input-Dependent Compute Analysis↩︎

Reaction-time (RT) in HVS measures the duration to respond to a visual stimulus, reflecting neural and cognitive processes in perception, attention, and decision-making, and varies with task complexity, object number and similarity, and uncertainties like occlusions or viewpoint changes, generally increasing with higher difficulty. We use the reaction-time analogy as a qualitative way to describe input-dependent computational load in FDT, not as a measured comparison to human response times. Because FDT dynamically selects fixation points, different images require different numbers of processed tokens and therefore different GMACs. To illustrate this behavior, we divide validation samples into balanced easy, medium, and hard groups according to their average number of fixations. 4 reports accuracy across these groups, and 8 visualizes representative samples and their compute requirements. Ultimately, these results demonstrate that FDT naturally scales its computational effort in response to image complexity, mirroring the reaction-time dynamics of the human visual system.

4.8 Model Decisions: Visualization and Insights↩︎

We visualize model fixation maps and attention maps as HVS-inspired diagnostics of selective processing. The FDT fixation module generates maps that highlight fixation probabilities across image tokens. To illustrate the model’s blockwise fixation selections, we identify the most likely token positions as fixation points. 9 shows these points across transformer block depth: arrows and green-to-yellow color transitions indicate block order, and circles mark fixation points with a fixed display size chosen for legibility rather than to encode fixation strength or probability. Focus intensity adapts to scenes with multiple objects through diverse sampling. Tokens are selected via Gumbel-Softmax and hard label techniques, producing binary maps. Averaging maps across blocks yields overall fixation maps (9), showing how the model prioritizes informative content while minimizing irrelevant background. Additionally, attention rollout [34] visualizes focus on the classification token, revealing that while fixation maps cover broad informative areas, attention maps concentrate on discriminative regions for class identification. These visualizations qualitatively support the intended selective-processing behavior.

5 Conclusion↩︎

We introduced the Foveated Dynamic Transformer (FDT), a novel architecture inspired by human visual system mechanisms, enhancing computational efficiency and robustness against adversarial attacks, natural corruption, and shortcut learning. Our results on the ImageNet100 dataset show that, at selected budgets, FDT can achieve higher clean accuracy, lower expected MACs, and stronger robustness metrics than the baseline architecture. This research contributes to biologically inspired computational models, integrating human visual principles into deep learning architectures. The FDT balances high performance with computational efficiency in the evaluated settings, highlighting its HVS-inspired capability to focus computation on informative image regions. Future work could evaluate FDT in domains such as video processing and augmented reality, where dynamic foveation may reduce computational demands while maintaining performance, and explore its deployment in real-world, resource-limited scenarios, underscoring the value of bioinspired approaches in developing efficient and robust AI systems.

6 Limitations↩︎

Although FDT achieves substantial MAC savings relative to the baseline, reduced MACs do not necessarily translate to lower wall-clock latency: the foveation pipeline requires token re-arrangement, channel splits and concat/merge operations to form multiscale Q, K, V tensors, which introduce memory-bound overheads that can dominate runtime. Our primary goal, however, is not to optimize latency, but to demonstrate how HVS-inspired design choices can improve robustness and compute efficiency. Inference latency ultimately depends more on platform and software stack than on MACs alone, and addressing these overheads through kernel-level optimizations or specialized implementations is left as future work to better align theoretical efficiency with measured latency.

FDT should also be interpreted as HVS-inspired rather than as a faithful biological model. In the current architecture, all spatial locations receive the same multi-scale feature construction, so there is no explicit peripheral degradation tied to retinal eccentricity. Multiple fixation regions are selected simultaneously within a single feedforward pass rather than through a temporal sequence of saccades, the model has no recurrence or mechanism for revisiting earlier locations, and the selected fixation set can change from block to block. These differences mean that foveation and fixation serve as design motivations for adaptive computation and token selection, not as claims of biological fidelity.

7 Broader Impact↩︎

The introduction of the Foveated Dynamic Transformer (FDT) represents a step toward more adaptive artificial vision systems, offering enhanced robustness and efficiency in the evaluated ImageNet100 settings. The FDT’s robustness to adversarial attacks and natural corruption without explicit adversarial training could motivate future studies in safety-sensitive domains such as autonomous driving, medical imaging, and surveillance, where accuracy and reliability are critical. Additionally, the model’s reduced expected MACs may support future deployment work in environments with limited computational resources. As an HVS-inspired design, FDT may foster interdisciplinary collaboration between AI researchers and neuroscientists while motivating future work on adaptive visual processing. Ultimately, the FDT’s approach could inspire new research directions, advancing both artificial intelligence and cognitive science.

8 Extended Related Work↩︎

Researchers have also examined foveation from complementary computational and perceptual perspectives. [19] investigated how foveated image transformations influence downstream visual representations in two-stage machine vision models, focusing on the role of texture-based peripheral coding. In a similar direction, [10] introduced a foveated object detector that applies image magnification to preserve high-resolution detail at points of interest while maintaining a compact canvas size.

A separate line of work has advanced efficiency in transformer-based vision models through token slimming, pruning, and related reduction strategies. Several methods explore how to discard uninformative visual tokens while retaining task-relevant structure ([20]; [21]; [14]; [22]; [23]). Building on this trend, [24] propose progressively removing less informative tokens during the forward pass to reduce computational overhead. [25] introduce Zero-TPrune, a zero-shot token pruning framework that leverages structural properties of the transformer attention graph to identify redundant tokens without additional training. [26] present a synergistic patch pruning approach that integrates both intra- and inter-layer importance signals to eliminate unnecessary patches across layers. Complementing these methods, [27] propose Focus-DETR, a DETR variant that employs dual attention and token scoring to emphasize informative regions and reduce inference cost while preserving detection accuracy.

Although these token-slimming approaches improve computational efficiency by selectively removing tokens or patches, they are driven primarily by engineering considerations and do not attempt to model visual attention from a biologically grounded perspective.

9 Adversarial Attack Qualitative Analysis↩︎

To complement the quantitative evaluation, a visual comparison is provided between adversarial examples generated for the FDT model and those for the DeiT baseline. This qualitative analysis offers a more intuitive understanding of the differences in robustness between the two models.

Figure 10 illustrates the perturbation patterns produced using the PGD-L2 attack with \(\epsilon=1\) over 100 optimization steps. This PGD-L2 visualization is separate from the \(\ell_\infty\) PGD evaluation row, which uses \(\epsilon=0.1\) as reported in [tab:attack95hparams]. The figure includes multiple visualization strategies to enhance interpretability. The second and third rows display the absolute values of the perturbations, scaled by a factor of 50 to improve visibility. The fourth and fifth rows present shifted versions of the perturbation maps, where pixel values are offset by 128, a standard approach for visualizing both positive and negative perturbation magnitudes. These shifted images are scaled by a factor of 20.

In this qualitative visualization, perturbations for FDT appear more concentrated in semantically relevant image regions, whereas those for DeiT appear more diffuse and less structured. This pattern is consistent with the quantitative robustness results, but it should be read as illustrative rather than as a separate robustness metric.

3.0pt

10 Experimental Settings↩︎

To evaluate FDT, we conduct image classification experiments on the ImageNet-100 dataset, a subset of the ImageNet-1k dataset containing 100 randomly chosen classes with 1300 training images and 50 validation images per class. We report top-1 accuracy using a single 224x224 crop and compare FDT to DeiT [35] using the same training settings, including AdamW optimization for 300 epochs with cosine learning rate decay and 20 epochs of linear warm-up, batch size of 512, and V100 GPUs for training and evaluation. The initial learning rate, weight decay, and momentum are set to 0.001, 0.05, and 0.9, respectively. Unless otherwise stated, all experiments use a budget of \(\beta=0.5\). All results are reported as the mean and one standard deviation of three differently initialized runs. For training Tiny, Small, and Base models, one, two, and four V100 GPUs with 32GB of memory are used, respectively. Evaluations of the models are done on a single V100 GPU.

10.0.0.1 Adversarial Robustness.

The robustness evaluation includes a Gaussian-noise (GN) baseline and 12 adversarial attacks: CW [4], PGD-L2 and PGD-\(\ell_\infty\) [36], FGSM [37], TPGD [38], FFGSM [39], EOTPGD [40], RFGSM [41], APGD and APGDT [42], BIM [43], and MIFGSM [44]. We used TorchAttacks v3.3.0 [45], the latest public release available on Nov. 17, 2022. Unless otherwise noted in [tab:attack95hparams], attack arguments follow the TorchAttacks v3.3.0 defaults. The row labeled PGD corresponds to the CSV column PGD0.1 and uses an \(\ell_\infty\) budget of \(\epsilon=0.1\); PGDL2 is a distinct \(\ell_2\) attack with \(\epsilon=1.0\). All adversarial examples are generated against the inference-time FDT forward pass: the fixation map is computed by the deterministic FIX logit comparison in 1 , selected tokens are processed and merged back into the token stream, and the Gumbel-Softmax masked-attention surrogate used during training is not used for attack evaluation. As demonstrated in Table [tab:adversarial95attack95all95models] in the Appendix, although a lower budget results in some accuracy drops, it overall helps to produce a more robust model. These findings support that the FDT method effectively enhances robustness against adversarial attacks by focusing on relevant features and filtering out noise.

Table

Adversarial-evaluation hyperparameters. All attacks were implemented with TorchAttacks v3.3.0. ‘–’ indicates that the field is not applicable.

Table

Comparison of robustness against 12 adversarial attacks and one Gaussian-noise baseline for FDT and DeiT models trained on ImageNet-100 dataset. The models are labeled with their corresponding fixation budget hyperparameter (subscript) and model size (T, S, B for tiny, small, and base, respectively). The mean and one standard deviation of three runs with different initializations are reported.

Table

Comparison of robustness against 12 adversarial attacks and one Gaussian-noise baseline for FDT and DeiT models trained on Tinted ImageNet-100 dataset. The models are labeled with their corresponding fixation budget hyperparameter (subscript) and model size (T, S, B for tiny, small, and base, respectively). The mean and one standard deviation of three runs with different initializations are reported.

References↩︎