1 Introduction↩︎

Recently, the impregnable position of convolutional neural networks (CNNs) in computer vision seems to be weakened by the emerging hierarchical Vision Transformer families [1]–[4]. As one representative, Swin Transformer [3] and its variants (e.g., [5], [6]) with an interleaved non-overlapped window-based self-attention & cross-window token mixing paradigm are able to achieve state-of-the-art performance in image recognition, and demonstrate excellent transferability on various downstream computer vision tasks such as object detection and scene parsing. Therefore it is meaningful to conduct an in-depth study on this new architecture family, and analyze what makes it so strong.

9.8 pt

Methodologically, Swin Transformer first partitions feature maps to a series of non-overlapped local windows, and uses multi-head self-attention (MHSA) layers to aggregate information in each window individually. Then, instead of using a dense sliding window paradigm for cross-window token mixing like CNNs [8]–[10], Swin Transformer proposes to shift windows between consecutive layers. By alternating these two operations, each token is able to interact with all other tokens, and the overall architecture can obtain very strong capacities.

Most successors of Swin Transformer mainly focus on replacing the shifted-window operation with other kinds of cross-window communication, such as the spatial token shuffle operation from [5], and the messenger tokens exchange proposed in [6]. While the use of window-based MHSA is usually taken for granted and treated as the de-facto standard for token fusion within each local window.

In this manuscript, we ponder the question: “What Makes for Hierarchical Vision Transformer?”. Specifically, we investigate whether MHSA is the only choice to aggregate information for the Swin model family, and the effects of different cross-window communication schemes such as spatial token shuffle and auxiliary messager tokens exchange. Previous practice from computer vision suggests that MHSA is good at capture dense and long-range contextual information [11], [12] in scene parsing tasks [13], [14]. Intuitively, it is somewhat too aggressive to use MHSA to model contextual relation inside a local window with only \(7 \times 7 = 49\) spatial tokens.

This motivates us to replace MHSA layers with linear mapping, one of the most common & simplest components in neural architecture design, in three representative hierarchical Vision Transformer instantiations, i.e., Swin Transformer [3], Shuffle Transformer [5], and MSG Transformer [6]. The resulting proof-of-concept model is termed as . We find that with embarrassingly simple linear mapping layers is sufficient for local content aggregation, and is able to achieve very competitive performance in ImageNet-\(1k\) image recognition benchmark [7]. Moreover, can better leverage the pre-trained representations from image recognition and demonstrate excellent transferability in downstream dense prediction tasks such as object detection & instance segmentation.

Furthermore, we experiment with other variants for information aggregation inside each local window, e.g., depth-wise linear mapping with separable weights (denoted as “DW Linear” in Tab. 1), and multi-layer perceptrons with intermediate activation functions (denoted as “MLP” in Tab. 1). We also conduct a study on different cross-window communication approaches such as spatial token shuffle (denoted as “Shuffle” in Tab. 1) and auxiliary messager tokens exchange (denoted as “MSG” in Tab. 1). As shown in Tab. 1, we find different content aggregation layers all give similar competitive results under the same cross-window communication approaches, and vice versa: different cross-window communication methods also achieve similar strong performance under the same content aggregation layer.

Based on the available evidence, we hypothesize that the macro architecture of Swin model families, i.e., interleaved non-overlapped window-based token mixing & cross-window communications, other than specific aggregation layers such as MHSA or specific means of cross-window communication such as shifted-window or spatial shuffle, may be more responsible for their strong performance, and is the real challenger to the ubiquitous CNN’s dense sliding window paradigm.

Please note that this manuscript is not an attempt to show that simple linear or MLP layers are superior to MHSA. On the contrary, we find MHSA is better than linear mapping & MLP in terms of accuracy with even fewer budgets (see Tab. 7). Our goal is to abstract away from specific aggregation layers as well as cross-window communication approaches, and highlight the importance and contribution of the macro architecture of the Swin Transformer family. We hope our work can encourage the community to rethink the role of attention in neural architecture design, and shed a little light on future studies of general visual representation learning.

2 Background and Related Work↩︎

Highly mature and robust training recipes [15] enable standard Transformer architecture [16], [17] directly inherited from natural language processing to attain excellent performance in the image recognition task even with limited data & model sizes [18], [19]. Standard Transformer models sequence-to-sequence relationship in a pair-wise manner with minimal prior via global scaled dot-product multi-head self-attention (MHSA), which scales quadratically with the sequence length. Therefore standard Transformer suffers from the scaling problem in spatial dimensions and fails to process high-resolution inputs with varying sizes in computer vision downstream tasks.

To more efficiently apply Vision Transformers to other downstream tasks in computer vision such as object detection, instance segmentation, and scene parsing, three key issues need to be solved: (1) involving hierarchical architectures to establish multi-scale feature representations for better handling of large variations in scales. (2) reducing memory & computation costs from global MHSA to efficiently process high-resolution inputs with varying token lengths, and (3) introducing appropriate inductive biases & prior knowledge of the target task for better performance.

To mitigate the aforementioned issues,  [2], [4], [20] process features with multi-resolution stages using spatial pooling operations instead of in a columnar manner.  [1], [3] further propose to compute MHSA in weakly-overlapped or non-overlapped local windows. After that, many follow-up hierarchical local window-based Vision Transformers emerge and challenge the hegemonic position of CNN in computer vision [5], [6], [21].

To demystify the relation between CNN and hierarchical Vision Transformer, [22] study the inhomogeneous depth-wise convolution under modern training & optimization recipe from [3], and demonstrates that CNN can achieve similar competitive performance compared with the Swin Transformer family in various vision tasks. Furthermore,  [23] and  [24] study the convolution-attention hybrid architecture. Combining the strengths from both camps, convolution-attention hybrid architecture can achieve state-of-the-art performance under different resource constraints across various datasets.

Previous studies show that simple multi-layer perceptrons (MLPs) architectures are competitive with CNNs in digit recognition [25], [26], keyword spotting [27] and handwritting recognition [28]. Recently, a series of works [29]–[31] revisit the architecture based exclusively on columnar structured MLPs in image recognition tasks under modern training and transfer learning recipes.

As cursorily summarized in Tab. 2, there are still two “missing pieces” remain, i.e., the CNN with columnar architectures, and the MLP with hierarchical architectures.  [22] touches the former topic with the columnar architecture proposed in [32]. This manuscript conducts a primitive study to the latter one: a straightforward, simple, yet must-know model in computer vision. We argue it is inevitable to investigate the potential of hierarchical linear mapping & MLP structures now, and we hope the proposed proof-of-concept model can encourage the community to rethink the role between macro model design methodologies and specific network building blocks.

4.2 pt

3 What Makes for Hierarchical Vision Transformer?↩︎

We first briefly review the Swin Transformer family in Sec. 3.1, and then introduce the proposed in Sec. 3.2.

3.1 The Swin Architecture Family↩︎

Methodologically, Swin Transformer [3] processes high-resolution input hierarchically using multi-head self-attention (MHSA) within non-overlapped local windows. Specifically, MHSA is used as the aggregation layer to fuse content information of spatial tokens inside each window. Since the non-overlapped partition scheme lacks connection across windows, Swin Transformer proposes to use shifted-window operations between every two successive window-based MHSA layers to encourage cross-window communications. Hierarchical architecture design is also adopted to produce multi-resolution representations for better handling of large scale & size variations in visual entities.

Most successors of Swin Transformer mainly focus on replacing shifted-window operations with other kinds of cross-window communications such as spatial token shuffle [5] or information exchange based on auxiliary messager tokens [6], while keeping other components unchanged.

Overall, Swin Transformer and its variants all adopt the MHSA as the aggregation layer for spatial token fusion, and use non-overlapped window-based token mixing & cross-window communications in an alternating fashion with hierarchical representations as the macro architecture. We refer readers to  [3], [5], [6] for more details of the Swin Transformer family architectures investigated in this manuscript.

3.2 A Proof-of-concept Model:↩︎

Despite being greatly successful in various tasks, we question whether MHSA is the only choice to aggregate information for the Swin Transformer family. To this end, we attempt to use linear mapping, one of the simplest components in neural architecture design, as a touchstone & probe to reveal that the macro architecture (interleaved non-overlapped window-based token mixing & cross-window communications) seems to be more responsible for Swin model families’ strong performance other than specific aggregation layers such as MHSA.

7.575 pt

7.8pt

We choose three representative and publicly available instantiations from the Swin model family, i.e., Swin Transformer [3], Shuffle Transformer [5], and MSG Transformer [6]. We directly replace their window-based MHSA layers with layers described in Algorithm 1, and align other components and configurations with Swin Transformer [3].

Specifically, layer takes partitioned image features with shape (\(\mathtt{B, C, ws^2}\)) as inputs, where \(\mathtt{B}\) is the total number of partitioned windows, \(\mathtt{C}\) is the number of channels and \(\mathtt{ws}\) is the window size. Since directly performing dense linear mapping from flattened inputs with shape (\(\mathtt{B, C \times ws^2}\)) to outputs with the same shape (\(\mathtt{B, C \times ws^2}\)) is computationally infeasible, we divide the input tensor to several groups where each group has \(\mathtt{ws^2}\) tokens with \(\mathtt{gs}\) channels (i.e., the group size is \(\mathtt{gs}\)). To further reduce FLOPs and parameters budgets, linear mappings are performed along tokens’ height and width dimensions separately, which is similar to the merit of [12] and [33]. Therefore, the number of inputs for linear mapping is only \(\mathtt{gs \times ws}\). Finally, the transformed representations along two axes are fused via element-wise addition followed by a point-wise linear projection layer.

In our default instantiation, the weights and biases of \(\mathtt{lin\_map\_h}\)() and \(\mathtt{lin\_map\_w}\)() in Alg. 1 are shared across different groups. Linear mappings with separate parameters for different groups are denoted as depth-wise linear mapping (“DW Linear” in Tab. 1 and Tab. 8) in this manuscript. Our controlled experiments demonstrate that using separate weight cannot bring further improvements in linear mapping given similar model sizes, which echos the observation in previous multi-head v.s. single-head self-attention studies [34], [35].

To some extent, is one of the simplest possible instantiations of architectures with interleaved non-overlapped window-based token mixing & cross-window communications scheme. Despite being simple, the layer is more lightweight than the MHSA layer. Therefore the lack in capacity compared with MHSA can be compensated by deeper or wider architecture design given similar FLOPs & parameters budgets, and the resulting architecture is still able to achieve competitive performance (Sec. 4.2).

It is noteworthy that the proposed layer enables a fully linear mapping & MLP architecture to directly process high-resolution input images with arbitrary shapes. This property allows MLP-like architectures to be easily transferred to different computer vision downstream tasks, which is lack in previous studies [29], [30], [36]. In Tab. 6 we demonstrate that the transfer learning performance of is on a par with Swin Transformer in object detection and instance segmentation even with relatively worse supervised pre-trained representations on ImageNet-\(1k\) [7].

Overall, the merit of is to abstract away from specific aggregation layers as well as cross-window communication schemes, and highlights the importance of the macro architecture, which seems overlooked in previous research on hierarchical Vision Transformer.

4 Experiments↩︎

We first give the general experimental setup in Sec. 4.1, and then report the pre-training, scaling, and transfer learning performance of in Sec. 4.2. The model analysis and ablation study are finally conducted in Sec. 4.3.

4.1 Setup↩︎

4.1.0.1 Pre-train Settings.

The experiments are conducted on the public available codebase of [3], [5], [6] and the \(\mathtt{timm}\) library [37].

During pre-training, all models are trained and evaluated on ImageNet-\(1k\) [7] benchmark following the setup in [3], [15]. We train models with \(300\) epochs on ImageNet-\(1k\) for main results. For model analysis and ablation study, we study different content aggregation layers with model width and model depth³ same as Swin Transformer-Tiny (i.e., model width \(= 96\), model depth \(= \{2, 2, 6, 2\}\)) using \(200\) epochs training schedule on ImageNet-\(1k\) unless specified.

The input resolution is \(224 \times 224\) and the window size is \(7 \times 7\) for all models in all experiments. For a clearer study of different aggregation layers in non-overlapped windows, we remove all densely slid conv-layers in the network stem and each block of [5] in this manuscript.

Model throughput data during inference are measured using a single Titan Xp GPU with batch size \(= 64\) with input resolution \(= 224 \times 224\). Model FLOPs during inference are measured with batch size \(= 1\) with input resolution \(= 224 \times 224\).

16.0 pt

4.2pt

4.1.0.2 Transfer Learning Settings.

The experiments are conducted on the public available codebase of [3] and the \(\mathtt{mmdetection}\) library [41].

We study the transfer learning performance of ImageNet-\(1k\) \(300\)-epoch supervised pre-trained models in the challenging MS-COCO [42] object detection and instance segmentation benchmarks using the Mask R-CNN [40] framework. We fine-tune the pre-trained with standard \(1 \times\) schedule [40] on MS-COCO \(\mathtt{train}\) split and report the transfer learning results on MS-COCO \(\mathtt{val}\) split following the training and testing configurations from [3].

Model FLOPs during inference are measured with batch size \(= 1\) and input resolution \(= 1280 \times 800\).

4.2 Main Results↩︎

4.2.0.1 Results of on ImageNet-1k.

As shown in Tab. 3, given limited FLOPs and parameters budgets, tiny-sized models are able to achieve competitive performance on ImageNet-\(1k\) [7] image recognition benchmark with three different cross-window communication paradigm, i.e., shifted-window [3], spatial token shuffle [5], and auxiliary messager tokens exchange [6].

Along with the results in Tab. 1, it is noteworthy that (1) shifted-window, spatial token shuffle, and auxiliary messager tokens exchange all give similar strong results under the same aggregation layer, and moreover, (2) different aggregation layers are all quite competitive under the same cross-window token mixing approach.

All these results support our proposal: the macro architecture of the Swin model family, other than specific aggregation layers or specific means of cross-window communication, may be more responsible for its strong performance.

4.8pt

3.pt

4.8pt

4.2.0.2 Scaling -Tiny.

There is little literature available on the scaling properties of hierarchical MLP-like models. Here we demonstrate that the tiny-sized model is scalable.

We choose Swin -Tiny as the model scaling start point. Width scaling [43], [44] is adopted while the model depth and input resolution are kept unchanged. The training and testing configurations are aligned with [3]. Results in Tab. 4 show that both -Small & -Base can be successfully optimized, converged, and consistently benefit from more computations and larger model size.

Finding appropriate model scaling laws tailored for as well as other MLP variants is non-trivial, since the model complexity of MLP-like architectures is a linear combination of two different parts: (1) the content aggregation layer part for depth-wise or group-wise spatial token mixing only, and (2) the feed-forward network part for point-wise or token-wise feature transformation only, which is different from the complexity of CNNs. Consequently, previous successful practice in CNNs scaling cannot apply to in a principled way and can be only verified one-by-one experimentally. We leave the study of sophisticated model scaling laws on for future work.

4.2.0.3 Comparisons with Other MLP Variants.

We summarize some recently proposed MLP-like architectures in Tab. 5. The proposed demonstrates superior performance with fewer FLOPs and parameters. The window partitioning operation and hierarchical representations introduce \(2\mathrm{D}\) locality bias and invariance to . Therefore it is not a surprise that is more efficient and competitive than global & columnar MLP architectures such as MLP-Mixer and ResMLP by leveraging these design priors.

Swin-Mixer is a recently proposed MLP-like architecture with hierarchical representations based on the Swin Transformer family. We demonstrate that both the tiny-sized and small-sized can outperform the corresponding Swin-Mixer counterparts even with fewer computation budgets.

4.2.0.4 Transfer Learning Performance of .

There is little literature available on the transferability of MLP-like architecture to downstream dense prediction tasks such as object detection and instance segmentation. Most available MLP variants using global kernels for spatial token mixing, therefore their spatial kernel sizes are fixed and highly correlated to the input resolution during pre-training, which largely limits the applications on high-resolution inputs with varying shapes and sizes during transfer learning.

The proposed can naturally overcome this issue via window-based spatial token mixing inherited from the Swin model family. is able to directly process arbitrary resolutions zero-padded to be divisible by the window size.

Results in Tab. 6 demonstrate that the transfer learning performance of is on a par with Swin Transformer in MS-COCO object detection and instance segmentation benchmarks even using relatively worse supervised pre-trained representations on ImageNet-\(1k\). These promising results indicate that could be able to better leverage the pre-trained representations for transfer learning.

4.3 Analysis and Ablation Study↩︎

In this section, we study the impact of model width & model depth configurations, weight sharing properties of linear mapping layers, as well as the number of groups. Overall, we conclude that is quite robust to different model choices and configurations thanks to the strong macro architecture.

4.3.0.1 Going Wider or Deeper?

We study the model width and model depth configurations for the tiny-sized model. A high-performance tiny-sized model can also be served as a promising start point for model scaling.

Since a single layer is much lighter than a single window-based MHSA layer from [3] in terms of both parameters and computations, we need to adjust the model width and depth of to align with the budgets for Swin Transformer-Tiny. The resulting models are termed as Swin -Tiny (Wide) and Swin -Tiny (Deep).

As shown in Tab. 7, the wider model seems to be more speed friendly while the deeper model is a lot more parameters & FLOPs efficient. We choose Swin -Tiny (Deep) as our default tiny-sized model instantiation.

4.3.0.2 Linear or Depth-wise Linear?

In this manuscript, depth-wise linear (DW Linear) layers refer to linear mapping with separate or non-shared weights & biases for each group. Therefore the model parameters increase while the theoretical FLOPs are kept unchanged when using DW linear layers instead of shared linear layers. As shown in Tab. 8, DW linear layers bring no significant improvement, which echos the observation in previous multi-head v.s. single-head self-attention studies [34], [35] to some extent. Therefore we choose to share linear weights & biases across different groups as our default instantiation for better parameters efficiency.

4.3.0.3 Linear Mapping or MLP?

In Tab. 9, we investigate the impact of using simple linear mapping layers and more sophisticated MLP layers for content aggravation inside each non-overlapped window. The results suggest that using MLPs with intermediate \(\mathtt{GELU}\)() activation functions [45] cannot bring further improvements.

This observation in our hierarchical is somewhat in line with the findings from the columnar counterpart [30], where ResMLP models also adopt simple patch-to-patch (or token-to-token) linear transformations instead of MLP layers in MLP-Mixer [29] for cross-patch communications.

4.3.0.4 Number of Groups (#Group) for Linear Layers.

In Tab. 10, we study the impact of different numbers of groups (#Groups) in linear layers. The weights & biases of linear layers are shared across groups. We find setting #Groups too large or too small is harmful to performance, while other choices yield similar results. In this manuscript, #Groups \(=32\) (i.e., \(\mathtt{gs} = 3 \times 2^{({\#}\mathtt{stage} - 1)}\) in Alg. 1) is chosen as the default configuration for all-sized models.

1.8pt

5 Conclusion↩︎

In this manuscript, we raise a crucial question that seems overlooked in previous research: “What Makes for Hierarchical Vision Transformer?”, and attempt to give an answer: the macro architecture design methodology may be more important than specific network layers and components. To this end, a proof-of-concept model named with embarrassingly lightweight & simple content aggregation layers is proposed as a touchstone or probe to support and validate our proposal. As a hierarchical MLP-like architecture, along with its variants also enables a fully linear mapping & MLP architecture to directly process high-resolution input images with arbitrary shapes, which makes it possible for investigating the transferability of linear mapping & MLP architectures on various computer vision downstream tasks other than image recognition such as object detection and scene parsing. We hope this manuscript can encourage the community to rethink the design of Vision Transformer, and shed a little light on future studies of general visual representation learning.

References↩︎