1 Introduction↩︎

Recent improvements in NLP tasks can be attributed to the Transformer [1] model. The model has led to better deeply contextualized representations [2], better machine translation systems [1], and language models [3], [4]. Despite their success, one main drawback of training these models is their computational cost, being a greatly limiting factor for many, with training times and memory usage ballooning as model sizes increase to attain better performance.

With this in mind, there has been recent interest in making the Transformer more parameter-efficient to reap its performance benefits while making the model more computationally efficient and able to scale better. Many approaches have focused on automating model design with neural architecture search that aim at finding more efficient Transformer variations using gradient descent [5]–[7]. As such, these techniques are expensive, requiring a significant amount of GPU hours to find good designs. In contrast to these approaches, the model by [8] has directly tackled model parameter reduction in the context of deeply contextualized word representations, still attaining similar (or better) performance. In this paper, we take a similar approach and look to explore whether these ideas can be applied to sequence-to-sequence models in a simple manner by designing the Subformer.

The Subformer incorporates two novel techniques: (1) SAFE (Self-Attentive Factorized Embeddings), in which we use a small self-attention layer to reduce embedding parameter count, and (2) Sandwich-style Parameter Sharing, in which we develop a simple and intuitive technique for parameter sharing to be effective in Transformer models.

We evaluate the Subformer on machine translation, abstractive summarization, and language modeling, showing that our model can achieve similar or better performance compared with a base/big Transformer with a \(\sim\)​40% parameter reduction and minimal modification to the original architecture, reinforcing the existing over-parameterization claims [7]–[9]. On WMT’14 EN-DE we achieve a BLEU score of 29.3, compared to Transformer-big’s 28.6 with 13M fewer parameters, and we also outperform the Transformer-XL model with a significant 3.6 lower perplexity and 37% fewer parameters.

2 The Subformer↩︎

Let us start by defining the notation to be used throughout the paper. We refer to the model dimension as \(d_m\), feed-forward projection dimension as \(\vec{d}_m\), the vocabulary size as \(V\), and the number of layers as \(L\). Note that, unlike standard Transformer models, in which the embedding dimension is kept the same as \(d_m\), we disentangle to embedding dimension to reduce parameter count. For this reason, we denote the embedding dimension to be \(d_e\).

2.1 SAFE: Self-Attentive Factorized Embeddings↩︎

We propose to reduce the number of parameters in our embedding layers, which can take up to 25% of the total parameter count (in the case of Transformer-base, [1]), using a small self-attention layer. Specifically, we look to reduce the embedding size by disentangling the model dimension from the embedding dimension, reducing the embedding dimension \(d_e\), and then projecting this to the model dimension \(d_m\) using a small self-attention sub-layer followed by a feed-forward module.

Given vocabulary size \(V\), the usage of a standard embedding layer would result in \(V \times d_m\) parameters. However, considering that the power of Transformers lies in their ability to learn contextual representations with attention, using a smaller \(d_e\) for non-contextual embeddings and then projecting to \(d_m\) is intuitively an effective method for parameter reduction [8]. This results in a significant parameter reduction for values of \(d_e \ll d_m\).

When using SAFE, the embedding layer is composed of a regular token \(\rightarrow\) vector embedding matrix \({\boldsymbol{E}}\in {\mathbb{R}}^{V \times d_e}\). This is followed by projecting the embeddings (summed with the positional encodings [1], denoted by \(PE\)) to the model dimension \(d_m\) using SAFE. Once we have our SAFE embeddings, we feed them through the first model layer —the base of the sandwich. The output of this first layer is then projected to the sandwich dimension \(d_s\). Once fed through the shared sandwich layers, we then project the output back to the model dimension using an MLP. The output of the projection is then fed through the final model layer to produce the output vectors.

Current models [3], [4], [8] often use a single linear projection, i.e. \(V \times d_e + d_e \times d_m\). In contrast, we empirically show that simply contextualizing this projection with a small self-attention layer results in stronger performance with a minimal addition of parameters —especially in the encoder-decoder case, where the input embedding layer and output projection are often tied [10] (Table 1).

2.2 Sandwich-style Parameter Sharing↩︎

Weight sharing techniques, despite being surprisingly effective, have been relatively unexplored in the context of generative Transformers. However, this has been shown to be powerful for leveraging models with large capacity and less memory usage/computation [8], [11].

Given that the output of each Transformer layer depends directly on its two sub-layers —multiheaded attention and the feedforward module— when discussing alternatives for parameter sharing across transformer layers there are several options. As we aim to leverage the aforementioned properties of weight sharing, we performed preliminary experiments, investigating the capabilities of weight sharing in the following five settings: (1) All-shared Naively sharing all encoder and all decoder layers —that is including both of their sub-layers, following [8], [13]. (2) All-Shared (Independent FFN) Naively sharing all encoder and all decoder layers but allowing each layer \(l \in [2, \cdot, L]\) to have an independent feed-forward sub-layer. (3) All-Shared (except last) Sharing weights across layers \(l \in [1,\ldots, L-1]\) such that layer \(L\) remains independent. (4) Every 2 layers shared Sharing every two layers, i.e. \([1,2],[3,4],[5,6]\) in the case of a 6-layer transformer. (5) Sandwich Finally, we only share the middle or central layers (i.e. \(2 \leq l \leq L-1\)), leaving layers \(1\) and \(L\) to have independent sets of parameters.

Table 2 summarizes the results of our exploratory study. As can be seen, naive parameter sharing/tying approaches do not offer any advantages, hurting performance significantly (\(\sim\)​50%) when compared to the regular Transformer. However, our results also show that when combined properly, using Sandwich-style parameter sharing, we can attain a good balance of parameter reduction and performance. When compared to tasks such as pre-training deep contextualized word representations, generative tasks such as machine translation require informative token-level representations for each input token to be accurately translated. In this context, we surmise that the success of Sandwich-style parameter sharing on this sequence-to-sequence task is a result of it being able to have the input and output layers (arguably, the most important layers) be trained independently, allowing them to learn different operations than the sandwich layers.

2.2.0.1 Combined Architecture

Having introduced our proposed parameter reduction techniques, we will now explain the Subformer architecture. The Subformer is composed of four main components, for both the encoder and decoder: the embedding layer, the model layers, the sandwich module and the projection layers. We disentangle the sandwiched layer dimension from that of the model layer, allowing the sandwich layer width to be larger than the rest of the model. For this reason, we denote the dimension of the sandwiched layer to be \(d_s\) and its corresponding feed-forward dimension to be \(\vec{d}_s\).

3 Experimental Setup↩︎

We apply our method to a variety of sequence modeling tasks: neural machine translation, summarization, and language modeling. We compare Transformer-base and big [1] with the Subformer trained in the same setting. We also include simple sandwich-style parameter sharing (denoted as Sandwich-{base,big}) and the usage of only SAFE as an ablation of what these techniques do in their naive forms when decoupled. Additional implementation and training details with hyperparameter settings are in the appendix.

3.0.0.1 Machine Translation

We evaluate our model on two standard translation benchmarks: WMT’14 English-German (EN-DE) and WMT’16 English-Romanian (EN-RO). Following previous work, we evaluate all models using tokenized BLEU [14]. In order to better contextualize our results, we consider parameter-efficient models DeLighT [7] (contemporaneous work), and the Evolved Transformer [6].

3.0.0.2 Abstractive Summarization

We test the model’s ability to process long documents on the CNN-DailyMail summarization benchmark [15], [16], comprising over 280K news articles paired with multi-sentence summaries. We also compare effects of BART [17] pretraining (details in Appendix 7). For this task we contexutalize our results with specialized architectures such as Pointer-Generator Networks [18], and methods leveraging pretraining: RobertaShare [19], BertExtAbs [20], and BART [17]. We evaluate using ROUGE 1,2,L [21].

3.0.0.3 Language Modeling

We evaluate on the large-scale Wikitext-103 dataset [22]. Models are evaluated in terms of perplexity on the test portion. In order to better contextualize our results, we consider the QRNN [23], Transformer-XL [4] and Deep Equilibrium Model (DEQ) [24], which also employs parameter sharing.

4 Results↩︎

4.0.0.1 Machine Translation

Tables 3 and 4² summarize our results on machine translation. Firstly, we note that our Transformer baselines outperform [1] (base: 27.3 \(\rightarrow\) 27.7, big: 28.4 \(\rightarrow\) 28.6). We surmise that this is due to training for longer and with a larger batch size.

The Subformer outperforms our Transformer baselines when trained in the same setting, with similar or fewer parameters. Subformer-small reduces parameters by 40%, matching performance with our Transformer baselines. Subformer-base and mid, outperform our model significantly (0.4, 0.8 BLEU) when using a fewer/similar number of parameters. Furthermore, we note that Subformer-mid performs similarly to the Transformer-big model (210M params.) in Table 4, despite a 70% parameter reduction.

For our large set of models (Table 4), Sandwich-big achieves the same performance as our Transformer-big reimplementation, but with 40% fewer parameters. We believe this to be an indication of the capability of Sandwich-style parameter sharing as the encoder/decoder layers get wider, while also providing further evidence for the overparameterization of the Transformer. Subformer-large, with 197M parameters achieves a significant 0.7 BLEU score gain over Transformer-big, despite using 13M fewer parameters.

4.0.0.2 Language Modeling

Results for language modeling can be seen in Table 5. The Subformer uses adaptive input embeddings [3] instead of SAFE embeddings, following common practice. We also train two Transformer baselines with the same setup —one with the same amount of parameters and another with a similar parameter count to Transformer-XL— to provide better context for comparison. Task-specific techniques that can be applied during training, such as caching [4] or other methods applied during inference time [26], [27] can further improve all models so we do not focus on these.

As seen in Table 5, the Subformer outperforms the baselines by a significant margin (between 1.4 and 6.5 perplexity), with a significant reduction in parameters.

4.0.0.3 Abstractive Summarization

For the CNN/Daily Mail summarization task we use Subformer-base. We also pretrain a Transformer and Subformer model with the same architecture on Wikipedia (details in Appendix 7). As can be seen in Table 6, the Subformer outperforms two Transformer baselines with both the same parameter count and its respective Transformer-base configuration in both settings, demonstrating the Subformer’s performance on a variety of tasks and with longer sequences.

4.0.0.4 Discussion on Speed and Convergence

We found training time to consistently speed up by 10-30%, and inference speed to either be faster by 10-20% (keeping \(d_s=d_m\)) to be slightly slower by 10-30% (when \(d_s \gg d_m\)) (due to more operations, similar to [8]). The Subformer converges faster most likely due to fewer parameters to optimize. Given the fewer number of parameters, it can be expected for the models to converge with fewer iterations. We test this on the task of language modeling, where we found that the Subformer converged 65% faster than its Transformer counterpart, as shown in Table 7. We also measure inference speed on our machine translation models (Table 8).

5 Conclusion↩︎

In this paper we have presented the Subformer, a parameter-efficient Transformer-based generative model primarily based on two simple parameter factorization/sharing techniques. Our empirical results on three sequence modeling tasks show that the Subformer can achieve similar or better performance compared with a base/big Transformer with a \(\sim\)​40% parameter reduction. Furthermore, the simplicity of our approach allows the Subformer to be used in conjunction with other parameter reduction techniques in the literature, for even smaller but performant models. We hope our work incites interest in parameter sharing techniques for an even wider range of Transformer models, ultimately helping reduce their computational cost in general.

Ethical Considerations↩︎

This work has impact in the field of natural language processing, and develops a more efficient approach for learning performant generative models. As with much of language technology has the potential to be both used for good and used maliciously. We also experiment with pretraining, learning representations in an unsupervised way, which is likely to capture and amplify biases found in the data. However, our approach has a potential positive impact given the lower cost and energy expenditure needed to train our proposed model.

Acknowledgments↩︎

We thank Jorge Balazs, Yusuke Iwasawa, Jungo Kasai, Cristian Rodriguez-Opazo, Alfredo Solano, Yutaro Yamada, and Victor Zhong for their helpful feedback and discussions over this work. MR is grateful to the Masason Foundation for their support.

6 Designing the Subformer↩︎

6.0.0.1 Naming

The Subformer is a play on words, referencing its small size - i.e. sub-, as well as the Sandwich-style parameter sharing technique, referencing the type of sandwich.

6.0.0.2 Architecture

When using SAFE, our parameter count would result in \(V \times d_e + \ 5d_e^2 + d_e \times d_m\) parameters, where the first term represents the embedding layer, the value \(5d_e^2\) groups the query, key, and value projections and 2 output feed-forward layers, and \(d_e \times d_m\) represents the linear projection from the embedding dimension to the model dimension. As mentioned in the paper, this results in a significant parameter reduction for values of \(d_e \ll d_m\).

As we tie the decoder’s output projection layer (returning a distribution over the vocabulary) with the decoder’s input embedding matrix, we project the decoder’s last hidden state (with dimension \(d_m\)) to \(d_e\) using a two layer MLP. Also, when we perform encoder attention in the decoder’s Sandwich Module, we simply linearly project the query from the decoder from \(d_s\) to \(d_m\) and then project it back to \(d_s\) once the attention operation is complete.

6.0.0.3 Memory Footprint

Table [tb:mem] summarizes the memory footprint of our proposed techniques. In this table, the benefits of Sandwich-style parameter sharing can be seen as the number of independent layers is controlled to be \(L \leq 3\), however, Transformers generally need to be deeper (with a standard of \(L = 6\)) to learn more meaningful representations with the parameter count scaling linearly with respect to the layer count. Similarly, the benefits of disentangling the model dimension from the embedding dimension can be seen as well. Due to the parameter reduction attained by these techniques, the models can be trained in memory-constrained scenarios with a larger batch size.

7 Data and Training Details↩︎

Training was done on 8 GPUs on a single DGX-1 Machine, while training on 16 GPUs was done using multiple compute nodes of a compute cluster. We train all base/small models on 8 NVIDIA Tesla V100 GPUs. For all big/large models, we train on 16 NVIDIA Tesla V100 GPUs. All models were trained with mixed precision [28] and are implemented in PyTorch [29] using our modification of fairseq [30].

7.0.0.1 Machine Translation

We train using 8192 tokens per GPU an update frequency of 2, for small, base models. For large models, we train on 16 GPUs with 4096 tokens per GPU with an update frequency of 2. We follow the training setup of [31]: we use the same weight initialization scheme as BERT [2], sampling weights from \(\mathcal{N}(0,0.02)\), initializing biases to zero and setting layer normalization parameters \(\beta\) and \(\gamma\) to be \(0\) and \(1\), respectively. For regularization we use the best of \([0.1,0.2,0.3]\) dropout, weight decay of \(0.01\), while using label-smoothed cross entropy loss with \(\epsilon = 0.1\). We train using an effective batch size of 128K tokens. The models are trained using Adam [32], with hyper-parameters \(\beta = (0.9,0.999)\) and \(\epsilon = 10^{-6}\). We warm up the learning rate to a peak of \(5 \times 10^{-4}\) within 10K iterations and then decay the learning rate with the inverse square root schedule. When creating the final model, we use the checkpoint with the lowest loss on the development set and generate using a beam size of 5 [1], tuning the length penalty of \(\alpha \in [0.0,0.2,\)…\(,2.0]\) in the validation set. We perform early stopping, training for a maximum of 250K iterations.

We use the following settings for our models: (1) Subformer-small has \(d_m = 512\), \(d_s = 768\), \(d_{e} = 256\) and \(L=8\), (2) Subformer-base has \(d_m = 512\), \(d_s = 1024\), \(\vec{d}_s = 3072\), \(d_e = 320\), (3) Subformer-mid has \(d_m = 768\), \(d_s = 768\), \(d_e = 350\) and (4) Subformer-large has \(d_m = 1024\), \(d_s = 2048\) and \(d_e = 512\). For WMT’16 EN-RO, our small model has \(d_m = 320\), \(d_s = 512\) and \(d_{e} = 192\) and our base model has \(d_m = 512\), \(d_s = 640\), and \(d_e = 384\).

In terms of datasets, we make use of the same pre-processed data used by [31] for WMT’14 EN-DE with a 32K BPE [33] vocabulary and during evaluation we perform de-hyphenation [1]. We use the same data as [34] for WMT’16 EN-RO with a 35K BPE vocabulary.

7.0.0.2 Abstractive Summarization

We follow [35] and use the official ROUGE-1.5.5.pl script with parameters -m -a -n 2. As mentioned in the paper, our model configuration is the same as Subformer-base, but we set \(d_e = 256\). Articles are truncated to 400 tokens [18] and we use a BPE vocabulary of 32K types [35]. We follow the training schedule of [35]. During inference, we tune generation length in the range of {40, 50, 60} and use tri-gram blocking, following standard practice. When pretraining, we pretrain on Wikipedia (14GB) for 100K iterations, using a batch size of 512K tokens. We use a learning rate of 7e-4, warmed up over 10K iterations.

7.0.0.3 Language Modeling

When training our language models, we use 8 GPUs with 3072 tokens per GPU and an update frequency of 3, following [3]. Models are trained using Nesterov’s accelerated gradient optimizer [36], warming up the learning rate to 1.0 for 16K iterations, and then annealing for 270K iterations using a cosine annealing schedule. We use three configurations: (1) \(d_m = 768, \vec{d}_m = 3072, d_s = 1536, \vec{d}_s = 6144\), (2) \(d_m = 768, \vec{d}_m = 4096\) and \(d_s = 2048, \vec{d}_s = 6144\) and (3) \(d_m = 1024, \vec{d}_m = 4096\) and \(d_s = 2048, \vec{d}_s = 6144\). All models use \(L = 12\). Our considered dataset, Wikitext-103, contains 103M tokens and has a vocabulary of nearly 270K.

8 Extended Related Work↩︎

8.0.0.1 Improving Transformers

Given the effectiveness of the Transformer, improving the architecture has been of much interest to the NLP community. Within this domain, one branch of research concerns the reduction of the quadratic complexity (w.r.t. sequence length) of the Transformer’s core self-attention mechanism [11], [37], pushing it down to linear or log-linear complexity. The second branch of research regards improving the expressiveness of Transformer models, by using more layers [38], or by improving the architecture [6], [11]. The third branch of research regards improving the parameter efficiency of Transformers. Approaches towards this goal include neural architecture search approaches [5], [6], where new Transformer-based architectures are learned using gradient descent, more manually crafted approaches [7], [13], as well as weight-sharing approaches [8], [11]. The work most similar to ours is ALBERT [8] in which complete weight sharing is used to pre-train deep contextualized word representations [2], [39]. Different from this work, we focus on common NLP generative/sequence-to-sequence tasks versus large-scale pre-training and develop an approach to increase model capacity while reducing parameter footprint tailored to this setting.

8.0.0.2 Compressing Transformers

We find prior work on pruning and quantizing Transformer models to reduce their size with a focus on sequence-to-sequence settings like machine translation [40], on encoder-based methods like BERT [41], [42] or with a more generic scope in mind [43], [44]. Our approach is orthogonal to these since we directly aim at reducing the number of parameters of Transformer models by proposing architecture modifications and weight sharing techniques.

8.0.0.3 Reducing Embedding Dimensionality in Sequence Models

As embeddings can substantially increase the parameter count as the vocabulary size increases, especially in sequence modeling scenarios, embedding reduction techniques have been proposed, including using a linear projection to project to a lower dimension [3], [4] or using combinations of block sparse transformations [7], [12]. We propose a self-attention based projection layer, SAFE, which we empirically show to outperform the aforementioned linear projection methods with a similar parameter count.

References↩︎