1 Introduction↩︎

Recent research has focused on devising new unsupervised pretraining methods from unlabeled data that involves some form of language modeling, primarily autoregressive [1], [2], masked [3]–[5] and generalized [2], [6], [7], with much success on downstream tasks. Under the hood, most of these methods use transformers [8] for encoding text sequences, which allows them to learn powerful contextual word representations that have been used widely for building models in NLP. However, this does not hold for sentence representations derived from pretrained transformer language models based on a special token or basic pooling operations. To this end, representation learning methods have been designed to better capture semantic information from pretrained transformer language models, e.g., using Siamese networks trained with a triplet loss [9] or transforming the desired sentence distribution to a Gaussian distribution through normalizing flows [10].

Existing sentence representations directly derived from pretrained language models or learned by specialized methods cannot guarantee perfect reconstruction of the input, a property that can enhance the structure of their semantic space and enable their use for controlled generation tasks. For the latter, a few recent studies have looked into ways to steer generation of pretrained language models towards a particular style [11], [12], although they require following the gradient during the sampling process and rely on style text classifiers which might not be always available. The latent space of a text autoencoder allows one to perform controlled text generation by directly manipulating sentence representations using basic numerical operations [13]. Yet, how to convert pretrained transformer language models to autoencoders with such properties still remains unexplored.

To fill in this gap, we introduce Autobot, a new autoencoder model for learning sentence “bottleneck” (i.e., fixed-size) representations from pretrained transformers that is useful for similarity, generation, and classification, displayed in Figure 1. Our model has two unique components: (i) a transformation that uses dot product attention to dynamically pool semantic information from the pretrained model’s hidden states into a sentence bottleneck representation, and (ii) a shallow transformer decoder that is modified to operate based on the bottleneck representation. Instead of training our autoencoder from scratch, we directly finetune it using an input reconstruction objective on the unlabeled data on which the original pretrained transformer was trained. We keep the underlying pretrained transformer encoder fixed, which makes it more efficient than training from scratch and proves beneficial even if we compare to pretrained transformers trained for an equal number of steps.

Our evaluation on representative sentence similarity, classification, and generation tasks demonstrates that the resulting sentence representations are compact, better capture semantic similarity at the sentence-level than strong sentence representation methods [9], and can be used for controlled generation tasks. Lastly, our model performs almost on par with the large RoBERTa model [4] even though it only introduces 1.6\(\%\) additional parameters relative to the base RoBERTa model.

2 Model: Autobot↩︎

Taking inspiration from recent research on text autoencoders [14]–[16], we extend standard autoregressive text autoencoders, which have been predominantly based on recurrent networks, to a transformer-based architecture and integrate them with pretrained language models; here we focus on RoBERTa [4].

Autoencoders generally follow the encoder-decoder model structure to reconstruct their input with the constraint that the encoder produces a single, fixed-length hidden representation \(\text{enc}(\boldsymbol{x}) = \mathbf{z}\): \[\text{AE}(\boldsymbol{x}) = \text{dec}(\text{enc}(\boldsymbol{x})) = \boldsymbol{x}'.\] Here, we focus on denoising autoencoders that aim to reconstruct a perturbed version of the input [15], [17], which is compatible with many of the pretrained language models that are based on masked language modeling. In our experiments, we use the same masking procedure as [3] to perturb the input.

2.1 Encoder↩︎

Standard approaches use encoders that reduce the input to a single representation \(\mathbf{z}\). To use a pretrained transformer for this purpose we need to reduce its output hidden representations \(\mathbf{H}\) after processing the input to a single vector. Since using the special token representation or basic pooling methods have been shown sub-optimal in prior work [9], here we opt to keep the original encoder fixed and train a transformation \(\beta\) that will learn to compress \(\mathbf{H}\) into a single representation \(\mathbf{z} = \beta(\mathbf{H}; \mathbf{\theta})\), with \(\theta\) being an additional set of parameters to be learned during finetuning. We choose \(\beta\) to be a multi-head attention mechanism that takes as input the keys \(\mathbf{K}\) and values \(\mathbf{V}\) corresponding to the final representations \(\mathbf{H}\) from the pretrained model and a query vector \(\mathbf{q}\) corresponding to a context vector \(\mathbf{u}\) that we choose to be the CLS vector from the pretrained model: \[\begin{align} \beta(\mathbf{H}; \mathbf{\theta}) &= \text{MultiHead}(\mathbf{q}, \mathbf{K}, \mathbf{V}) \end{align}\] where the parameters to be learned, \(\theta\), include the weights that are used to transform the query, keys, and values which amount to \(3 d^2\) with \(d\) being the dimensionality of each head (\(d = 64\) in our experiments).

2.2 Decoder↩︎

The cross-attention layer in the Transformer decoder architecture by [8] expects hidden representations for every token input from the encoder in order for each output candidate to attend to each input token. In the situation where only a single representation comes from the encoder, we have \[\text{Attention}(\mathbf{Q}, \mathbf{z}^\top \mathbf{W}_K, \mathbf{z}^\top \mathbf{W}_V) = \mathbf{z}^\top \mathbf{W}_V\] Note that the queries \(\mathbf{Q}\), which come from the previous masked self-attention layer, are not taken into account, and each step in the decoder will receive the exact same \(\mathbf{z}^\top \mathbf{W}_V\) as a result. In order to mitigate this, we propose a gating method inspired by [18]. Concretely, let \(\mathbf{Q_t}\) be the \(t\)th query representation. Then, the \(t\)th output \(\mathbf{o}_t\) of the cross-attention layer is computed as follows \[\mathbf{g}_t = \sigma(\mathbf{G} \mathbf{Q}_t + \mathbf{G'} \mathbf{z});\;\; \mathbf{o_t} = \mathbf{g}_t \odot \mathbf{z}^\top\mathbf{W}_V\] where \(\sigma(\cdot)\) is the sigmoid activation function and \(\mathbf{G}\) and \(\mathbf{G}'\) are the parameters of the transformation for the gate. One can view the role of the gate as determining the amount of per-element information from the linear transformation of the latent representation to keep for the current layer and timestep. Preliminary experiments found this method beneficial for generation.

2.2.0.1 Training considerations

To avoid training our model from scratch, we finetune it for 100K optimization steps on a pretraining dataset using the base RoBERTa model [4] on the encoder side and a single layer decoder side for efficiency purposes [19]. The model is trained using an input reconstruction loss by minimizing the negative log-likelihood computed over the reconstructed inputs. Note that only the parameters of the sentence bottleneck and the decoder are learned; the encoder parameters are kept fixed.

3 Experiments↩︎

To assess the quality of the sentence representations learned by our model we evaluate on sentence similarity (Section 3.2), classification (Section 3.3), and generation tasks (Section 3.4).

3.1 Settings↩︎

3.1.0.1 Datasets

Since the RoBERTa dataset is not publicly available, we use for pretraining the exact same dataset as BERT [3], which is composed of BooksCorpus [20] and English Wikipedia. For sentence similarity, we use the Natural Language Inference (NLI) dataset [21] for finetuning and evaluate on the Semantic Textual Similarity (STS) dataset [22], following [23]. For classification, we use mainly single-sentence datasets from the GLUE benchmark [24], namely Stanford Sentiment Treebank (SST) and Corpus of Linguistic Acceptability (CoLA) datasets, but we also report the average performance on the remaining datasets. For generation, we use the Yelp reviews dataset [25].

3.1.0.2 Baselines

For sentence similarity, we compare to SBERT which is a competitive method for deriving informative sentence representations from pretrained language models [9]. They obtain sentence representations by using simple pooling methods over BERT representations such as mean and max (instead of the CLS token representation) then finetuning the whole pretrained model using Siamese networks on a combination of natural language inference data. To compare with them on sentence similarity, we incorporate our model within their framework and follow their settings and training/evaluation protocol (details in Appendix 6.2).

For sentence classification, we compare our model to RoBERTa-base and RoBERTa-large models [4]. Note that BART [26] achieves similar results to RoBERTa, so a similar comparison can be made. For sentence generation tasks, we compare to a strong and efficient style transfer method by [15], which is a recurrent network-based denoising text autoencoder on in domain data. The style transfer is achieved through vector arithmetic, namely computing a “sentiment vector” \(\mathbf{v}\) by taking the vector difference between 100 negative and positive sentences, then evaluating by taking an input sentence, encoding it, adding a multiple of the sentiment vector to the sentence representation, then decoding the resulting representation. In addition to the denoising auto encoder (DAE) of [15], we include more sophisticated methods for style transfer that are more computationally expensive such as fast gradient iterative modification (FGIM) of [27] and Emb2Emb of [16] for reference.

3.2 Sentence Similarity↩︎

The results on the sentence similarity task are displayed in Table 1. Due to resource constraints and unreported results by prior work, we report our model only with RoBERTa-base. We can observe that Autobot applied to RoBERTa-base significantly outperforms other supervised base transformer methods. Additionally, Autobot approaches the performance of large transformers while having a minimal parameter overhead of 1.6%. We also find that Autobot without any supervision (Autobot-base unsup.) outperforms all of the unsupervised methods, and most notably improves upon average BERT embeddings by 26.1%. This demonstrates that our approach is effective in both supervised and unsupervised settings.

We find in Table 2 that using the proposed sentence bottleneck based on learned context provides noticeable gains over using simpler pooling methods from prior work. We suspect this is due to the additional flexibility provided by our bottleneck acting as “weighted pooling” by attending over all tokens to compute the final representation, as opposed to equal contribution of all tokens regardless of the input.

3.3 Sentence Classification↩︎

The results on single-sentence classification tasks and other tasks from the GLUE benchmark are displayed in Table 3. We find that Autobot provides a noticable performance increase on single-sentence tasks, specifically on the CoLA datasets when using both the RoBERTa-base and RoBERTa-large models. Additionally, we also find that Autobot, when fed both sentences concatenated for dual sentence GLUE tasks, maintains the original performance of the underlying pretrained encoder.

Hence, our model improves the quality of the sentence representations from pretrained transformer models without hurting their performance.

3.4 Sentence Generation↩︎

For sentence generation, we focus on the sentiment transfer task proposed by [15] both with and without further training on in-domain data from Yelp. When finetuning, we perform an additional 10K optimization steps using the Yelp dataset. Note that all the baselines require training on in-domain data, while this is optional for our model. In Figure 2 we find that the Autobot model not exposed to the Yelp dataset during finetuning performed on par with the DAE that was trained specifically on Yelp. Additionally, Autobot outperforms the DAE in the above-40 percent accuracy range when finetuned on in-domain data. We include autobot results with partial finetuning of the encoder in the appendix, which we find considerably improves the Self-BLEU metric. Since AUTOBOT uses vector arithmetic, inference is as fast as the DAE and over twice that of other methods.

4 Related Work↩︎

Reconstructing text with autoencoders is an active area of research that has lead to several advancements such as denoising [17], variational [28]–[30], adversarial [31], [32], and regularized [33] autoencoders. They have been found especially useful in controlled text generation [34]–[36], especially in sentiment style transfer [16], [25].

The encoder-decoder structure for obtaining representations has been used in pretraining [26], sentence infilling [37], and multilingual [38] scenarios. In particular, [26] treat denoising as translation task to perform pretraining from scratch, but their approach does not induce a sentence representation space with generative properties. In contrast, our method makes use of a frozen pretrained transformer to learn a shallow, sentence bottleneck autoencoder on top.

5 Conclusion↩︎

We proposed an approach that converts a pretrained transformer language model into a sentence-level autoencoder that is able to reconstruct its pretraining data. The resulting model improves the performance of the pretrained model on sentence-level tasks while maintaining its performance on multi-sentence tasks. In addition, the new sentence representations are suitable for efficient conditional text generation such as sentiment transfer without the need for training on in-domain data.

Acknowledgments↩︎

The authors thank Jungo Kasai and the anonymous reviewers for their helpful feedback. Nikolaos Pappas was supported by the Swiss National Science Foundation grant P400P2_183911.

6 Reproducibility↩︎

6.1 Experimental Setup↩︎

6.1.0.1 Computing Infrastructure

For all of our experiments involving base models, we use a computation cluster with 5 NVIDIA RTX 2080 TI GPU, 11GB GPU memory, and 128GB RAM. For large models, we use a computation cluster with 4 NVIDIA TITAN RTX GPUs, 24GB GPU memory and 256GB RAM.

6.1.0.2 Implementation

We will make our implementation available on Github.² We used Python 3.7, PyTorch 1.6.0, and Sentence Transformers 0.3.7. We use modified versions of Fairseq 0.9.0 and Transformers 3.3.1. We obtain our datasets from the citations specified in the main paper.

6.1.0.3 Autobot Training

We extract the sentences from the BooksCorpus and English Wikipedia datasets to recreate the BERT dataset, and use RoBERTa-base’s pretrained tokenizer for tokenization. We report our hyperparameters for Autobot-base in Table 4. Our decoder only has one single layer, and RoBERTa-base remains fixed during finetuning.

6.2 Sentence Representations↩︎

We use the Sentence Transformers framework for training and evaluation of Autobot. We use the default settings in their framework to train on NLI, and evaluate using the Spearman correlation of the cosine similarity. During NLI finetuning, we only use the encoder and bottlneck, with the bottleneck representation used as the sentence representation, and allow for all parameters to be finetuned.

6.3 Sentence Generation↩︎

We use a modified version of Fairseq’s generation code for encoder-decoder models to perform vector arithmetic for sentiment transfer. We follow the instructions of [16] to finetune a sentiment classifier using DistilBERT from the Huggingface transformers library.

For the Autobot models finetuned to the Yelp dataset, we follow the exact same steps as Appendix 6.1.0.3 except beginning with the Autobot-base model, using the Yelp training set, and performing 10k optimization steps.

6.4 Sentence Classification↩︎

We use the Huggingface library to perform sentence classification using Autobot. During finetuning, we only use the encoder and bottleneck, with the bottleneck representation used as a CLS representation, and allow for all parameters to be finetuned. We perform a hyperparameter search similar to that of RoBERTa by comparing development performances when using \(\{\)​1e-5, 2e-5, 3e-5\(\}\) for the learning rate.

7 Additional Results↩︎

We provide additional results in addition to our experiments below.

7.1 Autoencoding Steps↩︎

We perform an ablation study on the effect of autoencoding finetuning steps of the underlying pretrained encoder during autoencoding on the downstream sentence representation performance. We provide the detailed performances of performing Table 4 when using a learning rate of 1e-3 in Table 5.

7.2 Finetunable Encoder Layers↩︎

We perform an ablation study on the effect of finetuning the underlying pretrained encoder during autoencoding on downstream sentence representation performance. We provide the detailed performances of performing Table 4 with the optimal parameters, but varying how many of the last layers of RoBERTa-base to finetune. Results are in Table 6

7.3 Finetunable Encoder Generation↩︎

We provide an appended generation table from Section 3.4 to include the generation results we obtained by allowing the top three layers of RoBERTa-base to be finetuned during autoencoding on the style generation task. The results are shown in Figure 3. The same model as used in Appendix 7.2 is used.

7.4 Style Transfer Results↩︎

We provide Table 7 that reports results on the Yelp sentiment transfer test set from the generation table in Section 3.4, appending to the table [16]. We outline the relative time differences during inference. We can observe that our model not only provides competitive speed-quality tradeoff.

7.5 Detailed Sentence Classification Results↩︎

Section 3.3 provides a summary of the GLUE results, while outlining the specific single-sentence classification performances. We provide the results for each task in Table 8

References↩︎