1 Introduction↩︎

Measuring sentence similarity [2] is an important task in natural language processing, and has found many uses including paraphrase detection [3], bitext mining [4], language modelling [5], question-answering [6], and as reward functions or evaluation metrics for language generation tasks [7]. Within this context, fast and light-weight methods are particularly useful as they make it easy to compute similarity over the ever-increasing volumes of web text available. For instance, we may want to mine a hundred million parallel sentences [8] or use a semantic similarity reward when fine-tuning language generation models on tens of millions of training examples. These tasks are much more feasible when using approaches that are fast, can be run on CPU, and use little RAM, allowing for increased batch size.

This need for fast inference is one motivation for using sentence embeddings. Sentence embeddings allow the search for similar sentences to be linear in the number of sentences, or even sub-linear when using highly optimized tools like Faiss [9] that allow for efficient nearest neighbor search. This is contrast to models, like cross-attention models, which are quadratic during inference as they require both of the texts being compared as inputs. As we show in this paper, our simple and interpretable word-averaging sentence embedding models [10]–[12], are orders of magnitude faster to compute than prior embedding approaches while simultaneously possessing significantly stronger performance on monolingual and cross-lingual semantic similarity tasks. Since we are simply averaging embeddings and have no neural architecture, any models based on neural architectures, especially large pretrained neural architectures which are increasingly used, will not be as fast as the models described in this paper. Lastly, we also show that this approach is competitive with LASER [13], a state-of-the-art multilingual model, on mining bitext and has stronger performance on cross-lingual semantic similarity, while having inference speeds that are twice as fast on GPU and orders of magnitude faster on CPU.

We make several contributions in this paper that go beyond our prior work. Firstly, we reformat the code to support training models on tens of millions of sentence pairs efficiently and with low RAM usage. Secondly, we train an English model on 25.85 million paraphrase pairs from ParaNMT [11], a paraphrase corpus we previously constructed automatically from bitext. We then train models directly on X-English bitext for Arabic, German, Spanish, French, Russian, Turkish, and Chinese, producing models that are able to distinguish both paraphrases in English and their respective languages as well as cross-lingual X-English paraphrases. Even though all models are able to model semantic similarity in English, we find that training on ParaNMT specifically leads to stronger models as it is easier to filter the data to remove noise and sentence pairs with little to no diversity. We refer to our models as Paragram-SP, abbreviated as P-SP,² referring to how the models are based on averaging subword units generated by sentencepiece [14]. We make all of these models available to the community for use on downstream tasks.

We also add functionality to our implementation. Besides the support for efficient, low-memory training on tens of million of sentence pairs described above, we add code to support (1) reading in a list of sentences and producing a saved numpy array of the sentence embeddings; (2) reading in a list of sentence pairs and producing cosine similarity scores; and (3) downloading and preprocessing evaluation data, bitext, and paraphrase data. For bitext and paraphrase data, we provide support for training using either text files or HDF5 files.

Lastly, this paper contains new experiments showcasing the limits of these scaled-up models and detailed comparisons with prior work on a suite of semantic similarity tasks in a variety of languages. We release our code and models to the community in the hope that they will be found useful for research and applications, as well as using them as a base to build stronger, faster models covering more of the languages of the world.

2 Related Work↩︎

2.1 English Semantic Similarity↩︎

Our learning and evaluation setting is the same as that of our earlier work that seeks to learn paraphrastic sentence embeddings that can be used for downstream tasks [10], [11], [15]–[17]. We trained models on noisy paraphrase pairs and evaluated them primarily on semantic textual similarity (STS) tasks. More recently, we made use of parallel bitext for training paraphrastic representations for other languages as well that are also able to model cross-lingual semantic similarity [7], [18]. Prior work in learning general sentence embeddings has used autoencoders [19], [20], encoder-decoder architectures [21], [22], and other sources of supervision and learning frameworks [23]–[26].

For English semantic similarity, we compare to well known sentence embedding models such as InferSent [27], GenSen [28], the Universal Sentence Encoder (USE) [29], as well as BERT [30].³ We use the pretrained BERT model in two ways to create a sentence embedding. The first way is to concatenate the hidden states for the CLS token in the last four layers. The second way is to concatenate the hidden states of all word tokens in the last four layers and mean pool these representations. Both methods result in a 4096 dimension embedding. We also compare to a more recently released model called Sentence-BERT [1]. This model is similar to InferSent in that it is trained on natural language inference data (SNLI; [31]). However, instead of using pretrained word embeddings, they fine-tune BERT in a way to induce sentence embeddings. Lastly, we also compare to the unsupervised version of SimCSE [32], which fine-tunes a pretrained encoder on contrastive pairs, where positive pairs are obtained by using dropout on a single input sentence.

2.2 Cross-Lingual Semantic Similarity and Semantic Similarity in Non-English Languages↩︎

Most previous work for cross-lingual representations has focused on models based on encoders from neural machine translation [4], [33], [34] or deep architectures using contrastive losses [35]–[37]. Recently, other approaches using large Transformer [38] have been proposed, trained on vast quantities of text [39]–[41]. We primarily focus our comparison for these settings on LASER [13], a model trained for semantic similarity across more than 100 languages. Their model uses an LSTM encoder-decoder trained on hundreds of millions of parallel sentences. They achieve state-of-the-art performance on a variety of multilingual sentence embeddings tasks including bitext mining. We also compare to LaBSE [42], a contrastive model trained on six billion parallel pairs across languages and was also trained on monolingual text using a masked language modelling objective.

3 Methods↩︎

We first describe our objective function and then describe our encoder.

3.0.0.1 Training.

The training data consists of a sequence of parallel sentence pairs \((s_i, t_i)\) in source and target languages respectively. Note that for training our English model, the source and target languages are both English as we are able to make use of an existing paraphrase corpus. For each sentence pair, we randomly choose a negative target sentence \(t_i'\) during training that is not a translation or paraphrase of \(s_i\). Our objective is to have source and target sentences be more similar than source and negative target examples by a margin \(\delta\): \[\begin{align} \min_{\theta_\textrm{src}, \theta_\textrm{tgt}} \sum_i\Big[\delta - &f_{\theta}(s_i,t_i) + f_{\theta}(s_i,t_i'))\Big]_+ \label{eq:obj} \end{align}\tag{1}\] where the similarity function is defined as: \[\begin{align} f_\theta(s,t) = \textrm{cos}\Big(&g(s;\theta_\textrm{src}), g(t; \theta_\textrm{tgt})\Big) \end{align}\] where \(g\) is the sentence encoder with parameters for each language \(\theta = (\theta_\textrm{src}, \theta_\textrm{tgt})\). To select \(t_i'\) we choose the most similar sentence in some set according to the current model parameters, i.e., the one with the highest cosine similarity. We found we could achieve the strongest performance by tying all parameters together for each language, more precisely, \(\theta_\textrm{src}\) and \(\theta_\textrm{tgt}\) are the same.

3.0.0.2 Negative Sampling.

Negative examples are selected from the sentences in the batch from the opposing language when training with bitext and from any sentence in the batch when using paraphrase data. In all cases, we choose the negative example with the highest cosine similarity to the given sentence \(s\), ensuring that the negative is not in fact paired with \(s\) in the batch. To select even more difficult negative examples that aid training, we use the mega-batching procedure of [11], which aggregates \(M\) mini-batches to create one “mega-batch” and selects negative examples from this mega-batch. Once each pair in the mega-batch has a negative example, the mega-batch is split back up into \(M\) mini-batches for training. Additionally, we anneal the mega-batch size by slowly increasing it during training. This yields improved performance by a significant margin.

3.0.0.3 Encoder.

Our sentence encoder \(g\) simply averages the embeddings of subword units generated by sentencepiece [14]; we refer to our model as Paragram-SP, abbreviated as P-SP. This means that the sentence piece embeddings themselves are the only learned parameters of this model.

4 Code and Usage↩︎

We added a number of features to the code base to improve performance and make it easier to use. First, we added code to support easier inference. Examples of using the code programmatically to embed sentences and score sentence pairs (using cosine similarity) are shown in Figure 1.

Our code base also supports functionality that allows one to read in a list of sentences and produce a saved numpy array of the sentence embeddings. We also included functionality that allows one to read in a list of sentence pairs and produce the sentence pairs along with their cosine similarity scores in an output file. These scripts allow our models to be used without any programming for the two most common use cases: embedding sentences and scoring sentence pairs. Examples of their usage with a trained model are shown in Figure 2.

Secondly, we added a training mode using HDF5⁴ format, allowing training data to remain on disk during training. This leads to a significant reduction in RAM usage during training, which is especially true when using more than 10 million training examples. Efficient training can now be done on CPU only using only a few gigabytes of RAM.

Lastly, we also added code for preprocessing data, including scripts to download and evaluate on the STS data (English, non-English, and cross-lingual), as well as code to download and process bitext and ParaNMT automatically. For bitext, our scripts download the data, filter the data by length,⁵ lowercase, remove duplicates, train a sentencepiece model, encode the data with the sentencepiece model, shuffle the data, and process the data into HDF5 format for efficient use. For ParaNMT, our scripts download the data, use a language classifier to filter out non-English sentences⁶ [45], filter the data by paraphrase score, trigram overlap, and length,⁷ train a sentencepiece model, encode the data with the sentencepiece model, and process the data into HDF5 format. Examples are shown in Figure 3.

5 Experiments↩︎

5.1 Experimental Setup↩︎

5.1.0.1 Data.

For our English model, we train on selected sentence pairs from ParaNMT [11]. We filter the corpus by only including sentence pairs where the paraphrase score for the two sentences is \(\geq 0.4\). We additionally filtered sentence pairs by their trigram overlap [17], which is calculated by counting trigrams in the two sentences, and then dividing the number of shared trigrams by the total number in the sentence with fewer tokens. We only include sentence pairs where the trigram overlap score is \(\leq 0.7\). The paraphrase score is calculated by averaging paragram-phrase embeddings [10] for the two sentences in each pair and then computing their cosine similarity. The purpose of the lower threshold is to remove noise while the higher threshold is meant to remove paraphrases that are too similar.

Our training data is a mixture of Open Subtitles 2018⁸ [46], Tanzil corpus⁹ [47], Europarl¹⁰ for Spanish, Global Voices¹¹ [47], and the MultiUN corpus.¹² We follow the same distribution for our languages of interest across data sources as [13] for a fair comparison. One exception, though, is we do not include training data from Tatoeba¹³ [47] as they do, since this domain is also in the bitext mining evaluation set. The amount of data used to train each of our models is shown in Table 1.

5.1.0.2 Hyperparameters.

For all models, we fix the batch size to 128, margin \(\delta\) to 0.4, and the annealing rate to 150.¹⁴ We set the size of the sentencepiece vocabulary to 50,000, using a shared vocabulary for the models trained on bitext. If a word is not in vocabulary, we simply exclude it, unless the text only consists of unknown words in which case we use a single unknown-word token. We optimize our models using Adam [48] with a learning rate of 0.001 and train models for 25 epochs.

For training on the bilingual corpora, we tune each model on the 250 example 2017 English STS task [49]. We vary dropout on the embeddings over \(\{0, 0.1, 0.3\}\) and the mega-batch size \(M\) over \(\{60, 100, 140\}\).

For training on ParaNMT, we fix the hyperparameters in our model due to the increased data size making tuning more expensive. We use a mega-batch size \(M\) of 100 and set the dropout on the embeddings to 0.0.

5.2 Evaluation↩︎

We evaluate sentence embeddings using the SemEval semantic textual similarity (STS) tasks from 2012 to 2016 [2], [50]–[53] as was done initially for sentence embeddings in [10]. Given two sentences, the aim of the STS tasks is to predict their similarity on a 0-5 scale, where 0 indicates the sentences are on different topics and 5 means they are completely equivalent. As our test set, we report the average Pearson’s \(r\) over each year of the STS tasks from 2012-2016 as is convention.

Most work evaluating accuracy on STS tasks has averaged the Pearson’s \(r\) over each individual dataset for each year of the STS competition. However, [1] computed Spearman’s \(\rho\) over concatenated datasets for each year of the STS competition. To be consistent with previous work, we re-ran their model and calculated results using the standard method, and thus our results are not the same as those reported [1]. However, we also include results using their approach for completeness. One other difference between these two ways of calculating the results is the inclusion of the SMT dataset of the 2013 task, which we also exclude when replicating the approach in [1].

For cross-lingual semantic similarity and semantic similarity in non-English languages, we evaluate on the STS tasks from SemEval 2017. This evaluation contains Arabic-Arabic, Arabic-English, Spanish-Spanish, Spanish-English, and Turkish-English datasets. The datasets were created by translating one or both pairs of an English STS pair into Arabic (ar), Spanish (es), or Turkish (tr). Following convention, we report results with Pearson’s \(r\) for all systems, but also include results in Spearman’s \(\rho\) for LASER, LaBSE, and P-SP.

We also evaluate on the Tatoeba bitext mining task introduced by [13]. The dataset consists of up to 1,000 English-aligned sentence pairs for over 100 languages. The aim of the task is to find the nearest neighbor for each sentence in the other language according to cosine similarity. Performance is measured by computing the error rate.

6 Results↩︎

6.0.0.1 English Semantic Similarity.

The results for our English semantic similarity evaluation are shown in Table 2. Our P-SP model has the best performance across each year of the task, significantly outperforming all prior work. We outperform methods that use large pre-trained models including Sentence-BERT which is supervised, as it is trained on NLI data [31].

We also include results from SimCSE [32]. We compare to the unsupervised version, since our model is also unsupervised. We evaluate using the Spearman’s \(\rho\) of the concatenation of the datasets for each year, and find our average performance over the 2012-2016 datasets to be 76.9, compared to 77.4 and 77.9 for the RoBERTa-base [54] and RoBERTa-large versions of SimCSE. While our performance is slightly lower, we note that they tune their model on the dev set of the STS Benchmark [49], which contains a subset of the data from STS tasks which we use for evaluation. Therefore, they are tuning on a subset of the evaluation data, and it is unclear how tuning on this test data affects model performance.

6.0.0.2 Cross-Lingual Semantic Similarity.

The results for the non-English and cross-lingual semantic similarity evaluation are shown in Table 3. From the results, our model again outperforms all prior work using sentence embeddings. The only systems that have better performance are the top (non-embedding based) systems from SemEval 2017 for Spanish-English.¹⁵

6.0.0.3 Bitext Mining.

1.5pt

The results on the Tatoeba bitext mining task from [13] are shown in Table 4. The results show that our embeddings are competitive, but have slightly higher error rates than LASER. The models are so close that the difference in error rate for the two models across the 6 evaluations is 0.2, corresponding to a difference of about 2 mismatched sentence pairs per dataset. LaBSE performs a bit better, but was trained on much more data then both LASER and our method. We also compare to mBART, XLM-R, and CRISS.¹⁶

This bitext mining result is in contrast to the results on cross-lingual semantic similarity, suggesting that our embeddings account for a less literal semantic similarity, making them more adept at detecting paraphrases but slightly weaker at identifying translations. It is also worth noting that LASER was trained on Tatoeba data outside the test sets, which could also account for some of the slight improvement over our model.

7 Speed Analysis↩︎

We analyze the speed of our models as well as selected popular sentence embedding models from prior work. To evaluate inference speed, we measure the time required to embed 120,000 sentences from the Toronto Book Corpus [55]. Preprocessing of sentences is not factored into the timing, and each method sorts the sentences by length prior to computing the embeddings to reduce padding and extra computation. We use a batch size of 64 for each model. The number of sentences embedded per second is shown in Table 5.

From the results, we see that our model is easily the fastest on GPU, sometimes by an order of magnitude. Interestingly, using a single core of CPU, we achieve similar speeds to inference on GPU, which is not the case for any other model. Moreover, we repeated the experiment, this time using 32 cores and achieved a speed of 15,316 sentences/second. This is even faster than when using a GPU and indicates that our model can effectively be used at scale when GPUs are not available. It also suggests our model would be appropriate for use on embedded devices.

8 Conclusion↩︎

In this paper, we present a system for the learning and inference of paraphrastic sentence embeddings in any language for which there is paraphrase or bilingual parallel data. Additionally, we release our trained sentence embedding models in English, as well as Arabic, German, Spanish, French, Russian, Turkish, and Chinese. These models are trained on tens of million of sentence pairs resulting in models that achieve state-of-the-art performance on unsupervised English semantic similarity and are state-of-the-art or competitive on non-English semantic similarity, cross-lingual semantic similarity, and bitext mining.

Moreover, our models are significantly faster than prior work owing to their simple architecture. They can also be run on CPU with little to no loss in speed from running them on GPU—-something that no strong models from prior work are able to do. Lastly, we release our code that has been modified to make training and inference easier, with support for training on large corpora, preprocessing paraphrase and bilingual corpora and evaluation data, as well as scripts for easy inference that can generate embeddings or semantic similarity scores for sentences supplied in a text file.

References↩︎