1 Introduction↩︎

Pre-trained language models [2], [3] have been proven valuable tools for contextual representation extraction. Many studies have shown their effectiveness in discovering linguistic structures [4], which is useful for a wide variety of NLP tasks [5]–[7]. These positive results led to further exploration of (i) cross-lingual pre-training [1], [8], [9] through the use of multiple mono-lingual and parallel resources, and (ii) visual pre-training where large-scale image captioning corpora are used to induce grounded vision & language representations [10]–[14]. The latter is usually achieved by extending the masked language modelling (MLM) objective [3] with auxiliary vision & language tasks such as masked region classification and image sentence matching.

In this paper, we present the first attempt to bring together cross-lingual and visual pre-training. Our visual translation language modelling (VTLM) objective combines the translation language modelling (TLM) [1] with masked region classification (MRC) [13], [15] to learn grounded cross-lingual representations. Unlike most of the prior work that use classification or retrieval based downstream evaluation, we focus on the generative task of multimodal machine translation (MMT), where images accompany captions during translation [16]. Once pre-trained, we transfer the VTLM encoder to a Transformer-based  [17] MMT and fine-tune it for the MMT task. To our knowledge, this is also the first attempt of pre-training & fine-tuning for MMT, where the current state of the art mostly relies on training multimodal sequence-to-sequence systems from scratch [18]–[23].

Our findings highlight the effectiveness of cross-lingual visual pre-training: when fine-tuned on the English\(\rightarrow\)German direction of the Multi30k dataset [24], our MMT model surpasses our constrained MMT baseline by about \(10\) BLEU and \(8\) METEOR points. The rest of the paper is organised as follows: \(\S\)2 describes our pre-training and fine-tuning protocol, \(\S\)3 presents our quantitative and qualitative analyses, and \(\S\)4 concludes the paper with pointers for future work.

2 Method↩︎

We propose Visual Translation Language Modelling (VTLM) objective to learn multimodal cross-lingual representations. In what follows, we first describe the TLM objective [1] and then introduce the modifications required to extend it to VTLM.

2.1 Translation language modelling↩︎

The TLM objective is based on Transformer networks and assumes the availability of parallel corpora during training. It defines the input \(x\) as the concatenation of \(m\)-length source language sentence \(s^{(1)}_{1:m}\) and \(n\)-length target language sentence \(s^{(2)}_{1:n}\): \[x=\left[s^{(1)}_1,\cdots,s^{(1)}_m,s^{(2)}_1,\cdots,s^{(2)}_n\right]\label{eq:tlm95inp}\tag{1}\]

For a given input, TLM follows [3], and selects a random set of input tokens \(y=\{s^{(l)}_1,\dots,s^{(l)}_k\}\) for masking. Let us denote the masked input sequence with \(\tilde{x}\), and the ground-truth targets for masked positions with \(\hat{y}\). TLM employs the masked language modelling (MLM) objective to maximise the log-probability of correct labels \(\hat{y}\), conditioned on the masked input \(\tilde{x}\): \[\mathcal{L} = {\frac{1}{\abs{\mathcal{X}}}}\sum_{x\in\mathcal{X}}\log\Pr(\hat{y}{\vert}\tilde{x}; \theta)\label{eq:loss}\tag{2}\] where \(\theta\) are the model parameters. We keep the standard hyper-parameters for masking, i.e. \(15\%\) of inputs are randomly selected for masking, from which \(80\%\) are replaced with the [MASK] token, \(10\%\) are replaced with random tokens from the vocabulary, and \(10\%\) are left intact.

2.2 Visual translation language modelling↩︎

VTLM extends the TLM by adding the visual modality alongside the translation pairs (Figure 1). Therefore, we assume the availability of sentence pair & image triplets and redefine the input as: \[x\mathrm{=}\left[s^{(1)}_1,\cdots,s^{(1)}_m,s^{(2)}_1,\cdots,s^{(2)}_n,\textcolor{purple}{v_1,\cdots,v_o}\right]\label{eq:vtlm95inp}\tag{3}\] where \(\{v_1,\cdots,v_o\}\) are features extracted from a Faster R-CNN model [25] pre-trained on the Open Images dataset [26].¹ Specifically, we extract convolutional feature maps from \(o=36\) most confident regions, and average pool each of them to obtain a region-specific feature vector \(v_i \in \mathbb{R}^{1536}\). Each region \(i\) is also associated with a detection label \(\hat{v}_i\) provided by the extractor. Before encoding, the feature vectors and their bounding box coordinates are projected into the language embedding space.

The final model processes translation pairs and projected region features in a single-stream fashion [12], [13], and combines the TLM loss with the masked region classification (MRC) loss as follows: \[\mathcal{L} = {\frac{1}{\abs{\mathcal{X}}}}\sum_{x\in\mathcal{X}}\log\Pr(\{\hat{y}, \textcolor{purple}{\hat{v}}\}{\vert}\tilde{x}; \theta)\label{eq:vtlm95loss}\tag{4}\]

2.2.0.1 Masking.

\(15\%\) random masking ratio is applied separately to both language and visual streams, and the \(\hat{v}\) above now denotes the correct region labels for the masked feature positions. Different from previous work that zeroes out masked regions [11], [13], VTLM replaces their projected feature vectors with the [MASK] token embedding.² Similar to textual masking, \(10\%\) of the random masking amounts to using regional features randomly sampled from all images in the batch, and the remaining \(10\%\) of regions are left intact.

2.3 Pre-training↩︎

VTLM requires a three-way parallel multimodal corpus, which does not exist in large-scale. To address this, we extend³ the Conceptual Captions (CC) [27] dataset with German translations. CC is a large-scale collection of \(\sim\)​3.3M images retrieved from the Internet, with noisy alt-text captions in English. The translation of English captions into German was automatically performed using an existing NMT model [28] provided⁴ in the Fairseq [29] toolkit. Since some of the images are no longer accessible, the final corpus’ size is reduced to \(\sim\)​3.1M triplets. We used byte pair encoding (BPE) [30] to learn a joint 50k BPE model on the CC dataset. The pre-training was conducted for \(1.5\)M steps, using a single RTX2080-Ti GPU, and best checkpoints were selected with respect to validation set accuracy.

2.3.0.1 Settings.

We use a small version of the TLM [1]⁵ and set the model dimension, feed-forward layer dimension, number of layers and number of attention heads to \(d=512\), \(f=2048\), \(l=6\) and \(h=8\), respectively. We randomly initialise model parameters, instead of using pre-trained LM checkpoints such as BERT or XLM. We use Adam [31] with the mini-batch size and the learning rate set to \(64\) and \(0.0001\), respectively. The dropout [32] rate is set to \(0.1\) in all layers. The pre-training is done for \(1.5\)M steps using a single RTX2080-Ti GPU, and best checkpoints are selected with respect to validation accuracy.

2.4 Baseline MT models and fine-tuning↩︎

Our experimental protocol consists of initialising the encoder and the decoder of Transformer-based NMT and MMT models with weights from TLM/VTLM, and fine-tuning them with a smaller learning rate. The architectural difference between the NMT and the MMT models is that the latter encodes \(36\) regional visual features as part of the source sequence, similar to the VTLM (\(\S\) 2.2). As a natural baseline, we train constrained (trained only on the MT dataset) models without transferring weights from the pre-trained TLM/VTLM models. We refer to these models as from-scratch. For the fine-tuning experiments, we train three runs with different seeds. For evaluation, we use the models with the lowest validation set perplexity to decode translations with beam size equal to 8.

2.4.0.1 Dataset.

We use the standard MMT corpus Multi30k [24] for both fine-tuning and from-scratch runs. It contains 30k image descriptions from Flickr30k [33] and their human translations in German for training, along with three test sets of 1K samples each: the original and the most in-domain 2016 test set, as well as 2017 and COCO test sets created using images and descriptions collected from sources other than Flickr.

2.4.0.2 Settings.

For fine-tuning, we use the same hyper-parameters as the pre-training phase, apart from decreasing the learning rate to \(1e\rm{-}5\). For MT models that are trained from scratch, we increase the dropout rate to \(0.4\) and linearly warm up the learning rate from \(1e\rm{-}7\) to \(1e\rm{-}4\) during the first 4,000 iterations. Inverse square-root annealing is applied after 4,000 iterations.

3 Results↩︎

3.1 Machine translation↩︎

Table 1 reports Meteor and Bleu scores across three different test sets of Multi30k. First, we observe that the MMT system trained from scratch is consistently worse than its NMT counterpart. However, the gap disappears when pre-trained TLM/VTLM checkpoints are fine-tuned for MT. This suggests that pre-training may be necessary for single-stream multimodal encoding, where the number of regions (\(36\)) outnumbers the avg. number of source tokens (\(13\) for Multi30k).

Second, we see that the best performances are obtained when models are first pre-trained on the three-way parallel Conceptual Captions (CC) dataset. To validate this further, we train a baseline NMT on the concatenation of Multi30k and CC (NMT+CC) and an MMT that uses only Multi30k for both pre-training and fine-tuning. The results clearly show that these systems lag behind the ones pre-trained on CC.

We also experimented with an alternative pre-training strategy where we still have the MRC task to predict the object labels of randomly selected visual regions but the input regional features for those positions are not actually replaced with [MASK] . In other words, we let the model predict the object labels using a cross-lingual and multimodal input where only input words are randomly masked. Interestingly, this alternative MMT (Table 1) reveals that not masking visual regions during pre-training yields slightly better results overall. Overall, MMT fine-tuning on VTLM sets a new state of the art across all Multi30k test sets.⁶ We leave the exploration of visual region masking for the MRC task as future work and proceed with the alternative variant in the following experiments.

3.1.0.1 Encoder attention parameters.

When fine-tuning the TLM for MT, the default XLM implementation randomly initialises the decoder’s missing encoder attention parameters. In our experiments, we noticed that copying those parameters from the TLM self-attention layers substantially improves the results up to \(2.2\) BLEU.

3.2 Explicit masking↩︎

Here, we will evaluate the extent to which the visual information is taken into account (i) when TLM/VTLM predicts masked tokens, and (ii) when the fine-tuned NMT and MMT models are forced to translate source sentences with missing visual entities. For the latter, we use Flickr30k entities [35] to mask head nouns in 2016 test set sentences, similar to [36].

3.2.0.1 Last-word masking.

In this experiment, we measure the target word prediction accuracy, when last tokens⁷ of input caption pairs are systematically masked during evaluation. Table 2 suggests that the visual information is much more helpful (i.e. up to 6% accuracy improvement) when last tokens are masked in both English and German captions. However, if one caption is available, it provides enough context for cross-lingual prediction. Finally, when we shuffle (+shuf) the test set features to introduce incongruence [37], we see that the VTLM model deteriorates substantially. This confirms that the accuracy improvements are not due to side-effects of experimentation noise, such as regularisation or random seed related effects.

3.2.0.2 Entity masking in MT.

We devise two ways of masking entities i.e. we either replace them with the [MASK] token or remove them entirely so that the masking phenomena is not known to the model. The results in Table 3 show that MMT models can recover the missing source context to some extent, only when they are pre-trained using the proposed VTLM objective. In other words, the grounding ability can only be acquired when visual modality is present for both pre-training and fine-tuning. The gap between Mask and Remove also seems to highlight the importance of reserving a source position even it is corrupted/masked.

3.3 Visual attention in MMT↩︎

Here we take the MMT decoder’s cross-attention layers and measure the attention mass they attribute to regional features in the input embeddings. Although the encoder’s self-attention layers produce increasingly mixed contextual embeddings as we move towards the top layers, [38] show that the final layer states still encode corresponding input embeddings to some extent. With this assumption at hand, Figure 2 shows the average attention mass attributed to the first \(36\) (visual) top-layer encoding states, by each cross-attention layer in the decoder. We find these results to be in agreement with the quantitative metrics (Table 1), with VTLM-MMT assigning substantially more attention to these positions, compared to TLM-MMT and MMT from scratch.

4 Conclusions↩︎

We proposed a novel cross-lingual visual pre-training approach and tested its efficacy for multimodal machine translation. Our pre-training approach extends the TLM framework [1] with regional features and performs masked language modelling and masked region classification on a three-way parallel corpus. We show that this leads to substantial improvements compared to multimodal machine translation with cross-lingual pre-training only or without pre-training at all. As future work, we consider exploring more informed masking strategies for visual regions and investigating the impact of visual masking probability for the MRC pre-training task for downstream MMT performance.

Acknowledgments↩︎

This work was supported in part by TUBA GEBIP fellowship awarded to Erkut Erdem, and the MMVC project funded by TUBITAK and the British Council via the Newton Fund Institutional Links grant programme (grant ID 219E054 and 352343575). Lucia Specia, Pranava Madhyastha and Ozan Caglayan also received support from MultiMT project (H2020 ERC Starting Grant No. 678017) and Lucia Specia from the Air Force Office of Scientific Research (under award number FA8655-20-1-7006).

References↩︎