1 Introduction↩︎

Pre-trained vision-and-language (V&L) models improve the performance of tasks that require an understanding of the V&L grounding, including visual question answering [1], referring expression comprehension [2], and image-text matching (ITM) [3]. Recent V&L tasks, such as multi-modal reading comprehension [4]–[7] and dialogue [8]–[10], require a deeper NLU as well as the grounding. Extending pre-trained language models (LMs) is an option for those tasks as this allows V&L models to inherit language knowledge from their source LMs. The typical extending consists of visual pre-training and structure such as the stream type; the single-stream inserts vision tokens into the input sequence of the LM, and the dual-stream uses another sequence for early visual encoding.

One of the remaining challenges is to understand how such extensions affect the pre-trained language knowledge. For example, [11] proposed the dual-stream model where part of the goal was to protect the learned LMs. The authors focused on evaluation with V&L tasks and did not evaluate their models with language-only tasks. [12] evaluated the extent of language knowledge loss in the single/dual-stream models against the source LM using language-only tasks. However, the difference between single-stream and dual-stream models was unclear because the pre-training was also different in their models.

In this paper, we investigate the effect of visual extensions in V&L models on language-only tasks¹. [13] proposed a framework to unify transformer-based V&L models and compared some single/dual-stream models in the same setup. Based on their work, our study shows how these structural differences affect the performance of NLU using the GLUE [14] tasks.

In our experiments, fine-tuning of pre-trained V&L models shows that both single/dual-stream models perform worse than the source LM and that single-stream models perform slightly better than dual-stream models. Further, we fine-tune the models created by only structural modifications without pre-training. We observe that the single/dual modification alone has little effect on the GLUE scores, indicating the performance degradation is primarily caused by pre-training. We also see how the V&L models changed from the source LM by analyzing the changes in the model parameters and the problem sets that each model can solve. Our results suggest that it would be more effective to adopt a single stream, and devise pre-training strategies for maintaining language knowledge.

2 Controlled V&L Models↩︎

In this section, we describe the pre-trained V&L models used in our experiments. [13] proposed a framework for V&L models that consider a sequence of tokens in sentences as language information, and a sequence of recognized object regions as visual information. In their framework, they reproduced five existing models, VisualBERT [15], Uniter [16], VL-BERT [17], ViLBERT [11], and LXMERT [18], and made their controlled versions by modifying some parts for a fairer and easier comparison. We use these controlled versions.

2.1 Structural Modification↩︎

We describe streams and embeddings, which are the basic factors of the model structures. We summarize the model structures in the controlled setup used in this experiment in Table 1.

2.1.0.1 Streams.

V&L models can be divided into two categories based on how the vision and language sequences are encoded. Single-stream models, VisualBERT, Uniter, and VL-BERT, jointly process the vision and language sequences in a single encoder. Dual-stream models, ViLBERT and LXMERT, encode those sequences separately before encoding them jointly. ViLBERT is an early example of the dual-stream models and was proposed mainly to account for the differences in abstraction levels between vision and language, and to protect learned language models. In the controlled setup of [13], the stream type is identical to the original one in all models.

2.1.0.2 Embeddings.

The major difference in embeddings is the use of global visual feature. The original VisualBERT, Uniter, and LXMERT do not use the global visual feature. ViLBERT has a token that represents the global visual feature at the beginning of vision sequences. VL-BERT inserts the global visual feature to the last of vision sequences and also adds the global visual feature to each token embedding in the language sequence. Object location is also expressed differently. The original VL-BERT and LXMERT use four attributes (left, top, right, bottom). In addition to the four attributes, the original ViLBERT uses area, and the original Uniter uses width, height, and area. VisualBERT does not use location information².

The controlled setup is based on the structure of ViLBERT. For the global image feature, the setup inserts the average of vision tokens to the head of the vision sequence for all models. In addition to inserting the global visual feature, the controlled VL-BERT adds it to the respective tokens in the language sequence. For location, VisualBERT’s setup that do not use location information remain the same, while the other models use the five attributes. The five attributes are normalized by width or height. Another point is the token type for the vision tokens. In the controlled setup, the token type is not added for ViLBERT and LXMERT because they have separate streams. Of the single-stream models, VisualBERT and Uniter use BERT’s token type ID to specify vision tokens, while VL-BERT adds a new embedding to represent vision tokens.

2.2 Pre-training↩︎

We summarize the pre-training used in the controlled setup to train the five model structures described above. Note that we omit the detail of the pre-training used in each original paper here.

The five models were pre-trained on Google’s Conceptual Captions [19] corpus, which was collected from Web images and their alt-text HTML attributes. The corpus was filtered before training, and the size was approximately 2.7 M pairs as a result. Three tasks, masked language modelling (MLM), masked object classification (MOC), and ITM, were made from image-text pairs in the corpus. Given an image-text pair, the model predicts masked language tokens for MLM, the object class of masked vision tokens for MOC, and whether the pair is correct or not for ITM.

The weights of the five models were initialized with the pre-trained weights of BERT\(_{\rm BASE}\) if the corresponding weights were in BERT\(_{\rm BASE}\); otherwise (e.g., the weights of the vision encoder in dual-stream models), they were initialized randomly.

3 Experiment with GLUE↩︎

3.1 Datasets↩︎

The GLUE benchmark [14] is a collection of diverse tasks for studying NLU systems. It consists of nine tasks: CoLA [20], SST-2 [21], MRPC [22], QQP³, STS-B [23], MNLI [24], QNLI [25], RTE [26]–[29], and WNLI [30]. STS-B is a single-valued regression task, and the others are classification tasks. We train the controlled pre-trained models on the training sets and evaluate them with the development sets. Figure 1 (left) shows the number of the training sentences in the corpora and their word overlap between the corpus used in the V&L pre-training.

3.2 Implementation Details↩︎

We fine-tuned pre-trained models published by [13] ⁴. To use a script for the GLUE benchmark, we modified the model codes for Huggingface’s Transformers [31]⁵. We used the BERT-uncased tokenizer to tokenize sentences.

3.2.0.1 Image inputs.

Because the GLUE tasks have no image input, we used a black image (of \(224 \times 224\) pixels) in our experiments. We followed the method of [13] for image processing; we input the images to the Faster R-CNN object detector [32] trained for the Bottom-Up and Top-Down model [33], and used the top 36 detected results (bounding boxes and feature vectors) as vision tokens⁶. We used the average of the vision tokens as a global visual token. Those vision tokens were fixed and used for both training and evaluation in all models.

In this study, we tried image completion with black images for tasks where no image is provided as a simple way to preserve the input format used in pre-training. However, there are many possible methods for complementing the image input. For example, a method as simple as the present one can use other images, a noise input, or learnable parameters. Examining the impact of image input completion methods remains as future work.

3.2.0.2 Head for classification.

We adopted the method used in [13] for V&L tasks. We used a learnable linear layer to calculate the likelihood of document classes, such as entailment/neutral/contradiction. We input the element-wise product of two vectors made from the model’s output sequence into the linear layer. For those two vectors, we pooled the portions of the model’s output sequence that correspond to the vision input and to the language input, respectively, by taking the first token of each portion. This corresponds to taking the outputs of the [CLS] token (in the language sequence) and the global visual token.

3.2.0.3 Hyperparameters for fine-tuning.

We used a batch size of 64 and Adam for optimization. The learning rate was initialized at 2e-5 and decreased linearly. We trained for five epochs, evaluating the loss on the dev sets at the end of each epoch. Finally, we adopted the model with the lowest loss.

3.3 Overall Result↩︎

Table 2 shows the results of the GLUE benchmark. In our experiment, we fine-tuned five V&L models and their source language model–BERT\(_{\rm BASE}\). We also cited the BiLSTM baseline from the GLUE paper. The Glue avg of five V&L models decrease compared to BERT\(_{\rm BASE}\). We can see a trend where the single-stream models perform slightly better than the dual-stream models. Note that this trend is consistent with the results of [12] for linguistic probing of the original Uniter and LXMERT. Although the difference is small, this suggests that the single-stream models can maintain more of BERT\(_{\rm BASE}\)’s knowledge.

3.3.0.1 Performance of each task.

V&L models perform lower than the BiLSTM baseline for some tasks, including MRPC, RTE, and WNLI. Figure 1 (right) shows the correlation between the word overlap between the corpus for pre-training and the GLUE task corpora and the GLUE score. We can see a positive correlation between those two variables. Although we do not conclude clearly because word overlap and the number of training data also correlate, word overlap could have a large impact on task performance.

4 Analysis↩︎

4.1 Amount of Change in Parameters↩︎

We expected the model inference to be closer to BERT’s inference if a model has parameters closer to BERT. Therefore, we calculated the cosine similarity of the corresponding parameters between pre-trained models and BERT to indicate the degree to which the parameters had changed. Table 3 shows the averaged cosine similarity. We flattened parameters and calculated their similarity as vectors. We can see that the parameters of the single-stream and dual-stream models changed by the same extent. This suggests that separating streams alone may not be sufficient for knowledge maintenance.

4.2 Breakdown of Classification Results↩︎

Table 4 shows the results of aggregating the GLUE classification task problems into four categories: solvable by both BERT\(_{\rm BASE}\) and V&L models, BERT\(_{\rm BASE}\) only, V&L model only, and neither model. We defined success in a given problem as answering correctly in at least two out of three experimental runs. To make Table 4, we first calculated the tables of successful models for each GLUE task and V&L model and second averaged the tables for the tasks. For all five models, there are approximately 5% of problems that they only can solve. This category shows the positive impact of V&L pre-training on NLU. Problems that both models can solve tended to be more common for the single-stream models. This supports the finding that these models retain more language knowledge.

The difference of corpora for the last pre-training between BERT (mainly English Wikipedia) and the V&L models (images’ alt-texts) might affect the complexity of the sentences in the problem sets that can be solved only by BERT and only by the V&L models. Thus, we analyzed the distributions of some metrics (sentence length, readability). However, we found no significant difference between the two sets in each model. We show the distributions in Appendix 8.

4.3 Language and V&L Tasks↩︎

The last column of Table 2 shows the V&L scores for the V&L models. We calculated these scores by averaging the results on the five V&L tasks reported in [13]. Their tasks cover four groups widely used to test V&L models: VQA, image–text retrieval, referring expressions, and multi-modal verification. Comparing the V&L and GLUE scores, we cans see that no model is best in both respects at the same time. There is room for improvement in the V&L extension.

4.4 Structural Modification or Pre-training: Which Has the Greater Impact?↩︎

To further analyze the impact of structural modification, we fine-tuned models with only structural modifications (Mod. only). Table 5 shows a comparison between the GLUE scores of the Mod-only models and the full models (Mod+V&L-PT). Except for VL\(_{\rm CTRL}\), the Mod-only models achieve a score comparable to BERT\(_{\rm BASE}\), and the GLUE score decreases for the Mod+V&L-PT models. The fact that the structural modification preserves the score of the GLUE tasks in most cases suggests that the main factor for the drop in the GLUE tasks is V&L pre-training. This observation emphasizes the impact of pre-training on maintaining the language knowledge. Note that a possible reason for the exception of VL\(_{\rm CTRL}\) is that the global visual feature added to the language embeddings may break the language knowledge.

5 Discussion and Conclusion↩︎

The number of V&L model works that focus on both V&L tasks and language-only tasks has increased [34]–[37]. [34] proposed a V&L neural architecture and trained it on a language model in a visual context. They demonstrated that their architecture outperforms its equivalent trained on language alone in perplexity and stated that language is inseparable from its physical context. Although it is not clear whether methods that improve the perplexity of language modeling can also apply to maintain the performance of downstream tasks, the strategy of improving models with reference to human cognition would be an important direction. More recently, [36] achieved better performance on language-only tasks than their base model with pre-training on three types of corpora (text, image, and image–text pairs) at the same time. [35] reported that adding separated extractors for vision and language on top of a single-stream encoder can help maintain language knowledge.

In this paper, we fine-tuned V&L models extended from a language model (LM) to an NLU benchmark to compare their NLU performance. We used five V&L models, including single-stream and dual-stream models, pre-trained in the same setup. The benchmark scores of those models decreased compared with their source LM. We also found that the single-stream models tended to retain (slightly) more language knowledge than the dual-stream models, and that the main cause of the drop in the NLU tasks can be pre-training. Our observations suggest that adopting a single stream and devising pre-training strategies could be effective, at least for preserving the language knowledge.

Acknowledgements↩︎

We would like to thank Takuma Udagawa, Taku Sakamoto, and the anonymous reviewers for their insightful comments. This work was supported by JSPS KAKENHI Grant Number 21H03502.

6 Dataset Statistics↩︎

7 Additional Data for Overall Results↩︎

Table ¿tbl:table:overall46sd? shows the SDs to the averaged scores of V&L models on the GLUE tasks’ development sets.

8 Additional Data for Analysis↩︎

We show the distributions of sentence length and readability mentioned in Section 4.2 in Figure 2 and Figure 3, respectively.

References↩︎