1 Introduction↩︎

Joint visual-semantic embeddings (VSE) are central to the success of many vision-language tasks, including cross-modal search and retrieval [1]–[3], visual question answering [4], [5], multimodal machine translation [6], [7], etc. Learning VSE requires extensive understanding of the content in individual modalities and an in-depth alignment strategy to associate the complementary information from multiple views.

With the availability of large-scale parallel English-Image corpora [8], [9], a rich line of research has advanced learning VSE under the monolingual setup. Most recent works [1], [2], [10]–[14] leverage triplet ranking losses to align English sentences and images in the joint embedding space. In VSE++ [15], Faghri et al. improve VSE by emphasizing hard negative samples. Recent advancement in VSE models explores methods to enrich the English-Image corpora. propose to augment dataset with textual contrastive adversarial samples to combat adversarial attacks. Recently,   utilize textual semantics of regional objects and adversarial domain adaptation for learning VSE under low-resource constraints.

An emerging trend generalizes learning VSE in the multilingual scenario.   learn \(M\)-view representations when parallel data is available only between one pivot view and the rest of views. PIVOT [16] extends the work from  to use images as the pivot view for learning multilingual multimodal representations.   further confirms the benefits of multilingual training.

Our work is motivated by  but has important differences. First, to disentangle the alignments in the joint embedding space, we employ visual object detection and multi-head attention to selectively align salient visual objects with textual phrases, resulting in visually-grounded multilingual multimodal representations. Second, as multi-head attention [17] is appealing for its efficiency and ability to jointly attend to information form different perspectives, we propose to further encourage the diversity among attention heads to learn an improved visual-semantic embedding space. Figure 1 illustrates our gradient updates promoting diversity. The proposed model achieves state-of-the-art performance in the multilingual sentence-image matching tasks in Multi30K [18] and the semantic textual similarity task [19]–[21].

2 Related Works↩︎

Classic attention mechanisms have been addressed for learning VSE. These mechanisms can be broadly categorized by the types of {Query, Key, Value} as discussed in  . For intra-modal attention, {Query, Key, Value} are within the same modality. In DAN [22], the content in each modalities is iteratively attended through multiple steps with intra-modal attention. In SCAN [23], inter-modal attention is performed between regional visual features from  and text semantics. The inference time complexity is \(O(MN)\) (for generating \(M\) query representations for a size \(N\) datatset). In contrast to the prior works, we leverage intra-modal multi-head attention, which can be easily parallelized compared to DAN and is with a preferred \(O(M)\) inference time complexity compared to SCAN.

Inspired by the idea of attention regularization in , for learning VSE, we propose a new margin-based diversity loss to encourage a margin between attended outputs over multiple attention heads. Multi-head attention diversity within the same modality and across modalities are jointly considered in our model.

3 The Proposed Model↩︎

Figure 1 illustrates the overview of the proposed model. Given a set of images as the pivoting points with the associate English and German¹ descriptions or captions, the proposed VSE model aims to learn a multilingual multimodal embedding space in which the encoded representations \((v, e, g)\) of a paired instance \((i, x^e, x^g)\) are closely aligned to each other than non-paired ones.

Encoders: For a sampled \((i, x^e, x^g)\) pair, we first encode the tokens in the English sentence \(x^e=\{x_1^e, \dots, x_N^e\}\) and the tokens in the German sentence \(x^g=\{x_1^g, \dots, x_N^g\}\) through the word embedding matrices followed by two bi-directional LSTMs. The outputs of the textual encoders are \(e=\{e_1, \dots, e_N\}, e_n \in \mathbb{R}^H\) for English and \(g=\{g_1, \dots, g_N\}, g_n \in \mathbb{R}^H\) for German, where \(N\) is the max sentence length and \(H\) is the dimension of the shared embedding space. For the image, we leverage a Faster-RCNN [24] network with a ResNet [25] backbone to detect and encode salient visual objects in the image. With a trainable one-layered perceptron to transform visual features into the shared embedding space, we encode the image as \(v = \{v_1,\dots,v_M\}, v_m \in \mathbb{R}^H\), where \(M\) is the maximum amount of visual objects in an image.

Multi-head attention with diversity: We employ \(K\)-head attention networks to attend to the visual objects in an image as well as the textual semantics in a sentence then generate fixed-length image/sentence representations for alignment. Specifically, the \(k\)-th attended German sentence representation \(g^k\) is computed by: \[\begin{align} a_{i}^k &= \text{softmax}(\text{tanh}(W^k_{c_g} c_g^k)^\top \text{tanh}(W^k_g g_i))\\ g^k &= \sum_{i=1}^{N} a_i^k g_i\label{mattn}, \end{align}\tag{1}\] where \(a_i^k\) is the attention weight, \(W^k_g \in \mathbb{R}^{H \times H_{attn}} , W^k_{c_g} \in \mathbb{R}^{H_c \times H_{attn}}\) is the learnable transformation matrix for German. \(c_g^k \in \mathbb{R}^{H_c}\) is the learnable \(H_c\)-dimensional contextual vector for distilling important semantics from German sentences. The final German sentence representation is the concatenation of \(K\)-head attention outputs \(g=[g^0\|g^1\| \dots \| g^K]\). Similar for encoding the English sentence \(e=[e^0\|e^1\| \dots \| e^K]\) and the image \(v=[v^0\|v^1\| \dots \| v^K]\).

With \(\{V,E,G\}\) where \(v \in V, e \in E, g \in G\) as the set of attended fixed-length image and sentence representations in a sampled batch, we use the widely-used hinge-based triplet ranking loss with hard negative mining  [15] to align instances in the visual-semantic embedding space. Taking Image-English instances \(\{V,E\}\) as an example, we leverage the triplet correlation loss defined as: \[\begin{align} l_{\theta}(V,E) =& \sum_{p} \big[\alpha - s(v_p,e_p) + s(v_p,\hat{e}_p)\big]_+ \\ + &\sum_{q} \big[\alpha - s(v_q,e_q) + s(\hat{v}_q,e_q)\big]_+, \end{align}\] where \(\alpha\) is the correlation margin between positive and negative pairs, \([.]_+=\text{max}(0,.)\) is the hinge function, and \(s(a,b)=\frac{a^Tb}{\|a\|\|b\|}\) is the cosine similarity. \(p\) and \(q\) are the indexes of the images and sentences in the batch. \(\hat{e}_p=\text{argmax}_q s(v_p,e_{q \neq p})\) and \(\hat{v}_q=\text{argmax}_p s(v_{p \neq q}, e_q)\) are the hard negatives. When the triplet loss decreases, the paired images and German sentences are drawn closer down to a margin \(\alpha\) than the hardest non-paired ones. Our model aligns \(\{V,E\}\), \(\{V,G\}\) and \(\{E,G\}\) in the joint embedding space for learning multilingual multimodal representations with the sampled \(\{V,E,G\}\) batch. We formulate the overall triplet loss as: \[\label{loss95p} l_{\theta}(V,E,G)= l_{\theta}(V,G) + l_{\theta}(V,E) + \gamma l_{\theta}(G,E).\tag{2}\] Note that the hyper-parameter \(\gamma\) controls the contribution of \(l_\theta(G,E)\) since (\(e\), \(g\)) may not be a translation pair even though (\(e\), \(v\)) and (\(g\), \(v\)) are image-caption pairs.

One of the desired properties of multi-head attention is its ability to jointly attend to and encode different information in the embedding space. However, there is no mechanism to support that these attention heads indeed capture diverse information. To encourage the diversity among \(K\) attention heads for instances within and across modalities, we propose a new simple yet effective margin-based diversity loss. As an example, the multi-head attention diversity loss between the sampled images and the English sentences (i.e.  diversity across-modalities) is defined as: \[l^{D}_{\theta}(V,E)=\sum_{p} \sum_k \sum_{r} \big[\alpha_{D} - s(v_p^k,e_p^{k \neq r})]_+ \\\]

As illustrated with the red arrows for update in Figure 1, the merit behind this diversity objective is to increase the distance (up to a diversity margin \(\alpha_D\)) between attended embeddings from different attention heads for an instance itself or its cross-modal parallel instances. As a result, the diversity objective explicitly encourage multi-head attention to concentrate on different aspects of information sparsely located in the joint embedding space to promote fine-grained alignments between multilingual textual semantics and visual objects. With the fact that the shared embedding space is multilingual and multimodal, for improving both intra-modal/lingual and inter-modal/lingual diversity, we model the overall diversity loss as: \[\begin{align}\label{loss95d} l^{D}_{\theta}(V,E,G)&= l^{D}_{\theta}(V,V) + l^{D}_{\theta}(G,G) + l^{D}_{\theta}(E,E) \\ &+ l^{D}_{\theta}(V,E) + l^{D}_{\theta}(V,G) + l^{D}_{\theta}(G,E), \end{align}\tag{3}\] where the first three terms are intra-modal/lingual and the rest are cross-modal/lingual. With Eq. 2 and Eq. 3 , we formalize the final model loss as: \[l_{\theta}^{All}(V,E,G)=l_{\theta}(V,E,G)+\beta l^{D}_{\theta}(V,E,G),~\label{loss95all}\tag{4}\] where \(\beta\) is the weighting parameter which balances the diversity loss and the triple ranking loss. We train the model by minimizing \(l_{\theta}^{All}(V,E,G)\).

4 Experiments↩︎

Following , we evaluate on the multilingual sentence-image matching tasks in Multi30K [18] and the semantic textual similarity task [19], [21].

4.1 Experiment Setup↩︎

We use the model in  which is a Faster-RCNN [24] network pre-trained on the MS-COCO [8] dataset and fine-tuned on the Visual Genome [27] dataset to detect salient visual objects and extract their corresponding features. 1,600 types of objects are detectable. We then pack and represent each image as a \(36\times2048\) feature matrix where 36 is the maximum amount of salient visual objects in an image and 2048 is the dimension of the flattened last pooling layer in the ResNet [25] backbone of Faster-RCNN.

For the text processing, we lower-case, tokenize, and then truncate the maximum sentence length to 100. We use 300-dim word embedding matrices initialized either randomly or with pre-trained multilingual embeddings. (We use the multilingual version of FastText [28]). We also experiment incorporating the last layer of contextualized multilingual BERT embeddings [29] to replace the word embedding matrices as the textual input features for the bi-directional LSTMs.

For training, we sample batches of size 128 and train 20 epochs on the training set of Multi30K. We use the Adam [30] optimizer with \(2\times10^{-4}\) learning rate then \(2\times10^{-5}\) after 15-th epoch. Models with the best summation of validation R@1,5,10 are selected to generate image and sentence embeddings for testing. Weight decay is set to \(10^{-6}\) and gradients larger than \(2.0\) are clipped. We use 3-head attention (\(K=3\)) and the embedding dimension \(H=512\). The same dimension is shared by all the context vectors in the attention modules. Other hyper-parameters are set as follows: \(\alpha=0.2, \alpha_{D}=0.1, \beta=1.0\) and \(\gamma=0.6\).

4.2 Multilingual Sentence-Image Matching↩︎

We evaluate the proposed model in the multilingual sentence-image matching (retrieval) tasks on Multi30K: (i) Searching images with text queries (Sentence-to-Image). (ii) Ranking descriptions with image queries (Image-to-Sentence). English and German are considered.

Multi30K [18] is the multilingual extension of Flickr30K [9]. The training, validation, and testing split contain 29K, 1K, and 1K images respectively. Two types of annotations are available in Multi30K: (i) One parallel English-German translation for each image and (ii) five independently collected English and five German descriptions/captions for each image. We use the later. Note that the German and English descriptions are not translations of each other and may describe an image differently.

As other prior works, we use recall at \(k\) (R@\(k\)) to measure the standard ranking-based retrieval performance. Given a query, R@\(k\) calculates the percentage of test instances for which the correct one can be found in the top-\(k\) retrieved instances. Higher R@\(k\) is preferred.

Table 1 presents the results on the Multi30K testing set. The VSE baselines in the first five rows are trained with English and German descriptions independently. In contrast, PIVOT [16] and the proposed model are capable of handling multilingual input queries with single model. For a fair comparison with PIVOT, we also report the result of swapping Faster-RCNN with VGG as the visual feature encoder in our model.

As can be seen, the proposed models successfully obtain state-of-the-art results, outperforming other baselines by a significant margin. German-Image matching benefit more from joint training with English-Image pairs. The models with pre-trained multilingual embeddings and contextualized embeddings achieve better performance in comparison to randomly initialized word embeddings, especially for German. One explanation is that the degradation from German singletons is alleviated by the multi-task training with English and the pre-trained embeddings. While the model with BERT performs better in English, FastText is preferred for German-Image matching.

4.3 Semantic Textual Similarity Results↩︎

For semantic textual similarity (STS) tasks, we evaluate on the video task from STS-2012  [19] and the image tasks from STS-2014-15 [20], [21]. The video descriptions are from the MSR video description corpus [31] and the image descriptions are from the PASCAL dataset [32]. In STS, a system takes two input sentences and output a semantic similarity ranging from [0,5]. Following , we directly use the model trained on Multi30K to generate sentence embeddings then scaled the cosine similarity between the two embeddings as the prediction.

Table 2 lists the standard Pearson correlation coefficients \(r\) between the system predictions and the STS gold-standard scores. We report the best scores achieved by paraphrastic embeddings [26] (text only) and the VSE models in the previous section. Note that the compared VSE models are all with RNN as the text encoder and no STS data is used for training. Our models achieve the best performance and the pre-trained word embeddings are preferred.

4.4 Qualitative Results and Grounding↩︎

In Figure 2 we samples some qualitative multilingual text-to-image matching results. In most cases our model successfully retrieve the one and only one correct image. Figure 3 depicts the t-SNE visualization of the learned visually grounded multilingual embeddings of the \((v,e,g)\) pairs pivoted on \(v\) in the Multi30K testing set. As evidenced, although the English and German sentences describe different aspects of the image, our model correctly aligns the shared semantics (e.g. (“man”, “mann”), (“hat”, “wollmütze”)) in the embedding space. Notably, the embeddings are visually-grounded as our model associate the multilingual phrases with exact visual objects (e.g. glasses and ears). We consider learning grounded multilingual multimodal dictionary as the promising next step.

As limitations we notice that actions and small objects are harder to align. Additionally, the alignments tends to be noun-phrase/object-based whereas spatial relationships (e.g. “on”, “over”) and quantifiers remain not well-aligned. Resolving these limitations will be our future work.

5 Conclusion↩︎

We have presented a novel VSE model facilitating multi-head attention with diversity to align different types of textual semantics and visual objects for learning grounded multilingual multimodal representations. The proposed model obtains state-of-the-art results in the multilingual sentence-image matching task and the semantic textual similarity task on two benchmark datasets.

Acknowledgement↩︎

This research is supported in part by the DARPA grants FA8750-18-2-0018 and FA8750-19-2-0501 under AIDA and LwLL program. It is also supported by the IARPA grant via DOI/IBC number D17PC00340. We would like to thank the anonymous reviewers for their constructive suggestions.

References↩︎