1 Introduction↩︎

Large enterprises have several teams to create their content for the purpose of marketing, campaigning, or even maintaining a brand presence. Multimodal assets, particularly images are an integral part of this. The scale and the speed at which one needs to create and update content, especially to ensure personalization requires several resources, which in turn acts as a hindrance to the success of the enterprise. Opportunistic updates are critical for success in the current competitive marketing and advertising scenario. Ensuring that every multimodal asset associated with any piece of enterprise content has an appropriate caption and tag is not possible given the scale. While apparently a nuanced aspect of any image, the caption serves as the key information carrier of what the image is all about, in turn ensuring the right recall (ability to find) for the content that contains this image. If an image has a well-formed caption, that captures the context accurately – it also makes the document accessible. Making enterprise content accessible is an important metric for large organizations as they strive to be inclusive. The scale and the quick turn-around time demanded for the content creation cycle results in the lack of correct tagging and captions of images (multimodal assets), in turn an eventual lost of revenue and a target customer base. We propose a method towards automated context-aware captioning of images targeted to reduce the tedium and this critical gap in the enterprise authoring and content creation process.

Large-scale pretraining of language models ([1]; [2]) has witnessed great success in many downstream NLP tasks. This success has inspired multi modal pretraining for image-text, image-only, and video-text tasks. Currently, building unified models that jointly learn multiple vision-language tasks is gaining a lot of attention and has shown promising results on many VL tasks ([3], [4], [5], [6]).

The existing unified Vision-language models focus on tasks like image captioning [7], visual question answering [8], visual entailment [9], and image-text retrieval [10] that consider the image as a standalone entity. However, images are typically accompanied by text that adds additional meanings which are not utilized in these tasks as shown in Figure 1. Also, the same image can mean different things in different contexts. For example, a picture of a football player being emotional can mean they are celebrating a goal or are disappointed with their shot, depending on the context. Hence it is essential to consider the context of the image for understanding it completely.

Traditional image captioning models do not use contextual information. In news image captioning [11], the generated caption contains information extracted from both the news article and the image. The news image captioning task is a special subtask of context assisted image captioning task that uses the news article as the contextual information about the image. In our work, the task names- news image captioning and context assisted image captioning are hence used interchangeably. Existing pretrained VL models lack the ability to use contextual information as the pretraining tasks do not contain long text associated with image-text pairs. We propose a new unified VL model based on the One For All (OFA) model, with a focus on using the contextual information associated with the image for real-world problems like news image captioning.

As there are no existing VL classification task that uses contextual information, we introduce a new VL task called ‘Contextual Visual Entailment’. Visual entailment [9] is a refined image-text matching task that checks for the entailment of the caption with the premise image. Visual entailment deals with only the descriptive characteristics of the image. In our contextual visual entailment task, both the image and the context of the image are treated as the premise, and the entailment of the caption is predicted with respect to both.

Our contributions are:

• A new unified VL model pretrained for keyword extraction, contextual visual entailment, and news image captioning with a focus on using contextual information which has not been explored before.

• State-of-the-art results on the GoodNews and NYTimes800k datasets with an improvement of \(\mathbf{8.34}\) CIDEr points on the GoodNews dataset.

• A novel VL classification task where the context information surrounding the image is utilized for detecting the entailment of the caption with the image.

• Release of two datasets¹- a large synthetic dataset consisting of \(\mathbf{1.1M}\) Image-Caption pairs with context and a more challenging dataset with manually annotated negative samples consisting of \(\mathbf{2.2K}\) instances for the proposed contextual visual entailment task.

2 Related Work↩︎

Image Captioning was initially conceived as a caption retrieval or template filling task. It involved matching the query image with a predefined set of captions or identifying the objects in the image to place them in predefined templates ([12]; [13]; [14]). The advancements made with deep learning based techniques in machine translation inspired the community to adopt similar techniques for image captioning where images were fed to the encoder and the decoder generated caption as a sequence of words ([12]; [13]; [14]). Attention allows decoder to focus on different parts of the input and hence it was incorporated to generate words focused on important regions of the image in both sequential models ([15]; [16]; [17]; [18]) and transformer based models [19]. In recent years, the models are trained on huge datasets with several millions of image-text pairs for image captioning ([20]; [21]; [22]). However, in all these works images are treated as standalone entities and their context is not taken into account.

News Image Captioning deals with the generation of captions for news images. The news articles contain the context of the image and they are taken into account during the caption generation process. [11] propose an encoder-decoder model with attention over both image and news article encodings to generate news image captions. It generates captions with placeholders for named entities and fills those placeholders by choosing named entities from the news article. [23] model it as a query-based summarization problem where the news image acts as the query and the news article is the source text to be summarized. [24] use transformers with separate encoders for extracting image features, object features and faces present in the image. The decoder receives the input from all three encoders to generate caption. [25] use a visual selective layer that learns to align the image features with the text in the news article to generate captions. [26] discuss the journalistic guidelines followed while writing news image captions in journals and incorporate them in the generation process. [27] use prompt tuning to finetune pretrained models for news image captioning.

Unified Vision-Language (VL) modeling is a new paradigm that involves creating a unified framework for multiple vision-language tasks, allowing models to be trained on a range of datasets constructed for a range of tasks. ViLBERT [28] extends the BERT [1] architecture to work with visual inputs. [4] propose a multi-task training approach with 12 VL datasets on 4 broad tasks. VL-T5 [5] combines multiple VL tasks as text generation tasks using pretrained models for image features. UniT [29] unifies cross-modal tasks by using a modality specific encoder and a shared decoder. UFO [6] proposes to use the same transformer architecture as the encoder for both image and text in VL tasks. UniTAB [30] supports VL tasks with bounding boxes by encoding the text and box output sequences to shared token sequences. OFA [3] abstracts all VL tasks into sequence-to-sequence problems.

Existing unified VL models do not consider the context of the image in their pretraining tasks. Our unified model is pretrained with tasks that include contextual information and hence it achieves state-of-the-art results on news image captioning datasets.

3 Dataset↩︎

We pretrain our model on a large pretraining dataset and evaluate its performance on benchmark datasets for news image captioning.

3.1 Pretraining Datasets↩︎

We use Visual News [25] and KPTimes [31] datasets for constructing our pretraining datasets.

Visual News dataset was compiled by collecting news articles from four news agencies: The Guardian, BBC, USA Today, and The Washington Post. It only includes the articles with high resolution images and where the caption length is between \(5\) and \(31\) words. It is diverse with differences in properties like average caption length, article length, and distribution of named entities across news agencies.

KPTimes dataset was constructed by crawling over 0.5 million news articles, mainly from New York Times. The metadata associated with field- "news_keywords" and "keywords" form the gold standard keyphrases . The three pretraining datasets are constructed from these datasets.

News Image Captioning: We removed duplicate captions, and news articles without images, and captions from Visual News dataset. The cleaned dataset consists of \(11,97,000\) data instances with image, news article, and the caption. These are split as \(11,17,697\) for training, \(40,000\) for validation, and \(40,000\) for test sets respectively.

Contextual Visual Entailment is a binary classification problem, so it is required to construct both positive and negative pairing of the image, and caption with the context. For the data instances where the caption entails the image and the context, the original image, the caption, and the context from the training split of the above news image captioning dataset are used (P). We use the following operations to generate the inconsistent pairs in our dataset:

1. Choose a random caption different from the correct caption (N-I).

2. Replace the named entities in the correct caption with named entities from randomly chosen caption (N-II). For example, the caption ‘John Garrison performing in Berlin, April 2015’ will be changed to ‘Mark Pattinson performing in London, April 2015’.

3. Keep the named entities of the original caption intact but replace the remaining content with a random caption that has the same type and the same number of named entities (N-III). For example, the caption ‘John Garrison performing in Berlin, April 2015’ will be changed to ‘John Garrison waiting in queue for filing tax returns in Berlin, April 2015’.

Named entity recognition is done with SpaCy [32] in our experiments. SpaCy allows the detection of \(18\) different named entities. We only use the named entities labeled as ‘PERSON’, ‘FAC’, ‘ORG’, ‘GPE’, ‘LOC’, and ‘EVENT’ that represent a person, building/airport, organization, geopolitical entities, location, and event respectively, as they occur more frequently.

The N-I class of negative captions will have different information and different named entities from the original caption. The N-II class will have same information as the original caption but will contain different named entities. The N-III class of captions will have same named entities but will convey different information. The final dataset has \(1005925\), \(55884\), and \(55884\) instances in the train, validation and test split respectively. We also create a separate manually annotated challenging dataset for evaluation.

In addition to synthetically creating a dataset for pretraining, we create and release a manually annotated challenging dataset for the task of contextual visual entailment consisting of \(2.2K\) data instances. The negative captions in this dataset are created manually by changing a word or a small phrase from the original caption, such that its meaning changes significantly without much difference in the sentence structure. For example, ‘Supporters marched peacefully during the protest’ will be changed to ‘Supporters marched violently during the protest’. The negative examples created in these ways will ensure that the models need to learn the relationship between image, caption, and context to identify the entailment correctly. This is used to test the model’s knowledge of image-caption entailment at a more finer level.

Annotation Details: The annotations were performed by two annotators proficient in English. One is a master’s student and the other is a bachelor’s student in Computer Science and Engineering. They were provided with examples of negative captions before annotation. The image links, caption, and context were shared in Google Sheets for annotation. The annotators were only asked to select a word or phrase from the caption and replace it with a new word or phrase. The modified captions were exchanged and verified by each other.

3.1.1 Keyword Extraction↩︎

The dataset for the keyword extraction task is constructed from KPTimes dataset after removing duplicate news articles. The news article forms the input to the system and the sequence of keywords form the output. The final dataset has \(259902\), \(10000\), and \(10000\) data instances in the train, validation and test split respectively. The training data from these three datasets are combined to generate the pretraining dataset with 2.3M data instances.

3.2 Benchmark Datasets↩︎

The performance of models trained on the pretraining datasets is evaluated on two benchmark datasets- GoodNews [11] and NYTimes800K [24]. We follow the train, validation, and test splits from the original work for both datasets. The GoodNews dataset has \(424, 000\), \(18, 000\), and \(23, 000\) in training, validation, and test split respectively. The NYTimes800K dataset has \(763,000\) training, \(8000\) validation, and \(22,000\) test instances in the dataset.

4 Our Model↩︎

Unified Vision-Language (VL) modeling has shown great promise in multiple VL tasks. Hence, we use a unified model for all three tasks- context assisted image captioning, contextual visual entailment, and keyword extraction. Figure 2 shows an overview of our unified VL pretraining strategy. We use the OFA\(_{Large}\) [3] architecture. OFA is a task and modality agnostic model that unifies all vision-language, vision-only, and language-only tasks using a sequence-to-sequence learning framework. We use ResNet152 [33] and VQGAN [34] to obtain visual tokens for the given image. The text (context and caption) is tokenized by byte-pair encoding (BPE). A single unified vocabulary is used for both visual and linguistic tokens. Transformers are used as encoders and decoders and all vision-language tasks are abstracted to seq-to-seq conversion tasks with specific instructions created for each task, similar to the OFA pretraining.

The pretraining of our model involves three tasks- News image captioning, contextual visual entailment, and keyword generation. For news image captioning, we convert the image, caption, and context into a sequence of input tokens and generate the caption as a sequence of tokens conditioned on these input tokens. For keyword generation, the news article is tokenized as the input sequence and the keywords are generated by the model as the output sequence. Contextual visual entailment is a classification task, so the input sequence to the model is the image, caption, and context tokens and the model is trained to generate ‘Yes’ or ‘No’ as the output indicating if the caption entails the image and context or not respectively.

The model is trained to reduce the cross entropy loss. For the input sequence x consisting of visual and text tokens and output y, the loss is given as: \[\mathcal{L} = \sum_{i}^{|y|}logP_{\theta}(y_i| y_{<i}, x)\] where \(y_i\) is the text token to be predicted and \(y_{i-1}\), \(y_{i-2}\) are tokens predicted so far.

5 Experiments↩︎

The experimental details for pretraining and finetuning for context assisted image captioning are discussed here.

5.1 Pretraining↩︎

We used OFA\(_{Large}\) architecture for pretraining our model. The model has 472M parameters with 12 encoder and 12 decoder layers. The weights were initialized with the publicly available OFA\(_{Large}\) checkpoint to retain the knowledge from other VL tasks. The model was pretrained on the 2.3M data instances from the pretraining datasets.

All 3 tasks were abstracted into sequence-to-sequence task. For the instances of news image captioning dataset, the instruction was “What does the image describe based on the text <context> ?", where <context> holds the tokens from the news article. For contextual visual entailment, the instruction was”Is the text <caption> consistent with the image and the text <context> ?", where <caption> and <context> contain the text tokens from the caption and the context. For the keyword extraction task, the instruction given was “What are the keywords in the article <context>?".

5.2 Context Assisted Image Captioning↩︎

Our unified model pretrained model for the three tasks was finetuned on GoodNews and NYTimes800K datasets for the task of context assisted image captioning. The image resolution was fixed at \(384 * 384\) and the news article was clipped to \(512\) tokens. The maximum caption length was fixed at \(30\). We use a batch size of \(8\) for training. We train the model with early stopping and choose the model that achieves the best CIDEr score on the validation set. The best-performing model is then tested on the unseen test data and the results are summarized in Table 1.

5.3 Training Details↩︎

The experiments were done with the OFA\(_{Large}\) architecture. For both pretraining and finetuning, the image resolution was fixed at \(384*384\). The input token length was restricted to \(512\) tokens while the output was restricted to \(30\) tokens. The dropout ratio was set to 0.1. We used Adam optimizer [35] with \(0.9\) and \(0.999\) as the \(\beta\) values with \(\epsilon = 1e-08\) and warm-up ratio was set as \(0.06\). We used an initial learning rate of \(1e-5\) with polynomial decay. We used a beam size of \(10\) during the test inference with temperature \(0.98\). We also used mixed precision training to speed up the training process.

5.4 Frozen Image Encoder + Frozen LLM↩︎

LLMs and large-scale pretrained VL models have shown great zero-shot performance in many downstream applications. We use BLIP-2 [36] for getting zero-shot image captions for the GoodNews dataset. These captions are generated without contextual information and are descriptive in nature. These captions are passed to a LLM along with contextual information to generate context assisted image captions. We use text-davinci-003 model in the GPT-3 family [37]. The prompt for generating the caption was "Add contextual information to the caption. Caption: <sample caption> Context: <sample context>". We randomly sampled a caption, and context pair from the training dataset of the GoodNews dataset and used it as an example in the prompt. The contextual captions are predicted for caption and context pair in the test set. The results are discussed in Table 2.

5.5 Ablation Study↩︎

In order to analayse the importance of the three pretraining tasks we used, we pretrained the OFA model using three different subsets of the pretraining tasks. We pretrained the model with only “Captioning and Contextual Visual Entailment" tasks, with only”Captioning and Keyword Extraction" task and with only “Captioning" task and compared their performance with the model trained on all the three tasks.

Previous works in news image captioning ([25]; [26]; [27]) have shown that extracting named entities from the context and feeding them to the decoder helps generate correct named entities in the caption. Hence, we also try injecting named entities into the prompt while finetuning the model. We use SpaCy for identifying and extracting named entities. We update the prompt as "what does the image describe about the names <named entities> based on the text <context>?" during finetuning. We clipped the named entity tokens to \(64\) and restricted the context to \(512\) tokens as done in previous experiments. We also perform experiments without using the contextual information with our pretrained model in the traditional image captioning setting, to analyze the usefulness of the contextual information. The results are summarized in Table 2.

6 Results and Analysis↩︎

Our model achieves state-of-the-art results on both GoodNews and NYTimes800K datasets. The OFA model finetuned on benchmark datasets also shows good performance. This shows the ability of OFA to adapt to new tasks and the correctness of our instructions for finetuning. However, it can be seen that due to the lack context information in the pretraining tasks used in OFA, the model doesn’t produce substantially better results compared to the current SOTA models. Our model pretrained on the three tasks shows a \(5.14\) CIDEr score improvement over the OFA model on the GoodNews dataset which is an \(8.34\) CIDEr score improvement over the current SOTA model. The model also achieves a SOTA result of \(66.41\) CIDEr score on the NYTimes800K dataset.

The average length of news articles in GoodNews and NYTimes800K dataset are 451 and 974 words respectively. The larger article length in NYTimes800K dataset is the reason for the CIDEr scores being closer to the current SOTA as the context length in our experiments is restricted to 512 tokens. We also obtain comparable performance in precision and recall of named entity generation despite not feeding the named entities directly to the model like in the previous works.

The BLIP-2 model has shown great promise in zero-shot image caption generation. We use BLIP-2 to generate descriptive captions and feed those captions as input to the GPT-3 model along with the context to generate the final context assisted caption. The BLIP-2 + GPT-3 model generates fluent captions but it does not contain the relevant information based on the image features as indicated by the poor performance on evaluation metrics in Table 2. This indicates that it is essential to train with both image and context together.

Our pretraining tasks indirectly direct the model to capture named entity information from the context. However, earlier works on news image captioning show that extracting named entities and feeding them directly to the model can help it generate better captions with correct named entities. Our pretrained model showed a slight decrease in performance when named entity information is presented to it in the prompt. This is because the named entity tokens take up valuable space in the 512 tokens allowed for the context, leading to information loss gained from the context. Also, since only 64 tokens are allowed for named entities, not all the important entities in the news article are presented in the prompt and it disadvantages the model in both ways.

We also pretrained the OFA model with a subset of the three pretraining tasks to identify the importance of each task in pretraining. The models pretrained with a combination of two tasks and with only the captioning task performed poorly compared to the model pretrained on the three tasks. This shows the importance of training with all three tasks. Between the two two-task pretrained models, the model that used contextual visual entail task performed better, indicating the usefulness of the task we introduced.

7 Summary and Conclusion↩︎

In this work, we proposed a new unified VL model that uses contextual information of images that has not been utilized in pretraining before. We introduce a new VL classification task called contextual visual entailment and pretrain a model with three subtasks that uses long text along with image and caption. Our model achieves new state-of-the-results on benchmark datasets for news image captioning and highlights the importance of using contextual information in pretraining.

In the future, we aim to deploy our model to allow context-aware caption generation which could be used in enterprise authoring and many other content creation processes.

8 Appendix↩︎

8.1 Dataset Details↩︎

Table 3 provides the summary of the three datasets used for pretraining.

8.2 Additional Experiments↩︎

Our pretrained model achieves state of the results on news image captioning task. In addition, it performs very well on the other two pretraining tasks. The contextual visual entailment is a new task introuduced by our work and hence we propose baselines for comparing the results of our model. We compare our model’s performance on keyword extraction against standard works.

8.2.1 Contextual Visual Entailment↩︎

We propose a two baselines for contextual visual entailment, where the image and text features are extracted from pretrained networks. The features are obtained from a pretrained CLIP (Contrastive Language–Image Pre-training) model. CLIP [22] was trained on large scale image-text corpus to minimize contrastive loss such that the text embedding and the image embedding will have higher cosine similarity if the text describes the image perfectly and low when the text incorrectly describes the image.

CLIP and FNN model↩︎

In our CLIP embedding based models, the representation for image, caption, and context is obtained from a pretrained CLIP model.

The CLIP and FNN model, uses a simple early fusion strategy in which the image, caption, and context embeddings from CLIP are concatenated and fused with feed-forward neural networks. The three input embeddings are concatenated and passed to a two-layer feedforward neural network for combining the information. An output layer predicts the entailment label. The experiments are repeated without the context information and then again without the image features as input to study their impact on entailment detection.

CLIP and Transformer model↩︎

The fusion of image-text information using transformers has helped achieve good performance on many standard vision-language tasks ([38]; [39]; [3] ). The CLIP and Transformer model uses transformer [40] encoders with multi-head attention to combine these multimodal information. A transformer layer receives the input features from the image, caption, and context and generates the combined representation for the information. The outputs from the transformer layers are pooled to a single dense layer followed by a classification layer to perform the final binary classification.

We summarize the results of our experiments on the manually annotated contextual visual entailment data in Table 4 and compare it with the results produced by our pretrained model.

8.2.2 Keyword Extraction↩︎

We use our pretrained model to perform keyword extraction as a sequence to sequence task where the output is the set of keyword tokens. We use similar hyperparameters to caption generation for keyword generation. We finetune the pretrained model for \(5\) epochs and report the results on KPTimes dataset in Table 5.

8.3 Examples↩︎

Figure 3 shows examples from the GoodNews dataset and compares the caption generated by each model.

References↩︎