1 Introduction↩︎

The automatic identification of metaphors in corpora is an active area of research in NLP [1]–[6], and consequently, previous works have proposed the construction, curation and testing of metaphor indentification datasets in various forms [1], [7], also stemming from previous psycholinguistic research [8], [9]. A common feature of metaphor identification tasks is that they are usually framed as binary classification, where a potentially metaphorical expression (PME) and a context are provided as input, and a system has to determine whether the given PME is used metaphorically or not. For instance, dark is used metaphorically in the sentence ‘The latest developments move us closer to a dark age’, but not in ‘I like dark colors’. While this setting is attractive for testing supervised systems, it also simplifies the task, introducing a real risk of not testing metaphoricity, but instead spurious correlations that might lead to an artificially correct solutions.

Analyses of this sort have often proven critical in understanding the relationship between a proposed task and whether performance on associated datasets can be directly linked to performance on the task itself. Notable examples include the work of [10], who showed that, in the task of lexical relation modeling, the unreasonably high performance of supervised systems could be attributed to lexical memorization, that is, the presence of large number of prototypical cases in the dataset (e.g. animal for the hypernymy relation) that made it simple for models to “detect the relation” without having to consider both words in the pair. This issue has been identified also in word analogies [11]–[13] and, beyond lexical semantics, in natural language inference (NLI). In NLI, given a hypothesis and a premise, a system must determine whether the premise entails, contradicts or is neutral with respect to the hypothesis. Previous works have shown that supervised models could rely on superficial factors, e.g., in the SNLI dataset [14], hypothesis-only models are surprisingly competitive [15], [16], a trend also observed in medical NLI datasets [17].

In this paper, we report experimental results which suggest that (1) language models can identify metaphorical expressions with great accuracy without even seeing the given expression; and that (2) a model only seeing the metaphorical expression with no context performs very competitively, in both cases close to the model with complete information. A crucial distinction in this phenomenon, however, is that this is only observed in datasets not sampled from a natural distribution, and that whenever such distribution is preserved, these shortcuts are not as effective. We thus propose a simple sampling procedure from a naturally-distributed corpus that mitigates this issue.

2 Metaphor Identification Datasets↩︎

This section presents the list of existing English-language metaphor identification datasets used in our experiments. All the datasets are summarized in Table 1.

2.1 Psycholinguistic datasets↩︎

Datasets created for Psycholinguistics and Cognitive Science served experiments on the perception of metaphors. They are composed of sentences written or rephrased for the purpose of controlling characteristics such as length and word frequency. This line of study for the English language started with [18]¹, and was pursued by [19] and [8], who released a corpus of verbal and nominal PMEs (CARD_N and CARD_V). [9] extended this work for sentences of the form A is-a B.

2.2 NLP datasets↩︎

2.2.0.1 Sentence context.

The TroFi dataset published by [7] has often been used to evaluate metaphor identification NLP systems, being a notable example. The collection started with 50 predefined labeled target verbs. Then, sentences containing one of those verbs were extracted from the Wall Street Journal (WSJ) corpus, and subsequently labeled. [1] used TroFi to train a classifier for verbal metaphor identification that integrates features from WordNet [20]. As part of their research, they released a balanced test set of 222 sentences with verbal PME, and a new corpus of adjective-noun pairs (TSV_V and TSV_AN). Only clear examples with strong inter-annotator agreements were included in the test set. [21] studied the correlation between metaphors and emotions. This was achieved by building a new corpus of verbal metaphors from WordNet glosses of polysemous predicates, enriched with synset annotations and emotionality (MOH). [22] released the large LLC corpus of metaphors covering several part-of-speech and longer metaphoric expressions, scored on metaphoricity and emotions scales, and enriched with source domain information. The dataset construction relies partly on automatic metaphor extraction tools, with a manual validation of the output, resulting inevitably in a bias of the considered instances. [23] created a small test set of 60 instances to evaluate a metaphoricity score model (DUNN), with the same verb appearing with different levels of metaphoricity in three sentences sets. Similarly, [24] released a small corpus of verbal PME to evaluate the MERMAID model (CHAK).

2.2.0.2 Adjective-nouns.

[25] developed sets of labeled adjective-noun pairs of concrete-abstract associations (NEU) and [26] released a much larger dataset of frequent adjective noun pairs for a study on the compositional properties of metaphors (GUT).

2.3 Potential Idiomatic Expressions (PIEs)↩︎

Idioms, such as to rock the boat, are multiword metaphoric expressions that became lexicalized in a language. Similarly to the corpus construction methodology used for TroFi, idiomatic expressions datasets are usually built from an initial list of PIEs, which are then used to extract sentences from corpora, mostly from the British National Corpus. Several datasets focusing on different types of PIE have been released: [27] for verb noun constructs (VNC), [28] for various multiword expressions (IDIX), [29] on prepositional verb constructs (PVC), [30] in two different tracks (SE2013), and [31] for nominal compounds (MAD). [32] released the large MAGPIE dataset containing PIEs of various syntactic constructs, following an initial release of a smaller PIE corpus.

2.4 Sampled from annotated corpora: The case of the Amsterdam corpus↩︎

The VU Amsterdam Corpus [33], is a collection of documents from the British National Corpus (BNC-baby), labeled following the metaphor identification MIPVU protocol. Each word of the documents was considered by the annotators and marked when metaphoric. The dataset covers over 190,000 lexical units. Because very conventional metaphors were labeled as figurative, several versions of the corpus have later been released containing metaphoricity or novelty scores. Other modifications have been made to frame the corpus into NLP system inputs for figurative language identification. For example, [34] enriched the data with novelty scores (DO) and [35] released two versions of VUAC for a shared task of metaphor identification, with different sets of Part-of-Speech (PoS) tags considered (ST1 for verbs, ST2 for all PoS tags). The datasets are designed for classification of single tokens in the context of a sentence. Parde and Nielsen ([36]; [37]) also worked on metaphor novelty in VUAC, releasing a version of potential metaphoric source-target pairs among the syntactic dependencies of a metaphoric word.

2.4.0.1 Our sampling method.

Framing the VUAC for binary classification requires to sample literal instances of PME from the corpus. Any token that is not labeled as metaphoric in the VUAC can be considered literal. To avoid PME sampling biases, we sample literal instances from the set of expressions that also occur as metaphors, whenever possible. When the same word sequence is not found, we rely on identical lemma sequences, and finally on identical PoS sequences (we referred to this VUAC sampling as VUAC_BO).

3 Evaluation↩︎

The evaluation is aimed at understanding to what extent metaphor identification datasets are affected by construction or sampling biases.

3.1 Experimental setting↩︎

3.1.0.1 Data.

For the initial experiments (Section 3.2), we rely on the original data splits of existing metaphor identification datasets (see Table 1).² In Section 3.3 we also provide an extended analysis to assess the impact of baselines in different data splits with 5-folds cross-validation.

3.1.0.2 Model and training.

As our model for all the experiments, we rely on BERT-base [38]. Note that the goal of the experiments is not to provide the best possible model, but rather to show how a supervised model with incomplete information can attain a performance similar to the model with all the information. We use the HuggingFace transformers library and models, adding a classification layer on top of the pre-trained BERT model. For hyperparameter optimisation, we rely on the Bayesian Optimization with Hyperband (BOHB) algorithm [39] with 50 trials, available in RayTune [40]. The hyperparameters search space is set to a batch size equal to 4, 8 or 16; the learning rate in a ranging from 5e-7 to 5e-5; the number of epochs within 1 to 12; and the random seeds taking three possible values (1, 2 and 3).

3.1.0.3 Baselines.

In order to test our hypothesis, we test two baselines with incomplete information that effectively hide the context or the information about the metaphorical expression being tested. We will use the following sentence as our running example, with dark being the PME of the sentence: The latest developments move us closer to a dark age. We test the following three configurations according to the input shown to the model:

1. Default: The latest developments move us closer to a <PME>dark</PME> age.³

2. Baseline 1 – Only PME: dark.

3. Baseline 2 – Masked PME: The latest developments move us closer to a <masked> age.

3.1.0.4 Evaluation metrics.

Macro-F1 is chosen as our objective evaluation metric on the validation set during hyper-parameter optimization and is the main metric used in our experiment. Our choice of Macro-F1 was because it gives a synthetic view of the performances of the models for both classes, and datasets with very different label distributions. Accuracy, precision, recall and F1 results for the metaphor class can be found in Appendix 6.

3.2 Results↩︎

The main results are displayed in Table 2. In general, gaps in performance should only be compared within datasets, and are not comparable across datasets because of different sampling techniques. For instance, the balance between metaphorical expressions may be vastly different in different datasets, and the performance gap may also reflect this.

As can be observed, the results of models with incomplete information are extremely competitive, even in the case when only a PME is included. The results also show how different datasets suffer from diverse types of bias. For instance, the only PME baseline appears to be insufficient in NLP datasets, with the masked baseline being close to the default setting. In contrast, in datasets such as MAD, PIE or MAGPIE, the only PME baseline is more competitive (with relative drops often lower than 5%), even surpassing the default upperbound in PIE.

3.3 Analysis: Random and Lexical Splits↩︎

3.3.0.1 Setting.

Similarly to , in this experiment we consider datasets with random and lexical splits. A random split is simply a random allocation of instances for the training and test sets. In a lexical split, however, we ensure that a target word (the PME in our case) in the test set is not included in the training set. For a better generalisation, the experiments for this analysis were repeated on five different splits using 5-fold cross-validation.⁴ For each fold, 70% of the instances in the training set, 10% in the validation set and 20% in the test set.

3.3.0.2 Results.

Table 3 shows the results for the random and lexical splits. The trends observed in these splits are similar to the original splits. This now includes the datasets coming from the psycholinguistic literature, with minimal performance drops of the baseline masking the PME (lower than 5% in all cases). When the natural distribution is not used, the gap between the baselines and the model seeing all the information is in some cases small, with the performance of these baselines being non-trivial. This is not the case in all the datasets, such as MOH, which differs in its construction method from most of the other datasets.

3.3.0.3 Random vs lexical splits.

There are cases in which the random split may mislead the model, especially when the number of examples is limited. For example, one dataset may contain only literal examples in the training set for a PME, and the model may wrongly conclude that all instances in the test set of that PME are literal, when this may not be the case. This would not happen in the lexical split. For instance, the DUNN dataset was created with three instances per PME, one literal and two metaphorical. Similarly, for CARD there are two instances per PME, one literal and one metaphorical.

3.3.0.4 The case of VUAC.

VUAC has the particularity of following a natural distribution, and has been sampled differently by different researchers to frame it into a binary classification task. Despite its simplicity, our VUAC sampling approach (i.e., VUAC_BO) appears to be the generally robust, with over 10 points of difference between the results of the default configuration and the best baseline, both for the random and lexical splits.

4 Conclusion↩︎

Getting inspiration from previous studies analysing artifacts and biases from NLP datasets, we delved into the field of metaphor identification. By proposing baselines with incomplete information that hide the PME or the context, we show how the performance achieved by a supervised model is closed to the model that uses complete information. This highlights the type of bias that a model is picking up at training time, which differ from what a well-trained model would be expected to learn. Finally, we show that this problem is generally observed in datasets that are artificially constructed, as carefully sampling from datasets (VUAC_BO) stemming from a fully-annotated corpus alleviates this issue.

Limitations↩︎

Since this is a preliminary study on the study of biases in metaphor identification datasets, this comes with its own limitations that could be addressed in future work. For instance, we evaluate the models in a set of pre-defined splits that may include their own biases. We attempt at mitigating this by also including cross-validation settings on lexical and random splits, but this may not provide the full picture. For example, we do not study the effect of context duplication in this work (more details in the last point of Appendix 5). In terms of the models evaluated, given computational limitations, we focus on a single language model. Our goal is not to achieve the best possible results, but it is likely that some of the conclusions may slightly differ for different supervised models, including other language models.

A human evaluation in the three settings would be interesting to better understand the inherent biases of the models due to the probability of each PIE to be used metaphorically or the probability of a context to be paired with a metaphor from dataset artifacts. This could help provide a more complete picture of the reasons behind our findings.

Ethics statement↩︎

Our analysis sheds light on the possible deficiencies of supervised settings and the spurious correlations that supervised models can learn from if the datasets are not carefully designed. As such, researchers should be careful in the claims we can make about such models, in particularly in relation to metaphor identification as we show in this paper. In fact, that there may be other artifacts and biases not considered in this paper.

Acknowledgments↩︎

Jose Camacho-Collados is supported by a UKRI Future Leaders Fellowship. We thank all the authors of the datasets used in this paper for kindly sharing them with us, including those that were not currently available online.

5 Dataset Preprocessing and Sampling↩︎

In the following we list a few individual dataset preprocessing and sampling details not included in the main paper.

1. A few instances of the TSV test set were found in the training set, we deduplicate them in the random and lexical splits of the dataset.

2. The original 200 instances of the TSV_AN test set are provided within a full sentence context, which we ignore in our experiments because the training set only contains adjective-noun pairs and finetuned language models generally perform better when the test set is similar to the training set.

3. In the lexical splits columns of Table 3, TSV_ANand TSV_AN_L2 are two lexical splits of the TSV_AN instances respectively on the adjective and the noun. The same reading applies to the GUT dataset. The results for a single random split are shown duplicated in the table, for the two rows.

4. Some sentences of the WSJ appear several times in the TroFi dataset original version, we also deduplicate them in the random and lexical spits of the dataset.

5. The MAGPIE has an original lexical and random split, for which the results are presented in Table 1 (MAGPIE_L and MAGPIE_R). Different random and lexical splits appear in Table 3.

6. In Table 1, the statistics of the original MAD_FEWSHOT dataset are shown. Two other versions with less instances have been shared in [31]: MAD_ONESHOT and MAD_ZEROSHOT (equivalent to a lexical split). The results for the three original splits appear in Table 2. Our lexical and random splits of the MAD dataset are made from the instances of the few-shot version, that contains the largest number of instances among the three.

7. The PVC dataset contains too few distinct V-PREP instance for a relevant lexical split based on the two words, the split is made on the verb for each PME.

8. A few instances of the IDIX and VNC datasets contain minimal meaningful contextual information in the only-PME setting because they are multiword expressions of non consecutive words. For example, the idiom to rock the boat appears in the sentence as rock the political boat. Informative context inserted between the terms of a PIE occurs very rarely in those two datasets. Therefore, while they could have an effect in the final results, we decided not to not modify them and simply extracted or tagged the expressions by selecting the string starting and finishing with the PIE.

9. Context duplication: The CHAK dataset has the specificity of being made of triples, where nearly identical sentences appear three times. The sentences of a triple have the same context and three different substituted verbal PMEs, among which exactly one is literal and two are figurative. The MOH dataset, constructed from WordNet glosses, also contains several instances with identical context, and alternated verbal PMEs that are near synonyms of each other. Finally, multiple instances of the VUAC datasets may be constructed from the same sentence during the sampling procedure, because several words of each sentence may have been labeled as metaphoric during the original annotation process.

6 Additional evaluation metrics↩︎

Tables 4 and 5 provide additional metrics for the random and lexical split analyses, respectively (see Section 3.3). In particular, the tables include accuracy and precision, recall and F1 on the metaphor class. Moreover, for completeness we have included the accuracy results of a naive baseline that outputs the metaphor for all instances.

References↩︎