1 Introduction↩︎

Entity representations play a key role in numerous important problems including language modeling [2], dialogue generation [3], entity linking [4], and story generation [5]. One successful line of work on learning entity representations has been learning static embeddings: that is, assign a unique vector to each entity in the training data [4], [6], [7]. While these embeddings are useful in many applications, they have the obvious drawback of not accommodating unknown entities. Another limiting factor is the lack of an evaluation benchmark: it is often difficult to know which entity representations are better for which tasks.

We introduce EntEval: a carefully designed benchmark for holistically evaluating entity representations. It is a test suite of diverse tasks that require nontrivial understanding of entities, including entity typing, entity similarity, entity relation prediction, and entity disambiguation. Motivated by the recent success of contextualized word representations (henceforth: CWRs) from pretrained models [1], [8]–[11], we propose to encode the mention context or the description to dynamically represent an entity. In addition, we perform an in-depth comparison of ELMo and BERT-based embeddings and find that they show different characteristics on different tasks. We analyze each layer of the CWRs and make the following observations:

To further improve contextualized and descriptive entity representations (CER/DER), we leverage natural hyperlink annotations in Wikipedia. We identify effective objectives for incorporating the contextual information in hyperlinks and improve ELMo-based CWRs on a variety of entity related tasks.

2 Related Work↩︎

EntEval and the training objectives considered in this work are built on previous works that involve reasoning over entities. We give a brief overview of relevant works.

2.0.0.1 Entity linking/disambiguation.

Entity linking is a fundamental task in information extraction with a wealth of literature [12]–[18]. The goal of this task is to map a mention in context to the corresponding entity in a database. A natural approach is to learn entity representations that enable this mapping. Recent works focused on learning a fixed embedding for each entity using Wikipedia hyperlinks [6], [19], [20]. [4] additionally train context and description embeddings jointly, but this mainly aims to improve the quality of the fixed entity embeddings rather than using the context and description embeddings directly; we find that their context and description encoders perform poorly on EntEval tasks.

A closely related concurrent work by [21] jointly encodes a mention in context and an entity description from Wikipedia to perform zero-shot entity linking. In contrast, here we seek to pretrain a general purpose entity representations that can function well either given or not given entity descriptions or mention contexts.

Other entity-related tasks involve entity typing [22]–[28] and coreference resolution [29]–[33].

2.0.0.2 Evaluating pretrained representations.

Recent work has sought to evaluate the knowledge acquired by pretrained language models [34]–[42]. In this work, we focus on evaluating their capabilities in modeling entities.

Part of EntEval involves evaluating world knowledge about entities, relating them to fact checking [43]–[47], and commonsense learning [48]–[57]. Another related line of work is to integrate entity-related knowledge into the training of language models [58]–[60].

2.0.0.3 Contextualized word representations.

Contextualized word representations and pretrained language representation models, such as ELMo [1] and BERT [9], are powerful pretrained models that have been shown to be effective for a variety of downstream tasks such as text classification, sentence relation prediction, named entity recognition, and question answering. Recent work has sought to evaluate the knowledge acquired by such models [34]–[36], [38], [39], [41]. In this work, we focus on evaluating their capabilities in modeling entities.

3 EntEval↩︎

We are interested in two approaches: contextualized entity representations(henceforth: CER) and descriptive entity representations(henceforth: DER), both encoding fixed-length vector representations for entities.

The contextualized entity representations encodes an entity based on the context it appears regardless of whether the entity is seen before. The motivation behind contextualized entity representations is that we want an entity encoder that does not depend on entries in a knowledge base, but is capable of inferring knowledge about an entity from the context it appears.

As opposed to contextualized entity representations, descriptive entity representations do rely on entries in Wikipedia. We use a model-specific function \(f\) to obtain a fixed-length vector representation from the entity’s textual description.

To evaluate CERs and DERs, we propose a wide range of entity related tasks. Since our purpose is for examining the learned entity representations, we only use a linear classifier and freeze the entity representations when performing the following tasks. Unless otherwise noted, when the task involves a pair of entities, the input to the classifier are the entity representations \(x_1\) and \(x_2\), concatenated with their element-wise product and absolute difference: \([x_1, x_2, x_1 \odot x_2, |x_1 - x_2|]\). This input format has been used in SentEval [39].

The datasets used in EntEval tasks are summarized in table 1. It shows the number of instances in train/valid/test split for each dataset, and the number of target classes if this is a classification task. We describe the proposed tasks in the following subsections.

3.1 Entity Typing (ET)↩︎

The task of entity typing (ET) is to assign types to an entity given only the context of the entity mention. ET is context-sensitive, making it an effective approach to probe the knowledge of context encoded in pretrained representations. For example, in the sentence “Bill Gates has donated billions to eradicate malaria”, “Bill Gates” has the type of “philanthropist” instead of “inventor” [26].

In this task, we will contextualized entity representations, followed by a linear layer to make predictions. We use the annotated ultra-fine entity typing dataset of [26] with standard data splits. As shown in Figure 1, there can be multiple labels for an instance. We use binary log loss for training using all positive and negative entity types, and report \(F_1\) score. Thresholds are tuned based on validation set accuracy.

3.2 Coreference Arc Prediction (CAP)↩︎

Given two entities and the associated context, the task is to determine whether they refer to the same entity. Solving this task may require the knowledge of entities. For example, in the sentence “Revenues of $14.5 billion were posted by Dell₁. The company₁ ...”, there is no prior context of “Dell”, so having known “Dell” is a company instead of the people “Michael Dell” will surely benefit the model [61]. Unlike other tasks, coreference typically involves longer context. To restrict the effect of broad context, we only keep two groups of coreference arcs from smaller context. One includes mentions that are in the same sentence (“same”) for examining the model capability of encoding local context. The other includes mentions that are in consecutive sentences (“next”) for the broader context. We create this task from the PreCo dataset [62], which has mentions annotated even when they are not part of coreference chains. We filter out instances in which both mentions are pronouns. All non-coreferent mention pairs are considered to be negative samples.

To make this task more challenging, for each instance we compute cosine similarity of mentions by averaging GloVe word vectors. We group the instances into bins by cosine similarity, and randomly select the same number of positive and negative instances from each bin to ensure that models do not solve this task by simply comparing similarity of mention names.

We use the contextualized entity representations of the two mentions to infer coreference arcs with supervised training and report the averaged accuracy of “same” and “next”.

3.3 Entity Factuality Prediction (EFP)↩︎

The entity factuality prediction (EFP) task involves determining the correctness of statements regarding entities. We use the manually-annotated FEVER dataset [45] for this task. FEVER is a task to verify whether a statement is supported by evidences. The original FEVER dataset includes three classes, namely “Supports”, “Refutes”, and “NotEnoughInfo” and evidences are additionally available for each instance. As our purpose is to examine the knowledge encoded in entity representations, we discard the last category (“NotEnoughInfo”) and the evidence. In rare cases, instances in FEVER may include multiple entity mentions, so we randomly pick one. We randomly sample 10000, 2000, and 2000 instances for our training, validation, and test sets, respectively.

In this task, entity representations can be obtained either by contextualized entity representations or descriptive entity representations. In practice, we observe descriptive entity representations give better performance, which presumably is because these statements are more similar to descriptions than entity mentions. As shown in Figure 2, without providing additional evidences, solving this task requires knowledge of entities encoded in representations. We directly use entity representations as input to the classifier.

3.4 Contexualized Entity Relationship Prediction (CERP)↩︎

The task of contexualized entity relationship prediction (CERP) modeling determines the connection between two entities appeared in the same context.

We use sentences from ConceptNet [63] with automatically parsed mentions and templates used to construct the dataset. We filter out non-English concepts and relations such as ‘related’, ‘translation’, ‘synonym’, and ‘likely to find’ since we seek to evaluate more complicated knowledge of entities encoded in representations. We further filter out non-entity mentions and entities with type ‘DATE’, ‘TIME’, ‘PERCENT’, ‘MONEY’, ‘QUANTITY’, ‘ORDINAL’, and ‘CARDINAL’ according to SpaCy [64]. After filtering, we have 13374 assertions.

Negative samples are generated based on the following rules:

For example, let ‘A is B’ be the positive sample. (1) changes it to ‘C is B’ which serves as a negative sample and (2) changes it to ‘A is not B’ as another negative sample. (3) changes it to ‘C is not B’ as a positive example. In the end, we randomly sample 6000 instances from each class. This ends up yielding a 4000/4000/4000 train/dev/test dataset. As shown in Figure 3, this task cannot be solved by relying on surface form of sentences, instead it requires the input representations to encode knowledge of entities based on the context.

We use contextualized entity representations in this task.

3.5 Entity Similarity and Relatedness (ESR)↩︎

Given two entities with their descriptions from Wikipedia, the task is to determine their similarity or relatedness. After the entity descriptions are encoded into vector representations, we compute their cosine similarity as predictions. We use the KORE [66] and WikiSRS [67] datasets in this task. Since the original datasets only provide entity names, we automatically add Wikipedia descriptions to each entity and manually ensure that every entity is matched to a Wikipedia description. We use Spearman’s rank correlation coefficient between our computed cosine similarity and the gold standard similarity/relatedness scores to measure the performance of entity representations.

As KORE does not provide similarity scores of entity pairs, but simply ranks the candidate entities by their similarities to a target entity, we assign scores from 20 to 1 accordingly to each entity in the order of similarity. Table 2 shows an example from KORE. The fact that “Apple Inc.” is more related to “Steve Jobs” than “Microsoft” requires multiple steps of inference, which motivates this task. Since the predictor we use is cosine similarity, which does not introduce additional parameters, we directly use encoded representations on the test set without any supervised training.

3.6 Entity Relationship Typing (ERT)↩︎

As another popular resource for common knowledge, we consider using Freebase [68] for probing the encoded knowledge by classifying the types of relations between pair of entities. First, we extract entity relation tuples (entity1, relation, entity2) from Freebase and then filter out easy tuples based on training a classifier using averaged GloVe vectors of entity names as input, which leaves us 626 types of relations, including “internet.website.owner”, “film.film_art_director.films_art_directed”, and “comic_books.comic_book_series.genre”. We randomly sample 5 instances for each relation type to form our training set and 10 instances per type the for validation and test sets. We use Wikipedia descriptions for each entity in the pair whose relation we are predicting and we use descriptive entity representations for each entity with supervised training.

3.7 Named Entity Disambiguation (NED)↩︎

Named entity disambiguation is the task of linking a named-entity mention to its corresponding instance in a knowledge base such as Wikipedia. In this task, we consider CoNLL-YAGO (CoNLL; [69]) and Rare Entity Prediction (Rare; [70]).

For CoNLL-YAGO, following [69] and [6], we used the 27,816 mentions with valid entries in the knowledge base. For each entity mention \(\mathit{m}\) in its context, we generate a set of (at most) its top 30 candidate entities \(\mathit{C_m} = \{\mathit{c_j}\}\) using CrossWikis [71]. Some gold standard candidates \(c\) are not present in CrossWikis, so we set the prior probability \(p_\text{prior}(y)\) for those to 1e-6 and normalize the resulting priors for the candidate entities. When adding Wikipedia descriptions, we manually ensure gold standard mentions are attached to a description, however, we discard candidate mentions that cannot be aligned to a Wikipedia page. We use contextualized entity representations for entity mentions and use descriptive entity representations for candidate entities. Training minimizes binary log loss using all negative examples. At test time, we use \(\mathop{\mathrm{arg\,max}}_{c\in\mathit{C_m}}[p_\text{prior}(c)+p_\text{classifier}(c)]\) as the prediction. We note that directly using prior as predictions yields an accuracy of 58.2%.

[70] introduce the task of rare entity prediction. The task has a similar format to CoNLL-YAGO entity linking. Given a document with a blank in it, the task is to select an entity from a provided list of entities with descriptions. Only rare entities are used in this dataset so that performing well on the task requires the ability to effectively represent entity descriptions. We randomly select 10k/4k/4k examples to construct train/valid/test sets. For simplicity, we only keep instances with four candidate entities.

Figure 4 shows an example from CoNLL-YAGO, where the “China” in context has many deceptive meanings. Here the candidate “China” has exact string match of the entity name but it should not be selected as it is an after-game report on soccer. To match the entities, this task requires both effective contextualize entity representations and descriptive entity representation.

Practically, we encode the context using CER to be \(x_1\), and encode each entity description using DER to be \(x_2\), and pass \([x_1, x_2, x_1 \odot x_2, |x_1 - x_2|]\) to a linear model to predict whether it is the correct entity to fill in. The model is trained with cross entropy loss.

4 Methods↩︎

We first describe how we define encoders for contextualized entity representations(Section 4.1) and descriptive entity representations(Section 4.2), then we discuss how we train new encoders tailored to capture information from the hyperlink structure of Wikipedia (Section 4.3).

4.1 Encoders for Contextualized Entity Representations↩︎

For defining these encoders, we assume we have a sentence \(s = (w_1, \dots, w_T)\) where span \((w_i, \dots, w_j)\) refers to an entity mention. When using ELMo, we first encode the sentence: \((c_1, \dots, c_T) = \mathrm{ELMo}(w_1, \cdots, w_T)\), and we use the average of contextualized hidden states corresponding to the entity span as the contextualized entity representation. That is, \(f_\text{ELMo}(w_{1:T},i,j)=\frac{\sum_{k=i}^j c_k}{j-i+1}\).

With BERT, following [27], we concatenate the full sentence with the entity mention, starting with \(\mathrm{[CLS]}\) and separating the two by \(\mathrm{[SEP]}\), i.e., \(\mathrm{[CLS]}, w_1, \dots, w_T, \mathrm{[SEP]}, w_i, \dots, w_j, \mathrm{[SEP]}\). We encode the full sequence using BERT and use the output from the \(\mathrm{[CLS]}\) token as the entity mention representation.

4.2 Encoders for Descriptive Entity Representations↩︎

We encode an entity description by treating the entity description as a sentence, and use the average of the hidden states from ELMo as the entity description representation. With BERT, we use the output from the \(\mathrm{[CLS]}\) token as the description representation.

4.3 Hyperlink-Based Training↩︎

An entity mentioned in a Wikipedia article is often linked to its Wikipedia page, which provides a useful description of the mentioned entity.

The same Wikipedia page may correspond to many different entity mentions. Likewise, the same entity mention may refer to different Wikipedia pages depending on its context. For instance, as shown in Figure 5, based on the context, “France” is linked to the Wikipedia page of “France national football team” instead of the country. The specific entity in the knowledge base can be inferred from the context information. In such cases, we believe Wikipedia provides valuable complementary information to the current pretrained CWRs such as BERT and ELMo.

To incorporate such information during training, we automatically construct a hyperlink-enriched dataset from Wikipedia that we will refer to as WikiEnt. Prior work has used similar resources [4], [72], but we aim to standardize the process and will release the dataset.

The WikiEnt dataset consists of sentences with contextualized entity mentions and their corresponding descriptions obtained via hyperlinked Wikipedia pages. When processing descriptions, we only keep the first 100 word tokens at most as the description of a Wikipedia page; similar truncation has been done in prior work [4]. For context sentences, we remove those without hyperlinks from the training data and duplicate those with multiple hyperlinks. We also remove context sentences for which we cannot find matched Wikipedia descriptions. These processing steps result in a training set of approximately 85 million instances and over 3 million unique entities.

We define a hyperlink-based training objective and add it to ELMo. In particular, we use contextualized entity representations to decode the hyperlinked Wikipedia description, and also use the descriptive entity representations to decode the linked context. We use bag-of-words decoders in both decoding processes. More specifically, given a context sentence \(x_{1:T_x}\) with mention span \((i,j)\) and a description sentence \(y_{1:T_y}\), we use the same bidirectional language modeling loss \(l_\text{lang}(x_{1:T_x})+l_\text{lang}(y_{1:T_y})\) in ELMo where \[\begin{align} l_\text{lang}(u_{1:T})=-\sum_{t = 1}^T &\log p(u_{t+1}\vert u_1,\dots,u_{t}) + \\ &\log p(u_{t-1}\vert u_{t},\dots,u_{T}) \end{align}\]

and \(p\) is defined by the ELMo parameters. In addition, we define the two bag-of-words reconstruction losses: \[\begin{align} l_\text{ctx} &= -\sum_t \log q(x_t\vert f_\text{ELMo}(\mathrm{[BOD]} y_{1:T_y}, 1, T_y)) \\ l_\text{desc} &= -\sum_t \log q(y_t\vert f_\text{ELMo}(\mathrm{[BOC]} x_{1:T_x}, i, j)) \end{align}\] where \(\mathrm{[BOD]}\) and \(\mathrm{[BOC]}\) are special symbols prepended to sentences to distinguish descriptions from contexts. The distribution \(q\) is parameterized by a linear layer that transforms the conditioning embedding into weights over the vocabulary. The final training loss is \[\begin{align} l_\text{lang}(x_{1:T_x})+l_\text{lang}(y_{1:T_y})+l_\text{ctx}+l_\text{desc} \label{eq:loss} \end{align}\tag{1}\] Same as the original ELMo, each log loss is approximated with negative sampling [73]. We write EntELMo to denote the model trained by Eq. 1 . When using EntELMo for contextualized entity representations and descriptive entity representations, we use it analogously to ELMo.

5 Experiments↩︎

5.1 Setup↩︎

As a baseline for hyperlink-based training, we train EntELMo on the WikiEnt dataset with only a bidirectional language model loss. Due to the limitation of computational resources, both variants of EntELMo are trained for one epoch (3 weeks time) with smaller dimensions than ELMo. We set the hidden dimension of each directional long short-term memory network (LSTM; [74]) layer to be 600, and project it to 300 dimensions. The resulting vectors from each layer are thus 600 dimensional. We use 1024 as the negative sampling size for each positive word token. For bag-of-words reconstruction, we randomly sample at most 50 word tokens as positive samples from the the target word tokens. Other hyperparameters are the same as ELMo. EntELMo is implemented based on the official ELMo implementation.⁴

As a baseline for contextualized and descriptive entity representations, we use GloVe word averaging of the entity mention as the “contextualized” entity representation, and use word averaging of the truncated entity description text as its description representation. We also experiment two variants of EntELMo, namely EntELMo w/o \(l_\text{ctx}\) and EntELMo with \(l_\text{etn}\). For second variant, we replace \(l_\text{ctx}\) with \(l_\text{etn}\), where we only decode entity mentions instead of the whole context from descriptions. We lowercased all training data as well as the evaluation benchmarks.

We evaluate the transferrability of ELMo, EntELMo, and BERT by using trainable mixing weights for each layer. For ELMo and EntELMo, we follow the recommendation from [1] to first pass mixing weights through a softmax layer and then multiply the weighted-summed representations by a scalar. For BERT, we find it better to just use unnormalized mixing weights. In addition, we investigate per-layer performance for both models in Section 6.

5.2 Results↩︎

Table 3 shows the performance of our models on the EntEval tasks. Our findings are detailed below:

6 Analysis↩︎

6.0.0.1 Is descriptive entity representation necessary?

A natural question to ask is whether the entity description is needed, as for humans, the entity names carry sufficient amount of information for a lot of tasks. To answer this question, we experiment with encoding entity names by the descriptive entity encoder for ERT (entity relationship typing) and NED (named entity disambiguation) tasks. The results in Table 4 show that encoding the entity names by themselves already captures a great deal of knowledge regarding entities, especially for CoNLL-YAGO. However, in tasks like ERT, the entity descriptions are crucial as the names do not reveal enough information to categorize their relationships.

Table 5 reports the performance of different descriptive entity representations on the CoNLL-YAGO task. The three models all use ELMo as the context encoder. “ELMo” encodes the entity name with ELMo as descriptive encoder, while both [4] and Deep ED [19] use their trained static entity embeddings. ⁵ As [4] and Deep ED have different embedding sizes from ELMo, we add an extra linear layer after them to map to the same dimension. These two models are designed for entity linking, which gives them potential advantages. Even so, ELMo outperforms them both by a wide margin.

6.0.0.2 Per-Layer Analysis.

We evaluate each ELMo and EntELMo layer, i.e., the character CNN layer and two bidirectional LSTM layers, as well as each BERT layer on the EntEval tasks.

Figure 6 reveals that for ELMo models, the first and second LSTM layers capture most of the entity knowledge from context and descriptions. The BERT layers show more diversity. Lower layers perform better on ESR (entity similarity and relatedness), while for other tasks higher layers are more effective.

7 Conclusion↩︎

Our proposed EntEval test suite provides a standardized evaluation method for entity representations. We demonstrate that EntEval tasks can benefit from the success of contextualized word representations such as ELMo and BERT. Augmenting encoding-decoding loss leveraging natural hyperlinks from Wikipedia further improves ELMo on some EntEval tasks. As shown by our experimental results, the contextualized entity encoder benefits more from this hyperlink-based training objective, suggesting future works to prioritize encoding entity description from its mention context.

Acknowledgments↩︎

We thank Davis Yoshida for discussions at the early stages of this project, Kushal Arora and Jackie Chi Kit Cheung for answering our questions on the Rare Entity Prediction dataset, and Nitish Gupta for clarifying details about models from [4]. This research was supported in part by a Bloomberg data science research grant to K. Stratos and K. Gimpel.

References↩︎