1 Introduction↩︎

As a core language understanding task, named entity recognition (NER) faces rapid domain shifting. When transferring NER systems to new domains, one of the primary challenges is dealing with the mismatch of entity types [1]. For example, only 2 types are overlapped between I2B2 [2] and OntoNotes [3], which have 23 and 18 entity types, respectively. Unfortunately, annotating a new domain takes considerable time and efforts, let alone the domain knowledge required [4]. Few-shot NER is targeted in this scenario since it can transfer prior experience from resource-rich (source) domains to resource-scarce (target) domains.

Previous few-shot NER models [1], [4]–[6] generally formulate the task as a sequence labeling task and employ token-level prototypical networks [7]. These models first obtain token labels according to the most similar token-prototype pair and then obtain entities based on these labels, as Figure 1 (a) shows. The sequence labeling benefits from label dependency [4]. However, when it comes to few-shot NER models, the label dependency is off the table, because a few labeled data is way insufficient to learn the reliable dependency, and the label sets could vary from domain to domain. To tackle this, some methods try to transfer roughly estimated dependency. Hou et al. first learn the abstract label transition probabilities in source domains and then copy them to target domains. As Figure 2a shows, the abstract O\(\rightarrow\)I probability is copied to three targets directly (the red lines). However, this makes the target probability sum of O\(\rightarrow\)(all labels) end up with 160%. To avoid the possible probability overflows, Yang and Katiyar propose an even distribution method. As Figure 2b shows, the abstract O\(\rightarrow\)I probability is distributed evenly among the three targets (the green lines). However, this could lead to severe contradictions between the target probabilities and reality. For example, there are 4,983 DATE entities and only one EMAIL entity in the I2B2 test set, so the target probabilities of O\(\rightarrow\)I-DATE and O\(\rightarrow\)I-EMAIL should be clearly different.

Consequently, the current dependency transferring may lead to misclassifications due to the roughly estimated target transition probabilities, even though it sheds some light on few-shot NER.

In addition, the majority of prototypical models for few-shot NER [8], [9] obtain prototypes by averaging the embeddings of each class’s support examples, while Yoon et al. and Hou et al. demonstrate that such prototypes distribute closely in the embedding space, thus often causing misclassifications.

In this paper, we aim to tackle the above issues inherent in token-level prototypical models. To this end, we propose EP-Net, an sec:Entity-level Prototypical Network enhanced by dispersedly distributed prototypes, as Figure 1 (a) shows. EP-Net builds entity-level prototypes and considers text spans as candidate entities. Thus it can determine whether a span is an entity directly according to the most similar prototype to the span. This also eliminates the need for the label dependency. For example, EP-Net determines the “rain’’ and “tonight” are two entities, and their types are the Weather and Time, respectively (Figure 1 (a)).¹ In addition, to distribute these prototypes dispersedly, EP-Net trains them using a distance-based loss from scratch. And EP-Net aligns spans and prototypes in the same embedding space by utilizing a deep neural network to map span representations to the embedding spaces of prototypes.

In essence, EP-Net is a span-based model. Several span-based models [10]–[12] have been proposed for the supervised NER task. Our EP-Net differs from these models in two ways: (1) The EP-Net obtains entities based on the span-prototype similarity, while these models do so by classifying span representations. (2) The EP-Net works effectively with few labeled examples, whereas these models need a large number of labeled examples to guarantee good performance.

We evaluate our EP-Net on the tag set extension and domain transfer tasks, as well as the Few-NERD settings. Experimental results demonstrate that EP-Net consistently achieves new state-of-the-art overall performance. Qualitative analyses (\(\S\)5.5-5.6) and ablation studies (\(\S\)5.7) further validate the effectiveness of EP-Net.

In summary, we conclude the contributions as follows: (1) As far as we know, we are among the first to propose an entity-level prototypical network for few-shot NER. (2) We propose a prototype training strategy to augment the prototypical network with dispersedly distributed prototypes. (3) Our model achieves the current best overall performance on two evaluation tasks and the Few-NERD.

2 Related Work↩︎

Meta Learning. Meta learning aims to learn a general model that enables us to adapt to new tasks rapidly based on a few labeled examples [9]. One of the most typical metric learning methods is the prototypical network [7], which learns a prototype for each class and classifies an item based on item-prototype similarities. Metric learning has been widely investigated for NLP tasks, such as text classification [13]–[15], relation classification [16], [17] and NER [8]. However, these methods use the prototypes obtained by averaging the embeddings of support examples for each class, which are closely distributed. In contrast, our model uses dispersedly distributed prototypes obtained by supervised prototype training.

Few-shot NER. Previous few-shot NER models [6], [9] generally formulate the task as a sequence labeling task and propose to use the token-level prototypical network. Thus these models call for label dependency to guarantee good performance. However, it is hard to obtain exact dependency since the label sets vary greatly across domains. As an alternative, Hou et al. propose to transfer estimated dependency. They copy the learned abstract dependency from source to target domains, but the target dependency contradicts the probability definition. Yang and Katiyar propose StructShot, which improves the above dependency transferring by equally distributing the abstract dependency to target domains, whereas the target dependency contradicts the reality. Das et al. introduce Contrastive Learning to the StructShot, which inherits the estimated dependency transferring. We demonstrate that the roughly estimated dependency may harm model performance. In addition, prompt-based models [18]–[22] have been widely researched for this task recently, but the model performance heavily relies on the chosen prompts. Current with our work, Wang et al. also propose a span-level prototypical network to bypass label dependency, but their work is still hampered by closely distributed prototypes. In contrast, our model constructs dispersedly distributed entity-level prototypes, thus avoiding the roughly estimated label dependency and closely distributed prototypes.

3 Task Formulation and Setup↩︎

In this section, we formally define the task and then introduce the standard evaluation setup.

3.1 Few-shot NER↩︎

We define an unstructured sentence as a token sequence \(\mathcal{X}\) \(=\) \((x_1, x_2,…, x_n)\), and define entities annotated in \(\mathcal{X}\) as \({\mathcal{sec:E}} = [(e^{(1)}, t^{(1)}), ..., (e^{(k)}, t^{(k)})]\), where \(e^{(i)}\) and \(t^{(i)}\) denote entity text and entity type, respectively. A domain \(\mathcal{sec:D}=\{(\mathcal{X}^{(i)}, \mathcal{sec:E}^{(i)})\}_{i=1}^{N_\mathcal{sec:D}}\) is a set of (\(\mathcal{X}, \mathcal{sec:E}\)) pairs, and each \(\mathcal{sec:D}\) has a domain-specific entity type set \(\mathcal{T}_\mathcal{sec:D}=\{{t}_i\}^N_{i=1}\), and \(N\) is various across domains.

We achieve the few-shot task through three steps: Train, Adapt and Recognize. We first train EP-Net with the data of source domains \(\{\mathcal{sec:D}_1, \mathcal{sec:D}_2, . . .\}\). Then we then adapt the trained EP-Net to target domains {\(\mathcal{sec:D}’_1, \mathcal{sec:D}’_2,…\)} by fine-tuning it on support sets sampled from target domains. Finally, we recognize entities of query sets using the domain-adapted EP-Net. We formulate a support set as \(\mathcal{S}\) \(=\) \(\{(\mathcal{X}^{(i)}, \mathcal{sec:E}^{(i)})\}_{i=1}^{N_\mathcal{S}}\), where \(\mathcal{S}\) usually includes a few labeled examples (\(K\)-shot) of each entity type. For a target domain, we formally define the \(K\)-shot NER as follows: given a sentence \(\mathcal{X}\) and a \(K\)-shot support set, find the best entity set \({\mathcal{sec:E}}\) for \(\mathcal{X}\).

3.2 The Standard Evaluation Setup↩︎

To facilitate meaningful comparisons of results for future research, Yang and Katiyar propose a standard evaluation setup. The setup consists of the query set and support set constructions.

3.2.1 Query Set Construction↩︎

They argue that traditional construction methods sample different entity classes equally without considering entity distributions. For example, the I2B2 test set contains 4,983 DATE entities, while it only contains one EMAIL entity. Thus they propose to use the original test sets of standard NER datasets as the query sets, thus improving the reproducibility of future studies.

3.2.2 Support Set Construction↩︎

To construct support sets, they propose a Greedy Sampling algorithm to sample sentences from the dev sets of standard NER datasets. In particular, the algorithm samples sentences for entity classes in increasing order of their frequencies. We present the algorithm in Appendix 7.

4 Model↩︎

In this section, we first provide an overview of EP-Net in \(\S\)4.1, and then illustrate the model initializations in \(\S\)4.2 and discuss the model in \(\S\)4.3.

4.1 EP-Net↩︎

Figure 3 shows the overall architecture of EP-Net. Given a domain \(\mathcal{sec:D}\) \(=\) \(\{(\mathcal{X}^{(i)}, \mathcal{sec:E}^{(i)})\}_{i=1}^{N_\mathcal{sec:D}}\) and its entity type set \(\mathcal{T}_\mathcal{sec:D}=\{t_i\}_{i=1}^N\), we first initialize an entity-level prototype for each \(t_i\) (Figure 3-). \[\pmb{\it{\Phi}} = \{\pmb{\phi}_0, \pmb{\phi}_1, \pmb{\phi}_2,..., \pmb{\phi}_N\},\] where \(\pmb{\it{\Phi}}\in\mathbb{R}^{(N{+}1)*{d_1}}\), and \(d_1\) is the dimension of prototype representation. \(\pmb{\phi}_0\) is the prototype of the None type, and \(\pmb{\phi}_i \;(i >0)\) is the prototype of \(t_i\).

We design a distance-based loss \(\mathcal{L}_d\) to supervise the prototype training (Figure 3-), aiming to distribute these prototypes dispersedly in the embedding space (Figure 3-). We argue that the prototypes should be dispersedly distributed in an appropriate-sized embedding space, neither too large nor too small (\(\S\)5.5). Thus we first set a threshold \(\tau\) to limit the averaged prototype distance. Then we calculate the squared Euclidean distance between any two prototypes and obtain the averaged prototype distance (denoted as \(Euc(\pmb{\it{\Phi}})\)). Next, we construct the \(\mathcal{L}_d\) as follows. \[\begin{normalsize} \begin{align} {!}{Euc(\pmb{\it{\Phi}})} &= {!}{ \frac{\sum_{i=0}^{N}\sum_{j=0}^{N}\sum_{k=1}^{d} {(\pmb{\it{\phi}}_{i,k}-\pmb{\it{\phi}}_{j,k}})^2}{(N+1)^2}\small{,}} \\ \psi&=\left\{ \begin{array}{rcl} {!}{Euc(\pmb{\it{\Phi}})-\tau} & {\textrm{if}}\;\;\; {!}{Euc(\pmb{\it{\Phi}})\geq \tau,} \\ {!}{\tau-Euc(\pmb{\it{\Phi}})} & {\textrm{if}}\;\;\;{!}{Euc(\pmb{\it{\Phi}})<\tau,} \end{array}\right.\\ \mathcal{L}_d &= \log(\psi+1) \label{2c}. \end{align} \end{normalsize}\tag{1}\] The training goal is to achieve \(\psi\rightarrow0^+\), which equals to \({(\psi+1)}\rightarrow1^+\). Thus we design the \(\log(\cdot)\) loss (Eq.@eq:2c ), where the smaller the \(\mathcal{L}_d\), the more the \((\psi+1)\rightarrow1^+\).

Next, given a sentence \(\mathcal{X}=\{x_i\}_{i=1}^{n}\) of the domain \(\mathcal{sec:D}\), we first obtain text spans (denoted as \(s\), Figure 3-): \[\label{eq3} s =\{x_i, x_{i+1},..., x_{i+j}\} \;{!}{s.t. \;1 \leq i \leq i+j \leq n,}\tag{2}\]

where the span length \(j\)+\(1\) is limited by a threshold \(\epsilon\): \(j\)+\(1\leq\epsilon\). To obtain the span representation (Figure 3-), we first use BERT [23] to generate the embedding sequence of \(\mathcal{X}\). \[\boldsymbol{\em H}_\mathcal{X} = \{\pmb{h}_1, \pmb{h}_2,…, \pmb{h}_n\},\]

where \(\boldsymbol{\em H}_\mathcal{X}\in \mathbb{R}^{n*d_2}\), and \(d_2\) is the BERT embedding dimension. \(\pmb{h}_i\) is the BERT embedding of \(x_i\). We use \(\boldsymbol{\em H}_s\) to denote the BERT embedding sequence of span \(s\). \[\boldsymbol{\em H}_{s} = \{\pmb{h}_i, \pmb{h}_{i+1},…, \pmb{h}_{i+j}\}.\]

We obtain the span representation (denoted as \(\boldsymbol{\em sec:E}_s\)) by concatenating the max-pooling of \(\boldsymbol{\em H}_s\) (denoted as \(\tilde{\boldsymbol{\em H}}_s\)) and the span length embedding. \[\begin{normalsize} \begin{align} \tilde{\boldsymbol{\em H}}_s &={!}{ [\max(\pmb{h}_{i,1},..., \pmb{h}_{i+j,1}),..., \max(\pmb{h}_{i,d_1},..., \pmb{h}_{i+j,d_1})],} \\ \boldsymbol{\em sec:E}_{s} &= [\tilde{\boldsymbol{\em H}}_s; \pmb{w}_{j+1}]\label{6b}, \end{align} \end{normalsize}\tag{3}\] where \(\tilde{\boldsymbol{\em H}}_s\in\mathbb{R}^{d_2}\), \(\boldsymbol{\em sec:E}_{s} \in \mathbb{R}^{d_2+d_3}\). \(\pmb{w}_{j+1}\) is the length embedding trained for spans with a length \(j\)+\(1\) and \(d_3\) is the embedding dimension. Due to the fact that \(\boldsymbol{\em sec:E}_{s}\) and prototype representations (\(\pmb{\it{\Phi}}\)) are not in the same embedding space, we project \(\boldsymbol{\em sec:E}_{s}\) to the embedding space of \(\pmb{\it{\Phi}}\) using a multi-layer Feed Forward Network (FFN)² and denote the aligned span representation as \(\tilde{\boldsymbol{\em sec:E}}_s\) (Figure 3-). \[\label{eq7} \tilde{\boldsymbol{\em sec:E}}_{s} = \boldsymbol{\em sec:E}_{s} \boldsymbol{\em W} + \boldsymbol{\em b},\tag{4}\] where \(\tilde{\boldsymbol{\em sec:E}}_{s} \in \mathbb{R}^{d_1}\), \(\boldsymbol{\em W}\) and \(\boldsymbol{\em b}\) are FFN parameters. Next, for the span \(s\), we calculate the similarity between it and each prototype \({\pmb{\phi}_i}\in \pmb{\it\Phi}\) using the squared Euclidean distance. \[\label{eq8} \textrm{sim}(s, {\pmb\phi_i})_{i=0}^N = \sum_{j=1}^d(\tilde{\boldsymbol{\em sec:E}}_{s,j}-\pmb\phi_{i,j})^2.\tag{5}\]

As a shorter distance denotes a better similarity, we classify entities based the shortest distance (Figure 3-): \[\label{eq9} {t}_s = \mathrm{arg}\;\underset{\pmb{\phi}_i \in \pmb{\it\Phi}}{\mathrm{min}} \;\textrm{sim}(s, {\pmb{\it\phi_i}})_{i=0}^N,\tag{6}\] where \({t}_s\in\mathcal{T}_\mathcal{sec:D}\) is the type classified for the span \(s\).

To construct an entity classification loss, we first take the \(-\textrm{sim}(s, {\pmb\phi_i})_{i=0}^N\) as classification logits, thus the best similarity has the largest logit. We then normalize these logits using the softmax function. Finally, we construct a cross-entropy loss \(\mathcal{L}_s\). \[\tag{7} \begin{normalsize} \begin{align} \pmb{\hat{y}}_{s,i} &= \frac{\exp^{-\textrm{sim}(s, \pmb\phi_i)}}{\sum_{j=0}^{N}{\exp^{-\textrm{sim}(s, \pmb\phi_j)}}}, \\ \mathcal{L}_s &= -\frac{1}{M_{s}} \sum_{j=1}^{M_{s}}\sum_{i=0}^{N}{\pmb y}^j_{{s,i}}\log\hat{\pmb y}^j_{{s,i}}\tag{8}, \end{align} \end{normalsize}\] where \(\{\pmb{{y}}_{{s}}, \pmb{{\hat{y}}}_{{s}}\}\in\mathbb{R}^{N+1}\), and \(\pmb{{y}}_{{s}}\) is the one-hot vector of the gold type for the span \({{s}}\). \({M_{s}}\) is the number of span instances.

During model training, we optimize model parameters by minimizing the following joint loss. \[\mathcal{L}(\it{W};{\theta}) = \mathcal{L}_d + \mathcal{L}_s.\]

4.2 Initializations↩︎

4.2.1 Train Initialization↩︎

In the Train step, given a source domain \(\mathcal{sec:D}\) with an entity type set \(\mathcal{T}_\mathcal{sec:D}\) \(=\) \(\{{t}_i\}^N_{i=1}\), we randomly initialize the entity-level prototypes \(\pmb{\it{\Phi}}=\{\pmb{\phi}_i\}_{i=0}^N\).We assign \(\pmb{\phi}_0\) to the None type and \(\pmb{\phi}_i\) to \(t_i\). To guarantee that we can adapt EP-Net to target domains that have more types than the domain \(\mathcal{sec:D}\), we actually initialize \(\pmb{\it{\Phi}}=\{\pmb{\phi}_i\}_{i=0}^{100}\), where EP-Net can be adapted to any target domains with entity types less than 100. Moreover, the \(N\) can be set to an even larger value if necessary. By doing so, the prototypes \(\{\pmb{\phi}_i\}_{i=N+1}^{100}\) are unassigned, but we can still distribute them dispersedly through training them using the loss \(\mathcal{L}_d\) (\(\S\)5.6). We use the bert-base-cased model in the embedding Layer.³

4.2.2 Adapt and Recognize Initializations↩︎

In the Adapt step, given a target domain \(\mathcal{D'}\) with an entity type set \(\mathcal{T}_{\mathcal{sec:D}'}\) \(=\) \(\{{{t}'_i}\}^{N'}_{i=1}\) and the EP-Net trained in the Train step, we first assign a prototype of the trained \(\pmb{\it{\Phi}}=\{\pmb{\phi}_i\}_{i=0}^{100}\) to each \(t'_i\). In particular, we assign \(\pmb{\phi}_0\) to the None type. And if there are types that are overlapped between \(\mathcal{T}_\mathcal{sec:D}\) and \(\mathcal{T}_\mathcal{D'}\) (i.e., \(\mathcal{T}_\mathcal{sec:D} \cap \mathcal{T}_\mathcal{D'}\neq \varnothing\)), for each overlapped type, we reuse the prototype assigned in the Train step. For other types in \(\mathcal{T}_\mathcal{D'}\), we randomly assign an unassigned prototype in \(\pmb{\it{\Phi}}\) to it, and we first choose the prototype that is ever assigned in the Train step. Then, we adapt EP-Net to the domain \(\mathcal{sec:D}'\) by fine-tuning it on support sets sampled from \(\mathcal{D'}\).

However, Fine-tuning the model with small support sets runs the risk of overfitting. To avoid this, we propose to use the following strategies: (1) We freeze the BERT and solely fine-tune the assigned prototypes and the multi-layer FFN. (2) We use an early stopping criterion, where we continue fine-tuning our model until the loss starts to increase. (3) We set upper limits for fine-tuning steps, where the model will stop when reaching the limits even though the loss continues decreasing. With the above strategies, we demonstrate that only a few fine-tuning steps on these examples can make rapid progress without overfitting.

In the Recognize step, we use the domain-adapted EP-Net to recognize entities in the query set of \(\mathcal{D'}\) directly.

4.3 Model Discussion↩︎

In the Train step, the randomly initialized prototypes cannot represent entity types at first. Through the joint model training with the \(\mathcal{L}(\it{W};{\theta})\), EP-Net establishes correlations between entity types and their assigned prototypes. Moreover, the multi-layer FFN can also be trained to cluster similar spans around related prototypes in the embedding space. As Figure 3- shows, the “rain” is mapped to be closer to the Weather than other prototypes.

To precisely simulate the few-shot scenario, we are not permitted to count the entity length of target domains. Thus we set the span length threshold \(\epsilon\) to an empirical value of 10 based on source domains. For example, 99.89% of the entities in the OntoNotes have lengths under 10.

We propose a heuristic method for removing overlapped entities classified by EP-Net. Specifically, we keep the one with the best span-prototype similarity of those overlapped entities and drop the others.

Concurrently, Wang et al. propose a span-level model – ESD. We summarize how our EP-Net differs from the ESD as follows: (1) Our EP-Net fine-tunes on support sets while the ESD solely uses them for similarity calculation without fine-tuning. Ma et al. claim that the fine-tuning method is far more effective in using the limited information in support sets. (2) The ESD obtains class prototypes with embeddings of the same classes in support sets, thus suffering from closely distributed prototypes [4]. By contrast, our EP-Net avoids this by training dispersedly distributed prototypes from scratch.

5 Experiments↩︎

5.1 Evaluation Tasks↩︎

We evaluate EP-Net on two evaluation tasks and the Few-NERD settings using 1- and 5-shot settings. Limited by space, we solely report the key points here and discuss more details in Appendix 8.

Tag Set Extension. This task aims to evaluate models for recognizing new types of entities in existing domains. Yang and Katiyar divide the 18 entity types of the OntoNotes [3] into three target sets, i.e., Group A, sec:B and sec:C, to simulate this scenario. Models are evaluated on one target set while being trained on the others.

Domain Transfer. This task aims to evaluate models for adapting to a different domain. Yang and Katiyar propose to use the general domain as the source domain and test model on medical, news, and social domains.

Few-NERD Settings. Few-NERD [24] is a large-scale dataset for few-shot NER. It consists of two different settings: Intra and Inter. The Intra divides the train/dev/test according to coarse-grained types. The Inter divides the train/dev/test according to fine-grained types. Thus the coarse-grained entity types are shared. The Intra is more challenging as the restrictions of sharing coarse-grained types.

5.2 Datasets and Baselines↩︎

For a fair comparison, we use the same datasets and baselines reported in [1], [24], [25]. Specifically, we use OntoNotes (general domain), CoNLL 2003 [26] (news domain), I2B2 2014 [2] (medical domain) and WNUT 2017 [27] (social domain) for the tag set extension and domain transfer tasks.

We compare the performance of EP-Net with previous best models, including: Prototypical Network (ProtoNet) [7], ProtoNet+P&D [4] , NNShot and StructShot [1] and CONTaiNER [25]. We represent more baseline details in Appendix 9.

5.3 Implementation Details↩︎

In all experiments, we optimize EP-Net using AdamW with a learning rate of 5\(e\)-5 and set \(d_1\) and \(d_3\) to 512 and 25, respectively. \(d_2\) is 768 when using the BERT base model. We set 3 layers for the multi-layer FFN, the distance threshold \(\tau\) to 2, and the train batch size to 2 and 8 in 1- and 5-shot experiments, respectively. We set the distance threshold \(\tau\) to 2 and 3 for 1- and 5-shot experiments, respectively. Moreover, we investigate the model performance against different \(\tau\) values in Appendix 10. Following supervised span-based work [28], we sample spans of the None type during model training and set the sampled count to 20 and 40 in 1- and 5-shot experiments, respectively. Following [1], [25], we sample 5 support sets and report the mean and standard deviation of the F1 scores in each experiment.

5.4 Main Results↩︎

We report experimental results for 1- and 5-shot settings in Table 1 and Table 2, respectively. We have the following observations.

(1) In terms of the overall metric (i.e., Avg.), EP-Net consistently outperforms the listed baselines on the two tasks and Few-NERD, delivering +1.5% to +7.0% averaged F1 gains. Moreover, EP-Net improves up to +11.4% F1 scores on 1-shot Group B. We attribute these gains to the advantages of the proposed entity-level prototypical network.

(2) On the 5-shot Group C, EP-Net is inferior to CONTaiNER by 8.5% F1 scores. Detailed error analysis indicates that the group’s DATE type should bear the primary responsibility. Of the 4,178 entities in the test set, 1,536 are DATE entities, in which there are up to 429 different expressions, such as “week”, “this week”, “last week”, “2 weeks”, “2 - week” etc. However, the 5-shot setting solely enables us to sample very few various expressions, leading to the poor performance in DATE entities. For example, if a support set solely samples the “week”, it is hard for EP-Net to recognize entities like “this week” and “last week”.

In addition, we conduct episode evaluations on Few-NERD and report the results in Appendix 11.

5.5 Visualization↩︎

We use the 1-shot Group A experiment to investigate prototype distributions. Specifically, in the Train step we initialize the prototype set \(\pmb{\it{\Phi}}=\{\pmb\phi\}_{i=0}^{100}\) and assign \(\{\pmb\phi\}_{i=0}^{12}\) to the None type and the 12 pre-defined entity types of the source domain. In the Adapt step, we assign \(\{\pmb\phi\}_{i=0}^{6}\) to the None type and the 6 pre-defined entity types of the Group A.

We report the visualization results in Figure 4. From Figure 4b, we observe that all prototypes are dispersedly distributed because the Euclidean distance between any two prototypes is approximate 2. Therefore, we conclude that EP-Net can distribute the prototypes dispersedly through the prototype training. From Figure 4a, we see that the distances between the None type and other assigned prototypes are generally larger than other distances. We attribute it to the fact that the None type does not represent any unified semantic meaning, thus the None spans actually correspond to a variety of semantic spaces, requiring the None prototype to keep away from other prototypes to alleviate the misclassification problem.

Moreover, we realize another entity-level prototypical network with conventional prototypes⁴, and refer to it as CP-Net. We do not train the conventional prototypes with the loss \(\mathcal{L}_d\) but fine-tune it during the model training. We report more details of CP-Net in Appendix 12.

We visualize the distributions of our prototypes and conventional prototypes in Figure 5. To be specific, we use prototypes obtained in the Recognized step of both models. We observe that: (1) Our prototypes are distributed much more dispersedly than the conventional prototypes. (2) Our None prototype is more distant from other prototypes, whereas the conventional None prototype stays close to other conventional prototypes. These results indicate that our prototypes enable us to alleviate the misclassifications caused by closely distributed prototypes.

5.6 How does the Dispersedly Distributed Prototyes Enhance the EP-Net?↩︎

We run the 1-shot Group A experiment with EP-Net and CP-Net to conduct the investigation. We first compare the F1 scores of the two models. The results show our EP-Net outperforms CP-Net by +9.6% F1 scores, verifying the effectiveness of the dispersedly distributed prototypes.

In addition, we use t-SNE [29] to reduce the dimension of span representations obtained in the Recognize step of EP-Net and CP-Net and visualize these representations in Figure 6. We can see that our EP-Net clusters span representations of the same entity class while dispersing span representations of different entity classes obviously, which we attribute to the usage of dispersedly distributed prototypes. Based on the above fact, we conclude that our EP-Net can greatly alleviate the misclassifications caused by closely distributed prototypes.

5.7 Ablation Study↩︎

We conduct ablation studies to investigate the significance of model components and report the results in Table 3. Specifically, (1) In the “- Entity\(-\)level prototype”, we ablate the entity-level prototypes and use token-level prototypes instead. Moreover, we use the copying method (Figure 2) to transfer the label dependency. The ablation results show that the F1 scores drop from 5.1% to 7.2%, validating the advantages of entity-level prototypes. (2) In the “\(-\) Prototype training”, we remove the loss \(\mathcal{L}_d\) from the \(\mathcal{L}(\it{W};{\theta})\), thus the prototypes are not trained being dispersedly distributed. The decreasing F1 scores (5.8% to 11.8%) demonstrate that EP-Net significantly benefits from the dispersedly distributed prototypes. (3) In the “\(-\)Euclidean distance”, we use the cosine similarity to measure span-prototype similarities instead. We see that the Euclidean similarity consistently surpasses the cosine similarity, revealing that a proper measure is vital to guarantee good performance, which is consistent with the conclusion in [7].

6 Conclusion↩︎

In this paper, we propose an entity-level prototypical network for few-shot NER (EP-Net). And we augment EP-Net with dispersedly distributed prototypes. The entity-level prototypes enable EP-Net to avoid suffering from the roughly estimated label dependency brought by abstract dependency transferring. Moreover, EP-Net distributes the prototypes dispersedly via supervised prototype training and maps spans to the embedding space of the prototypes to eliminate the alignment biases. Experimental results on two evaluation tasks and the Few-NERD settings demonstrate that EP-Net beats the previously published models, creating new state-of-the-art overall performance. Extensive analyses further validate the model’s effectiveness.

Appendix

7 The Greedy Sampling Algorithm↩︎

8 Details of the Evaluation Task↩︎

8.1 Tag Set Extension↩︎

The Group A, B, and C split from the OntoNotes dataset are as follows.

In this task, we evaluate our EP-Net on each group while training our model on the other two groups. In each experiment, we modify the training set by replacing all entity types in the target type set with the None type. Hence, these target types are no longer observed during training. We use the modified training set for model training in the Train step. Similarly, we modify the dev and test sets to only include entity types contained in the target type set. We use the Greedy Sampling Algorithm to sample multiple support sets from the dev set for model adaption.

8.2 Domain Transfer↩︎

In this task, we train our EP-Net on the standard training set of the OntoNotes dataset and evaluate our model on the standard test sets of I2B2, CoNLL, and WNUT. In addition, we sample support sets for model adaption from the standard dev sets of the above three datasets.

8.3 Few-NERD Settings↩︎

FEW-NERD [24] is the first dataset specially constructed for few-shot NER and is one of the largest human-annotated NER datasets. It consists of 8 coarse-grained entity types and 66 fine-grained entity types. The dataset contains two sub-sets, name Intra and Inter.

We use the standard evaluation (\(\S\)3.2) and the episode evaluation to evaluate the performance of our EP-Net. For the standard evaluation, we conduct experiments on Intra and Inter, respectively. We first use the training set to train our EP-Net and then sample support sets from the test set for the model adaptation and evaluate our model on the remaining test set. For the episode evaluation, we use the exact evaluation setting proposed by [24].

9 Baseline Details↩︎

Following the established line of work [1], [24], [25], we compare EP-Net with the following competitive models.

For a fair comparison, we use the results of the ProtoNet, ProtoNet+P&D, NNShot, and StructShot reported in [1], and the results of CONTaiNER reported in [25].

In addition, we run the ProtoNet, ProtoNet+P&D, NNShot, and StructShot on Few-NERD using the standard evaluation setup (\(\S\)3.2, 8.3).

10 Performance against Prototype Distance Threshold (\(\tau\))↩︎

We conduct 1- and 5-shot experiments to explore the performance against different \(\tau\) values. Since validation sets are unavailable in the few-shot scenario, we randomly sample 20% of the query sets for the explorations. We report the results in Figure 7, where we set the \(\tau\) value from 1 to 10, respectively. We can observe that: (1) The F1 scores generally first increase and then decrease when the \(\tau\) value consistently increases. (2) Except for the Group C and Intra, our EP-Net performs the best in the 1-shot experiments when setting the \(\tau\) to 2. (3) Except for the Group A and Intra, our EP-Net performs the best in the 5-shot experiments when setting the \(\tau\) to 3.

The above results validate our argument that the prototypes should be distributed in an appropriate-sized embedding space, neither too large nor too small (\(\S\)4.1). For simplicity, we set the \(\tau\) to 2 and 3 in all the other 1- and 5-shot experiments, respectively.

11 Episode Evaluation on Few-NERD↩︎

We evaluate our EP-Net on Few-NERD with the episode evaluation setting and compare our model with previous state-of-the-art models, including ProtoBERT [24], NNShot, StructShot, CONTaiNER, and ESD [30]. We would like to mention that the ESD is a concurrent span-based few-shot NER model to ours.

We report the results in Table 4 and Table 5, where we take the results of ProtoBERT, NNShot, and StructShot reported in [24], and the results of CONTaiNER and ESD reported in their original papers. We can see that:

The above results demonstrate the effectiveness of the proposed EP-Net. And compared to ESD, our model is more efficient in the few-shot scenario when entities share less coarse-grained information (the Intra).⁵

Compared to our simple concatenation method (Eq.@eq:6b ) to obtain span representations, ESD proposes to use Inter Span Attention (ISA) and Cross Span Attention (CSA) to enhance the span representations. We believe that the ISA and CSA enable ESD to encode the shared coarse-grained information into span representations sufficiently, which helps ESD obtain the current state-of-the-art performance on the Inter dataset.

12 CP-Net↩︎

We propose the CP-Net as a comparable model to our EP-Net. CP-Net is also an entity-level prototypical network, but it uses conventional prototypes obtained by averaging the embeddings of type’s examples. Similar to EP-Net, CP-Net also uses the BERT model as an embedding generator. In addition, it uses the sampling strategy discussed in \(\S\)5.3 to randomly sample None spans. CP-Net consists of two steps, namely Train and Recognize.

In the Train step, we train CP-Net with the source domain data. To be specific, we obtain the entity-level prototypes by averaging the embeddings of type’s examples in the training set. Moreover, we obtain span representations with the same method of EP-Net (Eq.@eq:eq3 4 ), as well as the method to calculate span-prototype similarity (Eq.@eq:eq8 6 ). During the model training, we use the training loss \(\mathcal{L}_s\) (Eq.@eq:eq10b ) to fine-tune the BERT model.

In the Recognize step, we use the fine-tuned BERT model as the embedding generator and obtain the entity-level prototypes by averaging the embeddings of each type’s examples in the support sets. Then we obtain the type of each span according to the best similarity between the span and the prototypes.

The CP-Net differs from our EP-Net in the following two ways.

References↩︎