Augmenting NER Datasets with LLMs: Towards Automated and Refined Annotation

Yuji Naraki 1
LegalOn Technologies
Tokyo, Japan
yuji.naraki@legalontech.jp
Ryosuke Yamaki\(^*\)
Ritsumeikan University / ProPlace Inc
Shiga / Tokyo, Japan
yamaki.ryosuke@em.ci.ritsumei.ac.jp
Yoshikazu Ikeda
Osaka University / ProPlace Inc
Osaka / Tokyo, Japan
y-ikeda@sigmath.es.osaka-u.ac.jp Takafumi Horie
Ritsumeikan University
Shiga, Japan
horie.takafumi@em.ci.ritsumei.ac.jp
Hiroki Naganuma
Mila, Université de Montréal / ProPlace Inc.
Montreal, Canada / Tokyo, Japan
hiroki.naganuma@proplace.jp


Abstract

In the field of Natural Language Processing (NLP), Named Entity Recognition (NER) is recognized as a critical technology, employed across a wide array of applications. Traditional methodologies for annotating datasets for NER models are challenged by high costs and variations in dataset quality. This research introduces a novel hybrid annotation approach that synergizes human effort with the capabilities of Large Language Models (LLMs). This approach not only aims to ameliorate the noise inherent in manual annotations, such as omissions, thereby enhancing the performance of NER models, but also achieves this in a cost-effective manner. Additionally, by employing a label mixing strategy, it addresses the issue of class imbalance encountered in LLM-based annotations. Through an analysis across multiple datasets, this method has been consistently shown to provide superior performance compared to traditional annotation methods, even under constrained budget conditions. This study illuminates the potential of leveraging LLMs to improve dataset quality, introduces a novel technique to mitigate class imbalances, and demonstrates the feasibility of achieving high-performance NER in a cost-effective way.

1 Introduction↩︎

In the field of Natural Language Processing (NLP), Named Entity Recognition (NER) is a critical task that involves identifying and classifying named entities in a given text into predefined categories, including persons, organizations, locations, temporal expressions, quantities, monetary values, and percentages [1]. NER is integral to a wide range of applications, from basic information retrieval and content classification to more complex tasks like question answering and machine translation [2][4]. The enhancement of NER model performance not only directly contributes to the improvement of broad NLP applications but also plays a crucial role in enhancing our understanding and engagement with text-based data [5], [6].

In recent years, NER methods leveraging the in-context learning capabilities of Large Language Models (LLMs) have been proposed, demonstrating high performance close to that of models specialized in NER by carefully selecting the examples provided to the LLMs [7]. This indicates the effectiveness of utilizing LLMs’ advanced text comprehension abilities for NER tasks. On the other hand, inference with LLMs requires significant computational resources and involves lengthy latency before obtaining inference results, making them unsuitable for applications requiring the rapid processing of huge amounts of text data. Therefore, we believe it is crucial to develop approaches that leverage the language understanding capabilities of LLMs while requiring fewer computational resources and achieving faster NER task processing. Consequently, we focus on improving the quality of datasets used for training NER models with the aid of LLMs, aiming to enhance the performance of smaller, faster, dedicated NER models.

The performance and effectiveness of NER models rely heavily on the quality of annotations in the datasets used during their training [8]. Currently, the majority of datasets employed for training NER models are annotated manually by humans. This manual annotation process is fraught with several challenges, such as inconsistent quality of annotations, the presence of missed and incorrect annotations, significant time and financial costs, and the need for ethical considerations when dealing with text data that may contain offensive content, among others [9], [10].

In this study, we propose a novel approach to address these issues, particularly the problems of missed annotations and the time and financial costs associated with manual annotation, intending to enhance the quality of datasets and thereby strengthen the performance and effectiveness of NER models. Our proposed method leverages LLMs to supplement the missed annotations in datasets that have been manually annotated (where the presence of missed annotations is assumed) as shown in . By utilizing a hybrid approach that combines manual annotations with annotations generated by large LLMs, we aim to recover from the noise introduced by missed annotations in the dataset, thereby enhancing the quality of the datasets used for training NER models at a low cost and in an automated manner.

Figure 1: LLM-based annotation and label mixing.

In our proposed method, a detailed analysis of the annotations by LLMs revealed that a single expression could be assigned multiple named entity labels. Also, an imbalance in the data volume per label exists, which in turn causes performance disparities among labels in the NER model. This is particularly critical in the “Miscellaneous” category, where such inequalities are most pronounced. To address these issues, we implemented a label mixing technique for the multiple assigned named entity labels, thereby ensuring the robustness of the NER model by mixing these labels.

In the experiments, the CoNLL03 [11] and WikiGold [12] datasets were enhanced using our proposed hybrid annotation method combining manual human efforts and LLMs, and the improved datasets were then used to train a BERT-based [13] NER specialized model. The experimental results demonstrated that our proposed approach could achieve high performance in training NER models with datasets containing noise such as missed annotations, despite requiring minimal financial cost.

The contributions of this study can be summarized in three main points:

  • We propose a hybrid annotation method combining manual human efforts and LLMs for NER datasets, automatically recovering from noise such as missed annotations contained within the datasets ().

  • By incurring only a minimal financial cost, we enhance the performance of NER-specialized models ().

  • In response to the issue of imbalance among named entity labels arising from LLM-based annotations, we propose a label mixing technique, ensuring the robustness of the NER model ().

2 Related Works↩︎

2.1 Reliability of NER datasets↩︎

The reliability of datasets used in NER tasks has been scrutinized. Studies, such as the one presented in [14], highlight the inherent errors in these datasets. These inaccuracies range from mislabeled entities to inconsistent categorization, impacting the overall performance and reliability of NER systems [8], [15].

2.2 LLMs for NER tasks↩︎

[7] attempted to utilize GPT, a large-scale language model (LLM) [16], [17], for annotation tasks and validated its effectiveness. Additionally, they tested the output format and improved annotation accuracy using few-shot learning. LLMs’ advantages for NER tasks are high consistency and efficiency. Conventional methods trained with manual annotation tend to vary due to annotator subjectivity and fatigue. In addition, the annotation cost for LLMs is much cheaper than that of human annotators. The disadvantage of their work is that the computational cost of inference is very high each time. While their approach directly solves the NER task, which differs, our approach includes augmenting and creating the NER dataset from scratch.

2.3 Challenges and Remedies in Automatic Annotation↩︎

There are several challenges in automatic annotation. First, [18] proposes ACLM (Attention-map aware keyword selection for Conditional Language Model fine-tuning), an attention map-based data expansion method to address the problem of data imbalance. It should be noted, however, that this method does not take advantage of LLMs. Next, [19] proposes using multiple prompt templates to address the problem of multiple label assignments to the same entity. Furthermore, DiffusionNER in [20] and [21] address the issues of noise and generative bias by utilizing causality-based methods. To address these issues, we propose a hybrid approach that combines manual annotation with automatic LLMs annotation.

Other related studies in NER that must be mentioned are [22] for a comprehensive review of NER deep learning methods, [23] addresses noise from external knowledge bases, and [24] propose MINER (Mutual Information based Named Entity Recognition) for out-of-vocabulary support.

While the above diverse approaches address the NER challenge, this study proposes a unique methodology that focuses on improving the reliability of the dataset itself and leads to better accuracy for NER tasks.

3 Methods↩︎


In linguistic data automation using LLM, the GPT-NER methodology, as delineated in [7], emerges as a paradigmatic example. This method profoundly influences our study, harnessing LLMs to facilitate automated annotation. Central to the GPT-NER strategy is the LLM’s capability to discern and earmark specific entities within the text, encapsulating these segments with distinct character strings, specifically “@@” and “##”. For illustrative purposes, given an input such as “I live in Tokyo”, wherein the objective is to pinpoint LOC (location) entities, the LLM’s output would be “I live in @Tokyo?##”. This delineation effectively signifies the accurate detection of “Tokyo” as a location entity, with the “@@” and “##” serving as the entity’s boundaries.

Additionally, this approach empowers the LLM to apprehend both the intrinsic properties of entities and their interplay with the input-output schema, achieved through the integration of few-shot examples. In alignment with this methodology, we have formulated the ensuing prompt structure.

Instructions for the Large Language Model You are an excellent linguist.

Identify [target entity type] entities in given text.

If the text contains no [target entity type] entities, replicate the input text without any changes.

Strictly adhere to the definition provided below.

Definition of [target entity type]:

\(\cdots\)

Within this structured prompt design, the [target entity type] slot accommodates various entity types, including but not limited to PER, ORG, LOC, and MISC. Concurrently, the [definition target entity type] slot is allocated for the explicit definition of these enumerated entity types. Moreover, the [Example \(K\) Input] and [Example \(K\) Output] slots are respectively reserved for the input text and its corresponding output text, collectively functioning as few-shot examples.

As this prompt dictates, the entity recognition process is methodically applied to each target entity type within the text targeted for annotation. In scenarios where this strategy involves multiple iterations of annotation for each type of entity, a single word may be assigned multiple different entity types. To resolve instances of such overlapping entities, the simplest method we adopt is to select one entity from those multiple assignments randomly.

The criterion for selecting few-shot examples is inspired by the GPT-NER approach, entailing the selection of examples from validation data. This selection is predicated on the cosine similarity between token embeddings present in the input text.

To enhance the quality of the NER dataset, we created Hybrid annotations by merging annotations from LLMs with those generated manually. This merging process involved carefully analyzing the annotations from both sources, ensuring consistency and accuracy. The merged data combines all entity labels from the manual data and those from LLM annotation that do not overlap the same position as those from the manual data. Since the entity labels of the manual data are expected to be of higher quality than the ones from LLM annotation, they are merged in preference to the ones from LLM annotation. The merged dataset offers a more robust and diverse set of annotations, mitigating biases and errors that might exist in either of the individual sources.

Figure 2: Comparison of model performance when using noised and datasets annotated by Hybrid approach.


Addressing the issue, as mentioned earlier, of multiple labels being assigned to a single word, we extended the Mixup [25] technique, traditionally used in image processing, to NER tasks. Mixup is a data augmentation method that improves model performance and robustness by mixing two inputs and outputs. Our approach involves blending two different entity labels for a single token to create new annotations. This method is formally represented as follows:

\[y_{\text{mix}} = \lambda \times y_i + (1 - \lambda) \times y_j\]

Where \(y_i\) and \(y_j\) are two labels annotated NER tags, and \(\lambda \in [0,1]\) is a parameter determining the mixing ratio. Following the mixup technique [25], \(\lambda\) is drawn from the beta distribution (\(\alpha=\beta=0.2\)). Compared to randomly selecting one label from multiple labels assigned to a single word, this technique contributes to a more nuanced understanding of tokens that could belong to multiple entity types, enhancing the dataset’s richness and classification flexibility.

4 Experiments↩︎

Table 1: Comparison between different data count balances from manual and LLM-based data under the fixed budgets.
Budget #Manual #LLM F1 Score
$38 87 0 0.02
70 702 0.81
35 2106 0.84
0 3510 0.85
$152 351 0 0.63
280 2808 0.85
140 8424 0.87
0 14041 0.86
$608 1404 0 0.86
1263 5616 0.87
1123 11232 0.87

4.1 Setup↩︎

In our research, we conducted experiments using two distinct datasets. Firstly, we employed the CoNLL03 dataset [11], recognized as a standard benchmark in the field of NER. Additionally, acknowledging recent advancements in this field models that have led to a saturation in performance improvements on the CoNLL03 dataset, we also included the WikiGold dataset [12]. The WikiGold dataset was specifically chosen due to its higher level of challenge and the observation that performance levels achieved on it have not yet paralleled those on the CoNLL03 dataset. This was split into training, validation, and test sets in a 7:1:2 ratio for the experiments.

For the automated annotation process, we leveraged ChatGPT (gpt-3.5-turbo-0613) API2 as LLM, and we set the number of few-shot examples at 32.

Finally, to validate the effectiveness of our annotation approach, we fine-tuned a sequence classification model based on a pre-trained BERT [13], utilizing the datasets annotated through this methodology. The evaluation of the NER task is in line with [11], and we present weighted average F1 scores in this paper.

A detailed description of the hardware and software configurations is also provided in Appendix 8 to ensure reproducibility.

4.2 Results↩︎

We assessed the capability of LLM-based annotations to recover from noisy data. [26] pointed out some annotation errors, such as missing annotations or label switching, in the CoNLL03 dataset [11]. We prepared noised data that reproduces this problem by removing annotations for some entity annotations. We show that it can be improved by combining LLM-based annotation in the previously described manner. We experimented with 20%, 40%, 60%, 80% entity annotations removed. By comparing the noised datasets to those enhanced through our proposed combining methodology, we established a baseline for improvement. Figure 2 presents a comparative analysis of these datasets, highlighting the efficiency of our approach to noise reduction. It was shown that even when noise is severe, it can be significantly improved by combining LLM-based annotations.

Figure 3: Number of annotations for each entity type in the manual, noised, and LLM-based datasets.

We evaluated our approach under identical budget constraints by drawing parallels with the study of  [27]. Our experiments were conducted on three different budgets. We created composite datasets including different numbers of samples from manual and LLM-based annotations. The middle budget ($152) is the cost of LLM-based annotation of all training data of the CoNLL03 dataset, and the costs of a quarter and a quadruple of it are assumed for small ($38) and large ($608) budgets, respectively. The relationship between the balance of the number of data in each budget and performance is shown in Table 1.

When the budget is small ($38), training is not possible with only manual data due to the small amount of data. Table 1 shows that it is thought that performance can be improved by using LLM-based annotations to increase the training data. For the middle budget ($152), the model’s performance was reduced using only the manual data; using just 2,808 LLM-based annotations showed a significant improvement. In addition, the performance of the Hybrid annotations was higher than that of LLM-based alone, suggesting that the use of both annotations contributes to the performance. For the large budget ($608), the performance is sufficient with 1,404 manual data, but a slight performance improvement was confirmed by reducing the manual data and using LLM-based as well. The results show that the composite datasets can increase the number of data at a low cost and improve the performance of downstream tasks under all budgets.

shows that there is a difference in the number of annotations for each entity type. This imbalance issue is even greater with annotation by LLM. Also, the performance difference of LLM-based annotation per entity type is large compared to manual annotation. To tackle the class imbalance issue, we motivated the use of our proposal, label mixing. We present the comparison between LLM-based annotation and label mixing in . Label mixing reduced the performance difference between entity types. Therefore, we believe that soft labeling of LLM-based annotations according to possible entities rather than one-hot hard labels will prevent model training in a biased manner.

Figure 4: Improvement of each entity type and micro average with label mixing.

Table 2: The number of entities overlapping among classes in the CoNLL03 and WikiGold dataset.
CoNLL03 WikiGold
2-7 ORG LOC MISC ORG LOC MISC
PER 72 37 15 13 5 1
ORG - 180 59 - 44 41
LOC - - 158 - - 5

In addition to the analysis above, we further explore the impact of LLM-based annotations and label mixing on the CoNLL03 and WikiGold datasets, particularly focusing on the number of entities that overlap between two classes. presents a detailed quantification of these overlapping entities between different entity types when annotated by LLM. The fact that the entities overlapping between classes constitute only 521 cases (2.65%) of the CoNLL03 dataset and 109 cases (6.80%) of the WikiGold dataset, and yet the application of label mixing to these specific data points results in a performance improvement across the dataset, is a suggestive observation. This observation implies that the strategic application of label mixing to even a small subset of data can improve the performance balance between entity types. Moreover, the fact that the number of entities overlapping between classes such as LOC and MISC with ORG is comparatively higher than other combinations, while there is less overlap between PER and MISC, indicates a potential challenge in the discriminatory capability of LLM-based annotations for the former combinations. These observations support the effectiveness of label mixing as a technique to buffer these shortcomings, suggesting that it serves as a valuable strategy for improving the robustness and performance of NER models.

5 Discussion and Conclusion↩︎

Our study extends the use of LLMs beyond annotation tasks. We identified and addressed new NER annotation challenges, proposing novel solutions. Our integration of LLM-based annotations with manual annotation, especially in datasets with noise or limited samples, significantly enhances downstream task performance. This method leverages LLMs’ evolving strengths [28], [29] and introduces label mixing to address biases and imbalances from LLM-based annotations. Our work illuminates LLMs’ role in enhancing NER, underscoring the need for ongoing adaptation and improvement in annotation methodologies.

6 Ethical Considerations↩︎

In our research, we focus not on generating text, but on the automation of annotations. This approach places concerns about creating potentially harmful sentences outside our scope of study. However, while our primary objective is to annotate entities with labels, it is essential to consider the implications of certain annotations, such as the toxicity of entities, especially when these annotations pertain to specific demographic groups.

This risk stems from biases inherent in the LLMs used for annotation, necessitating continuous monitoring.

7 Limitations↩︎

Finally, we would like to mention the limitations of our work. As the first limitation, we have only validated our proposed methods with relatively small datasets, CoNLL03 and Wikigold, thus requiring validation with larger NER datasets, such as OntoNotes 5.0 [30].

Another limitation is that when using the ChatGPT (gpt-35-turbo-0613) API, some regions and organizations are not generated due to their alignment. It is conceivable that this kind of alignment-based distribution bias could affect NER task performance.

The final limitation involves the LLM-based annotation method for NER tasks, which requires computational costs proportional to the number of entity types. Thus, algorithms independent of entity type count are needed.

Appendix↩︎

8 Details of Experiment Setup↩︎

In our experiments, we used virtual machines on Google Cloud. The detailed machine specifications are as follows:

  • Machine type: n1 standard 4

  • Memory: 15 GB

  • virtual CPU: Intel Broadwell x 4

  • GPU: NVIDIA T4 x 1

The configurations of BERT used for the performance evaluation are as follows:

  • Pre-trained model: bert-model-unbased 3

  • Learning rate: \(1e-05\)

  • Upper limit for norm of gradients: \(10\)

  • Batch size: \(64\) for training, \(32\) for inference

  • Max length of token: \(128\)

Each training was terminated when the F1 score on the validation set remained unchanged for three consecutive epochs.

In NER, labels must be assigned at the word level, but BERT’s tokenizer often splits a single word into multiple tokens. Therefore, in this study, when a word is split into multiple tokens, when a word is split into multiple tokens, the model adopts the label for the first token as the label for the entire word.

9 Datasets↩︎

The CoNLL-2003 dataset is designed to recognize four types of named entities, including person names, place names, organization names, and other miscellaneous entities. The data file consists of four columns: word, part-of-speech tag, syntactic chunk tag, and named entity tag, with each item separated by a space. The named entity and syntactic chunk tags use the I-TYPE format 4, and the B-TYPE 5 tag is introduced to indicate the start of a new phrase.

The Wikigold dataset is a manually annotated corpus for named entity recognition consisting of a small sample of Wikipedia articles. Words are labeled for each of the four named entity classes (LOC, MISC, ORG, and PER) of CONLL-03 described above.

References↩︎

[1]
Behrang Mohit. Named entity recognition. In Natural language processing of semitic languages, pp. 221–245. Springer, 2014.
[2]
Jiafeng Guo, Gu Xu, Xueqi Cheng, and Hang Li. Named entity recognition in query. In Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’09, pp. 267–274, New York, NY, USA, 2009. Association for Computing Machinery. ISBN 9781605584836. . URL https://doi.org/10.1145/1571941.1571989.
[3]
Diego Mollá Aliod, Menno van Zaanen, and Daniel Smith. Named entity recognition for question answering. In Australasian Language Technology Association Workshop, 2006. URL https://api.semanticscholar.org/CorpusID:14715684.
[4]
Bogdan Babych and Anthony F. Hartley. Improving machine translation quality with automatic named entity recognition. Proceedings of the 7th International EAMT workshop on MT and other Language Technology Tools, Improving MT through other Language Technology Tools Resources and Tools for Building MT - EAMT ’03, 2003. URL https://api.semanticscholar.org/CorpusID:6516117.
[5]
Pengxiang Cheng and Katrin Erk. Attending to entities for better text understanding. ArXiv, abs/1911.04361, 2019. URL https://api.semanticscholar.org/CorpusID:207852851.
[6]
Oren Etzioni, Michael J. Cafarella, Doug Downey, Ana-Maria Popescu, Tal Shaked, Stephen Soderland, Daniel S. Weld, and Alexander Yates. Unsupervised named-entity extraction from the web: An experimental study. Artif. Intell., 165: 91–134, 2005. URL https://api.semanticscholar.org/CorpusID:7162988.
[7]
Shuhe Wang, Xiaofei Sun, Xiaoya Li, Rongbin Ouyang, Fei Wu, Tianwei Zhang, Jiwei Li, and Guoyin Wang. Gpt-ner: Named entity recognition via large language models. ArXiv, abs/2304.10428, 2023. URL https://api.semanticscholar.org/CorpusID:258236561.
[8]
Ruslan Mitkov, Constantin Orasan, and Richard J. Evans. The importance of annotated corpora for nlp: the cases of anaphora resolution and clause splitting. 2000. URL https://api.semanticscholar.org/CorpusID:14801671.
[9]
Arzoo Katiyar and Claire Cardie. Nested named entity recognition revisited. In North American Chapter of the Association for Computational Linguistics, 2018. URL https://api.semanticscholar.org/CorpusID:44170949.
[10]
Karën Fort, Gilles Adda, and Kevin Bretonnel Cohen. Amazon mechanical turk: Gold mine or coal mine? Computational Linguistics, pp. 413–420, 2011.
[11]
Erik F. Tjong Kim Sang and Fien De Meulder. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pp. 142–147, 2003. URL https://aclanthology.org/W03-0419.
[12]
Dominic Balasuriya, Nicky Ringland, Joel Nothman, Tara Murphy, and James R. Curran. Named entity recognition in Wikipedia. In Proceedings of the 2009 Workshop on The Peoples Web Meets NLP: Collaboratively Constructed Semantic Resources (Peoples Web), pp. 10–18, Suntec, Singapore, August 2009. Association for Computational Linguistics. URL https://aclanthology.org/W09-3302.
[13]
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. : Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4171–4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. . URL https://aclanthology.org/N19-1423.
[14]
Ridong Han, Tao Peng, Chaohao Yang, Benyou Wang, Lu Liu, and Xiang Wan. Is information extraction solved by chatgpt? an analysis of performance, evaluation criteria, robustness and errors. ArXiv, abs/2305.14450, 2023. URL https://api.semanticscholar.org/CorpusID:258865200.
[15]
Shubhanshu Mishra, Sijun He, and Luca Belli. Assessing demographic bias in named entity recognition. arXiv preprint arXiv:2008.03415, 2020.
[16]
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020.
[17]
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022.
[18]
Sreyan Ghosh, Utkarsh Tyagi, Manan Suri, Sonal Kumar, S Ramaneswaran, and Dinesh Manocha. Aclm: A selective-denoising based generative data augmentation approach for low-resource complex ner. arXiv preprint arXiv:2306.00928, 2023.
[19]
Yongliang Shen, Zeqi Tan, Shuhui Wu, Wenqi Zhang, Rongsheng Zhang, Yadong Xi, Weiming Lu, and Yueting Zhuang. Promptner: Prompt locating and typing for named entity recognition. arXiv preprint arXiv:2305.17104, 2023.
[20]
Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Diffusionner: Boundary diffusion for named entity recognition. arXiv preprint arXiv:2305.13298, 2023.
[21]
Shuai Zhang, Yongliang Shen, Zeqi Tan, Yiquan Wu, and Weiming Lu. De-bias for generative extraction in unified ner task. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 808–818, 2022.
[22]
Jing Li, Aixin Sun, Jianglei Han, and Chenliang Li. A survey on deep learning for named entity recognition. IEEE transactions on knowledge and data engineering, 34 (1): 50–70, 2020.
[23]
Kun Liu, Yao Fu, Chuanqi Tan, Mosha Chen, Ningyu Zhang, Songfang Huang, and Sheng Gao. Noisy-labeled ner with confidence estimation. arXiv preprint arXiv:2104.04318, 2021.
[24]
Xiao Wang, Shihan Dou, Limao Xiong, Yicheng Zou, Qi Zhang, Tao Gui, Liang Qiao, Zhanzhan Cheng, and Xuanjing Huang. Miner: Improving out-of-vocabulary named entity recognition from an information theoretic perspective. arXiv preprint arXiv:2204.04391, 2022.
[25]
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018. URL https://openreview.net/forum?id=r1Ddp1-Rb.
[26]
Ridong Han, Tao Peng, Chaohao Yang, Benyou Wang, Lu Liu, and Xiang Wan. Is information extraction solved by chatgpt? an analysis of performance, evaluation criteria, robustness and errors, 2023.
[27]
Bosheng Ding, Chengwei Qin, Linlin Liu, Yew Ken Chia, Boyang Li, Shafiq Joty, and Lidong Bing. Is GPT-3 a good data annotator? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 11173–11195, Toronto, Canada, July 2023. Association for Computational Linguistics. . URL https://aclanthology.org/2023.acl-long.626.
[28]
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. A survey of large language models. arXiv preprint arXiv:2303.18223, 2023.
[29]
Bonan Min, Hayley Ross, Elior Sulem, Amir Pouran Ben Veyseh, Thien Huu Nguyen, Oscar Sainz, Eneko Agirre, Ilana Heintz, and Dan Roth. Recent advances in natural language processing via large pre-trained language models: A survey. ACM Computing Surveys, 56 (2): 1–40, 2023.
[30]
Ralph Weischedel, Martha Palmer, Mitchell Marcus, Eduard Hovy, Sameer Pradhan, Lance Ramshaw, Nianwen Xue, Ann Taylor, Jeff Kaufman, Michelle Franchini, Mohammed El-Bachouti, Robert Belvin, and Ann Houston. , 2013. URL https://hdl.handle.net/11272.1/AB2/MKJJ2R.

  1. equal contributions↩︎

  2. https://chat.openai.com↩︎

  3. https://huggingface.co/bert-base-uncased↩︎

  4. This tag means "Inside" and is assigned to words that are within an entity of a specific type (TYPE). That is, this tag is used when a word is part of a specific entity and indicates the type of that entity. For example, "I-PER" indicates that the word is part of a person’s name (Person).↩︎

  5. The B in B-TYPE means "Beginning" and is the tag given to the first word of the second entity when directly followed by an entity of the same type. This tag is used to indicate that a new entity is beginning and to distinguish it from the previous entity. For example, "B-PER" means that the new person’s name begins with this word.↩︎