1 Introduction↩︎

Event Extraction (EE) aims to identify structural event information from text, which contains two subtasks [1], i.e., Event Detection (ED) and Event Argument Extraction (EAE). The former identifies the trigger words of events (event triggers) and their corresponding types, while the latter extracts arguments and their associated roles based on a trigger and its event type. EE has demonstrated its value across a variety of domains, including finance [2], biomedical research [3], and Cyber-security [4]. Each of these domains has its specific event types, which jointly form a large event schema containing massive types. It leads researchers [5] in this area to pursue a general-purpose system capable of extracting events with massive types in different domains.

In doing so, the basic challenge is the lack of an effective and efficient annotation method to construct datasets. As illustrated in Figure 1, existing datasets can be divided into two types based on their annotation methods: human-annotated and distant supervision-based ones. Human annotation is generally effective but inefficient, requiring annotators to understand long guidelines and undergo specialized training. Consequently, the obtained datasets are often limited in terms of both event type and scale. Considering that existing semantic frame knowledge bases, e.g., FrameNet [6], Propbank [7], contain massive types of predicates, distant supervision-based methods [1], [8], [9] automatically annotate triggers and arguments if they have been annotated in the knowledge bases, offering a more efficient alternative. For example, GLEN [1], the largest ED dataset to date, uses Propbank and Wikidata [10] to annotate event triggers across more than 3,000 types. However, these datasets often suffer from noise in three aspects: 1) Unreasonable Trigger Annotation: Some predicates in the semantic frame are not considered as events in the view of EE, leading to the incorrect annotation of irrelevant words as triggers. For example, the adverb “voluntarily” is annotated as a trigger in GLEN. 2) Coarse-Grained Type Annotation: Due to the hierarchical structure of event types, a predicate often has types with multiple granularities. cmethods struggle to assign precise fine-grained types to these predicates. For example, GLEN annotates several potential types (crime, property crime, etc) for the trigger “crime”. 3) Missing Argument Annotation: Unlike triggers, where candidate triggers are relatively limited, the number of potential arguments is much larger and cannot be fully enumerated by existing knowledge bases. Thus, distant supervision-based methods often miss argument annotations when the arguments are not included in the knowledge bases.

Large Language Models (LLMs) have recently achieved significant performance improvements across many Natural Language Processing (NLP) tasks and emerge as a promising approach for annotating EE datasets [11]. One key challenge, however, is to mitigate the annotation bias inherent in a specific LLM. Motivated by this, we propose an LLM-based collaborative annotation method. Based on the results from distant supervision-based methods, it first performs event trigger filtering by removing irrelevant triggers. This is followed by event type refinement, which assigns more fine-grained event types for triggers based on context. Finally, it identifies the roles of arguments associated with each event and refines the original annotation with human annotation rules. After each step, multiple LLMs collaborate to generate annotations, and then a voting phase is used to unify the annotation preferences across LLMs. Finally, we obtain a new dataset, called EEMT, with over 200,000 annotated samples, covering 3,465 event types and 6,297 argument role types, which is the largest EE dataset regarding event type and scale, to the best of our knowledge.

To adapt LLMs for EE with massive types, we propose a Partitioning EE method for LLMs called LLM-PEE, which addresses the prompt length limitation when handling massive types. LLM-PEE begins by recalling the top-k most similar event types, then divides these types into several partitions, which are assembled into prompts of LLMs. Based on these partitioning prompts, LLMs extract event triggers and their corresponding arguments. Experimental results on the EEMT dataset demonstrate that LLM-PEE outperforms the state-of-the-art models by 5.4% in event detection and 6.1% in argument extraction. Besides, LLM-PEE achieves up to 12.9% F1 improvement compared to mainstream LLMs in the zero-shot setting, demonstrating its strong generalization capabilities.

Our contributions can be summarized as follows:

• We propose an LLM-based collaborative annotation method for EE with massive types, where multiple LLMs automatically annotate events collaboratively.

• Based on the above method, we construct the EEMT dataset, which is the largest EE dataset to date in terms of both its coverage and scale.

• We propose an LLM-based Partitioning Extraction method, which significantly improves EE under supervised and zero-shot settings.

2 Related Work↩︎

2.0.0.1 Event Extraction Dataset.

In human annotated datasets, ACE 2005 [12] is the most commonly used dataset, including 33 event types and 22 roles. Followed by this, Rich ERE [13] is proposed to further enhance the scale of the EE dataset. MAVEN-arg[14] is constructed based on MAVEN[15], the current largest EAE dataset annotated by human experts with 162 event types and 612 roles. However, due to the high cost of annotating events, human annotators can not further extend schema and data scale. For datasets from distant supervision-based methods, LSEE [8] is constructed from FrameNet [6] and WikiPedia [16] to automatically annotate events. GLEN [1] is the largest ED dataset including 3465 types and 200,000 samples. However, the dataset from distant supervision suffers from the low annotation quality.

2.0.0.2 Event Extraction Method.

EE methods can be divided into two kinds: classification-based and generation-based ones. Classification-based methods [1], [17] tend to formulate EE as a token classification or a sequential labeling task. Generation-based methods [18]–[20] aim to generate the text containing a structured event, while these methods require manual schema-specific templates, which are difficult to be adapted to massive types. Nowadays, due to the strong generation abilities of LLMs, they have been widely used in EE, like InstructUIE [21], KnowCoder [22], AlignXIE [23]. Besides, some works focus on few-shot or zero-shot settings such as Code4Struct [24] and CodeIE [25], which employ code format to extract information.

3 LLM-based Collaborative Annotation↩︎

3.1 Annotation Method↩︎

Given that LLMs demonstrate strong capabilities in understanding natural language, we incorporate them into the EE annotation process. To improve efficiency, we use LLMs to perform annotation based on trigger annotations from distant supervision, rather than starting from scratch. By collaborating across multiple LLMs using offset alignment and voting, we achieve consistent and precise annotation results, making the process more effective.

Specifically, as shown in Figure 2, the proposed annotation method consists of four steps: (1) Event trigger pre-annotation annotates the triggers and their potential types based on distant supervised methods; (2) Event trigger filtering filters the unreasonable event triggers via LLMs; (3) Event type refinement maps the trigger to more fine-grained event types via LLMs; (4) Event argument annotation further annotates the arguments via LLMs. The detailed prompts and voting strategies are listed in Appendix 9.2 and Appendix 9.3. In the first step, this paper takes the distant supervision-based methods based on Propbank [6] and Wikidata [10], as applied in GLEN [1], for example. The proposed annotation method can be easily extended to other Distant supervision-based methods and knowledge bases.

3.1.0.1 Event Trigger Pre-annotation.

Due to the massive types in the schema, directly using LLMs to annotate events is inefficient, as it requires including extensive guidelines in the prompt of LLMs. To improve the efficiency of the annotation process, we perform event trigger pre-annotation using distant supervision. For example, in the distant supervision-based method used in GLEN, a sentence is first processed by annotating words as triggers if they are included in the Propbank predicate annotations. Then, the event type (referred to as roleset in Propbank) of each trigger is mapped to Wikidata QNode types via DWD Overlay [5]. After this step, we obtain the initial annotations of event triggers and their corresponding candidate event types.

3.1.0.2 Event Trigger Filtering.

The event annotation guidelines in semantic frame knowledge bases, such as Propbank, differ from the event definitions in mainstream EE datasets [12]. For example, some adverbs or adjectives are treated as event triggers in Propbank. This gap leads to the issue of unreasonable trigger annotation, as discussed earlier. Thus, this step leverages LLMs to filter out invalid triggers. By observing these differences in event definition and describing them as guidelines, we instruct LLMs to evaluate the validity of event annotations. To help the LLMs better comprehend the task, carefully selected examples are also included in the prompt.

Due to potential differences and biases in the filtering process provided by different LLMs, a voting strategy is applied to obtain the final results. Specifically, we aggregate the valid events identified by each LLM. An event is considered valid if the majority of LLMs support it. In cases where a tie occurs during the voting process, we will instruct each LLM to re-annotate the case repeatedly until the majority of LLMs support the result.

3.1.0.3 Event Type Refinement.

In the original GLEN annotation, 60% of triggers have more than one event type. Distantly supervised methods cannot assign precise, fine-grained types to these triggers, as this requires more accurate event definitions and a deeper semantic understanding of the sentence. Therefore, this step uses LLMs to refine the event types to a more fine-grained level automatically.

We formalize the event type refinement task as a multiple-choice problem for the LLMs. Specifically, this step takes the candidate event types from Step 1 and their corresponding descriptions as input. It then instructs the LLMs to select the event types most accurately align with the event triggers and sentences. Additionally, we include a “None of them” option for cases where none of the candidate event types align with the event trigger.

Each LLM performs this task independently, and the final fine-grained event type is determined by selecting the candidate with the highest number of votes from different LLMs. In the event of a tie during the voting process, each LLM is instructed to re-annotate the case. However, since the most probable fine-grained types typically converge within one or two candidates, such ties are generally resolved within two rounds of iterative voting.

3.1.0.4 Event Argument Annotation.

After the fine-grained event type is selected, we need to annotate the event arguments based on the event type and its corresponding schemas. As mentioned above, distant supervised methods often fail to conduct argument annotations when the relevant arguments are absent from the knowledge bases. Thus, we adopt LLMs to annotate arguments and their roles in this step. However, annotating event arguments introduces two additional challenges for LLMs: 1) Roles Understanding: Event argument annotation requires LLMs to comprehensively understand the complex event schema, including event types and role definitions. Additionally, LLMs must be capable of analyzing syntactic structures within sentence spans and mapping each span to a specific role. 2) Bias in Span Offset: LLMs exhibit inherent bias in span offset when dealing with different styles of text. Different LLMs are more likely to give out different offsets in argument annotation.

Considering the additional challenges, we develop a set of guidelines for analyzing logical relationships in sentences and assigning roles accordingly. Then, LLMs are adopted to identify logical relationships within sentences, and map text spans to their corresponding roles. Additionally, we include examples of manual annotations in the prompts to help the LLM better understand the roles of events. To eliminate bias in span offset across different LLMs, we instruct the LLMs to refine and update the original annotation results, especially in span offset, using the rules derived from human observations. We refer to this process as Offset Alignment, which ensures greater consistency in the annotations generated by different LLMs. Following this alignment, we employ a voting strategy to determine the final argument annotations. For each argument in the annotated event, if a specific argument-role pair appears in more than half of LLMs’ annotation results, it is deemed valid. If the LLMs generate completely different annotations for certain roles, we employ GPT-4o to annotate the case given the original annotation from each LLM.

3.2 The Constructed EEMT Dataset↩︎

3.2.0.1 Dataset Construction.

As some datasets already pre-annotate event triggers, we reuse the GLEN dataset, currently the largest ED dataset with over 3,000 event types, to construct a more comprehensive and high-quality dataset. Specifically, we employ three mainstream, state-of-the-art LLMs for collaborative annotation: Deepseek-V3 [26], Qwen-Plus [27], and GPT-4o-mini [28]. We follow the data splits of GLEN for the training, development, and test sets. Additionally, we select 1,500 samples from the origin test set and create a human-annotated test set with the help of three graduate students specializing in NLP. More details are provided in Appendix 9.

The statistics of the EEMT dataset and the existing datasets are shown in Table 1. Compared to commonly used datasets, EEMT surpasses them in terms of the size of event types, role types, and overall data scale. Particularly, our dataset contains 10 times more event and role types than the largest human-annotated EE dataset, MAVEN-arg. Compared to GLEN, we filter out nearly 7.64% of unreasonable event triggers, refine the coarse-grained type annotation (which accounts for 61.3% in the origin GLEN dataset) into fine-grained ones, and annotate the arguments along with their roles. More detailed statistics are in Appendix 10.

3.2.0.2 Quality Assessment.

To evaluate the effectiveness of the proposed annotation methods, we assess the annotation quality of each of the three LLM-based steps on the previously mentioned human-annotated test set. The results are in the last column of Table 2. The F1 scores for all steps surpass 85%, with the event type refinement achieving an impressive F1 score of 96.2%. These results strongly validate the effectiveness of the annotation method. Additionally, we evaluate the results annotated by a single LLM. By leveraging collaborative annotation, the F1 score improves at each step, further enhancing the quality of the annotations.

4 LLM-based Partitioning Extraction↩︎

To adopt LLMs to EE with massive types, we propose an LLM-based partitioning extraction method for EE, called LLM-PEE. As shown in Figure 3, LLM-PEE consists of three key components: similarity-based type recall, type-partitioning prompting, and LLM-based event extraction. In the similarity-based type recall step, we reduce the number of candidate event types that may appear in a sentence to a small subset using a similarity-based model. Next, the type-partitioning prompt divides the subset into several partitions using three different strategies, thus further reducing the prompt length of each partition. Finally, based on the partitioning prompts, LLMs extract event triggers and their corresponding arguments.

4.0.0.1 Similarity-based Event Type Recalling.

Given a sentence \(s={s_1,..., s_i, ..., s_n}\) and a set of candidate event types \(\{e_1, ..., e_i, ..., e_m\}\), the similarity-based event type recalling step identifies the potential event types described by the sentence based on the similarity between the sentence and the event types. Following CDEAR [1], we employ ColBERT [29] as the encoder of sentence and event types. Based on the embeddings from encoders, we calculate the similarities and get \(k\) candidate event types for each sentence. More details are in Appendix 7.1.

4.0.0.2 Event Type Partitioning Prompting.

Existing LLM-based EE methods [30], [31] typically adopt prompting learning [32] for EE. Their input prompts generally contain three parts, i.e., event schema information, task description, and the input sentence. Although the number of event types is reduced to \(k\) after the event type recall step, the prompt describing the event schema information remains long, particularly for some open-source LLMs with relatively short context lengths. Moreover, some existing studies [33] have shown that as the prompt length increases, the difficulty of understanding the prompt also increases. Motivated by this, the event type partitioning prompting step splits the event types into smaller partitions to mitigate this issue. To determine which event types within a partition enhance LLM performance, we design three partitioning strategies based on the confidence scores obtained in the similarity-based event type recall step: 1) Random: The k event Types are evenly divided into N parts randomly; 2) Average: The k event types are evenly divided into N parts, ensuring that the sum of the confidences for event types in each part is as equal as possible. This strategy aims to ensure that the extraction difficulty is balanced across different partitions. 3) Level: Sort the k event types based on their confidences, and then evenly divide them into N parts. This strategy ensures different partitions have varying difficulty levels. We analyze different partition strategies in Appendix 8.1.

4.0.0.3 LLM-based EE.

With the prompt as input, we use LLMs to conduct extraction. Specifically, we conduct the two-stage extraction process following KnowCoder [22]. For the ED task, with the partitioning prompts as input, we generate the triggers and corresponding event types. For the EAE task, with the golden event types and triggers in the single sentence as input, LLMs predict potential arguments and their corresponding roles. The details of the extraction prompt including schema, instruction and completion are in Appendix 11.

5 Experiment↩︎

5.1 Experiment Setting↩︎

5.1.0.1 Evaluation Metrics.

Following GLEN [1] and KnowCoder [22], we use Trigger Identification (TI) F1 and Trigger Classification (TC) F1 to evaluate ED. For EAE, we use Argument Identification (AI) F1 and Argument Classification (AC) F1 .

5.1.0.2 Baselines.

For ED tasks, following GLEN, we employ four classification-based baselines(denoted as C.B. Method), including DMBERT [34], token-level classification and span-level classification, and CDEAR [1]. For EAE tasks, we employ two classification baselines CRF-Tagging, and Tag-Prime [35] and a generative method(denoted as G.B. Method) Bart-Gen [36]. We compare with three LLM-based baselines, i.e., InstructUIE, IEPILE and KnowCoder. They are all fine-tuned on the proposed EEMT dataset. We also evaluate the mainstream LLMs, i.e., Qwen-Plus, GPT-4o-mini and DeepSeek-V3. For fairness, we evaluate the three models without the offset alignment in EAE. Besides, we conduct further evaluation on other mainstream LLMs in Appendix 8.2.

5.1.0.3 Benchmark.

For supervised evaluation, we evaluate the methods both on LLM-based testset and human annotated testset. For zero-shot evaluation, we evaluate methods on ACE 2005 [12].

5.1.0.4 Implement Details.

We utilize LLaMA2-7B-Base [37] as the backbone and LLAMA-Factory [38] as the training framework. Specifically, LoRA [39] is used for efficient hyper-parameter tuning. We set the LoRA Rank to 8 and learning rates to 0.0003. The maximum sequence length is set to 2048, and the batch size is set to 256. Training is conducted for four epochs. We use the VLLM [40] to accelerate the inference, employing greedy search and a maximum output length of 500. We recall the top 15 similar event types in the similarity-based event type recalling stage and divide them into two partitions with level strategy.

5.2 Experiment Results↩︎

5.2.1 Supervised Evaluation↩︎

The results of the supervised evaluation are listed in Tables 3 and  4. We conduct a comprehensive analysis from the next four perspectives.

5.2.1.1 Analyses on the proposed Annotation Method.

The collaborative annotation method (denoted as CA) is presented in the first row of Tables 3 and  4. First, the collaborative annotation method achieves an F1 score of 85% even on the human-annotated test set, demonstrating the high effectiveness and superiority of our proposed annotation method. Furthermore, our collaborative annotation method outperforms the direct extraction performance of any single annotation model, thereby validating the rationality and efficacy of the design underlying our method. Specifically, in the ED task, there is a significant performance gap (90.1 v.s.56.2) between the collaborative annotation method and individual annotation models. This disparity arises because distant supervision is incorporated as auxiliary information during the annotation process, whereas directly performing event detection using individual LLMs remains highly challenging. In the EAE task, the performance of a single LLM significantly declines due to the absence of a collaborative mechanism and offset alignment. This observation aligns with the results reported in Table 2, further reinforcing the advantages of our method that mitigates bias and enhances the consistency of dataset styles.

5.2.1.2 Analyses on ED.

Table 3 presents the results of ED on the proposed EEMT dataset. Compared with the KnowCoder (generation method based on LLM), LLM-PEE outperforms 1.7% on TI and 5.4% on TC, respectively. These results demonstrate the effectiveness of our partitioned extraction framework in addressing the event detection with massive types. In comparison with CDEAR (designed for ED with massive types), LLM-PEE integrates prompting learning into the framework, effectively unleashing the potential of LLMs in event detection with massive types. Besides, without fine-tuning on the dataset, mainstream LLMs tend to over-generate events, which leads to high recall but low precision. LLM-PEE demonstrates superior event judgment accuracy, leading to a higher F1 score compared to the mainstream LLMs. Additionally, the relatively low performance on TC highlights the inherent difficulty of accurately identifying event types from a large-scale event schema, which remains a significant challenge for our dataset.

5.2.1.3 Analyses on EAE.

Table 4 presented the results of EAE on the proposed EEMT dataset. Compared to KnowCoder, LLM-PEE achieves improvements of 3.9% in AI and 6.2% in AC, respectively. In LLM-PEE, the schema representations of ED and EAE are consistent. We guess that this consistency enables the ED task to facilitate the EAE task, thereby deepening the model’s understanding of event types and roles and contributing to higher performance on both tasks. In comparison with the mainstream LLMs, after fine-tuning on the dataset, the argument extraction performance of LLM-PEE (trained on LLaMA2-7B) surpasses that of the original annotation LLMs. This improvement can be attributed to the high-quality dataset after the offset alignment and collaborative annotation. Additionally, classification-based methods often face challenges in accurately identifying the boundaries of arguments, particularly for continuous spans (e.g., multi-token arguments) in roles such as “purpose”, which result in lower performance in both AI and AC. That suggests generation-based methods might be more suitable for the complex EAE task.

5.2.1.4 Ablation Analysis.

We conduct ablation experiments on two key modules: similarity-based event type recalling (denoted as LLM-PEE w/o e.t.r) and event type partitioning prompting (denoted as LLM-PEE w/o t.p.p). The results are presented in the bottom two rows of Table 3. Since the EAE task paradigm needs to specify event types and triggers in advance, we focus our ablation analysis solely on the ED task. LLM-PEE w/o e.t.r exhibits a significant drop of 18.3% in TC. This result suggests that, without recalling the most similar event types, the model struggles to accurately distinguish the correct event type from the event set, particularly for unseen event types. Furthermore, LLM-PEE w/o t.p.p shows a performance decline of 5.1% in TC, further validating the effectiveness of the partitioning strategies in our framework. More detailed results of the partitioning strategies are provided in Appendix8.1. Additionally, to verify the effectiveness of LLM-PEE when applied to other EE datasets, we conduct an additional supervised experiment on ACE2005. The results and analyses are presented in Appendix 8.4.

5.2.2 Zero-Shot Evaluation↩︎

To assess the generalization capabilities of LLM-PEE to the unseen dataset, we conduct zero-shot experiments on the ACE 2005, which is the most commonly used EE dataset. The results are presented in Table 5. LLM-PEE outperforms other LLMs in TI, AI, and AC. Notably, in TI and AI, LLM-PEE demonstrates superior consistency in identifying spans. This improvement can be attributed to the high quality of the EEMT dataset, which significantly mitigates biases in span offsets and enhances the model’s ability to generalize across diverse event types. Although LLM-PEE exhibits slightly lower performance in TC compared to other methods, this is because our model predicts more fine-grained event types, which are not defined in ACE 2005. These findings convincingly demonstrate the effectiveness and advantages of LLM-PEE, underscoring its generalization capabilities in handling complex event extraction tasks.

6 Conclusion↩︎

In this paper, we proposed a new LLM-based collaborative annotation method. It refines trigger annotations from distant supervision and then performs argument annotation through collaboration among multiple LLMs. We then created the new EEMT dataset based on this annotation method, which is the largest EE dataset in terms of event types and data scale. To further adapt LLMs for EE with massive types, we introduce a Partitioning EE method for LLMs called LLM-PEE. The experimental results in both supervised and zero-shot settings demonstrate that LLM-PEE outperforms other baselines in ED and EAE tasks and even surpasses mainstream LLMs in terms of generalization capabilities in the zero-shot setting.

Limitations↩︎

We summarized the limitations of this work and looked at them as areas for future improvement.

• Hierarchical level of event extraction. We believe that an important factor restricting our model is that the event hierarchy is not sufficiently distinguished. We will explore how to improve the understanding of the hierarchical level of events by better event definition or positive and negative sample strategy.

• End-to-End event extraction. The proposed LLM-PEE method still divides the event extraction into two sub-tasks ED and EAE, we try to proceed directly with end-to-end event extraction based on LLMs.

• Document level event extraction. The proposed EEMT dataset is only annotated on sentence-level text, lacking document-level annotation. We hope to annotate large-scale events in document-level documents to further verify the model’s capability.

7 LLM-based Partitioning Extraction↩︎

7.1 Similarity-based Event Type Recalling↩︎

ColBERT consists of a BERT [41] layer, a convolution layer and an L2 normalization layer. In this paper, we use \(\operatorname{ColBERT}(\cdot)\) to denote the encoder. Specifically, the embedding list of all tokens in a sentence \(s\), \(h_s = [\mathbf{h}_1^s, \mathbf{h}_2^s, ...]\), is calculated as follows: \[\begin{align} \small h_s = \operatorname{ColBERT}(\text{\small{[CLS]}} \text{\small{[SENT]}} s_1, s_2, ... \text{\small{[SEP]}}), \end{align}\] where [SENT] is a special token indicating the object being encoded is a sentence.

For an event type \(e\), the corresponding embedding list of all tokens in the event type name, \(h_e = [\mathbf{h}_1^{e}, \mathbf{h}_2^{e}, ...]\), is calculated as follows: \[\begin{align} \small h_e = \operatorname{ColBERT}(\text{\small{[CLS]}} \text{\small{[EVENT]}} \tau_i \text{\small{[SEP]}}), \end{align}\] where [EVENT] is a special token indicating the object being encoded is an event type.

Then, the similarity score between sentence \(s\) and event type \(e\) is computed as the sum of the maximum similarity between the token embeddings in sentence and event type: \(\rho_{(s, e)} = \sum_{h_s}\max_{h_e}(\mathbf{h}_{i}^e, \mathbf{h}_{j}^s)\).

A similar margin loss to CDEAR is adopted for training, which ensures that the best candidate is scored higher than all negative samples.

\[\small \mathcal{L} = \frac{1}{N}\sum_s \sum_{e^-} \max \{ 0, (\tau - \max_{e \in C_y} \rho_{(e, s)}+ \rho_{(e^-, s)}) \}.\]

Base on the similarity score, we can get \(k\) candidate event types for each sentence.

8 Experiment↩︎

8.1 Influence of partitioning strategy↩︎

We conduct the experiment on ED to figure out the influence of partitioning strategy in Table 6. Our strategy outperforms than IEPLIE¹, because we incorporate the information of the sentence when recalling the similar event types and ensures consistency between the training and testing phases. Besides, it demonstrates that the Level strategy significantly enhances the precision of event extraction. We hypothesize that sorting samples in descending order of confidence enables the grouping of the most challenging-to-distinguish event types into the same partition. This approach facilitates the model’s ability to learn fine-grained distinctions between different event types, including their corresponding triggers and classifications. Notably, the inclusion of naturally occurring challenging negative samples within these partitions allows the model to better grasp the nuanced boundaries between similar or ambiguous event categories.

8.2 Further Evaluation on Mainstream LLMs↩︎

To further evaluate how the mainstream LLMs perform on our dataset, we conduct the extra evaluation on our human annotation benchmark. We use the LLAMA3.1-405B-instruct [42], Gemini-2.0-flash [43], GPT-4o, Qwen-Max to conduct further evaluation. The results are listed in  7.

The results suggests that without training, the mainstream LLMs exhibit almost the same annotation and extraction capabilities, which is a still challenging task for mainstream LLMs. Considering the effectiveness and efficiency, we employ the Qwen-Plus, GPT-4o-mini and DeepSeek-V3 as the final annotation LLMs.

8.3 Case Study↩︎

Since we use the GLEN dataset as the original propbank annotation, we further analyze whether more fine-grained event type can be predicted based on the EEMT, we randomly selected a subset of examples from the test set for verification. As shown in Table 8, we observed that compared to the original GLEN dataset, when trained on the EEMT, the model predictions shifted from “Assessment” to “Risk Assessment” and from “Occupation” to “Military occupation”, which better aligns the sentence. Upon analyzing the training data, we find that the key is the increase in the number of corresponding events in train set, the number of “Risk assesement” increases from 0 to 5, and the number of “Military occupation” increases from 1 to 4, which results in more accurate fine-grained event predictions.

8.4 Supervised Evaluation on other dataset↩︎

Considering the generality of LLM-PEE   on other datasets, we apply LLM-PEE on the supervised setting in the dataset ACE 2005 in Table 9.

Compared to other fine-tuned methods, LLM-PEE achieves state-of-the-art performance. However, the margin of advantage is narrower than when applied to the EEMT dataset, which contains a significantly larger number of event types. This reduction in relative performance can be attributed to the core design philosophy of our model, which primarily focuses on addressing the challenges posed by long prompts resulting from massive event types. When applied to datasets with fewer event types, such as ACE 2005, the inherent advantages of LLM-PEE are somewhat diminished, as the problem it was specifically designed to tackle becomes less pronounced.

9 Annotation Details↩︎

9.1 Annotation Setting↩︎

We use the DeepSeek-V3 [26], Qwen-Plus [27] and GPT-4o-mini as the annotation LLM. Considering the accuracy of the annotation and the divergence of the answers, we set the temperature to 0.5. Besides, we set a answer detection mechanism, if the obtained result can not be correctly parsed, the model is required to annotate the events agian.

9.2 Prompts Examples↩︎

We introduce the prompts we used in the Collaborative Annotation method including Event Triggers Filtering, Event Type Refinement and Event Argument Annotation(two-stage) in Table 11, 12, 13 and  14.

9.3 Voting Strategies Details↩︎

After different stages, we employ different voting strategies. We will introduce the voting details here.

For voting after Event Trigger Filtering, if there are more than two LLMs that consider the event to be reasonable, it will consider the event to be reasonable.

For voting after Event Type Refinement, we tally and vote on the most potential event type identified by each LLM, selecting the event with the highest number of votes as the corresponding fine-level event. In cases where a tie occurs, we apply a repeated voting procedure, capping the maximum number of voting rounds at five to prevent excessive iterations. However, in practice, ambiguous potential events were typically limited to two or three, with most reaching a definitive outcome within two voting rounds.

For voting after Event Argument Annotation, if an argument with its role appears more than or equal to half the annotation results from different LLMs, the argument is considered correct. Specifically, after voting, for the roles if LLMs provide three different annotation results, we instruct GPT-4o with three annotation results as a reference to generate the final annotation result. The offset alignment prompt with multiple input are listed in Table 15.

10 Dataset Statistics↩︎

10.1 The split of dataset↩︎

We introduce the spilt of our dataset in this section. For the train/dev/test set, we follow GLEN’s 90/5/5 setting, besides, to further improve the quality of our dataset. We manually annotate 1,500 cases as the human annotation set.

10.2 Arguments Distribution↩︎

Furthermore, we analyze the distribution of argument in Figure 4 and categorize the role types into five broad classes: “agent”, “entity”, “location”, “purpose”, and “others”. Agent, location, and entity are the three most frequently occurring argument roles, aligning with real-world distributions. This alignment facilitates knowledge sharing of similar roles across different event schemas, thereby reinforcing the validity of our dataset construction.

11 Prompt for Training↩︎

We present examples of training prompts for both EAE and ED task, including instruction and completion in figure 5 and  6.

More specifically, we employ the same Python class style prompt with KnowCoder [22], which includes the schema and instruction. Besides, we use the same output representation as the object of the certain class of event type. Following the KnowCoder, we adopt class comments to provide clear definitions of concepts, which includes the event definition and several samples from the training set. However, while the KnowCoder use different schema representation for ED and EAE tasks, we use the same representation, which could better facilitate the model understanding about the certain event type.

References↩︎