1 Introduction↩︎

Event extraction (EE) is a fundamental task in natural language processing and information extraction, aiming to identify, structure, and organize event-related knowledge from documents [1]. While the majority of existing EE research has focused on texts [2], [3], recent work has increasingly explored the integration of additional modalities [4], [5]. This shift is motivated by the growing prevalence of multimodal content in contemporary news media and online platforms [6], where images, videos, and audio provide complementary information that can support more accurate and comprehensive event understanding.

Prior research has investigated EE or closely related tasks within individual modalities [7]–[10], including text, images, video, and audio, or has leveraged cross-modal cues to address specific challenges such as ambiguity [11], [12]. However, evaluation mostly remains restricted to a single target modality. Multimedia event extraction (MEE) has recently attracted attention and adopts a holistic view by jointly extracting and evaluating events across multiple modalities, typically combining textual and visual inputs [6], [13], [14]. Despite this progress, existing MEE benchmarks remain limited and evaluation challenging, most notably due to annotation scarcity, lack of train splits, and increased evaluation complexity inherent to multimodal settings [6], [14].

Prior studies have demonstrated that even traditional textual EE suffers from substantial and often overlooked evaluation challenges [3], [15], [16], making it prone to hidden pitfalls. These include discrepancies in data and task assumptions as well as metric design choices that can distort model comparisons and fail to reflect real-world performance [3]. Crucially, extending textual EE to the multimodal setting not only inherits existing evaluation issues but also introduces additional pitfalls. These arise from factors such as data scarcity, heterogeneous modalities, and multi-stage pipelines commonly employed in MEE [6], [17]–[19]. As a consequence, inconsistent and under-specified evaluation settings can easily emerge, posing a potential obstacle to reliably assessing progress in MEE research.

Motivated by concerns about the reliability and comparability of current evaluations, this work systematically investigates hidden pitfalls and challenges in MEE evaluation, with the goal of raising awareness and encouraging a shift toward more rigorous evaluation practices. Through an in-depth analysis of the widely used M2E2 benchmark, we first identify three major categories with several issues: inconsistent data processing, inconsistent task assumptions, and relaxed evaluation settings. Building on this analysis, we introduce a more rigorous evaluation framework, StrictEval, and use it to examine how hidden pitfalls influence reported performance. Finally, we show that minor experimental design choices can substantially affect evaluation outcomes.

In summary, our contributions are twofold: (1) We conduct a systematic analysis of evaluation pitfalls and challenges in MEE and propose a more rigorous evaluation framework (StrictEval). (2) We systematically quantify how hidden evaluation pitfalls affect reported performance and reevaluate recent MEE approaches to highlight limitations.

2 Background and Related Work↩︎

2.1 Background↩︎

Textual EE is commonly formulated as a two-stage pipeline [1] consisting of event detection (ED) and event argument extraction (EAE). Event detection aims to identify event mentions, typically grounded to trigger spans, and classify them into predefined event types. Event argument extraction focuses on identifying argument spans and assigning them semantic roles conditioned on the detected event mentions. Analogously, visual EE decomposes the task into detecting events grounded in images and linking their associated semantic roles to visual regions, such as objects [20]. Building on these two research directions, MEE integrates textual and visual information to jointly extract events and their arguments across modalities [6], [13]. This multimodal integration introduces the additional subtask of cross-modal event coreference resolution, with the aim to unify event mentions from different modalities that refer to the same real-world event into a coherent multimedia event representation (see 1).

2.2 Related Work↩︎

2.2.0.1 Multimedia Event Extraction Benchmarks

While most EE benchmarks focus exclusively on text [3], [21]–[23], early multimodal extensions augment textual datasets with images, but evaluation remains limited to textual events [11], [12]. To overcome unimodal limitations, [6] introduce the first MEE benchmark, M2E2, which evaluates event and argument extraction for both texts and images. In addition, M2E2 includes cross-modal event coreferences, analogous to cross-document coreferences in text [24]. Subsequent work extend to images and videos, such as VM2E2 [13], CMMEvent [25], TVEE [26], and MultiVENT-G [14]. More recently, [5] propose a comprehensive benchmark covering textual, visual, and audio inputs by integrating datasets such as M2E2 and ACE with recorded speech. However, only M2E2 and MultiVENT-G publicly release complete data while other benchmarks are still closed-source [25], [26] or lack critical annotations [5], [13]. Moreover, complex annotation formats, abundance of train splits, and missing evaluation scripts further hinder reliable benchmarking.

2.2.0.2 Multimedia Event Extraction

Early approaches focus on cross-modal correlations and align visual and textual representations using large-scale unlabeled news corpora (e.g., VOA) [13], [17], [27], often in combination with contrastive learning objectives [6], [28], [29]. Subsequent studies explore complementary directions, including augmenting training data with synthetically generated image–text pairs [18], designing sophisticated multi-grained fusion mechanisms [30], [31], or leveraging multi-task learning with pseudo labeling strategies [19]. Other works narrow their focus to specific subtasks, such as ED [32] or EAE [33]. With recent advances in multimodal large language models (MLLMs), several instruction-following approaches have been proposed to enable more universal information extraction [4], [5], [34]–[36]. However, most of these methods primarily work with given image–text pairs and do not explicitly address broader MEE settings such as cross-modal event coreference resolution [4], [34], [35]. Notably, the majority of existing approaches are evaluated on the M2E2 benchmark, underscoring its role in advancing MEE research. Despite substantial progress, prior methods adopt diverse task formulations and evaluation protocols, which hinders fair comparison across different modeling approaches.

2.2.0.3 Evaluation Pitfalls

Recent studies have highlighted numerous issues in the evaluation of textual EE models, including inconsistent data assumptions, processing steps, output space discrepancies, and relaxed evaluation metrics [2], [3], [15], [16]. While these works clearly highlight significant differences in textual EE benchmarking, issues in the evaluation of visual and multimedia EE remains relatively underexplored.

3 Pitfalls and Challenges in Evaluation↩︎

Motivated by evaluation concerns for MEE, we first present our investigation setup (§3.1) and systematic analysis to identify evaluation issues (§3.2). We then provide a detailed analysis of common pitfalls across three major categories: data processing (§3.3), task assumptions (§3.4), and relaxed evaluation settings (§3.5). Lastly, building on the insights from this analysis, we introduce StrictEval as rigorous evaluation framework to address hidden pitfalls (§3.7).

3.1 Preliminaries↩︎

To examine the sources and impact of evaluation issues in MEE, we adopt the M2E2 benchmark [6]. Our choice is motivated by two main considerations: (1) M2E2 is publicly available and, to the best of our knowledge, the most widely used benchmark in MEE research. (2) As discussed in §2.2, alternative benchmarks are often incomplete [5], [13] or not accessible [25], [26].

3.1.0.1 M2E2 Dataset

In 1, we show the complete task and involved components. The M2E2 benchmark comprises 6,167 sentences and 1,014 images from 245 multimedia news documents collected from 108k Voice of America (VOA) documents. Overall, the events cover 8 event types and 15 argument roles, with 1297 textual and 391 visual events. Thereby, there exist 309 multimedia events which are coreferenced by 192 textual and 203 visual events. As no training data exists, the benchmark adopts ACE [21] for textual and imSitu [7], with object groundings from SWiG [20], for visual EE training. Annotation mappings to the M2E2 schema are provided by the original work [6].

3.1.0.2 M2E2 Evaluation

Following [6], evaluation is conducted separately for textual, visual, and multimedia EE with precision (P), recall (R), and F1 for the subtasks ED and EAE. In textual EE, an event mention is correct if its type and trigger offsets match the reference, while arguments must additionally match argument offsets and role types. For visual EE, a visual event mention is correct if its type matches the reference image, and a visual argument if its event type, role label, and bounding box match a reference argument with IoU \(>\) 0.5. Lastly, a multimedia event mention is correct if its event type and trigger offsets (or image) match the reference trigger (or image). The inherited textual and visual arguments are evaluated using the same criteria as in the textual and visual modality. However, in our preliminary analysis we observe inconsistencies in the evaluation criteria for multimedia events, which we discuss in detail in §3.4.

3.2 Systematic Analysis↩︎

We collect peer-reviewed MEE studies evaluated on M2E2 published between 2020 and 2025 through keyword and citation-based searches, resulting in 18 articles across multiple venues (e.g., ACL, ACM, AAAI). Of these, we analyze 15 studies and exclude three [5], [37], [38] due to reliance on custom train-test splits or newly introduced evaluation metrics, which hinder direct comparison. The complete set of methods and reported evaluation scores is summarized in 5. For each study, we review the article, supplementary materials, and, when available, its public codebase, with particular attention on the training and evaluation stages (see 1). We focus on data processing, experimental setups, and evaluation protocols [3], [16]. Through this analysis, we uncover three major categories of evaluation issues: inconsistent data processing, inconsistent task assumptions, and overly relaxed evaluation settings. These issues largely stem from the inherent complexity of MEE, which relies on external training datasets, multi-stage pipelines, and heterogeneous modalities. Nevertheless, such inconsistencies can lead to unfair comparisons and performance estimates that do not reflect real-world scenarios.

3.3 Inconsistent Data Processing↩︎

Due to the absence of standardized preprocessing and the reliance on external training datasets, we observe substantial variation in data assumptions across studies. These differences include training set construction, preprocessing, and postprocessing procedures.

3.3.0.1 [P1] Train Size Discrepancies

As described in §3.1, M2E2 relies on external datasets (e.g., ACE and SWiG) for training, which provide predefined train, development, and test splits. While the original M2E2 benchmark uses only train splits, subsequent work often incorporates other sets as additional training data. Consequently, models are optimized on differing numbers of samples (e.g., 75k vs. 100k images for SWiG).

3.3.0.2 [P2] Oracle Trigger Refinement

Due to distributional annotation differences between ACE and M2E2, most reported evaluation scores applies a postprocessing step that adjusts textual ED predictions using M2E2 ground truth annotations. For example, this script removes predicted event mentions with deadly as the trigger span. However, many studies do not clearly specify this experimental setting or report results exclusively with postprocessing applied. We argue that ground truth-based postprocessing does not reflect real-world conditions.

3.3.0.3 [P3] Verb Mapping Refinement

Because the label ontologies of SWiG and M2E2 differ, the original authors provide a verb and role mapping to align SWiG annotations with the M2E2 schema. We notice that some studies adopt refined versions of this mapping. For example, one refined mapping aligns 73 verbs rather than the original 67 verbs to the M2E2 event types. Such discrepancies in label alignment can lead to performance differences driven more by data engineering than by modeling improvements.

3.4 Inconsistent Task Assumptions↩︎

We identify inconsistencies in task assumptions. For instance, some studies evaluate events only present in texts and images [6], [17], [18], while others focus exclusively on multimedia events [4], [34], [35] (cf. 5). Similarly, some methods filter test data to exclude samples without events, whereas others evaluate on the full set. Consequently, reported results are often not directly comparable.

3.4.0.1 [P4] Test Subset Selection

Recent works restrict test-time predictions to sentences or images containing at least one event. This filtering reduces the number of sentences from 6167 to 1086 and images from 1014 to 391, while other work evaluate on the full set. Moreover, methods focusing on the multimedia events task often only evaluate on 309 image-text pairs derived from the event coreference annotations, further reducing the test set to 192 sentences and 203 images, respectively.

3.4.0.2 [P5] MEE Task Discrepancies

Due to the absence of standardized evaluation scripts for multimedia events, follow up work has adopted different task definitions. The original M2E2 benchmark considers a multimedia event correct if either the textual or visual event matches the reference and treats cross-modal coreference as a separate task. More recent work, however, introduce a stricter setting that additionally requires correct event coreference prediction as additional attachment. In contrast, some methods assume gold coreference links and evaluate only aligned image-text pairs. Despite these substantial differences in task formulation, results are often compared directly.

3.5 Relaxed Evaluation Settings↩︎

Similar to observations in TextEE [3], we find that MEE evaluation metrics are often imprecise due to relaxed matching criteria or missing structural constraints. In 2, we illustrate common issues and discuss them below:

3.5.0.1 [P6] Relaxed Textual Evaluation

Some studies ignore trigger offsets during textual EAE evaluation. When multiple events of the same type appear in a sentence, this relaxation allows arguments to be matched to any event of that type, potentially inflating reported performance such as discussed by [3].

3.5.0.2 [P7] Relaxed Visual Evaluation

Most prior work employs a many-to-many matching that considers a visual argument correct if it has the correct role and its IoU exceeds a threshold. However, this approach allows multiple predictions to match one gold argument (see 4), effectively rewarding recall-oriented models for visual EAE. To address this issue, we propose a one-to-one strategy via bipartite matching, that penalizes redundant predictions (see 7.2).

3.5.0.3 [P8] Relaxed Multimedia Evaluation

We also observe missing attachment constraints in multimedia ED evaluation. For instance, ignoring trigger offsets allows incorrect event predictions to be counted as correct at the sentence level, rather than requiring span-level accuracy (see 2). This relaxed assumption can substantially overestimate real-world performance.

3.6 Data Leakage↩︎

3.6.0.1 [P9] Test Data Leakage

M2E2 forms a subset of the collected VOA samples and several works utilize this image-caption dataset during training. We find that some of these works include images and captions from test documents in their training data. Although no ground-truth annotations are used, exposure to test images or captions can still result in information leakage.

3.7 StrictEval↩︎

Our analysis demonstrates that prior MEE evaluations vary widely in data processing, task assumptions, and matching criteria. To systematize these discrepancies, we annotate each study with the evaluation pitfalls it exhibits (P1–P9) in 1 and cluster them into six distinct setups (see 1). Based on this analysis, we propose StrictEval, a more rigorous evaluation framework that eliminates all identified pitfalls. StrictEval enforces (1) consistent data usage without oracle postprocessing or test exposure, (2) a clearly specified task definition evaluated on the full benchmark, and (3) strict matching criteria that preserve structural constraints across textual, visual, and multimedia predictions. As shown in 1, StrictEval represents the strictest setup with the goal of providing a reproducible evaluation framework designed to more faithfully reflect real-world performance.

4 Experiments and Analysis↩︎

The analysis in §3 reveals notable discrepancies and pitfalls in MEE evaluation, raising concerns about the extent to which evaluation design choices influence reported performance. Starting with StrictEval, we conduct a series of controlled experiments in which each evaluation factor is examined in isolation.

4.1 Experimental Setup↩︎

4.1.0.1 MEE Model

We adopt Single Task models to avoid complex architectural choices which ensures that performance differences stem from evaluation setups rather than model capacity. We train independent single-task models for each textual and visual subtask. Textual and visual ED are implemented as token-level and image-level classification models. For textual and visual EAE, we follow [17]–[19] and classify ground-truth textual entities and detected visual objects (via YOLO) into argument roles. Multimedia events are constructed through an event coreference resolution step [18], described below. Further implementation details are provided in Appendix 7.3 and 7.4.

4.1.0.2 Event Coreference Resolution

Following prior work [17]–[19], [33], we perform event coreference resolution by computing CLIP-based similarity scores between image–sentence pairs [39]. We note that StrictEval itself does not introduce a new coreference model, but instead relies on coreference predictions from existing approaches, which we compare under a unified evaluation setting. A textual and a visual event are merged when they share the same predicted event type and their image-text similarity exceeds a threshold of 20. We adopt this hyperparameter following [18], [19], [33]. The resulting multimedia event inherits all associated arguments from both modalities.

4.1.0.3 Evaluation Metrics

As outlined in §3.1, we report micro-averaged P, R, and F1. Unlike [6], we follow subsequent work and require multimedia events to match event mentions and their coreference links [17]–[19], [30], [33]. Unless otherwise specified, all experiments use our introduced evaluation framework StrictEval..

4.2 Unimodal Evaluation and Analysis↩︎

In this section, we focus on unimodal EE results. This allows us to isolate modality-specific behavior before examining the extraction of multimedia events. An overview of the unimodal results is presented in 2 and an extended analysis in Appendix 7.6.

4.2.0.1 Impact of Data Processing

Data processing choices lead to substantial performance variations. Applying the oracle trigger refinement step [P2] and verb refinements [P3] exhibits improvements of up to +13.9 F1 for textual and +7.0 F1 for visual ED. In contrast, incorporating the development sets of ACE and SWiG during training [P1] results in only marginal gains (up to +1.3 F1). These processing decisions also yield improvements of up to +6.8 and +3.1 F1 for textual and visual EAE. Importantly, these discrepancies extend to multimedia ED and EAE since both metrics depend on textual and visual predictions. These findings underscore the importance of consistent data processing, as new state-of-the-art results may not solely reflect advances in modeling techniques.

4.2.0.2 Impact of Task Assumptions

Inconsistent task assumptions, particularly test subset selection [P4], significantly boosts unimodal EE evaluation scores. Restricting training and evaluation to texts and images that contain at least one event instance increases F1 scores of up to +27.8 and +30.0 for textual and visual ED, respectively, and improves EAE scores by up to +14.5. These gains are primarily driven by higher precision, as no-event instances are excluded from evaluation. Notably, this issue propagates to downstream multimedia EE and is further amplified when evaluating on smaller subsets, such as the 309 gold event coreference pairs [P4]. However, these results do not reflect real-world performance, where sentences and images without any targeted events are prevalent.

4.2.0.3 Impact of Relaxed Evaluation

We further examine the effect of relaxed evaluation settings for textual and visual EAE, as described in §3.5. For textual EAE, ignoring the trigger span attachments [P6] introduces discrepancies of up to 6.4 F1. For visual EAE, many-to-many matching [P7] yields only marginal improvements (+0.3 F1), however, this highly depends on the object detector and confidence threshold. As shown in 6, object detectors trained on OpenImages often predict overlapping object categories (e.g., person and suit), which are counted as multiple correct predictions under relaxed evaluation (e.g., +8.0 F1 for yolo-x OI and \(\tau=0.5\)). This highlights that relaxed evaluation metrics can inflate the actual quality of EAE.

4.3 Multimedia Evaluation and Analysis↩︎

In this section, we empirically analyze how varying task assumptions and evaluation settings affect multimedia EE results. Real-world applications typically do not provide oracle image-text pairs or prior knowledge about whether a text or image contains a multimedia event.

4.3.0.1 Experimental Setup

To study the impact of task discrepancies [P5] and missing trigger attachments [P8], we evaluate three event coreference resolution strategies: threshold-based, greedy, and bipartite matching approaches proposed in prior work [17]–[19]. We further compare evaluations on the full dataset against subsets restricted to samples containing at least one multimedia event, and analyze the effect of applying the same subset selection during training.

4.3.0.2 Impact on Multimedia Results

3 presents the multimedia evaluation results. Consistent with textual evaluation, ignoring trigger offsets for textual events leads to an drop of up to 3.3 F1 for ED, indicating correct sentence-level predictions with incorrect trigger spans. Larger differences arise across task setups. Restricting evaluation to samples containing multimedia events leads to gains of up to +21.2 F1 for threshold-based methods, although such prior knowledge would not be available at deployment. Applying the same subset selection during training further amplifies this effect and reveals opposing trends between full and subset evaluations (see 3). Under realistic conditions, bipartite matching performs best, whereas threshold-based methods outperform it on the filtered subset (30.9 vs. F1), attributed to increased recall. Beyond multimedia ED, we also observe similar patterns for EAE (bottom row of 3), which is connected to ED performance due to the pipeline setup. Finally, experiments using gold image-text coreference annotations show the highest scores, highlighting that subset-based evaluation can substantially overestimate real-world performance.

5 Consistent Evaluation↩︎

Our analysis reveals several limitations in current MEE evaluation practices, however, so far ignores the combination of hidden pitfalls and advanced modeling techniques. Therefore, we reproduce and reevaluate recent methods [18], [19], [33], [35]² under their original evaluation settings and compare them with our proposed framework StrictEval. Notably, SSGPF supports only multimedia evaluation and assumes given image-text pairs. Reproduction details are provided in Appendix 7.5.

5.0.0.1 Reevaluation Results and Discrepancies

The results in 4 highlight discrepancies between the Original and StrictEval setting. First, evaluation scores change substantially in absolute performance levels. Specifically, textual and visual ED scores differ significantly, primarily due to trigger postprocessing [P2], verb refinement [P3], and test subset selection [P4] (cf. 2). This indicates that minor implementation choices can dominate reported gains. Second, removing these pitfalls consistently leads to lower EAE performance due to the pipeline setup. This can overestimate advances in multimedia EAE, which are often driven by increased ED performance. Third, we observe most degradation in multimedia ED and EAE, with drops of up to 56.8 and 33.6 F1, respectively. When neither test subset selection nor gold event coreference annotations are used, precision decreases dramatically, revealing that cross-modal event coreference resolution remains the largest challenge in a more realistic setting. Further intuition about these low scores and performance drops is provided in Appendix 7.6.

5.0.0.2 Implications for Evaluation and Future Work

Overall, these findings underscore the need for standardized and transparent evaluation protocols. We encourage the research community to report detailed evaluation choices, avoid reliance on gold annotations, and adopt unified evaluation pipelines to ensure fair comparison and reproducibility across MEE methods Our analysis further indicates that progress in multimedia event extraction requires a stronger focus on accurate cross-modal event coreference and semantic alignment, which remain underexplored in recent work. Finally, future advances would benefit from more comprehensive multimodal datasets with explicit coreference annotations, high-quality training data, and standardized splits to support robust and comparable evaluation.

6 Conclusion↩︎

In this work, we present the first systematic analysis of evaluation pitfalls and challenges in MEE and reveal substantial gaps between reported performance and a model’s actual ability to ground events across textual and visual modalities. Our analysis of the M2E2 benchmark uncovers three major sources of discrepancies: inconsistent data processing, inconsistent task assumptions, and overly relaxed evaluation settings. To address this, we propose the evaluation framework StrictEval, which enforces strict evaluation constraints for more challenging evaluation. Controlled experiments show that minor evaluation choices can significantly affect performance, highlighting that cross-modal event coreference resolution and precise MEE remain open challenges.

Limitations↩︎

In this work, we focus on analyzing evaluation pitfalls primarily based on the M2E2 benchmark, which, despite being the most widely used public benchmark for MEE, represents only a subset of possible settings. As a result, some identified issues and recommended practices may not fully generalize to newer benchmarks, annotation schemes, or domains beyond news media. Extending our analysis to additional datasets, modalities (e.g., videos or audio), and task formulations remains an important direction for future work. Moreover, our empirical analysis relies on a relatively simple MEE model and a limited set of reproduced recent methods. While this design choice allows us to isolate the impact of evaluation decisions, it does not capture the full diversity of modeling approaches, particularly recent instruction-following MLLMs. Although we expect the identified evaluation pitfalls to persist across architectures, their quantitative impact may vary with different modeling paradigms. Finally, our proposed evaluation framework adopts stricter evaluation settings and result into substantially lower absolute performance scores. While this may hinder direct comparisons with prior work, our goal is to promote more realistic and transparent evaluation that better reflects the real-world challenges of MEE.

Ethical Considerations↩︎

The results reported in this paper are intended to improve evaluation transparency in MEE and should not be interpreted as implying misconduct by prior work. The identified evaluation pitfalls are subtle and often related to underspecified benchmark evaluation standards, making them easy to be overlooked. Therefore, the motivation of this work is to raise awareness about these issues and to promote more reliable evaluation practices.

7 Appendix↩︎

7.1 Articles for Systematic Analysis↩︎

5 presents a comprehensive list of studies included in our systematic analysis, along with their reported evaluation scores. These works cover diverse modeling paradigms, ranging from early pipeline-based approaches to recent instruction-following and MLLMs.

7.2 One-to-One Matching Strategy↩︎

As described in §3.5, visual arguments are initially evaluated using a many-to-many matching strategy. In this setting, multiple predicted arguments may be matched to the same ground-truth argument (many-to-one) and a single predicted argument may match multiple ground-truth arguments (one-to-many). While this approach captures all overlapping predictions, it can inflate evaluation scores and does not enforce a strict mapping between predicted and gold arguments.

7.2.0.1 One-to-One Matching

To address this limitation, we adopt a one-to-one matching strategy based on the Hungarian algorithm [41]. Predicted and ground-truth arguments are modeled as nodes in a bipartite graph and a globally optimal matching is computed between the two sets. This formulation guarantees that each predicted argument is matched to at most one ground-truth argument, yielding a stricter and more interpretable evaluation.

7.2.0.2 Empirical Impact Analysis

In 6, we show the impact for object detectors YOLOv8 [42], [43] and Faster R-CNN [44], trained on COCO [45] and OpenImages [46], respectively. Models trained on COCO show robust performance under the stricter matching strategy, whereas results on OpenImages exhibit substantial differences. For example, we observe an F1 increase of +8.0 for YOLO-X OI at \(\tau=0.5\). This behavior can be attributed to overlapping object categories in OpenImages (e.g., person and suit) which are otherwise counted as multiple correct predictions under many-to-many matching (see 4).

7.3 Proposed Single Task Models↩︎

In this section, we describe the Single Task models used in the experiments reported in §4. Each subtask is modeled independently, including textual ED / EAE, visual ED / EAE, and Event Coreference Resolution.

7.3.0.1 Textual Event Extraction

For textual ED and EAE, we employ BERT [47] as the text encoder. Given an input sentence \(s\), BERT produces contextualized subtoken representations, which are mean-pooled to obtain token-level embeddings. Textual ED is formulated as a sequence labeling task where each token is classified into an event type using a linear classifier. For textual EAE, following [6], we assume gold entity mentions and perform role classification by mean-pooling the subtoken representations of each entity mention. Textual ED and EAE are trained as separate models using cross-entropy loss.

7.3.0.2 Visual Event Extraction

For visual ED and EAE, we adopt the CLIP [39] vision encoder. Given an input image \(i\), CLIP produces global and patch-level representations. Visual ED is treated as image-level classification by feeding the [CLS] token representation into a linear classifier. For visual EAE, we first detect objects using an offline detector [19]. Patch representations corresponding to each object are mean-pooled to form object embeddings, which are then classified into argument roles using a linear layer. Analogous to the textual tasks, Visual ED and EAE are trained independently with cross-entropy loss.

7.3.0.3 Multimedia Event Extraction

As described in §4.1, event coreference resolution between textual and visual events is performed using CLIP-based similarity scores. Following prior work [18], [19], [33], we construct a multimedia event when a text–image pair shares the same predicted event type and their similarity score exceeds a threshold of 20. The resulting multimedia event aggregates all associated textual and visual arguments. In addition to this heuristic approach, we also report results obtained using greedy and bipartite matching strategies [17].

7.3.0.4 Training

We train all models using a learning rate of \(1\times10^{-5}\) for encoder parameters and \(1\times10^{-4}\) for classifier parameters. Textual models are trained with a batch size of 16 for 20 epochs, while visual models use a batch size of 64 and are trained for 10 epochs. Model performance is evaluated at the end of each epoch on the ACE development set for textual tasks and the SWiG development set for visual tasks. We select the checkpoint with the best development set performance for final evaluation.

7.4 Implementation Details↩︎

All models are implemented using the Transformers [48] (v4.55.0) library in conjunction with PyTorch (v2.8.0). Unless otherwise specified, we use bert-base-uncased [47] with 222M parameters as text encoder and clip-vit-base-patch16 [39] with 85M parameters as vision encoder. Object detections are obtained using YOLOv8 [43]³ trained on COCO and detections with confidence scores below 0.8 are discarded. All experiments are conducted on NVIDIA A100 GPUs within a single compute node running CUDA 12.3. We run each experiment with three random seeds and report the average performance across runs.

7.5 Reproduction Details↩︎

In this section, we provide detailed reproduction information for the models discussed in §5, along with potential explanations for any observed differences in performance scores.

7.5.0.1 CAMEL

We use the official released code⁴ provided by [18]. Our reproduced F1 scores largely align with the reported results, except for multimedia ED (48.3 vs. 57.5 F1) and EAE (26.5 vs 33.2 F1). We attribute these discrepancies primarily to the absence of balanced visual ED training in our reproduction that reduced recall in favor of precision.

7.5.0.2 MMUTF

We use the official released code⁵ provided by [33]. Following the original paper, which mainly focuses on EAE, we use ED predictions from CAMEL. Our reproduced scores closely match the reported results with minor decreases in multimedia ED and EAE, which we also attribute to the lower recall of CAMEL predictions.

7.5.0.3 X-MTL

We use the official released code⁶ provided by [19]. Our reproduced results show only minor deviations from the originally reported results (e.g., 56.6 vs. 52.9 F1 for textual ED). We believe these differences are related to inconsistencies in the pseudo-labeled VOA image-caption dataset, caused by broken links during dataset construction.

7.5.0.4 SSGPF

We re-implement SSGPF using LLaVA-v1.5-7B as described in the paper⁷ [35]. The model assumes aligned image-sentence pairs and requires manually written event and role descriptions. We obtain comparable performance on multimedia ED (65.7 vs. 61.0 F1) but observe a substantial drop on EAE (36.0 vs. 14.1 F1), which we attribute to implementation differences in EAE evaluation, visual grounding (SEEM), and role descriptions. For StrictEval, we use the proposed fine-tuned cross-modal retrieval model to construct image-sentence pairs.

7.6 Analysis of StrictEval↩︎

We base our experiments on the human-annotated M2E2 dataset (Inter-Annotator Agreement of 81.2%) [6] and remove relaxed assumptions used in prior work to ensure more consistent comparison. The resulting substantially lower scores are primarily due to stricter evaluation protocols that expose false positives otherwise ignored, which we detail next. Using our proposed Single Task models, we provide further intuition for these effects.

7.6.0.1 Textual Evaluation

In textual evaluation, oracle trigger refinement⁸ and test subset selection remove a large number of incorrect predictions and negative samples. For example, the Single Task model predicts 3,969 text events, of which refinement removes 1,636 false positive events and 1,885 arguments prior to evaluation. Furthermore, our manual analysis suggests annotation discrepancies between ACE and M2E2, potentially contributing to the large number of false positive events.

7.6.0.2 Visual Evaluation

Similarly, test subset selection excludes 623 images without annotated events, restricting evaluation to only 391 positive samples. As a result, false positives from the excluded images are not counted. For instance, the Single Task model predicts 88 false positive visual events and 341 arguments on these omitted images. Evaluating only on positive samples therefore inflates precision by ignoring predictions on negative images.

7.6.0.3 Multimedia Evaluation

For multimedia evaluation, recent work often assumes access to ground truth image–sentence pairs or applies post-hoc filtering using gold sentence and image IDs (e.g., via coreference annotations). This substantially reduces the number of evaluated pairs and removes negative candidates. In contrast, StrictEval constructs and evaluates over all possible image–sentence pairs. Consequently, models must handle a much larger and noisier candidate space, leading to a significant increase in false positives. For our baseline models, post-hoc filtering excludes 1,118 false positive multimedia events, resulting into a substantial drop in precision and F1 scores.

7.7 Additional Experimental Results↩︎

This section supplements the multimedia results presented in §4.3 by providing EAE scores along with precision and recall. We report evaluations for different coreference resolution strategies: threshold-based (7 and 8), greedy matching (9), and bipartite matching (10).

References↩︎