1 Introduction↩︎

Relation extraction (RE) is an NLP task that distills factual information from text by identifying relationships between entities in the form of fact triplets (e.g.,⟨head, relation, tail⟩) [1]–[4]. RE facilitates various downstream applications such as knowledge graph construction, question answering, and information retrieval  [5]–[7]; however, creating datasets for training RE models is costly and challenging, requiring annotators to identify entities and relations across large sections of text [8], [9].

Recent efforts have explored adapting the RE task into a natural language inference (NLI) task, where the goal is to determine whether a given hypothesis logically follows from, contradicts, or is neutral with respect to a premise. This adaptation enables the use of relatively large NLI datasets to improve performance on an RE-adapted task [10]–[12]. RE-to-NLI works transform relation instances into premises paired with \(m\) class-indicative hypotheses where \(m\) is the number of relation classes in a dataset. A language model is trained to label premise-hypothesis pairs as entailed, contradicted, or neutral. We build on this work by introducing MetaEntail-RE, a novel NLI adaptation method that improves RE performance by leveraging three key enhancements: automatic feasible hypothesis filtering, meta-class analysis, and group-based prediction selection.

Feasible hypothesis filter: We first introduce a feasible hypothesis filter that automatically removes infeasible hypotheses based on domain knowledge derived from data. To develop this filter automatically, we approximate valid sets of entity-type pairs corresponding to each relation class by aggregating all relations in the training data. These approximated sets of valid type-pairs are then used to remove hypotheses that verbalize infeasible relationships. For instance, in the BioRED dataset [9], it is impossible for a gene to “bind” to a disease (i.e., the “bind” label is not applicable to gene-disease entity-type pairs). We therefore remove the “bind” hypothesis from all instances with gene-disease entity types. This filter improves training efficiency by reducing the number of NLI instances.

Meta-class analysis: In past RE-to-NLI works, if a premise does not entail a hypothesis, the corresponding NLI label assigned is “neutral” [10], [12]. However, this misses an implicit training signal we can gain by analyzing the semantics of a dataset’s relation classes. When assigning NLI labels to adapted RE instances, we distinguish between task-based mutual exclusivity and definition-based mutual exclusivity. Task-based mutual exclusivity is an artifact of the single-class classification task inherent to a dataset. Each input instance is annotated with a single relation class, thereby arbitrarily making all classes mutually exclusive. In contrast, definition-based mutual exclusivity is derived from definitions of relation classes. For example, within the BioRED dataset [9], the “positive correlation” class is definitionally mutually exclusive and contradictory to the “negative correlation” class [9].

If two classes are definitionally mutually exclusive, we apply the “contradict” label to the appropriate premise-hypothesis pair, thereby injecting additional information about the meta-relationship between relation classes which the model can exploit while learning relationship representations. Leveraging this insight, we can glean multiple informative training signals from a single relation instance when adapting the relation extraction task into the natural language inference task. We call this method meta-class analysis (MCA) and use it to determine the appropriate NLI labels for each premise-hypothesis pair. We show through ablation experiments that MCA leads to significant gains on an RE-adapted task.

Group-based prediction selection: Group-based prediction selection exploits the feature of RE-to-NLI adaptation in that each relation instance is converted into a group of premise-hypothesis pairs where each hypothesis verbalizes a relation class in the dataset. When evaluating cases where the model predicts multiple “entail” labels within a single group, we can select the most confident “entail” prediction and ignore other predictions. Our results demonstrate that this group-based prediction selection method leads to additional gains.

MetaEntail-RE as an RE-to-NLI adaptation method is technically domain agnostic; however, it is particularly well-suited for biomedical RE where associations often have opposing classes such as “positively correlated” and “negatively correlated” [9] or “agonist” and “antagonist” [13] enabling a rich MCA. We also find that associations in biomedical RE are often type-dependent compared to general domain RE, making the feasible hypothesis filter more effective at trimming infeasible hypotheses. Still, we extend our evaluations beyond the biomedical domain to determine how MetaEntail-RE fares on general domain RE datasets. Notably, we observe improvements in both domains, reinforcing the effectiveness and versatility of MetaEntail-RE. We summarize the main contributions of this work as the following:

• We introduce a novel RE-to-NLI adaptation method, MetaEntail-RE, and showcase its robustness and versatility in RE datasets from general and biomedical domains.

• We illustrate through ablation experiments the effectiveness of components of MetaEntail-RE.

• We openly provide all code, experimental settings, and datasets used to substantiate the claims made in this paper.²

2 Related Work↩︎

Traditionally, RE has been approached as a classification task, where input instances are classified as belonging to a relational class [1]–[4]. These methods have several drawbacks: they tend to generalize poorly [12], [14], and they heavily rely on relatively small and disjoint RE datasets. To account for these drawbacks, recent works have proposed clever adaptation methods to recast RE into adjacent NLP tasks, such as a question-answering [15] and NLI [10]–[12], [16]. Task adaptation presents an opportunity to leverage the relatively large datasets available for other tasks (e.g., SQuAD [17], MultiNLI [18], SNLI [19], etc.), which can be particularly advantageous in the context of biomedical RE where datasets are often limited.

[15] recast RE into a question-answering task by associating relation instances with one or more natural-language questions, resulting in predicted spans denoting class indicative text. [16] adapts general domain RE into an NLI task by using relation instances as premises where each premise is paired with a hypothesis generated by verbalizing a relation class. In doing so, they formulate a binary entailment task where they predict whether or not a premise entails the corresponding hypothesis.

[10] expands on [16] by incorporating a three-label classification objective where a model can predict entail, contradict, and neutral depending on the premise-hypothesis pair, bringing the task in line with a standard NLI formulation [20]. They manually generate hypothesis templates corresponding to each relation class in a dataset, and NLI labels are assigned based on the alignment of the premise-hypothesis pair. If the corresponding hypothesis is the verbalized version of the ground truth relation label, then “entail” is assigned as the NLI label for the instance. The “neutral” label is applied to positive class hypotheses which do not align with a given premise. The “contradict” label is applied in two cases: (1) if the premise is a positive relation instance (e.g., any class other than “no relation”), the “no-relation” hypothesis is labeled as “contradict,” and (2) if the premise is a negative instance (e.g., “no relation”), then all other positive class hypotheses are labeled as “contradict.” [10] fine-tune a language model pre-trained on the MultiNLI [18] dataset to predict generated NLI labels. They observe impressive results in zero- and few-shot scenarios on TACRED [21], a general domain, sentence-level RE dataset.

[12] explores cross-domain transfer learning, leveraging indirect supervision from general domain NLI datasets to improve biomedical RE-to-NLI adapted methods. Our work can be considered an extension of their proposed NBR method. However, we introduce a few key improvements: meta-class analysis, a feasible hypothesis filter, and group-based prediction selection. We also expand evaluations beyond sentence-level RE to include more challenging document-level RE [9], [22].

3 Problem Statement↩︎

Our problem is a hybridization of RE and NLI; as such, we describe both tasks, as well as the adapted RE-to-NLI task.

Relation Extraction (RE): RE takes inputs \(\{x_1, x_2,\ldots, x_n\} \in X_{\mathrm{\small RE}}\) where \(X_{\mathrm{\small RE}}\) is a corpus of sentences, paragraphs, or documents of size \(n\) and \(x_i\) is a singular instance containing an entity pair \(e_{i_{1}}\) and \(e_{i_{2}}\). Each input \(x_i\) has a corresponding label \(y_i\). Labels \(\{y_1, y_2,\ldots, y_n\} = Y_{\mathrm{\small RE}}\) belong to a set of \(m\) relation classes \(R = \{r_1, r_2,\ldots, r_m\}\). RE seeks to identify which class links the co-mentioned entities to form a fact triplet \(\langle e_{i_{1}}, y_i, e_{i_{2}} \rangle\), or, semantically, ⟨head, relation, tail⟩.

Natural Language Inference (NLI): NLI takes a premise \(p_i \in P\) and a hypothesis \(h_i \in H\), where \(P\) and \(H\) are the set of premises and hypotheses in a corpus, respectively, and seeks to determine whether the premise entails, contradicts, or is neutral to the respective hypothesis [19], [20]. Using \(\hat{y}_i\) to represent an NLI label applied to the \(i_{th}\) instance, \(\hat{y}_i \in \{entail, contradict, neutral\}\), and a single NLI example can be expressed as \(\langle p_i, \hat{y}_i, h_i \rangle\).

RE-to-NLI Adaptation: RE-to-NLI adaptation converts RE inputs and labels into premise-hypothesis pairs such that each input instance maps to \(|R|\) premise-hypotheses pairs: \((x_i,y_i)\rightarrow\{(p_i, \hat{y}_j, h_j)\}_{j=1}^{|R|}\). We decompose RE-to-NLI adaptation into the following sub-steps:

1. Premise generation, \(x_i\rightarrow p_i\): Input instances \(x_i \in X_{\mathrm{\small RE}}\) directly become premises \(p_i \in P^{|X_{\mathrm{\small RE}}|}\) where \(P\) is the collection of all premises generated from \(X_{\mathrm{\small RE}}\).

2. Hypothesis generation, \(H_{i} = \{h_j\}_{j=1}^{|R|}\): In the hypothesis generation step, a set of hypotheses \(H_{i}\) paired with each premise \(p_i\). This is achieved by first verbalizing relation classes in \(R\) into a set of \(m\) hypothesis templates \(T=\{t_1, t_2,\ldots, t_m\}\). Each hypothesis template contains head and tail entity placeholders, which are replaced by the head and tail entities found in the corresponding premise \(p_i\). The verbalizer function \(f_{verbalizer}(\cdot)\) takes each hypothesis template and entity pair in premise \(p_i\) to produce the set of hypotheses \(H_i = \{f_{verbalizer}(t_j, e_{i_{1}}, e_{i_{2}})\}_{j=1}^{|R|}\).

3. NLI label generation, \(\hat{Y} = \{\hat{y}_i\}_{i=1}^{|X_\mathrm{\small RE}| \times |R|}\): The set of NLI labels \(\hat{Y}\) is generated via a function which takes the original instance label \(y_i\) and the premise-hypothesis pair \(f_{target}(y_i, p_i, h_j) \rightarrow \hat{y}_j\) where NLI label \(\hat{y}_j = entail\) iff verbalized class-indicative hypothesis \(h_j\) aligns with the ground truth label \(y_i\), and, depending on the adaptation method, \(\hat{y}_j\) is assigned \(neutral\) or \(contradict\) for non-aligned hypotheses.

The RE-to-NLI task is to correctly predict entailed premise-hypothesis pairs where each entailed pair has a 1-to-1 mapping to the original RE label.

4 Methods↩︎

This section sequentially discusses the modules used in MetaEntail-RE(see Figure 1).

Premise Construction: Following [12], a relation instance \(x_i\) is transformed into a premise by replacing surface forms of the subject and object entities, \(e_1\) and \(e_2\), respectively, with their corresponding entity types, \(e_{1_{type}}\) and \(e_{2_{type}}\). Abstracting entity surface forms into entity types helps alleviate the long-tail nature of biomedical entities and encourages language models to learn from context instead of shallow heuristics [14]. The start and end spans of entity types are denoted with “\(@\)” and “\(\$\),” respectively.

Hypothesis Verbalizer: Past works have manually generated hypothesis templates for each relation class in a dataset which are then used, in turn, to generate hypotheses to pair with a given premise. A secondary contribution of MetaEntail-RE is that we reduce this human effort by leveraging LLMs to automatically generate the set of hypothesis templates \(\{t_1, t_2,\ldots,t_m\}\in T\), where \(m\) corresponds to the number of relation classes in a dataset. We prompt an LLM³ to verbalize each relation class using natural language and placeholders for subject and object entities (see Appendix 8.1 for more details). The placeholders within the hypothesis templates are replaced by the entity types, \(e_{1_{type}}\) and \(e_{2_{type}}\), found in the premise.

Feasible Hypothesis Filter: There is an implicit multiplicative effect of adapting RE into an NLI task where each relationship instance produces \(m\) class-indicative hypotheses resulting in \(|X_{\mathrm{\small RE}}| \times m\) premise-hypothesis pairs. To mitigate this effect, we develop a feasible hypothesis filter which automatically filters out improbable hypotheses by aggregating valid sets of entity-type pairs by relationship classes across all training data: \(E_{valid}=\left\{r_1 \mapsto S_1, r_2 \mapsto S_2, \ldots, r_m \mapsto S_m\right\}\) where \(r_j \in R\) for \(j=1,2, \ldots, m\) and each \(S_j\) is the set of tuples of entity-type pairs associated with all instances of relationship class \(r_j\).

Using this filter, we assess the feasibility of hypotheses given a pair of entity types: \(\hat{H}_i = \{h_j | (e_{1_{type}}, e_{2_{type}}) \in E_{valid}(r_{j}) \}_{j=1}^{|R|}\) where \(\hat{H}_i\) is a set of feasible hypotheses given the entity-type pair found in instance \(i\), and \(\hat{H}_i \subset H\) where \(H\) is the set of all possible hypotheses.

Since sets of feasible hypotheses are approximated using the training data’s relationships and entity-type pairs, the filter may remove valid hypotheses based on an entity-type pair and corresponding relation that exists only in the test set. For these instances, the entailed premise-hypothesis pair will not be presented to the model, leading to false negatives. However, in practice, we observe that this does not occur with the datasets we use for evaluation and should not occur as long as training data is sufficiently representative of the test data (i.e., the training data contains at least one relationship with a specific entity-type pair for every relation and entity-type pair found in the test set).

Meta-class Analysis (MCA): After applying the aforementioned feasible hypothesis filter, we leverage MCA to assign NLI labels, namely entail, neutral, and contradict, to the resultant premise-hypothesis pairs. To do this, we first construct definition-based mutually exclusive meta-relationships between relation classes. For example, in the ChemProt dataset, the “up regulator” class is, by definition, mutually exclusive to the “down regulator” class. For datasets with a negative class (e.g., “no relation”), the negative class is mutually exclusive to all positive classes and vice-versa. With this analysis, we construct NLI labels in the following way:

1. Entail: Premise-hypothesis pairs are labeled “entail” when the hypothesis \(h_j\) aligns with the verbalized ground truth label \(y_i\).

2. Neutral: If the original instance expresses a positive class (i.e., any class other than the “no relation” class), then all non-exclusive class hypotheses are labeled as “neutral.”

3. Contradict: The “contradict” label is assigned to hypotheses that verbalize definitionally exclusive classes.

See Appendix 8.4 for tables showing how relation labels map to NLI labels using MCA.

LLM Fine-tuning: With generated premise-hypothesis pairs, we train a discriminative language model, namely BioLinkBERT_large [23], to predict NLI labels. We concatenate premise-hypothesis pairs as the input to the language model and send the resultant representation of the special [CLS] token through a fully connected layer, which is trained using cross-entropy loss: \[\label{eq:cross95entropy} \begin{align} \mathcal{L}_{\mathrm{CE}}=-\sum_{i=1}^{m} y_{o,i} \cdot \log \left(p\left(y_{o,i}\right)\right) \end{align}\qquad{(1)}\] where \(y\) is a binary indicator that is \(1\) if and only if \(i\) is the correct classification for observation \(o\), \(p(y_{o,i})\) is the softmax probability that observation \(o\) is of class \(i\), and \(m\) is the number of classes.

Group-based Prediction Selection: Given the multiplicative effect of adapting RE-to-NLI where one relation instance results in a group of up to \(m\) premise-hypothesis pairs, we can employ a group selection method to select the most confident entail prediction. If the model predicts two or more entailed instances within a group of premise-hypothesis pairs, we use the softmax probability from Equation ?? as a proxy for model confidence [24] and select the prediction with the highest confidence. We allow the model to naturally abstain from making a prediction by predicting “neutral” for all premise-hypothesis pairs in a group.

5 Experiments↩︎

5.1 Datasets↩︎

We include a spread of experiments on various biomedical RE datasets. BioRED is a document-level RE dataset featuring eight relation classes [9]. BioRED also provides an orthogonal and binary “Novel” class, which annotates whether an instance expresses a novel finding. BC5CDR is a document-level RE dataset featuring binary relations between chemical and disease entities [22]. DDI13 is a drug-drug interaction dataset with four relation classes [25], and ChemProt is a chemical-protein dataset featuring five relation classes [13]. GAD is a gene-disease dataset with binary relations [26]. We only include GAD in our main experiment for comparative purposes to past works. We believe that the GAD dataset should be retired from future works due to significant label accuracy issues, which the authors acknowledge.⁴

ChemProt and BioRED, unlike the other datasets in our experiments, do not annotate negative instances, leading to ambiguity in handling unannotated data. These unannotated instances can be addressed in three ways: (1) by treating them uniformly as a negative class, (2) by considering them as potential members of novel, unannotated classes, or (3) by using a generalized approach that considers unannotated instances as a mix of negative and novel classes. To avoid this subjectivity, we focus only on annotated instances for training and evaluation, applying this approach consistently across all datasets to ensure a fair comparison of methods.

As mentioned in Section 1, our method is designed to leverage features of biomedical domain RE, namely the prevalence of definitionally exclusive classes and the importance of entity types vis-à-vis feasible relationships. However, we also seek to assess our method beyond the biomedical domain and extend our experiments to general domain datasets ReTACRED [27] and SemEval-2010 Task 8 [28]. ReTACRED is a re-annotated version of TACRED [21] and features 40 relation classes—significantly more classes than any of the biomedical datasets we tested. SemEval-2010 is a sentence-level RE dataset with ten relation classes.

5.2 Baselines↩︎

5.2.1 Traditional Multi-Class Classification↩︎

We select leading biomedical language models and train them using a traditional RE multi-class approach where models directly predict relation classes. BioM-ALBERT_xxlarge and BioM-BERT_large [29] are transformer architectures adapted into the biomedical domain by using a custom biomedical vocabulary and pre-training on PubMed abstracts [30] and PubMed Central articles [31]. BioMed RoBERTa_base [32] features the RoBERTa architecture [33] adapted to the biomedical domain via continued pre-training on papers from the S2OR Corpus [34]. PubMedBERT_base [35] and BioLinkBERT_large [23] are BERT [36] variants. The former is trained on PubMed abstracts with a custom biomedical vocabulary. The latter is trained with two self-supervised objectives: masked language modeling and document relation prediction.

5.2.2 NLI Adapted Models↩︎

NBR is a biomedical domain RE-to-NLI method that leverages BioLinkBERT_large as a backbone language model. Like our method, NBR converts relation instances and labels into premise-hypothesis pairs. Key differences between NBR and our method are that NBR does not use MCA or feasible hypothesis filtering, and they leverage a ranking loss training objective to rank entailed premise-hypothesis pairs over non-entailed pairs.

The RE-to-NLI adaptation method used in MetaEntail-RE is architecture-agnostic, so we also experiment with auto-regressive architectures. We conduct the following experiment using identical data and methods to those discussed in Section 4; the only difference is the final training step.

We fine-tune Phi-2 (2.7B) and Phi-3 (3.8B).⁵ For Phi-2 and Phi-3, we construct a seq-to-seq task and fine-tune the models to generate an NLI label for each premise-hypothesis pair. For more information about training Phi-2 and Phi-3, see Appendix 8.2.2.

We also seek to assess the performance of large, frontier auto-regressive language models, GPT 3.5 [37] and GPT 4 [38],⁶ leveraging few-shot, in-context learning. For more on the prompts we use to solicit predictions from GPT 3.5 and GPT 4, see Appendix 8.2.3.

For all NLI-adapted models, only entailed premise-hypothesis pairs map directly to the original RE training instance. Thus, we only keep NLI instances labeled or predicted as entailed when mapping instances back into the original RE labels for evaluation. This ensures a fair comparison across adapted and non-adapted methods.

5.3 General Domain Experiments↩︎

For our general domain experiments, we use DeBERTaV3_large [39] and RoBERTa-MNLI_large [33]. DeBERTaV3 is an improved version of BERT that uses replaced token detection, a more sample-efficient pre-training objective. RoBERTa-MNLI_large is the RoBERTa architecture fine-tuned on the MNLI corpus [18].⁷

We make slight modifications to the general domain version of MetaEntail-RE. We use RoBERTa-MNLI_large as the backbone language model, and we do not leverage surface-form abstraction for entity types (i.e., we leave the original entities as they appear in the text and do not replace them with their corresponding types). Entity surface form abstraction is a method developed for the long-tail nature of biomedical entities [14]. Also, some general domain RE datasets, such as SemEval-2010 Task 8, do not provide annotated entity type information.

6 Results↩︎

max width=1.0

We observe an interesting comparison between the BioLinkBERT_large model and MetaEntail-RE. Both experiments share the same backbone language model, yet the performance of our MetaEntail-RE method is significantly higher providing evidence of the effectiveness of adapting the RE task into one of textual entailment. We hypothesize that the boost in performance primarily comes from the additional data abstraction RE-to-NLI introduces by training the model to recognize entailed premise-hypothesis pairs instead of directly predicting suppositional classes. By combining RE-to-NLI adaptation with surface-form entity abstraction, the model is less prone to memorizing entities and shallow heuristics of relation classes; instead, it must understand the context and the natural language interplay between a premise and hypothesis. Furthermore, the boost in performance between the NBR model and MetaEntail-RE highlights the effectiveness of leveraging MCA, feasible hypothesis filtering, and group-based prediction selection.

Within the biomedical domain experiments, the NLI-adapted auto-regressive models generally underperform compared to the discriminative models. Predictably, the larger Phi-3 outperforms Phi-2 and fine-tuning smaller auto-regressive models outperforms larger models, GPT 3.5 and GPT 4, leveraging few-shot in-context learning. This aligns with findings from [40] that LLMs using in-context learning underperform relative to smaller, fine-tuned language models on information extraction tasks.

We observe better performance from auto-regressive architectures in the general domain. The performance from Phi-3 approaches that of MetaEntail-RE on both ReTACRED and SemEval-2010 Task 8 datasets which is promising for auto-regressive models, in general. We leave fine-tuning larger auto-regressive models to future work but expect additional gains to be made, potentially overtaking the discriminative models.

max width=.48

6.1 Ablation Experiments↩︎

max width=.48

We conduct ablations to better understand MetaEntail-RE’s performance gains by removing modules and reporting the performance. Note that the performance of BioLinkBERT_large in Table ¿tbl:tab:main95results? can be considered an ablation of MetaEntail-RE that does not leverage NLI adaptation or any additional modules since MetaEntail-RE uses the same backbone language model. For our ablations, we choose to examine the BioRED, ChemProt, and ReTACRED datasets because they feature more than two relation classes and contain one or more definition-based mutually exclusive relations as determined by MCA.

1. w/o Feasible Hypothesis Filter: We remove the feasible hypothesis filter, and, in doing so, each relation instance is converted into \(m\) premise-hypothesis pairs, with \(m\) being the number of classes in a dataset. This produced a moderate drop in performance on BioRED (\(m=8\)). Since the feasible hypothesis filter is based on entity type pairs, it is not available (N/A) for datasets such as ChemProt, which only feature a single entity type pair (namely, chemical and gene) associated with every relation class. However, the feasible hypothesis filter is essential in model convergence when a dataset consists of many relation classes, such as ReTACRED (\(m=40\)). The model did not converge on ReTACRED without the feasible hypothesis filter, likely due to the overwhelming number of non-informative “neutral” premise-hypothesis pairs used in training.

2. w/o Meta-class Analysis: Removing MCA and using “neutral” as the NLI label for all non-entailed premise-hypothesis pairs led to a considerable drop in performance, indicating the benefit of training the model with the additional training signal obtained via MCA. Note that in this ablation experiment, we maintain mutual exclusive NLI labels between positive and negative (i.e., “no relation”) classes.

3. w/o Group Prediction Selection: Without this module, we select all entailed predictions regardless of how many entail predictions are made within a group of premise-hypothesis pairs. Doing this allows the model to predict multiple classes for a single relation instance. This ablation experiment led to a drop in performance across all datasets but most significantly on BioRED, which we suspect results from the closeness in BioRED’s “positively correlated” and “associated” relation classes, as “associated” can sometimes be considered a hypernym of “positively correlated,” leading the model to predict entail for both of the corresponding hypotheses.

6.2 Meta-class Analysis Case Study↩︎

To further explore the impacts of leveraging MCA, we decompose results from ReTACRED, BioRED, and ChemProt by evaluating the change in Micro F1 scores (\(\Delta\)F1) for each class. We isolate the effect of MCA by training identical models with and without MCA-informed NLI labels and report the results in Figure 2.

MCA results in a net benefit in performance across classes and datasets, but the specific nature of these benefits varies. In ReTACRED, we observe notable improvements in the “member of” and “members” classes, which are definitionally exclusive. Conversely, some classes experience minor decreases in performance. For BioRED, we observe a slight drop in predictive performance for the “negative correlation” class, while all other classes get a significant boost. The largest performance gains are seen in classes that are not mutually exclusive, suggesting that the additional training signal from MCA aids the model in disentangling adjacent relation class representations. In ChemProt, we observe near-uniform, albeit relatively small, boosts in performance across all classes. This indicates that MCA has varied effects across disparate datasets. There is a net benefit but, interestingly, the exact nature of the benefit varies across datasets.

6.3 Additional Experiments↩︎

Given that RE-to-NLI adaptation leads to models predicting the same entail, neutral, contradict labels across disparate datasets, we naturally sought to investigate the potential of combining the relatively small and disjoint biomedical RE datasets into a single, unified task. Unfortunately, these experiments failed to produce significant performance gains, indicating that these biomedical datasets have limited synergistic effects when adapted to the NLI task (see Appendix 8.3).

7 Conclusion↩︎

The exploration of NLI techniques to enhance relation extraction has opened new avenues in natural language processing, and our study introduces MetaEntail-RE as an advancement in this area. By adapting the RE task into an NLI framework and incorporating innovative strategies such as meta-class analysis, feasible hypothesis filtering, and group-based prediction selection, MetaEntail-RE demonstrates remarkable improvements in RE performance. Our experiments, conducted across biomedical and general domain datasets, highlight the robustness and versatility of MetaEntail-RE. By openly sharing our code, experimental settings, and datasets, we aim to facilitate further research and development in this promising intersection of NLI and RE, paving the way for more sophisticated and accurate information extraction systems in diverse domains.

Limitations↩︎

MetaEntail-RE is not without its limitations. By verbalizing a hypothesis for each relation class, the training data is multiplied by the number of relation classes in the dataset, necessitating additional training resources. Our introduced module, the feasible hypothesis filter, relies heavily on accurate entity-type information. This information is crucial for the success of the adaptation process. However, the filtering process becomes ineffective if this information is unavailable or if numerous feasible hypotheses (e.g., 40+) exist for a given relation class and entity type pair. In these scenarios, the “entail” class becomes a minority class in a sea of “neutral” NLI instances, potentially causing the model to collapse to a trivial state of simply predicting “neutral” for every premise-hypothesis pair. Such a scenario would require the design of manually tuned sampling strategies or bespoke learning objectives to handle the overwhelming number of “neutral” premise-hypothesis pairs. We defer the exploration of such challenging settings to future research.

Additionally, in our study, meta-class analysis is performed manually, which introduces an extra layer of human effort. This manual effort involves reading annotation guidelines for a specific dataset to determine which relation classes are mutually exclusive based on their definitions. While this task is relatively quick and straightforward, it does require additional human involvement.

Ethics Statement↩︎

We do not anticipate any major ethical concerns; relation extraction is a fundamental problem in natural language processing. A minor consideration is the potential for introducing certain hidden biases into our results (i.e., performance regressions for some subset of the data despite overall performance gains). However, we did not observe any such issues in our experiments, and indeed these considerations seem low-risk for the specific datasets studied here because they are all published.

8 Appendix↩︎

8.1 Automatic Generation of Hypothesis Templates↩︎

To reduce human effort in our methods, we turn to LLMs, specifically GPT 3.5 [37], to automatically generate hypothesis templates. Some datasets, such as BC5CDR, GAD, and BioRED Novel, feature two classes, making the template generation process relatively trivial. The benefits of automating the generation of hypothesis templates are more significant for datasets such as ReTACRED, which feature 40 relation classes.

We use the following prompt where the ellipsis is replaced with the list of natural language relation classes (e.g., relation classes with underscores removed and spaces inserted) used in each dataset:

A special case arose for the DDI13 dataset where each relation instance describes a relation between two drugs. We referenced the verbalized hypotheses proposed by [12] and included instructions about describing two drug entities:

Table ¿tbl:tab:hypo95templates? contains the generated hypothesis templates for each dataset.

max width=1.0,totalheight=0.90,keepaspectratio

8.2 Baselines↩︎

8.2.1 GPU Resources↩︎

All baselines were trained on a single NVIDIA A100, and training times ranged from 1 to 12 hours, changing based on the size of the dataset and the number of parameters in the model.

8.2.2 Phi-2 and Phi-3↩︎

Since responses from auto-regressive models may sometimes include additional text, all responses are aligned to ground truth labels using partial string matching. We do this by searching for the matches of the first three letters in each NLI label (e.g., “ent” \(\rightarrow\) entail, “con” \(\rightarrow\) contradict, “neu” \(\rightarrow\) neutral). When a class cannot be matched, we assign “none,” which, during evaluation, is equivalent to the NLI label neutral.

For Phi-2, we use the following prompt to fine-tune the model on our task:

Phi-3 uses a similar prompt that differs only in format:

Both Phi-2 and Phi-3 were fine-tuned using the hyperparameters in Table ¿tbl:tab:phi295phi395hp?.

max width=.48

8.2.3 GPT 3.5 and GPT 4↩︎

GPT 3.5 and GPT 4 often perform better on tasks with the help of in-context learning [41], [42]. We construct a prompt that lists the NLI labels and offers four examples of premise-hypothesis pairs expressing each NLI label.

The following is the prompt we used for soliciting predictions for our tests:

For responses from GPT 3.5 and GPT 4, we use the same partial string matching used for Phi-2 and Phi-3 (Appendix 8.2.2) for evaluation.

8.2.4 Hyperparameters for MetaEntail-RE↩︎

Table ¿tbl:tab:meta95entail95re95hyperparams? contains the hyperparameters used to train MetaEntail-RE.

max width=.48

8.3 Task Unification Results↩︎

We explore unifying biomedical relation extraction datasets in hopes of boosting performance on a target dataset. We investigate two task-unification training methodologies: single-stage training and double-stage training. Single-stage training can be viewed as multi-task learning, where the model is trained simultaneously on multiple datasets and tested on a target dataset. Double-stage training can be viewed as an initial pre-training stage on all data except the target dataset, followed by fine-tuning and evaluation on the target dataset.

Unfortunately, we did not observe a significant performance boost across our task-unification experiments (see Table ¿tbl:tab:task95unification?), potentially indicating that these biomedical datasets do not provide complementary information when adapted into an NLI task. Generally, the two-stage training is more effective than the single-stage training, but both fail to realize significant performance gains on the target datasets. We leave investigating other task-unification methods for future works.

max width=1.0

8.4 Meta-class analysis↩︎

We conduct a meta-class analysis for each dataset used in Section 5. We leverage class definitions to determine sets of mutually exclusive classes. The following tables show how meta-class analysis converts original RE labels (row headers) into NLI labels. The \(h(class)\) column headers denote verbalized hypotheses using the corresponding class. For each table, we use the following denote NLI labels:

• 0 \(\rightarrow\) contradict

• 1 \(\rightarrow\) neutral

• 2 \(\rightarrow\) entail

References↩︎