1 Introduction↩︎

Semantic Role Labeling [1] aims to understand the role of words or phrases in a sentence. It has facilitated other natural language processing tasks including question answering [2], sentiment analysis [3], information extraction [4], and machine translation [5].

Semantic role labeling can take various forms, each associated with different datasets. Predicates can be coarsely divided into PropBank [6] senses, each with a core set of numbered semantic arguments (e.g., Arg0–Arg5). There are also modifier arguments (e.g., ArgMLoc) typically representing information such as the location, purpose, manner or time of an event. Alternatively, predicates can also be hierarchically clustered into VerbNet [7] classes according to similarities in their syntactic behavior. Each class admits a set of thematic roles (e.g. Agent, Theme) whose interpretations are consistent with all predicates within the class.

As a modeling problem, SRL requires associating argument types and phrases with respect to an identified predicate. The two labeling tasks (i.e., VerbNet SRL and PropBank SRL) are closely related; but they differ in their treatment of predicates and have disjoint label sets. Learning jointly can improve data efficiency across the different labeling SRL tasks.

A common formulation used to instantiate this idea in prior work is multitask learning [8], [9]: each label set is treated as a separate labeling task, and sometimes also modeled with inter-task feature interaction or consistency losses. While multitask learning often works well in such cases, the loss formulation represents a conservative view over label compatibilities of different tasks. At prediction time, subtask modules still run independently and are not constrained by each other. Consequently, decoded labels may violate structural constraints with respect to each other. In such settings, constrained inference [10], [11] has been found helpful. However, this raises the question of how to involve such inference during learning for better data efficiency. Furthermore, given the wider availability of PropBank-only data [12], [13], how to efficiently benefit from such data also remains a question.

In this paper, we argue that the two disjoint but compatible labeling tasks can be more effectively modeled as one task using their compatibility structures that are already explicitly defined in the form of Semlink [14]. Semlink offers mappings between various semantic ontologies including PropBank and VerbNet. [9] devised a deterministic conversion from PropBank label sequences to VerbNet ones using only the unambiguous mappings in Semlink. This conversion gives a test bed that has half of the predicates in CoNLL05 SRL dataset [15] with both VerbNet and PropBank jointly labeled.

Given this setting, we propose a simple and effective joint CRF model for the VerbNet SRL and PropBank SRL tasks. In addition to the joint CRF, we propose an inference constraint that uses compatible label structures defined in Semlink, and show that our constrained inference achieves higher overall SRL F1—the average of VerbNet and ProbBank F1 scores—than the current state-of-the-art. Indeed, when PropBank labels are observed, it achieves over \(99\) F1 on the VerbNet SRL, suggesting the possibility of an automated annotation helper. We show that our formulation naturally extends to a constrained marginal model that learns from the more abundant PropBank-only data in a semi-supervised setting. When learning and predicting with constraints, it achieves even better SRL F1 in out-of-domain generalization.²

2 Joint Task of Semantic Role Labeling↩︎

We consider modeling VerbNet (VN) and PropBank (PB) SRL as a joint labeling task. Given a sentence \(x\), we want to identify a set of predicates (e.g., verbs), and for each predicate, generate two sequences of labels, one for VerbNet arguments \(y^V\), the other for PropBank arguments \(y^P\). With respect to VN parsing, a predicate is associated with a VerbNet class that represents a group of verbs with shared semantic and syntactic behavior, thereby scoping a set of thematic roles for the class. Similarly, the predicate is associated with a PropBank sense tag that defines a set of PB core arguments along with their modifiers. A example is shown in Tab. 1.

We treat predicate classification and argument labeling as separate tasks and focus on the latter.³ Assuming predicates \(u\) and their associated VN classes \(\eta\) and PB senses \(\sigma\) are given along with \(x\), we can write the prediction problem as: \[\begin{align} (x, u, \eta, \sigma) \rightarrow (y^V, y^P) \label{eq:joint95task} \end{align}\tag{1}\]

2.1 VerbNet Completion↩︎

There is a much larger amount of PropBank-only data [12], [13] than jointly labeled data. Inferring VerbNet labels from observed PropBank labels, therefore, is a realistic use case. This corresponds to the modeling problem: \[\begin{align} (x, u, \eta, \sigma, y^P) \rightarrow y^V \label{eq:completion95task} \end{align}\tag{2}\] We refer to this scenario as completion mode. In this paper, we will focus on the joint task defined in Eq. 1 while also generalizing our approach to address the completion task in Eq. 2 .

2.2 Multitask learning and Its Limitations↩︎

When predicting multiple label sequences for SRL, a common approach is multitask learning using dedicated classifiers for each task that operate on a shared representation. The current state-of-the-art model [9] used an LSTM stacked on top of BERT [17] to model both PropBank and VerbNet. While each set of the semantic roles is modeled jointly with VerbNet predicates, the argument labeling of the two subtasks is still kept separate.

Separate modeling of VerbNet SRL and PropBank SRL has a clear disadvantage: subtask argument labels might disagree in three ways:

In Sec. 6, we show that a model with separate task classifiers, while having a close to state-of-the-art F1, can have a fair amount of argument assignment errors with respect to Semlink, especially for out-of-domain inputs.

3 A Joint CRF Model↩︎

To eliminate the errors discussed in Sec. 2.2, we propose to model the disjoint SRL tasks using a joint set of labels. This involves converting multitask modeling into a single sequence labeling task whose labels are pairs of PB and VN labels. Doing so not only eliminates the BIO inconsistency, but also exposes an interface for injecting Semlink constraints.

Our model uses RoBERTa [18] as the backbone to handle textual encoding, similarly to the SRL model of [19]. At a high level, we use a stack of linear layers with GELU activations [20] to encode tokens to be classified for a predicate. For the problem of predicting arguments of a predicate \(u\), we have an encoding vector \(\phi_{u, i}\) for the \(i\)-th word in the input text \(x\). \[\begin{align} e &= \texttt{map}(\mathrm{\small RoBERTa}(x)) \\ \phi_{u} &= \left\{f_{ua}\left([f_u(e_u), f_a(e_i)]\right), \forall_i\in x\right\} \label{eq:fua} \end{align}\tag{3}\] Here, \(\texttt{map}\) sums up word-piece embeddings to form a sequence of word-level embeddings, the functions \(f_u\) and \(f_a\) are both linear layers, and \(f_{ua}\) denotes a two-layer network with GELU activations in the hidden layer. We use a dedicated module of the form in Eq. 3 for the VN and PB subtasks. This gives us a sequence of vectors \(\phi_{u}^v\) for VN and a sequence of vectors \(\phi_{u}^p\) for PB.

Next, we project the VN and PB feature sequences into a \(|Y^V|\times|Y^P|\) label space: \[\begin{align} z_u &= \{g([\phi_{u, i}^V, \phi_{u, i}^P]), \forall_i\in x\} \label{eq:zvp} \end{align}\tag{4}\] Here, \(g\) is another two-layer GELU network followed by a linear projection that outputs \(|Y^V|\times|Y^P|\) scores, corresponding to VN-PB label pairs. The final result \(z_u\) denotes a sequence of VN-PB label scores for a specific predicate \(u\). In addition, we use a CRF as a standard first-order sequence model over \(z_u\) (treating it as emission scores), and use Viterbi decoding for inference. The training objective is to maximize: \[\begin{align} \log P(y^{VP}|x) &= s(y^{VP}, x) - \log Z(x) \label{eq:crf} \end{align}\tag{5}\] where \(s(\cdot)\) denotes the scoring function for a label sequence that adds up the emission and the transition scores, and the term \(Z(x)\) denotes the partition that sums exponentiated scores over all label sequences. The term \(y^{VP}\) denotes the label sequence that has VN and PB jointly labeled. We will refer to this model as the joint CRF, and the label sequence as the joint labels.

3.0.0.1 Reduced Joint Label Space.

We use the cross-product the two label sets, prefixed with a BIO prefix⁴. A brute-force cross product leads to a \(|Y^V|\times|Y^P|\) label space. In practice, it is important to keep the joint label space at a small scale for efficient computation, especially for the CRF module. Therefore, we condense it by first disallowing pairs of the form (B-*, I-*) and predicate-to-argument pairs. The former enforces that the VerbNet arguments do not start within ProbBank arguments, while the latter ensures that the predicate is not part of any argument. Next, we observe the co-occurence pattern of VN and PB arguments, disabling semantically invalid pairs such as (Theme, ArgM-Loc)⁵. This reduces the label space by an order of magnitude (from \(144\times 105=15,120\) to \(685\)).

3.0.0.2 Input Construction using Predicates.

We take inspiration from prior work [21]–[23] to explicitly put predicate features as part of the input to augment textual information. At the same time, we also seek to maintain a simple construction that can be easily adapted to a semi-supervised setting (i.e. compatible with PropBank-only data). To this end, we propose a simple solution that appends the PropBank senses of potential predicates to the original sentence \(x\): \[\begin{align} x_{\mathrm{\small WP}} &= [ \mathrm{\small Cls} \text{ } w_{1:T} \text{ } \mathrm{\small Sep} \text{ } \sigma_{1:N} \text{ } \mathrm{\small Sep}] \end{align}\] where \(w_{1:T}\) denotes the input words, and \(\sigma_{1:N}\) denotes the senses of the \(N\) predicates. In practice, we use the PropBank roleset IDs which consist of a pair of (lemma, sense)—e.g., run.01. Our models only take the encodings for \(w_{1:T}\) after the RoBERTa encoder and ignore the rest. We consider this design to be more efficient than prior work [9], [23] that dedicated text feature for each predicate. In our setup, the argument labeling for different predicates shares the same input, thus no need to run encoding multiple times for multiple predicates.

4 Semi-supervised Learning with PropBank-only Data↩︎

Compared to data with both VerbNet and PropBank fully annotated, there is more data with only PropBank labeled. The Semlink corpus helps in unambiguously mapping \(\sim56\%\) of the CoNLL05 data [9]. Therefore, a natural question is: can we use PropBank-only data to improve the joint task?

Here, we explore model variants based on the joint CRF architecture described in Sec. 3. We will focus on modeling for PropBank-only sentences.

4.1 Separate Classifiers for VN and PB↩︎

As a first baseline, we treat VN and PB as two separate label sequences during training. This is essentially a multitask setup where VN and PB targets use separate classifiers. We let these two classifiers share the same RoBERTa encoder, and have their own learnable weights for Eq. 3 5 .

4.2 Dedicated PropBank Classifier↩︎

Another option is to retain the joint CRF for the jointly labeled dataset and use an additional dedicated CRF for PB-only sentences. Note that this setup is different from the model in Sec. 4.1. As before, we let these two to share the same encoder and, for Eq. 3 5 , they have dedicated trainable weights.

During inference, we rely on the Viterbi decoding associated with the joint CRF module to make predictions. In our preliminary experiments, the joint CRF and the dedicated PropBank CRF achieve similar F1 on PropBank arguments.

4.3 Marginal CRF↩︎

For partially labeled sequences, we take inspiration from [11] to maximize the marginal distribution of those observed labels. In our joint CRF, the marginalization assumes uniform distribution over VN arguments that are paired with observed PB arguments. The learning objective is to maximize the probabilities of such label sequences as a whole: \[\begin{align} & \mathrm{\small Lse}_{y\in y_u^P} \left(s(y)\right) - \log Z(x) \label{eq:marginal} \\ & \quad\text{where } \mathrm{\small Lse}_y\left(\cdot\right) = \log \sum\limits_y \exp(\cdot)\nonumber \end{align}\tag{6}\] where \(y\in y_u^P\) denotes a potential joint label sequence with only PropBank arguments observed for a predicate \(u\). Scores of such label sequences are aggregated by the Lse operator. Note that the marginal CRF and the joint CRF (Eq. 5 ) use the same model architecture, just with a different loss.

4.4 Marginal Model with Semlink(Marginal\(_\mathrm{\small seml}\))↩︎

The log marginal probability in Eq. 6 assumes uniform distribution over a large label space. It included any arbitrary VerbNet arguments paired to the observed PropBank labels. In practice, we can narrow it down to only legitimate VN-PB argument pairs defined in Semlink. Such legitimate space is uniquely determined by a VerbNet class \(\eta\) and PropBank sense \(\sigma\). We will refer to label sequences that comply with this space as \(y_u^{\mathrm{\small seml}}\), and apply it on Eq. 6 : \[\begin{align} \mathrm{\small Lse}_{y\in y_u^P\cap y_u^{\mathrm{\small seml}} }s(y) - \mathrm{\small Lse}_{y\in y_u^{\mathrm{\small seml}}}s(y) \label{eq:marginal95semlink} \end{align}\tag{7}\] Note that this formulation essentially changes the global optimization into a local version which implicitly requires using Semlink at inference time. We will present the details of \(y_u^\mathrm{\small seml}\) in Sec. 5. Intuitively, it zeros out losses associated with joint label traces that violate Semlink constraints. During training, we found that it is important to apply this constraint to both B-* and I-* labels.

4.4.0.1 Where to apply \(y_u^{\mathrm{\small seml}}\)?

Technically, the summation over reduced label space can be applied at different places, such as the partition \(Z\) in Eq. 5 . We will report performances on this setting in Sec. 7.2. In short, plugging the label filter \(y_u^{\mathrm{\small seml}}\) to the joint CRF (therefore jointly labeled data) has little impact on F1 scores, thus we reserve it for the PropBank-only data (as in Eq. 7 ).

5 Inference with Semlink↩︎

Here we discuss the implementation of the \(y_u^{\mathrm{\small seml}}\). Remember that each pair of VerbNet class \(\eta\) and PropBank sense \(\sigma\) uniquely determines a set of joint argument labels for the predicate \(u\). For brevity, let us denote this set as \(\mathrm{\small seml}(u)\) (e.g., Tab. 1). Eventually, we want the Viterbi-decoded label sequence to comply with \(\mathrm{\small seml}(u)\). That is, \[\begin{align} \forall (l^V, l^P) \in & \mathrm{\small seml}(u) \rightarrow \nonumber \\ \forall i, & \Big[ \big( \forall_{l \notin \mathrm{\small seml}(u)} (y_i^V=l^V, l) \notin y_u^{VP} \big) \nonumber \\ & \wedge \big( \forall_{l \notin \mathrm{\small seml}(u)} (l, y_i^P=l^P) \notin y_u^{VP} \big) \Big] \label{eq:semlink95constraint} \end{align}\tag{8}\] where \(i\) denotes the location in a sentence, \(y_u^{VP}\) is the joint label sequence, consisting of \((y_i^V, y_i^P)\) pairs. The constraint in Eq. 8 translates as: if a VerbNet argument is present in the predicate \(u\)’s Semlink entry, we prevent it from aligning to any PropBank arguments not defined in Semlink; and the same applies to PropBank arguments.

This constraint can be easily implemented by a masking operation on the emission scores of the joint CRF, thus can be used at both training and inference time. During inference, it effectively ignores those label sequences with Semlink violations during Viterbi decoding: \[\begin{align} y^{VP}_u &= \arg\max_{y\in y_u^{\mathrm{\small seml}}} s(y) \label{eq:viterbi95semlink} \end{align}\tag{9}\]

In Sec. 6, we will show that using Eq. 9 always improves the overall SRL F1 scores.

5.0.0.1 VerbNet Label Completion.

For models based on our joint CRF, we mask out joint labels that are not defined in \(y^P\) during inference, similar to Eq. 7 . For models with separate VN and PB classifiers (in Sec. 4.1), we enforce the VN’s Viterbi decoding to only search arguments that are compatible with the gold \(y^P\) in Semlink. Furthermore, we always use the constraint (Eq. 8 ) in the completion mode.

6 Experiments↩︎

In this section, we aim to verify whether the compatibility structure between VerbNet and PropBank (in the form of Semlink) has a positive impact on their sequence labeling performances.

6.1 Data↩︎

We follow the prior state-of-the-art [9] in extracting VerbNet labels from the CoNLL05 dataset using the Semlink corpus. We use the same version of Semlink to extract the data for training and evaluation. Therefore, our F1 scores are directly comparable with theirs (denoted as Iwcs2021 in Table [tab:results95wp]) The resulting dataset accounts for about \(56\%\) of the CoNLL05 predicates, across training, development and test sets (including WSJ and Brown). We will refer to this data as the joint data column in Table [tab:results95wp]. For semi-supervised learning, we incorporate the rest of the PropBank-only predicates in the CoNLL05 training split. For development and testing, we use the splits in the joint dataset for fair comparison with prior work.

6.2 Training and Evaluation↩︎

We adopt the same fine-tuning strategy as in [19]—we fine-tune twice since this generally outperforms fine-tuning only once, even with the same number of total epochs. In the first round, we fine-tune our model for \(20\) epochs. In the second round, we restart the optimizer and learning rate scheduler, and fine-tune for \(5\) epochs. In both rounds, checkpoints with the highest average VN/PB development F1 are saved. For different model variants, we report the average F1 from models trained with \(3\) random seeds.

For the Semlink constraint, we use the official mapping between VN/PB arguments⁶. When involved in constrained inference, we use the gold VerbNet classes \(\eta\) and PropBank senses \(\sigma\).

For evaluation, in addition to the standard VN and PB F1 scores, we also report the percent of predicates with predictions that are inconsistent with Semlink, denoted by \(\rho\).

6.3 Performance on Semlink Extracted CoNLL05↩︎

We want to compare models trained on the joint dataset and variants (in Sec. 4) on the semi-supervised setup. Table [tab:results95wp] presents their performances along with Semlink violation rates in model predictions. Note that the ground truth joint data bears no violation at all (i.e., \(\rho=0\)).

6.3.0.1 Multitask involves Semlink violations.

Firstly, we show the limitations of multitask learning. While the architecture is simple, the testing scores mostly outperform the previous state-of-the-art, except the Brown PropBank F1. However, there is a fair percentage of predicates having structurally wrong predictions, especially in the Brown test set. With semi-supervised learning, VN and PB F1s are improved on the Brown set while slightly lowered on the WSJ VN. This also comes with a degraded Semlink error rate (\(3.43 \rightarrow 4.08\) on WSJ and \(8.71 \rightarrow 10.48\) on Brown). While Semlink inconsistency is not reported in [9], we believe that, due to the nature of multitask learning, Semlink errors are inevitable.

6.3.0.2 Joint CRF outperforms multitask learning.

A direct comparison is between the Multitask v.s. Joint. Our joint CRF obtains higher overall SRL F1 across the WSJ and the Brown sets. A similar observation applies to the semi-supervised setting where Multitask compares to Joint+CRF\(_\mathrm{\small pb}\). Most of such improvements are from the use of inference-time Semlink constraints.

6.3.0.3 Inference with Semlink improves SRL.

In Table [tab:semlink95inf], we do side-to-side comparison of using versus not using the Semlink structure during inference. We do so for each modeling variant. With constrained inference, models no longer have Semlink structural violations (\(\rho=0\)). And this results in a clear trend where using Semlink systematically improves the F1 scores. We hypothesize this is due to the reduced search space which makes the decoding easier. Likely due to the higher granularity of VerbNet argument types compared to PropBank, a majority of the improvements are on the VN F1s.

6.3.0.4 Does semi-supervised learning make a difference?

The answer is that it depends. For Multitask, using PropBank-only data traded off a bit on the overall WSJ F1 but improved the out-of-domain performances. Accompanied with this trade-off is the slightly higher inconsistency rate \(\rho\). The Joint+CRF\(_{PB}\) model tells an opposite story that the partially labeled data is favorable on the in-domain test but not so in the out-of-domain test. This observation is also consistent with both the Marginal CRF and constrained Marginal model (Marginal\(_\mathrm{\small Seml}\)). Furthermore, when performance improves, the margins on VN and PB are fairly distributed. Finally, we should note that, neither the Joint+CRF\(_\mathrm{\small pb}\) nor the Marginal have a better Brown F1 than the Joint, meaning that they did not use the PB-only data efficiently.

6.3.0.5 Impact of marginal CRF.

We compare the Joint+CRF\(_\mathrm{\small pb}\) to Marginal to see how a single CRF handles partially labeled data. The latter outperforms the former consistently by \(0.2\) on the in-domain test set but performed slightly worse on the out-of-domain Brown set. Comparing to the Joint model, it seems that naively applying marginal CRF leads to even worse generalization.

6.3.0.6 Constrained marginal model improves generalization.

We want to see if our constrained model can help learning. Modeling PropBank-only data with a separate classifier (i.e., the Multitask and Joint+CRF\(_\mathrm{\small pb}\)) failed to do so (although they indeed work better on the in-domain WSJ). In contrast, our constrained Marginal\(_\mathrm{\small Seml}\) apparently learns from the PropBank-only data more efficiently, achieving strong in-domain performance and substantially better out-of-domain generalization. This suggests that learning with constraints works better with partially labeled data. Interestingly though, it seems that the constraint is optional for fully annotated data since the Marginal\(_\mathrm{\small Seml}\) only enables the constraint on PB-only data. We verify this phenomenon in Sec. 7.2 with an ablation study.

6.3.0.7 Statistical Significance of Constrained Inference.

We measure statistical significance using a t-test implemented by [24] on predictions from models in Table [tab:semlink95inf]. For each model, we compare inference with and without Semlink constraints (Sec. 5). For a fair comparison, we limit predictions from the model trained with the same random seed in each test, and apply the test for all random seeds (3 in total). We observe that the improvements on VerbNet F1’s are universally significant. The p-values are far less than \(0.01\) across different testing data, models, and random seeds. This aligns to the observation in Table [tab:semlink95inf] that VN F1 has a substantial F1 boost while PB F1 improvements tend to be marginal.

To look closer, we examined the predictions of a Joint model on the Dev set (\(1,794\) predicates), and found that, after using Semlink during inference, \(51\) wrongly predicted predicates in VN SRL were corrected (i.e., improved predicate-wise F1), and no predicates received a degraded F1. However, for PropBank SRL, there were \(12\) predicates corrected by the constraint while \(6\) became errors.

6.4 VerbNet Label Completion from PropBank↩︎

As discussed in Sec. 1, we also aim to address the realistic use case of VerbNet completion. Table 2 summarizes the performances for VerbNet argument prediction when gold PropBank arguments are given. In the completion mode, the Joint model performs generally better than all the semi-supervised models. This phenomenon is likely because the Joint model is optimized for the probability \(P(y^V, y^P \mid x)\), while the semi-supervised models, in one way or the other, have a term for \(\sum_{y^V} P(y^V, y^P \mid x)\) on PB-only data. The latter term does not explicitly boost model’s discriminative capability on the unique ground truth.

In addition to the \(x_\mathrm{\small wp}\) input construction in Sec. 3.0.0.2, we propose a special construction \(x_{\mathrm{\small comp}}\) for the VN completion mode by using the PB arguments as text input.: \[\begin{align} x_{\mathrm{\small comp}} &= [ \mathrm{\small Cls} \text{ } w_{1:T} \text{ } \mathrm{\small Sep} \text{ y}_1^P... \text{ } \sigma_{v} \text{ y}_{v+1}^P... \text{ } \mathrm{\small Sep}] \label{eq:x95comp} \end{align}\tag{10}\] where \(y_i^P\) denotes the PropBank argument label for the \(i\)-th word. For the predicate word, we use the predicate feature (i.e. lemma and sense). Compared to \(x_\mathrm{\small wp}\), this formulation makes the computation less efficient as the input \(x_{\mathrm{\small comp}}\) is no longer shared across different predicates. However, it offers a more tailored input signal and delivers above \(99\) F1 on both WSJ and Brown.

7 Analysis↩︎

We report statistical metrics in Sec. 7.1. In Sec. 7.2, we analyze the use of constrained learning on the jointly labeled data.

7.1 Variance of SRL Models↩︎

A majority of F1s in Tab. [tab:results95wp] vary in a small range. Models trained on joint-only data show higher variance on the out-of-domain Brown test set. Among the semi-supervised models, the Marginal models exhibit high F1 variance on the Brown set while the Marginal\(_\mathrm{\small Seml}\) models work more stably.

7.2 Impact of Learning with Semlink Constraint on Joint Data↩︎

In Table [tab:semlink95config], we use the form of constrained learning in Eq. 7 but apply it on the joint CRF loss over the jointly labeled training data. Note that the constraint term only affects the denominator part in Eq. 5 . Overall, the effect of Semlink at training time seems small. On the WSJ test set, both VN and PB F1s are fairly close. The Brown test F1s have a drop, especially on VN, suggesting that constrained learning on the joint data is not needed.

7.3 Reliance on Gold Predicate Labels↩︎

While we focused on experiments that use given gold predicate labels (i.e., VN class \(\eta\) and PB sense \(\sigma\)), our models do not rely on them. Intuitively, there will be a performance degradation on argument labeling when predicate labels are not always accurate. To study this degradation curve, we experiment with randomly corrupted predicate classes/senses, to illustrate the performance dependency on predicate disambiguation. Specifically, we perturb the development split by randomly swapping a predicate’s VN class (or PB sense) with another one from the dataset and observe how well our best model performs. This setup essentially simulates a real-world testing scenario where predicate labels on either VN/PB are imperfect. Results in Tab. 3 suggest that F1 degradation happens in a smooth way due to error propagation.

8 Discussion↩︎

The use of constraints in SRL has a long history that mostly focuses on the PropBank SRL task. Earlier work investigated inference with constraints [25]–[27]. Other work developed constrained models for learning [28] ; or incorporated constraints with emerging neural models [10], [16], [19], [29], [30].

VerbNet SRL, on the other hand, is often studied as a comparison or a helper for PropBank SRL [31]. [32] showed that the mapping between VerbNet and PropBank can be used to disambiguate PropBank labels. It has been shown that model performance on VerbNet SRL is affected more by predicate features than PropBank SRL [33]. In a sense, our observation that VerbNet F1 gains larger improvements from Semlink is also consistent with prior work. Beyond comparison work, [34] explored the downstream impact of VerbNet SRL and showed promising uses in entity state tracking.

8.0.0.1 Multitask Learning

The closest work to this paper is [9]. Instead of modeling argument labels and predicate classes via multitasking, we adopted a simpler design and focused on joint SRL argument labeling. This comes with two benefits: 1) a focused design that models joint labels; 2) an easy extension for using marginal CRF for partially labeled data. Other technical differences include a generally better transformer (i.e., RoBERTa) instead of BERT [35], simpler input construction, and our proposal of the completion mode.

8.0.0.2 Marginal CRF

[11] explored the use of marginal CRF on disjoint label sequences in the biomedical domain. Disjoint label sequences are concatenated into one, thus requiring dedicated decoding to reduce inconsistency w.r.t. various structure patterns (e.g., aligned BIO pattern). In this paper, we took a step further by pairing label sequences to form a joint SRL task, allowing an easy interface for injecting decoding constraints.

8.0.0.3 Broader Impact

Recent advantages in large language models (LLM) have shown promising performance in semantic parsing tasks [36]–[38]. A well-established approach is via iterative prompting (e.g., Chain-of-Thought [39]) and potentially using in-domain examples for prompt construction. While such work bears many technical differences from this work, there are advantages that can potentially be shared. For instance, our direct use of a knowledge base (Semlink in our case) allowed for a guarantee of \(0\) violations; and LLM-based work is less reliant on training data. Another scenario is when treating semantic structures as explicit intermediate products, such as for language generation. Our joint modeling allows for \(\geq99\%\) accuracy in converting PropBank arguments to VN arguments. When using such labels for prompted inference, it can make fewer errors.

8.0.0.4 Conclusions

In this work, we presented a model that learns from compatible label sequences for the SRL task. The proposal includes a joint CRF design, extension for learning from partially labeled data, and reasoning and learning with Semlink constraints. On the VerbNet and PropBank benchmark based on CoNLL05, our models achieved state-of-the-art performance with especially strong out-of-domain generalization. For the newly proposed task of completing VerbNet arguments given PropBank labels, our models are near perfect, achieving over \(99\) F1 scores.

9 Limitations↩︎

9.0.0.1 Towards fully end-to-end parser.

Our model architecture is on the track of end-to-end SRL parser, but it still assumes gold predicate positions and predicate attributes are given. A fully end-to-end parser can take sentences in raw text and output disjoint label sequences. While doing so can make computation less efficient (e.g., requiring substantially larger memory for training), it can bring users convenience.

9.0.0.2 Involving document context.

[9] showed that using neighboring sentence prediction with transformer positively impacts parsing F1. In contrast, we assumed sentences in the corpus are independent.

9.0.0.3 Why does PropBank seem more difficult?

We hypothesized the reason to be less granularity in argument labels and more ambiguous label assignments. As mentioned in Sec. 3, prior work benefited from using dedicated label text/definition as an auxiliary input. We only used such features at the predicate level, implicitly trading off potential gains on PB F1 for more efficient computation.

9.0.0.4 Marginal model’s capacity at handling constraints.

In this paper, we focused on the Semlink constraint for compatible label sequences. There is a broad spectrum of SRL constraints in prior work [27], some of them do not easily fit in the marginalization formulation, such as the unique core role constraint.

Acknowledgement↩︎

We thank the reviewers for their helpful insights and comments. We gratefully acknowledge the support of NSF under grants #1801446 (SATC) and #1822877 (Cyberlearning), and of DARPA CwC (subcontracts from UIUC and SIFT), DARPA AIDA Award FA8750-18-2-0016 (RAMFIS), DARPA KAIROS FA8750-19-2-1004-A20-0047-S005 (RESIN, sub to RPI, PI: Heng Ji), as well as DTRA HDTRA1-16-1-0002/Project 1553695 (eTASC - Empirical Evidence for a Theoretical Approach to Semantic Components).

The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies of any government agency.

10 Appendix↩︎

10.1 Hyperparameters↩︎

As discussed in Sec. 6, we adopted a 2-stage fine-tuning for each of our models. In the first round, we fine-tune for \(20\) epochs with initial learning rate of \(3\times 10^{-5}\). In the second round, we fine-tune for another \(5\) epochs with fresh start on the optimizer. Across the two stages, we applied a dropout layer with rate \(0.5\) (which is preliminarily grid-searched from \([0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8]\)) before each of the linear layers in Sec. 3. The RoBERTa transformer uses the built-in default configurations which is implemented by [40].

The learning rate (\(\lambda\)) in the second round of fine-tuning varies by model. We preliminarily grid-searched from \([3\times 10^{-6}, 1\times 10^{-5}, 3\times 10^{-5}, 1\times 10^{-4}]\). Specifically, for each learning rate, we trained \(3\) random models and chose the one with the highest overall SRL F1 on the Dev set. We put their values in Table 4.

References↩︎