1 Introduction↩︎

The performance of natural language processing systems is dependent on the amount of training data, which is often scarce. To complement existing training data, supplementary data sources can be used. Especially data annotated for the same task from other sources can be beneficial to exploit. However, because of heterogeneity in language or domain this might lead to sub-optimal performance. In early work on combining training sources, data was selected at training time [1], [2] for a given test set. A more nuanced way to exploit heterogeneous data is to encode properties of the language as features [3].

Recently,   showed that encoding the language of an instance as an embedding in a neural model is beneficial for multi-lingual learning.² Follow-up work found that multiple datasets within the same language can also be combined by encoding their origin [5], [6], thereby implicitly learning useful commonalities, while still encoding dataset-specific knowledge. These dataset embeddings are employed in groups of datasets which usually range in size from 2 to 10 datasets. However, it remains unclear in which situations these dataset embeddings thrive best.

Furthermore, two often overseen issues with dataset embeddings are that they are commonly learned from the gold data-source labels attached to each training and test instance and it is assumed that the test data is from a distribution which is seen during training. In many real world situations these assumptions are clearly violated. A common strategy when the test data is drawn from a different distribution as the training datasets (zero-shot), is to use a manually assigned proxy treebank [7]–[9]. Recent work showed that for unseen datasets in mono-lingual setups [10], interpolated dataset embeddings can be used to improve performance for zero-shot settings. We use automatically predicted proxy data sources instead, and focus on mono-linugal as well as cross-lingual setups.

In this paper, we provide an extensive evaluation of the usefulness of dataset embeddings in existing setups and beyond. More concretely, we ask: 1) What are good indicators to predict the usefulness of dataset embeddings? 2) Can we effectively use dataset embeddings in the absence of gold data-source information?

2 Dataset Embeddings↩︎

Dataset embeddings enable conditioning of inputs on some property of the data when training on multiple sources. They are vector representations learned during model training, with the aim to capture distinctive properties of the sources into a continuous vector, without losing their heterogeneous characteristics. Given \(D\) data sources, technically, we learn a vector representation \(e(d)\) for each data source \(d \in D\) while training a single model from a group of sources. Every input instance marked with its dataset source \(d\). For each word \(w_i\) with \(i=0, ..., n\), the word embedding \(e(w_i)\) is concatenated with the dataset embedding \(e(d)\), and both are updated during training. Figure 1 shows the overall architecture of the model which employs a contextual encoder that uses the resulting embedding as input and outputs \(e(c_i)\) to be used for prediction.

2.1 Experimental Setup↩︎

In this work, we copy the exact setups from the UUParser 2.3 [7] and the Multi-Team tagger [6] because they were high ranking systems in two recent shared tasks [11], [12] and they both showed large gains by using dataset embeddings. The UUParser is an Arc-Hybrid  [13] BiLSTM [14] dependency parser, which exploits a dynamic oracle [15] and supports non-projective parsing through the use of a swap action [16]. The Multi-Team tagger performs morphological tagging [17] and lemmatization jointly; to this end, they use a shared BiLSTM encoder and feed the output of the tagging as input for the lemmatization, which is predicted as a sequence of characters. For efficiency reasons and simplicity, we disabled the use of external embeddings³ as well as POS embeddings for the UUParser.

It should be noted that besides the differences in the models and tasks, the setups also differ among several aspects; the version of UD data (2.3, 2.2) [18], type of dataset splitting used (Multi-Team always has train-dev-test), and most interestingly, the dataset grouping strategies.   manually designed dataset groups based on typological information, language-relatedness and empirical evidence;  instead propose pairs:every dataset is matched with one other dataset based on word overlap. For both of the models, we copy the exact language grouping as in the original papers⁴. For comparison of different grouping strategies, we refer to .

3.5pt

2.2 Data Source Prediction↩︎

In this work, we predict data source on the sentence level, because it matches the language switches at test-time and it improves the accuracy of the classification.⁵ We use a linear support vector classifier based on word and character n-grams (without tokenization). We use this approach here because of simplicity, efficiency and they have shown to reach competitive performance for text classification tasks [19]–[22]. We performed a grid search with \(n \in [1-7]\) and all sequential combinations (1-2, 1-3, etc.) for \(n\)-grams. For this hyper-parameter tuning, we used the eight datasets from , and found the most robust parameters to be 1-2 for words and 1-5 for characters. The obtained macro average F1 on all data pairs from  is 95.42, and on all data groups from  is 91.76. The performance difference can be explained by the number of datasets per group, which in the former setup is always two. To match the setup during testing, we obtain predicted dataset identifiers for the training data with 5-fold jack-knifing, and use these during training.

3 Results↩︎

We report results for all tasks in two main settings: in-dataset, for setups where we assume that input data is from a distribution present during training (3.1); and zero-shot, (3.2), a setup where this is not the case. For all reported experiments, we use Labeled Attachment Score (LAS) for parsing [11], F1 score for morphological tagging, and accuracy for lemmatization. We do not perform any tuning, and thus only report results on development data (if no dev-split is available we use test). As a control, we compare dataset embeddings to a simple Concat, training on concatenation of all the data sources from a dataset group without dataset embeddings. Reported results are average over 3 runs for the UUparser, for the Multi-Team tagger we did only a single run because of the computational costs (see Appendix for more details).

3.1 In-dataset Evaluation↩︎

The average results over all datasets are shown in Table 1, as well as the results for mono-lingual and multi-lingual dataset groups (the full results can be found in the appendix). These are the takeaways:

3.1.0.1 Gold

Overall, gold dataset embeddings provide substantial gains (Table 1: all). They outperform both base and concat on all 3 tasks, which confirms previous findings [6], [7]. Gains are largest for dependency parsing, followed by lemmatization and finally morphological tagging, where the increase is only 0.71.

3.1.0.2 Dataset group composition

Comparing the mono-lingual dataset groups with the multi-lingual groups, we can see that gold dataset embeddings improve results in both settings for 2/3 tasks. The only setup where gold dataset embeddings are not beneficial is for morphological tagging in multi-lingual groups, where the gains for lemmatization are also only marginal (+0.63 abs.compared to base). This may be attributed to the nature of the tasks, morphological tagging and lemmatization are more language-specific, making it difficult to transfer relevant information from another language. In Figure 2, we plot the performance increase from base to using gold dataset ID’s in relation to its dataset size. Unsurprisingly, the largest gains are obtained in smaller datasets (\(<\)​50,000 words) for all tasks. However, especially for the morphological tasks, the largest drops are also observed in this range, and mainly happen for low-resource languages which are paired with a distant language (e.g. Akkadian (akk_pisandub) and Irish (ga_idt)).

3.1.0.3 Gold vs Predicted

The pred columns in Table 1 shows that dataset embeddings are only beneficial (i.e. outperforming base) for lemmatization and dependency parsing when we do not have access to the gold dataset ids but use predicted ids instead. For lemmatization, the average increase compared to base is only 0.31 Acc., whereas for dependency parsing, it is +1.60 LAS. Pred is mostly beneficial in multi-lingual dataset groups, which is probably because the performance of the dataset classifier (Section 2.2) is higher in these cases.

3.2 Zero-shot evaluation↩︎

In many real-world situations, the basic assumption that data instances originate from the training data distribution is violated, and it becomes essential to find a good way of using data from other sources, like finding the best proxy source. To test whether dataset embeddings are still useful in a zero-shot setup, we perform experiments where we hold out the target dataset during training and then classify all development sentences into the other sets. We run this experiment only for dataset groups containing more than 2 datasets (11 groups and 53 datasets, for dependency parsing). As baseline we compare to a model trained on the concatenation of all other datasets from the group. This zero-shot setup is challenging. Results are expected to be overall lower.

Detailed results are reported in Appendix 7–Table 2 summarizes the main results.⁶ We aggregate over datasets for which another in-language dataset is available within its dataset group (“\(\exists\) same-lang”), and those where this is not the case. Overall, the performance increase has almost vanished, having only an 0.07 absolute increase in LAS (‘all’). This increase is void in cases no in-language data exists in the group. Only for datasets for which a same language dataset exists (\(\exists\) same-lang, that is, a treebank exists for the language but it comes from another distribution/domain), slight improvements are obtained. We conclude that dataset embeddings are not useful in setups when the test instances are from another distribution.

3.5pt

For demonstration purposes, we highlight the full results of two dataset groups in Table 3. The first (multi-lingual) group shows that dataset-embeddings are mainly beneficial for languages with multiple datasets, both in the in-domain and zero-shot setting. In this particular dataset group, prediction of the embeddings performs on-par with the gold labels, probably because of the high performance of the classifier. In contrast, in the second (mono-lingual) group, the classifier scores lower, and pred prediction performances are lower compared to gold. For this group, predicted dataset embeddings are outperformed by a simple dataset concatenation.

4 Conclusion↩︎

We provide an extensive evaluation of dataset embeddings in two large-scale settings where they were used successfully [6], [7]. In setups where in-distribution training data is available, we found dataset embeddings more useful in monolingual dataset groups, compared to cross-lingual ones. In general, performance gains were the largest for 1) datasets for which another same-language datasets was available during training 2) small datasets 3) datasets which were part of a large dataset group. However, with predicted id’s, their benefit is limited, contrary to gold information. When moving to zero-shot setups, the performance increases become negligible (except for some particular datasets). In particular, without in-source training data, dataset embeddings work in some cases when another treebank for the language exists; but this gain is not consistent and often small. Overall, we find dataset embeddings fail to be a viable adaptation method when no in-source data is available. Hence, in many realistic out-of-distribution setups, their benefit vanishes.

Acknowledgements↩︎

We would like to thank Gertjan van Noord for feedback on an earlier version of this paper. We thank NVIDIA, the HPC cluster at the ITU and the University of Groningen for computing resources. This research was supported by an Amazon Research Award, an STSM in the Multi3Generation COST action (CA18231), and grant 9063-00077B (Danmarks Frie Forskningsfond).

5 Reproducability report↩︎

The UUParser was run on two E5-2660 v3’s (40 threads total, we used only 30), and took on average approximately 20 hours on a single thread per model. For three random seeds, 16 dataset groups, and approximately 5 settings (4 from Section 3.1 +1 from Section 3.2 were we only used half of the groups in two settings), the total number of models is 240. So the total computation walltime was 160 hours (approximately a week).

The Multi-Team tagger was run on two Tesla V100’s (one model per V100, so two models were trained in parallel). On average this took approximately 3 hours. For this setup we had 104 dataset pairs, of which 82 were unique (if the pairs consist of the same two languages, we only trained one model). For the Multi-Team tagger, we ran four setups (Section 3.1), totaling to 328 models. The total computation time was 984 hours, which divided by 2 gpus resulted in a walltime of 492 hours (approximately three weeks).

Regarding the differences in the settings (base, concat, gold and pred), there were no clear trends in differences in run-time, even the base settings (where multiple models where trained for 1 dataset group) was equal in runtime compared to the other settings where one large model was trained.

It should be noted that the UUParser uses a maximum of 15,000 sentences per epoch, and the Multi-Team tagger 500,000 words (default settings), which makes training times substantially shorter (especially for the UUParser), and reduced the memory usage. Excluding external embeddings helped us reduce the running time and memory usage even further. For the UUParser, a maximum of 8GB ram is enough for training a single model ( 4GB on average), and the Multi-Team tagger requires a minimum of 8GB of GPU RAM.

For all the other settings and hyperparameters, we exactly replicated the original code from the   and  , and thus refer to their papers for experimental details. The only adaptation we made to the systems is that for the UUParser we added support for supplying the dataset information in the connlu misc-column (the adapted version is available in our repo).

6 Aggregates over all results↩︎

For easier analysis, we provide average scores over aggregates of datasets. To this end, we propose to use data-filters, and report average scores over specific subsets of the data. The results are shown in Table 4. The filters show aggregates over a) whether the training portion of the dataset is small (\(<\) 30,000 words) or large b) whether the dataset group to which this dataset belongs is mono-lingual or multi-lingual c) whether another dataset with the same language is available in the dataset group d) whether the svm classifier predicts the datasource id’s with an accuracy of \(>\)​95% accuracy e) whether the word overlap is larger then 10%.

7 Full results for dependency parsing↩︎

Table 5 shows the results of the UUParser [7] for each dataset, grouped by dataset groups. All results are the average over three runs. We do not report scores for datasets without in-source training data in the ‘parser setting’ columns (which corresponds to Section 3.1 of the paper), as training data is necessary for those settings.

For the ‘without train’ columns (corresponding to Section 3.2 of the paper), we do not include results for dataset groups of size two; this is because we leave one training set out, and try to predict for the corresponding development set in which set it belongs. For groups of size two, this classification is trivial and non-informative, as there is only one dataset left. The left-out datasets are not taken into account for the averages. The reported results are on development splits, except for datasets which did not have a development split available, there we used test (indicated with * in the table) as we do not perform any tuning.

8 Full results for morphological tagging and lemmatization↩︎

Table 7 shows the results of the Multi-Team tagger [6] on the development data for each dataset. Because of the computational costs, results are over a single run. The second column shows the ‘help-dataset’ that each dataset is paired with, based on word overlap.

Note that data sizes are different compared to Table 5 due to a re-split of the data by [12], and different UD versions (  used 2.3 whereas   used 2.2). Another effect of this re-split is that for all datasets, a train, development and test split is available. Also note that dataset prediction (svm) scores reported are on the train data; so if the score is 1.00, pred and gold can still have different scores because the dataset prediction was not equally accurate on the development data.

9 PCA-analysis of gold versus predicted treebank embeddings↩︎

To gain deeper insights in what is represented in treebank embeddings, we plotted the eight largest dataset groups of the UUParser setup into a PCA space [23]. This is done with the default sklearn settings [24]. Results of the gold spaces are shown in Figure 3 and the predicted spaces are plotted in Figure 4. For some groups, there are some clear differences, however for others the plots are highly similar. There seems to be no clear trend in the amount of differences and the performance shifts in Table 5.

References↩︎