1 Introduction↩︎

As online speech gets democratised, we see an ever-growing representation of non-English languages on online platforms. However, in stance detection multilingual resources are scarce [1]. While English datasets exist for various domains and in different sizes, non-English and multilingual datasets are often small (under a thousand examples [2]–[4]) and focus on narrow, potentially country- or culture-specific topics, such as a referendum [2], [5], a person [3], [6], or a notable event [7], with few exceptions [8].

Recently, notable progress was made in zero- and few-shot learning for natural language processing (NLP) using pattern-based training [9]–[11]. These approaches shed light on the ability of pre-trained models to perform in low-resource scenarios, making them an ideal option for modelling cross-lingual stance. Yet, previous work mostly focused on single-task and single-language scenarios. In contrast, here we study their multilingual performance, and their ability to transfer knowledge across tasks and datasets. Moreover, a limitation of these models, especially for pattern-exploiting training [10], is the need for label verbalisation, i.e., to identify single words describing labels. This can be inconvenient for label-rich and nuanced tasks such as stance detection. We overcome this limitation by introducing a label encoder.

Another line of research is transfer learning from different tasks and domains. Recent studies have shown that multi-task and multi-dataset learning can increase both the accuracy and the robustness of stance detection models [12], [13]. Nonetheless, pre-training should not necessarily be performed on the same task; in fact, it is important to select the auxiliary task to pre-train on carefully [14]. Additional or auxiliary data, albeit from a similar task, can also improve performance. An appealing candidate for stance detection is sentiment analysis, due to its semantic relationship to stance [15], [16].

Our work makes the following contributions:

• We present the largest study of cross-lingual stance detection, covering 15 datasets in 12 diverse languages from 6 language families.

• We explore the capabilities of pattern training both in a few-shot and in a full-resource cross-lingual setting.

• We introduce a novel label encoding mechanism to overcome the limitations of predicting multi-token labels and the need for verbalisation (single-token labels).

• We diverge from stance-to-stance transfer by proposing a novel semi-supervised approach to produce automatically labelled instances with a trained sentiment model, leading to sizeable improvements over strong baselines.

• We show that our newly introduced semi-supervised approach outperforms models fine-tuned on few shots from multiple cross-lingual datasets, while being competitive with pre-trained models on English stance datasets.

2 Method↩︎

We propose an end-to-end few-shot learning, and a novel noisy sentiment-based stance detection pre-training.

2.1 Few-Shot Pattern-Exploiting Learning (PET)↩︎

PET and its variants [10], [17], [18] have shown promising results when trained in a few-shot setting. They bridge the gap between downstream tasks like text classification and the pre-training of models by converting the dataset into a cloze-style question format that brings it closer to the masked language modelling objective. Using this technique, models with few hundred million parameters can outperform parameter-rich models such as GPT-3 [9] on various benchmark tasks [19] by fine-tuning on just 32 examples.¹ Our motivation for adopting this framework is threefold: (i) there has not been much prior work that puts these models under scrutiny in a cross-lingual setting, (ii) often, there is a data scarcity for many languages, which is also the case with stance datasets (only three of our datasets contain more than 2,000 training examples, see Section 3), (iii) the label inventories of different datasets are often shared or contain synonymous words such as ‘pro’, ‘in favour’, ‘support’, etc., which can be strong indicators for the model in both few-shot or full-resource setting.

2.2 Cross-lingual Stance Pattern Training↩︎

Figure 1 shows the architecture of our model. First, we use a simplified PET with a pre-trained language model to predict the likelihood of each label to fill a special mask token in a sentence-based template (see Prompt below). To obtain a suitable representation (label embeddings) for the labels, we use a ‘label encoder’ that averages the pooled vectors from the model’s token embeddings for each sub-word. Finally, we take the dot product of the label embeddings and the contextualised word embedding for the masked position to obtain the likelihood for each label to fit in.

2.2.0.1 Prompt

The prompt design is an important aspect of the pattern-exploiting training procedure. In our work, we select a prompt that describes the stance task, rather than a punctuation-based one used in previous work [10]. In particular, our prompt is shown below, where the special token changes based on the model choice:

[CLS] The stance of the following ____^CONTEXT is [MASK] the ____^TARGET.[SEP]

Prior work [20]–[22] has studied aspects of PET such as prompt design, tuning, and selection. Here, we focus on the training procedure, and we leave the exploration of these in a multilingual setting for future work.

2.2.0.2 Label Encoder

A well-known challenge in PET is the need for a fixed number of positions for the label, e.g., a single mask is needed for words present in the dictionary such as ‘Yes/No’; however, we need multiple positions to predict more complex ones with multiple tokens such as ‘Unrelated’. Moreover, if different labels have different lengths, the model needs to ignore some of the positions, e.g., to predict a padding inside the sentence. The label inventory commonly contains words tokenised into multiple tokens. [10] propose a simple verbalisation technique where the original labels are replaced with words that can be represented with a single token from the vocabulary, e.g., ‘Favour’ \(\xrightarrow{}\) ‘Yes’, ‘Against’ \(\xrightarrow{}\) ‘No’. Another possibility is to automatically detect such words, but this yields notable drop in performance compared to manual verbalisation by a domain expert [23].

Here, we propose a simple, yet effective, approach to overcome this problem. Instead of using a single token representation per label, we take the original label inventory and we tokenise all words, as shown in Figure 1. In the ‘Label inventory’ box, we see four labels common for stance tasks and their tokens (obtained by the XLM-R’s tokenizer) – {‘_against’}, {‘_discuss’, ‘ing’}, {‘_in’, ‘_favour’}, and {‘_un’, ‘related’, ‘_to’}. For each token of a label, we extract the vector representation from the MLM pre-trained model’s (e.g., XLM-R) token embeddings \(v^{L_t}_{TE} = TokEmb(L_t)\). Afterwards, we obtain the final label representation (\(LE_L\)) using an element-wise averaging for all \(v^{L_t}_{TE}\) (see Eq. 1 ).

\[LE_L = \frac{1}{N} \sum_{t=0}^N{TokEmb(L_t)}; \forall_{L} \in \{Labels\} \label{eq:tokemb}\tag{1}\]

Note that for single tokens, this method defaults to the original MLM task used in learning BERT-based models [24], [25]. The technique of averaging the embedding is shown to be effective with non-contextualised language models such as word2vec [26] and GloVe [27] for representing entire documents or for obtaining a token-level representation with fastText [28].

Finally, to obtain the label for each example, we take the dot product between the MLM representation for the masked token position, and each of the \(LE_L\) vectors. There is no need for padding, as both representations are of the same dimensionality by design  [29]. Here, we must note that we select the candidates only from the task-related labels; however, we treat the task as a multi-label one, as we describe in more detail below.

2.2.0.3 Training Objective

We use a standard binary-cross entropy (BCE) loss for each label, where for positive examples, we propagate 1, and for negative ones, we propagate 0. We do not use the original MLM cross-entropy over the entire dictionary, as this will force the model to recognise only certain words as the correct labels, whereas their synonyms are also a valid choice. Moreover, such a loss will prevent further knowledge transfer between tasks and will degraded the model’s ability to perform in a zero-shot setting.

\[\begin{gather} \begin{medsize} \displaystyle \mathcal{L}_{LE} = \sum_{y^{\prime}_\in y^{p}}{\text{BCE}(p(y^{\prime}|x), 1)} + \sum_{y^{\prime\prime} \in y^{n}}{\text{BCE}(p(y^{\prime\prime}|x), 0)} \end{medsize} \\ \mathcal{L} = \lambda \cdot \mathcal{L}_{LE} + (1-\lambda) \cdot \mathcal{L}_{MLM} \end{gather}\]

2.2.0.4 Positive and Negative Sampling

The label encoder allows for sampling of positive and negative examples at training. This can be useful for tasks such as stance detection, where label inventories can differ, but labels overlap semantically. Indeed, this holds for our datasets, as is apparent in Table  1 where we see semantically similar labels like support, agree, favor etc. across several datasets.

To obtain a set of synonyms for each label, we use two publicly available sources: (i) Google Dictionary suggestions²; and (ii) synsets of the English WordNet [30]. However, this is prone to noise, as a word can have multiple meanings, and building a high-quality lexicon would require a human annotator proficient in the target language. Thus, we use negative sampling, as unrelated words are also undesirable to predict by the model, rather than using these examples to enrich the positive labels lexicon.

2.3 Sentiment-Based Stance Pre-Training↩︎

We propose a novel semi-supervised method for pre-training stance detection models using annotations from a sentiment analysis model. This is motivated by the observation that these are two closely related tasks (the difference being that sentiment analysis does not have a target).³ To illustrate this, consider the sentence ‘I am so happy that Donald Trump lost the election.’, which has a positive sentiment, but when expressed towards a specific target, e.g., ‘Donald Trump’, then the expected label should be the opposite – negative, or more precisely against. This requires for the introduction of targets that can change the sentiment label. For further details how we produce corresponding datasets see Section 3.3.

We hypothesise that such pre-training could help bootstrap the model’s performance, especially in a low-resource setting, similarly to pre-training on cross-domain stance datasets. We use the same model and pattern as for fine-tuning the cross-lingual stance models, and we use a masked language modelling objective and negative sampling to improve the language model’s performance on one hand, and, on the other hand, to allow the model to associate synonyms as the label inventories are very diverse (see Table 1). We do not do positive sampling as it requires high-quality synonyms, which can only be obtained by manual annotations, while our goal is to design an end-to-end pipeline without a need for human interaction.

3 Datasets↩︎

We use three types of datasets: 15 cross-lingual stance datasets (see Table 1), English stance datasets, and raw Wikipedia data automatically annotated for stance. The cross-lingual ones are used for fine-tuning and evaluation, whereas the rest are only used for pre-training. Appendix 9 provides additional examples for the cross-lingual datasets shown in Table 7. Further quantitative analysis of the texts are also shown in the Appendix in Table 5 and Figure 2.

3.1 Cross-Lingual Stance Datasets↩︎

ans [31]. The Arabic News Stance corpus has paraphrased or contradicting news titles from several major news sources in the Middle East.
arabicfc [32] consists of claim-document pairs with true and false claims extracted from a news outlet and a fact checking website respectively. Topics include the Syrian War and other related Middle Eastern issues.
conref-ita [2] contains tweet-retweet-reply triplets along with their stance annotation pertaining to a polarising referendum held in Italy in December 2016 to amend the constitution.
czech [6] provides stance-annotated comments on a news server in Czech on a proposed Smoking ban in restaurants and the Czech president Miloš Zeman.
dast [33] includes stance annotations towards submissions on Danish subreddits covering various political topics.
e-fra, r-ita [3] consists of French tweets about the 2017 French presidential election and Italian ones about the 2016 Italian constitutional referendum.
hindi [7] has Hindi-English code-mixed tweets and their stance towards demonetisation of the Indian currency that took place in 2016.
ibereval [5] contains tweets in Spanish and Catalan about the Independence of Catalonia, collected as part of a shared task held at IberEval 2017.
nlpcc [34] contains posts from the Chinese micro-blogging site Sina Weibo about manually selected topics like the iPhoneSE or the open second child policy.
rustance [4] includes posts on Twitter and Russian-focused media outlets on topics relating to Russian politics. The extraction was done in 2017.
sardistance [35] includes textual and contextual information about tweets relating to the Sardines movementin Italy towards the end of 2019.
xstance [8] contains questions about topics relating to Swiss politics, answered by Swiss political candidates in French, Swiss German, or Italian, during elections held between 2011 and 2020.

3.2 English Stance Datasets↩︎

We use 16 English stance datasets from two recent large-scale studies of multi-task/multi-dataset stance detection [12], [13]. We followed the data preparation and the data pre-processing described in the aforementioned papers as is. The combined dataset contains more than 250K examples, 154K of which are used for training. The data comes from social media, news websites, debating forums, political debates, encyclopedias, and Web search engines, etc. The label inventory includes 24 unique labels. We refer the interested reader to the respective papers for further detail.

3.3 Sentiment-Based Stance Datasets↩︎

We use Wikipedia as a source of candidate examples for constructing our sentiment-based stance dataset due to its size and diversity of topics covered. To study the impact language has on pre-training, we construct two datasets: English (enWiki) and multilingual (mWiki). The latter includes examples from each of the languages covered by some of our datasets. In particular, we use the Wikipedia Python API to sample random Wiki articles. For the multilingual setup, for each language, we sampled 1,000 unique articles⁴ (non-overlapping among languages), a total of 11,000. For the English-only setup, we sampled the same number of articles.

Next, to obtain the contexts for the datasets, we split the articles (with headings removed) at the sentence-level using a language-specific sentence splitting model from Stanza [36]. Each context is then annotated with sentiment using XLM-T, an XLM-R-based sentiment model trained on Twitter data [37]. We use that model as it covers all the datasets’ languages, albeit from a different domain. It produces three labels – {positive, negative, neutral}, which we rename to {favor, against, discuss}, to match the label inventory common for stance tasks. To obtain a target–context pair, we assign a target for each context – either the ‘Title’ of the article, or, if there is a subheading, the concatenation of the two. To cover as much as possible of the stance label variety, we also include unrelated in the inventory, which we define as ‘a piece of text unrelated to the target’: for this, we randomly match targets and contexts from the existing tuples. The latter class also serves as a regulariser, preventing overfitting to the sentiment analysis task, as it includes examples with positive or negative contexts that are not classified as such. The resulting distribution is unrelated (60%), discuss (23%), against (10%), favor (7%). This aims to match the class imbalance common for stance tasks [4], [32], [38].

Finally, we augment 50% of the examples by replacing the target (title) with the first sentence from the abstract of the Wikipedia page. These new examples are added to the original dataset, keeping both the original and the augmented ones. Our aim is to also produce long examples such as user posts, descriptions of an events, etc., which are common targets for stance. The resulting dataset contains around 300K examples, which we split into 80% for training and 10% for development and testing each, ensuring that sentences from one article are only included in one of the data splits.

4 Experiments↩︎

4.0.0.1 Models

We evaluate three groups of models: (i) without any pre-training, i.e., baselines (see next); (ii) pre-trained on multiple English stance datasets (‘enstance’), using automatically labelled instances produced using a sentiment model (‘*Wiki’), see Section 2.3; and (iii) multi-dataset learning (MDL), i.e., we include \(N\) examples from each dataset into the training data. We train and evaluate on a single dataset, except in the case of MDL, where we train and evaluate on everything. We choose the best model based on the macro-averaged F1 on all datasets. All models use XLM-R_Base as their base.

4.0.0.2 Baselines

In addition to our proposed models (Section 2), we compare to a number of simple baselines:

Majority class baseline calculated from the distributions of the labels in each test set.

Random baseline Each test instance is assigned a target label at random with equal probability.

Logistic Regression A logistic regression trained using TF.IDF word unigrams. The input is the concatenation of separately produced vectors for the target and the context.

XLM-R A conventionally fine-tuned XLM-R_Base model predicting and back-propagating the errors though the special \(<\)s\(>\) token.

4.1 Quantitative Analysis↩︎

We first analyse the high-level few-shot performance of the proposed models using the averaged per-dataset F1 macro. Then, we zoom in on the dataset level and analyse the models in the two most extreme training scenarios: few-shot with 32 examples, and full-resource training.

4.1.0.1 Few-shot analysis

Table 2 shows results for different types of pre-training on top of the pattern-based model (Section 2). The top of the table lists baselines, followed by ablations of training techniques. More fine-grained, performance per dataset is shown in Table 3 in Appendix 10. We can see that the ‘Pattern’ model outperforms random baselines in all shots, except zero. Moreover, there is a steady increase in performance when adding more examples. The performance saturates at around 256 examples, with the difference between it and all being 1.3 points F1, whereas in subsequent pairs from previous columns the margin is 3.5 to 5 points.

The middle part of Table 2 ablates the stance pre-training on top of the pattern-based model. We first analyse the models trained using the artificial dataset from Wikipedia articles, automatically labelled with a multilingual sentiment model (Section 2.3). We study the effects of the language of the pre-training data by including two setups – enWiki that contains only English data, and mWiki with equally distributed data among all languages seen in the datasets. Both variants give a sizeable improvement over the baselines in all few-shot settings, especially in low-resource ones. The increase in F1 when using 32 examples is more than 6 points on average; and these positive effects are retained when training on all examples. The mWiki model outperforms the Pattern baseline by 4 points and the enWiki – 1.4 points respectively. The multilingual pre-trained model constantly scores higher than the English pre-trained one. Moreover, we see a tendency for the gap between the two to increase with the number of examples reaching 2.6 points in all.

For pre-training on English stance data (enstance), even with 32 examples, we see a large increase in performance of 11 points absolute over the Pattern baseline. This model is also competitive, within 3 points absolute on average, to the baseline trained on the whole dataset. Furthermore, the enstance model outperforms the pre-training with automatically labelled stance instances (en/mWiki). Nevertheless, the en/mWiki models stay within 3-5 points F1 in the all shots. The gap in performance is expected, as the enstance model is exposed to multiple stance definitions during its extensive pre-training, in contrast to the single one in the Wiki and its noisy labels. Finally, only enstance surpasses the random baselines even in the zero-shot setting, scoring 35.16 F1, demonstrating the difficulty of this task. We offer additional analyses of the zero-shot performance in Appendix 10.3.

The bottom part of Table 2 shows results for multi-dataset learning (MDL). Here, the models are trained on \(N\) examples from each dataset, instead of \(N\) from a single dataset. The first row presents the MDL Pattern model (without any pre-training). Here, we can see that in few-shot setting training on multiple datasets does not bring a significant performance gains compared to using examples from a single dataset. Nonetheless, when all the data is used for training, F1 notably increases, outperforming the English stance model. Furthermore, combining MDL with the multilingual sentiment-based stance pre-training (MDL mWiki) yields an even larger increase – almost 9 points F1 higher than Pattern, 5 points better than mWiki, and 1 point – enstance. We attribute the weaker performance on few-shot and the strong performance on full-resource learning of the MDL-based models to the diversity of the stance definitions and domains of the datasets, i.e., MDL fails to generalise and overfits the training data samples in the few-shot setting, however when more data is included, it serves as a regularizer, thus the model’s score improves. The phenomena is also seen in other studies on English stance [12], [13]. To some extent, the same regularisation effect comes from the pre-training on the artificial stance task, then the model needs to adjust its weights to the new definition, without having to learn the generic stance task from scratch.

4.1.0.2 Per-dataset analysis

Next, we analyse our experiments on the dataset level. In Table 3 we present a fine-grained evaluation for each dataset covering the two most extreme data regimes that we run our models in: (i) full-resource training and (ii) few-shot training with 32 examples. We want to emphasise that we do not include state-of-the-art (SOTA) results in Table 3 as the setup in most previous work differs from ours, e.g., the data splits do not match (see Appendix 9.1), or the use different metrics, etc. For more details about the SOTA refer to the Appendix 10.1. For completeness, we include two standard strong baseline models, i.e., Logistic Regression and a conventionally fine-tuned XLM-R. Both baselines are trained on every dataset separately using all of the data available in its training set.

From our results it is clear that even with all data available from training, a model that does not do any pre-training or knowledge transfer such as the Logistic Regression struggles with the cross-lingual stance detection tasks. Even though the model surpasses the random baselines, it falls over 14 points F1 short compared to both the XLM-R_Base and the Pattern model. In turn, the Pattern model is 1 point better than the XLM-R_Base outperforming the random baselines on all datasets. Interestingly, the XLM-R_Base model fails to beat the random baselines on hindi and rustance. We attribute this to the code-mixed nature of the former, and the small number of training examples (359) in the latter.

To further understand the results of the models bootstrapped with pre-training or multi-dataset learning, we analyse their per-dataset performance next. From Table 3 we can see that the MDL variants achieve the highest results on 8 out of the 15 datasets, enstance rank best on 6 and a single win is for mWiki on sardistance.

Examining the results achieved by the sentiment-based stance pre-training (en/mWiki) we see between 7 and 29 points absolute increase in terms of F1 over the Pattern baseline for several datasets – czech, conref-ita, e-fra and r-ita. A contradiction example are the two datasets dast and rustance, where we have a notable drop in F1 compared both to the Pattern and the enstance models. On one hand this can be attributed on the skewed label distribution, especially in the support(ing), deny and querying classes, on the other hand that also suggests the stance definition in these two datasets is different than the one adopted by us in the en/mWiki pre-training. In turn, enstance demonstrates a robust performance on all datasets, as it has been pre-trained on a variety of stance detection tasks.

A common characteristic uniting the datasets, where the MDL models achieve the highest F1, is the presence of at least one other dataset with similar topic and language: (i) conref-ita and r-ita are both Italian datasets about a referendum, (ii) ibereval contains tweets about the “Independence of Catalonia” in Catalan and Spanish, and (iii) xstance contains comments by candidates for elections in Switzerland. This suggests that multi-dataset learning is most beneficial when we have similar datasets.

Finally, we analyse the few-shot training with 32 examples. Here, the highest scoring model on 9 out 15 is the enstance one. This suggests that other models struggle to learn the stance definition from the cross-lingual datasets by learning from just 32 examples. This phenomena is particularly noticeable in datasets having a skewed label distribution with one or more of the classes being a small proportion of the dataset such as the two Arabic datasets (ans – other (2%), arabicfc – disagree (3%)). Nevertheless, en/mWiki models show steady sizeable improvements of 6 points F1 on average on all dataset. On the other hand, as in the full-resource setting, training on multiple datasets (MDL mWiki) boosts the performance of conref-ita and r-ita with 27 points F1 compared to the Pattern baseline. However, we must note that this holds true only when we pre-training on a stance task, as the MDL model has lower F1. That again is an argument in favour of our hypothesises: (i) few-shot training on multiple stance datasets fails to generalise, and (ii) combining datasets that cover the same topic and are in the language have the largest impact on the model’s score.

5 Discussion↩︎

Our fine-tuning with few instances improves over random and non-neural baselines such as Logistic Regression trained on all-shots, even by more than 20 points F1 on average when training on just 32 instances. However, such models, especially when trained on very few examples, suffer from large variance and instability. In particular, for cross-lingual stance, the pattern-based model’s standard deviation (\(\sigma\)) varies from 1.1 (conref-ita, nlpcc) to 8.9 (ibereval-ca), with 3.5 on average when trained on 32 examples. Pre-training improves stability by reducing the variance, e.g., en/mWiki have a \(\sigma\) 2.7 with minimum under 1, which is more than 5% relative change even when comparing to the highest F1 average achieved with 32 examples. The lowest \(\sigma\) is when the model is trained on all-shots, and especially in the MDL models with 1.7, and 1.4 for the mWiki variant.

This variability can be attributed to the known instabilities of large pre-trained language models [39], but this does not explain it all. Choosing a right set of data points is another extremely important factor in few-shot learning that calls for better selection of training data for pre-training and fine-tuning [40], [41].

Another important factor is the inconsistency of the tasks in the training data. This is visible from our MDL experiments, i.e., the tasks use a variety of definitions and labels. Even with more examples in the training set in comparison to single-task training (15x\(N\) examples), the models tend to overfit and struggle to generalise to the testing data. In turn, when sufficient resources are available, MDL yields sizeable improvements even without additional pre-training.

Having access to noisy sentiment-based stance data in the same languages helps, but transferring knowledge from a resource-rich language (e.g., English) on the same task (or set of task definitions) is even more beneficial, in contrast to the data’s (see Section 4.1) and label’s language (see Appendix 10.5). Moreover, when using noisy labels from an external model, there is always a risk of introducing additional bias due the training data and discrepancies in the task definition [42], [43]. We observed this for both the dast and the rustance datasets.

6 Related Work↩︎

6.0.0.1 Stance Detection

Recent studies on stance detection have shown that mixing cross-domain English data improves accuracy and robustness [12], [13]. They also indicated important challenges of cross-domain setups such as differences in stance definitions, annotation guidelines, and label inventories. Our cross-lingual setup adds two more challenges: (i) data scarcity in the target language, which requires learning from few examples, and (ii) need for better multilingual models with the ability for cross-lingual knowledge transfer.

6.0.0.2 Cross-Lingual Stance Detection

There have been many efforts to develop multilingual stance systems [5], [8], [44], [45]. However, most of them consider 2–3 languages, often from the same language family, thus only providing limited evidence for the potential of cross-lingual stance models to generalise across languages. A notable exception is [3], who work with 5 languages, but restrict their study to a single family of non-English languages and their domain to political topics only. Our work, on the other hand, spans 6 language families and multiple domains from news [31] to finance [8].

6.0.0.3 Stance and Sentiment

Sentiment Analysis has a long history of association with stance [46], [47]. Sentiment is often annotated in parallel to stance [48], [49] and has been used extensively as a feature [15], [16], [50] or as an auxiliary task [51], [52] for improving stance detection. Missing from these studies, however, is leveraging sentiment annotations to generate noisy stance examples, which we explore here: for English and in a multilingual setting.

6.0.0.4 Pattern-based Training

Recently, prompt or pattern-based training has emerged as an effective way of exploiting pre-trained language models for different tasks in few-shot settings [10], [53]. [9] introduced a large language model (i.e., GPT-3), which showed strong performance on several tasks through demonstrations of the task. [10], [17] proposed Pattern-Exploiting Training (PET), a novel approach using comparatively smaller masked language models through Cloze-style probing with task informed patterns. [18] build on PET, with an additional loss that allows them to circumvent the reliance on unsupervised data and ensembling. There have been studies on aspects of prompt-based methods such as performance in the absence of prompts [21], quantifying scale efficiency [54], learned continuous prompts [20], [22], [55] or gradient-based generated discrete prompts [56]. [57] offer a survey of prompt-based techniques. We focus on the few-shot advantages offered by PET methods and evaluating them in a cross-lingual setting.

7 Conclusion and Future Work↩︎

We have presented a holistic study of cross-lingual stance detection. We investigated PET with different (pre-)training procedures and extended it by proposing a novel label encoder that mitigates the need for translating labels into a single verbalisation. In addition to that, we introduced a novel methodology to produce artificial stance examples using a set of sentiment annotations. This yields sizeable improvements on 15 datasets over strong baselines, i.e., more than 6% F1 absolute in a low-shot and 4% F1 in a full-resource scenario. Finally, we study the impact of multi-dataset learning and pre-training with English stance data, which further boost the performance by 5% F1 absolute.

In future work, we plan to experiment with more sentiment-based models and stance task formulations, as well as different prompt engineering techniques.

8 Fine-Tuning and Hyper-Parameters↩︎

• All the neural-based models (i.e., XLM-R and its variants) are developed in Python using PyTorch [58] and the Transformers library [59].

• Logistic regression and TF.IDF are implemented using Scikit-learn [60].

• All neural-based models are trained for 5 epochs with batch size 16, using the AdamW optimiser [61], with weight decay 1e-8, \(\beta_1\) 0.9, \(\beta_2\) 0.999, \(\epsilon\) 1e-08.

• All models use class weights, in BCE only the positive class is weighted.

• We use a maximum sequence length of 150 tokens. Afterwards, we truncate only the target and the context sequences token by token, keeping the tokens from the pattern unchanged, starting from the longest sequence in the pair.

• For logistic regression we convert the text to lowercase, and limit the dictionary in the TF.IDF to 15,000 uni-grams. The vocabulary is fit on the concatenated target and context. The two are then transformed separately to form the individual vectors.

• All the hyper-parameters are tuned on the corresponding validation sets.

• In models, trained on a single dataset, we choose the best checkpoint based on its performance on the corresponding validation set. For multi-dataset training, we choose the checkpoint that performs best on the combined validation sets. The models are evaluated on every 200 steps.

• We run each setup three times and then average the scores. In few-shot setting, we use three different subsets from the training dataset with \(K\) instances, while keeping the same seed for the models, whereas in full-resource setting, we set different seeds for each run.

• Each single dataset experiment took roughly 20 minutes on a single NVIDIA V100 GPU using half precision.

8.0.0.1 Zero-shot

For our zero-shot experiments, we only pass the labels from the corresponding task’s inventory to the models to obtain a ranking, we then take the label with maximum probability.

8.0.0.2 Few-shot

The few-shot models are trained w/ LR 1e-05 and warmup 0.06. The number of training steps is: (i) 2,000 steps for single dataset training, (ii) 4,000 steps for multi-dataset learning.

8.0.0.3 Full-resource

The XLM-R, both \(<\)s\(>\) fine-tuned and pattern-based, models are trained w/ LR 3e-05 and warmup 0.06 for 8 epochs. The logistic regression model is trained using the Limited-memory BFGS until convergence.

8.0.0.4 Sentiment-Based Pre-Training

We train for 3 epochs with MLM of 12.5% and 2 negative samples per-label.

9 Dataset↩︎

In this section, we first discuss the datasets characteristics in terms of data splits, then we present the word-level statistics and finally analyse the transformed labels after tokenisation in terms of sentence pieces [62].

In order understand and demonstrate the relationship between the tasks, we proportionally sample 25,000 instances from each dataset and pass them though the \(<\)s\(>\) token of a XLM-R_Base model to obtain a sentence-level representation of each. The input has the following form: \(<\)s\(>\) context \(<\)/s\(>\) target. The resulting clusters are shown in Figure 2. As the model is not fine-tuned on any down-stream task, we see that each cluster covers only one language, e.g., ans and arabicfc, both in Arabic, are grouped in the right part of the plot, Spanish/Catalan (ibereval) are in the middle, and Italian (conref-ita, r-ita, sardistance) are on the left. Surprisingly, the two French ones — xstance-fr and e-fra) — are well-separated. We attribute this to their different domains, i.e., tweets about French politicians vs. stance of political candidates towards Swiss policies. Moreover, even datasets from the same source domain such as xstance-de and xstance-fr do not have any overlapping points. On the other hand, we see that the centroids of datasets in the same language and with related topics such as ibereval-ca and ibereval-es or conref-it and r-ita are also close in the latent space.

9.1 Data Splits↩︎

Table 4 shows the number of examples in the training, development, and test datasets. Due to their small size, some of the datasets did not provide any standard splits as their authors used cross-validation. The datasets that did provide such splits are ans, e-fra, r-ita, xstance. For the rest of the datasets, we produced our own splits with ratio 3:1:4 for the training, development, and testing, respectively. sardistance had only train/test split, and thus we further split the training set into train/dev using a ratio of 3:1. Finally, we used only the training set of ibereval, and Task A’s training data from nlpcc, as the testing data was not publicly available.

We generated the few-shot subsets by randomly sampling \(K\) examples from the training set. We allow example overlap between the 3 subsets, but we enforce the subset to have at least one instance from each class. We think that this setup is more realistic and recreates a real-world scenario, in contrast to sampling an equal number of examples per-class as is often done in previous work on few-shot learning.

9.2 Word Statistics↩︎

Next, in Table 5 we present the statistics in terms of number of words/tokens per dataset. The first column shows the average number of words (tokens). The statistics are obtained using language-specific tokenisers from Stanza [36]. In turn, the tokens statistics are calculated in terms of sentence pieces after applying XLM-R’s tokeniser. In subsequent columns, we describe the text lengths’ distribution by reporting the 25th and 50th percentile, and the max.

Most of the datasets contain short texts, i.e., under 100 words/pieces, expect for arabicfc that is built using retrieved documents from the Google search engine, and conref-ita whose contexts are triplets of tweets (tweet-retweet-reply).

9.3 Label Tokens↩︎

In Table 6, we include the labels and their tokenised forms. Each label is present in at least one stance dataset, either English or multilingual. We include the tokens obtained from XLM-R, as we use it for training the pattern-based models.

The results confirm that common words in base form such as positive, negative, neutral are tokenised into a single token. In turn, rare words such as endorse, conjugated verbs (supporting, querying), words with prefixes (unrelated), British spelling (favour), or multi-word expressions (argument for, in favor) are broken into multiple pieces.

3pt

10 Experiments↩︎

In this section, we cover additional experiments and state-of-the-art results. For the latter, we must emphasise that our setup, and respectively results, are not comparable with most of the previous work due to their data splits not being publicly available or their choice of evaluation metrics (see Appendices 9.1 and 10.1 for more details).

The rest of the section covers the model’s performance in different setups: sentiment-based stance task (Appendix 10.2), zero-shot inference (Appendix 10.3), per-dataset few-shot with 64–256 instances (Appendix 10.4, and finally an ablation study of prominent model components (Appendix 10.5).

10.1 State-of-the-Art↩︎

In Table 9 we present the previously reported state-of-the-art results on the cross-lingual datasets included in our study. The results for 12 out of 15 datasets, however, are not comparable to the ones we report (see Table 3) because of mismatch in the data split and metrics between our and the original setup. As highlighted in Section 9.1, our splits differ from the original ones in 10 out of 15 datasets due to unavailability of public development or test sets, making direct comparison infeasible. Furthermore, for r-ita and e-fra, the reported F1-macro results in previous work, are only calculated across the favor and against classes, ignoring the none class. We calculate our results across all classes as it more accurately represents a real-world setting, where many text pieces do not take a stance towards a specific target.

10.2 Sentiment-Based Stance↩︎

In Table 10 we present the results on the sentiment-based stance pre-training task in terms of F1. In particular, we train each model only on either English or multilingual data (see Section 3.3), and respectively evaluate on a testing dataset containing examples in the same language(s).

For the results shown, we can see that the difference in terms of F1_macro between the two setups is marginal, under 0.5 points. However, when analysing the fine-grained scores for each label, we observe some larger differences. More specifically, on the favor and against classes the English model is performing better, whereas the multilingual one scores higher on the unrelated instances. The performance on the discuss class for both models is more or less equal.

Nevertheless, our experiments show that the multilingual model performs better on the cross-lingual stance datasets (see Section 4.1). This suggests that the improvements comes from the language-specific pre-training, not from the model’s performance on the auxiliary sentiment-based stance task.

10.3 Zero-Shot Analysis↩︎

In Table 8 we present the results for zero-shot setting. We include three variants of the pattern-based models – one without any pre-training (Pattern), one pre-trained on the multilingual stance-based sentiment task (mWiki), and finally the model trained on English stance data (enstance).

First, we see that the Pattern model has no prior knowledge of how to solve the tasks given just a template. Moreover, the model always predicts a single class, not necessarily the majority one, therefore some of the scores exactly match the F1 from the Baseline (Max) row.

Next, we study the mWiki pre-trained model. In general, it does not outperform the baseline in terms of macro-averaged F1, however the model still shows positive results on some of the datasets, i.e., conref-ita, e-fra, nlpcc, r-ita, sardistance and xstance-fr. These improvements, over the random/majority baselines, are mostly due to the higher F1 scores in two of the labels – against and favor. We must note that this increase is not completely related to the naming of the labels as for other datasets, having similar label inventories such as the hindi and ibereval, all models fail to outperform the baseline.

Finally, the enstance variant successfully surpasses the baseline results on 9 out of the 15 datasets, and in turn achieves 5 points absolute higher F1 than the baseline. On one hand, this indicates the model’s potential to perform in a zero-shot setting, however, adding even as little as 32 examples can drastically improve performance (see Tables 2 and 3), i.e., enstance scores 50.4 (32-shots) vs. 35.2 (zero-shot). Moreover, even when we have datasets that share the same label inventories as pre-training tasks, there is no guarantee that the model would be able to transfer the knowledge in zero-shot setting, e.g., support, query, comment, discuss are present in dast and rustance from the cross-lingual datasets, and in the English datasets used for pre-training (e.g., the SemEval task: RumourEval [65]).

10.4 Per-Dataset Few-Shot Results↩︎

Here, we analyse the per-dataset few-shot results shown in Figure 3. Our experiments suggest that even with 32 examples, our models outperform the random/majority baseline on almost all datasets. Only exceptions being the models with no additional pre-training (Pattern, MDL), which fail to do so on xstance-de. This suggests that the xstance task is complex and is not trivial to learn from so few examples. The same models also score close to the baseline on its French version (xstance-fr), which in general is easier than the German variant (see Tables 3 and 9), further supports this observation.

Additionally, our experiments show that with increased the training instances, the positive growth in F1 is retained in all datasets. This is also visible from the F1 scores (macro-averaged over all datasets) in Table 2. Moreover, the Pattern model performs considerably worse than the en/mWiki pre-trained models and MDL, except on dast, and rustance. This again supports our hypothesis that the latter two datasets adopt different stance definitions compared to the other cross-lingual stance detection tasks.

Nevertheless, increasing the number of examples does not necessarily lead to an increase in performance in low-shot setting. In particular, for the enstance model, we see very small or no gain on several datasets, independent of the number of shots, i.e., arabicfc, rustance, and xstance-fr. Nonetheless, going up to 256 shots often leads to sizeable performance gains in the Pattern model by more than 15 points F1 (ans, conref-ita, ibereval-ca/es, nlpcc. We attribute this to the large class imbalance that requires the models to be able to learn a target class from one or two instances.

10.5 Ablations↩︎

In Table 11, we ablate different components from the pattern training. For a baseline model, we use a XLM-R_Base fine-tuned using patterns (i.e., Pattern). We present the results in 32–256 shots, where on each row we add one additional component to the baseline model. From the first two rows, we can see that adding two negatives, and masking some of the input tokens does not help. We attribute the former to the specificity of the label inventories and the introduced additional imbalance between positive to negative samples. For the latter, [10] also observed that the MLM loss does not bring additional performance gains when the number of shots increases (but still is beneficial in the low-shot setting). We hypothesise that such a small number of examples are not enough for the model to adapt to the tasks’ domains and languages, albeit previous work shows that further MLM pre-training on task-specific data helps [66].

Finally, we experiment with translating the label inventories into the corresponding language of the dataset.⁵ Using these translated labels, we see improvements in terms of F1 – between 0.36 and 1 point absolute over the Pattern baseline. However, with increased number of examples, the improvement diminishes. This suggests that the language of the labels is not a major factor when training the models.

References↩︎