1 Introduction↩︎

The pervasive problem of online hate speech (HS) has motivated the research community to investigate methods for mitigating hatred, such as hate speech detection [1], [2] and, more recently, hate moderation through counter narratives (CNs). CNs are non-negative responses to hateful messages providing fact-bound arguments or alternative viewpoints. Distinct from standard approaches to hate intervention by content moderation, counter narratives are preferable as they preserve the right to freedom of speech and encourage peaceful conversations [3], [4].

In particular, the NLP community has started exploring CN generation [5]–[9], also in multilingual settings [10] with the aim of helping non-governmental organizations (NGOs) fight HS. However, automatic generation and detection of counter narratives still face three important challenges. First, counter narratives vary considerably in manner of expression and strategies [11], see Table  1, posing difficulties to automatic CN evaluation and CN classification. Second, few corpora are available since domain experts are required to obtain high-quality data. Third, the only existing work on classifying counter narratives targeted just English [12].

In our view, automated mechanisms to distinguish CN characteristics are crucial for developing hate countering applications that can address personalization or for tasks such as counter narrative mining from user-generated content. Classifying counter narratives would also help to establish which counter narratives are more effective to target hate speech. Finally, it may also be useful for NGO operators to answer hate speech with diverse counter narratives (not just the most common ones) which could be mined from online sources.

To the best of our knowledge, this work is the first to tackle counter narrative classification from a multilingual perspective, predicting the types/strategies employed in a counter narrative message given a hateful message. Other contributions include: (1) the first empirical analysis of counter narrative characteristics on expert-based hate countering data targeting Islamophobia [13]; (2) strong baseline results for multilingual and multi-class counter narrative classification by experimenting with the large pre-trained model XLM-R [14]; and (3) a set of cross-lingual zero-shot experiments showing that huge improvements can be obtained by translating the training data from other languages to English, which is coherent with previous work on the multilinguality of language models [15]. This result also shows that common characteristics of some counter narrative types are carried across languages. Data splits and training scripts are available at https://github.com/yilingchung/multilingualCN-classification.

2 Counter Arguments v.s. Counter Narratives↩︎

While counter arguments and counter narratives have a similar purpose - to present an alternative stance to a statement - a counter argument is not necessarily a counter narrative, and vice versa. The main differences between the two are briefly discussed below, and point for the development of specific classification approaches for CNs.

2.0.0.1 Presence of an argument.

A counter argument is typically defined as an argument (equivalent to a claim or standpoint) contradicting the initial statement with supporting evidence or reasoning [16]–[18]. The presence of supported evidence or reasoning is essential for counter-argumentation and argumentative information retrieval, regardless of it being implicit or explicit (e.g., Statement: Nuclear energy should be used to replace fossil fuels. v.s. Counter argument: Renewable energy is a better option for replacing fossil fuels than unsustainable and expensive energy like nuclear power.). In contrast, counter narratives can be formulated without using arguments. For example, a counter narrative can be without supporting evidence (e.g., Hate speech: Islam is a plague. v.s. Counter narrative: How can you use such language to describe a religion of 1.6 billion people?).

2.0.0.2 Non-negative arguments.

Following guidelines¹ generally adopted by NGOs, an appropriate counter narrative should not contain hostile language, prejudice, and unlawful content. Using negative tones as responses may risk escalating the conversation and engaging in hateful sentiment. On the contrary, counter arguments are not bound by such constraints.

3 Counter Narrative Type Classification↩︎

Classifying counter narrative types is a new and challenging task. We hypothesize that underlying linguistic features in one language can facilitate the classification performance in other languages. Accordingly, we approach this task in a multilingual setting focusing on English, French and Italian. We experimented in the following learning settings: (1) monolingual, in which the train and test set are in the same language; (2) multilingual, in which we train in 3 languages and test on each of them; (3) zero-shot cross-lingual, where we train in one language and evaluate on the other two unseen languages; and, (4) zero-shot translated, similar to the multilingual experiment but translating the Italian and French training data into English.

3.1 Dataset↩︎

In our experiments we use CONAN [13], the only multilingual hate countering dataset currently available that has been niche-sourced to NGO operators. The dataset consists of 14k HS-CN pairs with counter narrative type annotation over English, French, and Italian. For each language, an original pair is augmented with two paraphrases of the original HS coupled with the original CN. In CONAN annotation, one CN could be assigned more than one CN type. As a first investigation into multilingual counter narrative classification, we select the pairs annotated with just one CN type for simplicity, discarding 25%, 17%, and 27% of samples for EN, FR and IT, respectively. Considering that the class balance is skewed, we narrow down the categories to majority classes over 3 languages: facts (35%), denouncing (13%), question (9%), hypocrisy (7%), and humor (5%). We then randomly sampled 80% and 20% of the dataset for the training and testing, ensuring that one HS and its two HS paraphrases paired with the same counter narrative are kept in the same split, so that the same CN does not appear by chance in both the training and testing sets. While the class imbalance poses a challenge, it reflects the practical scenario where certain types are less frequent.

Since CONAN contains only positive examples of CN, we further created 200 instances for the classes support and unrelated.² Support is a response that endorses the hate speech and unrelated is a text not connected to the hate speech in any sense. Clearly these two classes do not fall into the categories of CN and in fact they do not exist in CONAN. We include them to ensure that the models are exposed to varied non-CN text. Such setting can avoid model overfitting and increase the applicability of a system to the real world full of noisy content. The amount of these instances (200) is set to be close to the less populated class (humor). For the unrelated pairs, we randomly sampled data from Wikilingua³ [19], featuring topics unrelated to islamophobia. The support pairs consist of HS paired with each of the two paraphrases. Table 2 reports the distribution of training and testing examples per class across the three languages.

3.2 Models↩︎

The XLM-R language model [14], pre-trained on CommonCrawl for 100 languages, reports strong performance on several cross-lingual downstream tasks such as natural language inference or named entity recognition, and also on multilingual stance detection, close related to CN classification [20]. In this paper we leverage XLM-R to provide a strong baseline in the task of CN type classification in both monolingual and multilingual settings, using the Transformers library [21]. For every experiment, we fine-tuned the base version of XLM-R for 10 epochs with batch size 32 and 2e-5 learning rate.

4 Experimental Results↩︎

Similarly to stance detection and argument mining tasks [18], [22], we report average macro F1 score over the CN types to avoid obtaining very high scores simply deriving from the dominant classes in the dataset. Furthermore, we also follow previous literature on stance detection [22] and report only the performance over the relevant classes.

4.0.0.1 Monolingual results.

As shown by Table 3, monolingual models consistently yield the best performance in predicting the majority classes facts and question. As for denouncing, we obtain results above average for English, French and Italian (0.65, 0.58, and 0.51 respectively). Worst results are obtained for humor and hypocrisy. For Italian and French the model completely fails to identify humor, the most difficult and under-represented class; for English, the prediction is moderate (0.45). Lastly, the low results for the hypocrisy class seemed to be caused by the difficulty for the model in discriminating hypocrisy from facts. For example, 38% of the prediction errors for the hypocrisy class in English are caused by wrongly classifying hypocrisy instances as facts (more details are provided in the Appendix, Figure 1).

In a post-hoc manual analysis, annotators expressed difficulties in differentiating these two classes, difficulties illustrated by the examples provided in Table 4. This issue seems to be further confirmed by annotation statistics obtained from CONAN: among all the instances that contains the hypocrisy label, 45% were annotated with hypocrisy alone, 27% with hypocrisy and facts, while other multi-label cases were much lower. We hypothesize that hypocrisy could be a subclass of facts. After all, pointing out the contradiction in a hate statement may imply correcting misstatements via facts. Future work can try to merge two classes together to improve classification performance.

4.0.0.2 Multilingual results.

As it is reported in Table 3, multilingual learning obtains better overall results than monolingual variants. In general the model trained on all languages improve the per-type performance although we observe decreases in denouncing and hypocrisy for Italian. Also, humor is still challenging for each language with more than 50% of miss-categorization.

4.0.0.3 Zero-shot cross-lingual results.

For each target language, we provide the average F1 of zero-shot cross-lingual models trained on the other 2 source languages in Table 3. Although the results are lower than in the monolingual setting, results show that cross-lingual transfer can be a feasible strategy for classifying counter narratives for languages for which no training data is readily available, especially for the majority classes. Best results in this scenario are obtained when English is the source language, obtaining lower but close results with respect to the monolingual results in Italian and French (for more details about the results per language pair see Table 5 in the Appendix).

4.0.0.4 Zero-shot translated results.

Data augmentation through translation has been widely employed to improve classification performance [23], also in cross-lingual settings [20]. At the same time, the cross-lingual capabilities of Transformer models such as XLM-R and mBERT are being actively investigated [15], [24], [25]. Thus, we conduct an additional experiment adding, to the English training set, the manually translated Italian and French training data before testing on the target languages (Italian and French). The aim is to investigate if XLM-R benefits from fine-tuning on a high resource language (English) instead of combining English with other languages, such as Italian and French, which are not so well represented in pre-trained multilingual models [26]–[28]. By doing so, in this ‘zero-shot transl\(_{en}\)’ setting we aim to expose the model with semantic knowledge from some target languages without actually seeing those languages (Italian and French). Table 3 shows that we obtain a huge performance jump with respect to the multilingual results, with 13 points improvement in macro-F1 score for Italian and 23 points for French. Interestingly, the improvement is more impressive for the humor class, the most challenging of them all.

5 Related Work↩︎

Broadly speaking, counter narrative type classification is related to stance detection [23], [29], which is crucial for argument search [18]. In contrast to stance detection, that concentrates on binary or relatively simple classification – e.g., determine if an argument supports or contests a given topic [18], [30], [31] – we present a multi-class approach to counter narrative classification.

Counter narratives have been adopted as a direct and effective response to online hatred in several campaigns and on social media platforms including Twitter [32], [33], Facebook [4], and Youtube [12], [34]. Although it has been argued that hate speech detection can benefit from CN classification, there are very few studies on this regard, with only one previous work on classifying counter narrative types [12]. However, unlike our present work, they consider hostile language as one of the main types of counter narratives, which is explicitly discouraged by NGOs working on hatred intervention. Furthermore, we investigate multilingual and cross-lingual CN classification leveraging a SoTA pre-trained multilingual language model.

6 Conclusion↩︎

We present the first work on multilingual CN type classification. Our results show that: (i) the performance is promising for the majority classes (facts, question, denouncing); (ii) classifying humor and hypocrisy CNs is still challenging; (iii) combining training data from the three source languages improves performance over the monolingual evaluation; and (iv), the best overall results are obtained in the ‘zero-shot transl\(_{en}\)’ approach where the training data for Italian and French is translated to English. This shows that some knowledge about CNs is transferred across languages. While this is coherent with previous literature about multilingual language models, the exact source of such successful transfer across languages remains an open topic.

Acknowledgements↩︎

Rodrigo Agerri is funded by the RYC-2017–23647 fellowship and by the DeepReading (RTI2018-096846-B-C21, MCIU/AEI/FEDER, UE, Spanish Ministry of Science, Innovation and Universities) and DeepText (KK-2020/00088, Basque Government) projects.

7 Appendices↩︎

References↩︎