1 Introduction↩︎

Social media (SM) plays an increasingly important role in our lives. As of 2021, seven out of ten US adults use at least one social media platform like Facebook, Twitter, Instagram, or Pinterest [1]. That proportion likely underestimates SM use if we broaden the definition of SM to include user-generated content like restaurant reviews, news article comments, and forum discussions. The ever-growing trove of text produced by social media users is both a challenge and an opportunity for natural language processing (NLP). NLP models with a strong grasp of social media language could perform a variety of socially, economically, and politically important tasks. They could, for example, tackle automatic content moderation to improve the quality of online discourse, summarize restaurant reviews to simplify the decision-making process of hungry customers, and detect and stunt disinformation campaigns aimed at sowing societal instability.

However, modeling social media language is inherently challenging. Due to its informal, noisy, and fast-evolving nature, social media language on platforms such as Twitter [2] is different from the language found in books, news publications, and Wikipedia. Additionally, the tasks that organically arise from the social media domain (trend detection, emoji prediction, cyberbullying detection, online marketing, etc.) are qualitatively different from the tasks natural to the domain of standard written language (translation, entailment, grammar checking, etc.). Although the recipe of pretraining language models on massive conventional corpora has been successful in pushing the state-of-the-art of general language understanding [3]–[8], it is unclear if this recipe’s success will transfer to the social media domain. The reason for this lack of clarity is that general language understanding benchmarks [9]–[15] include neither SM data nor tasks and therefore do not measure social media language understanding.

While huge strides have been made in the social media language understanding literature to mitigate these challenges, existing work has spotty coverage of the full range of social media platforms and social media language understanding tasks. First, some related works studied vocabulary shift on a single platform (Twitter, [16]) and compared corpora of different genres [17], [18]. However, they did not directly compare social media language with conventional language. Similarly, many existing social media language understanding benchmarks either focus on a single platform [19], [20] or a single task [21]–[23] and thus cannot provide a holistic evaluation. Finally, while there exist language models pretrained on multiple platforms in specialized domains such as scientific literature [24], biomedical text [25], [26], and electronic health records [27], some of the most popular social media language models [2], [19], [28]–[31] are explicitly pretrained on data from a single social media platform. It is unclear whether these models generalize well across social media platforms.

Towards a more comprehensive understanding of social media language across multiple platforms and multiple application scenarios, we propose a new benchmark and a recipe for training language models that accounts for the divergence between social media language and conventional language. Specifically, we consider social media language understanding in English and make the following contributions:

• We conduct a time-aligned comparison between the vocabulary (token) distribution of (1) posts from Twitter and Reddit and (2) that of mC4 [32], a conventional text corpus used to pretrain language models in many existing works [32]–[34]. We observe a substantial difference between the two distributions and also find that social media language changes twice as fast as conventional language (3).

• We compile a Social MedIa Language Evaluation (SMILE) benchmark that includes social media language data from four platforms (Twitter, Reddit, Yelp, and Civil Comments) across both classification and generation tasks organically arising from the social media domain. This newly compiled benchmark coupled with an evaluation protocol is a well-rounded toolkit for the evaluation of an LM’s social media language understanding ([sec:benchmark]).

• We provide an effective recipe for training LMs for social media language understanding backed by a large-scale empirical study conducted using the SMILE  benchmark and training regimen for T5-based architectures [4]. Our study suggests that by training a custom tokenizer and pretraining the model from scratch using a corpus of both social media and conventional language, we can improve performance by 4.2 points compared to a similarly-sized baseline model (5). We carry out additional ablation studies in 6.

It is worth noting that very large language models (LLMs) like PaLM [6], [35] exhibit emergent few-shot capabilities in understanding and generating different styles, modes, and dialects of language. Thus the challenges of social media language understanding outlined above may be less pronounced for this class of models. To that end, we outline some interesting research directions in 7. The central focus of our study, however, is models that are three orders of magnitude smaller in terms of parameters—i.e. 220M vs. 540B for the largest PaLM model—and follow a pretraining-fine-tuning workflow (2) rather than the few-shot learning setup.

The rest of the paper is organized as follows. 2 provides necessary background. 3 presents our comparison between social media language and conventional language. In [sec:benchmark], we describe the compilation of the SMILE  benchmark. Using the SMILE  benchmark, we report the findings of our empirical study on learning language models for social media language understanding in 5. We carry out related ablation studies in 6 and discuss future work in 7. Finally, we conclude the paper in 8.

2 Background↩︎

We describe a typical end-to-end workflow for training models for language understanding. This workflow proceeds in three steps: data preparation, tokenizer setup, and model training. We present these three steps in detail, describing related works and highlighting why they may be insufficient for social media language understanding.

2.0.0.1 Data Preparation

In this step, we gather data needed to produce a language model from scratch. Two types of data are needed: training corpora and evaluation benchmarks.

• Training corpora typically consist of text crawled from the web with filters applied for data quality. For example, the C4 corpus [4], which is widely used to pretrain language models for general language understanding, contains 750GB of meticulously filtered web text free from offensive language, paragraphs that are too short, and sentences that do not end in a terminal punctuation mark. While such filters may make sense for general language understanding, it is unclear whether these design choices are as suited to social media language understanding: units of social media text tend to be short and poorly punctuated, and understanding offensive language likely key to downstream social media tasks like online safety. In fact, as a result of filtering, the C4 corpus does not include any Twitter data. We further study this disparity between social media language and conventional language in 3.

• Evaluation benchmarks typically contain a number of small-scale datasets with labels for a particular task, e.g., sentiment classification. These datasets typically have splits: a train split used to fine-tune a model and a test split used to evaluate the fine-tuned model’s performance on that task. In social media understanding, a popular evaluation benchmark is TweetEval [19], which consists of seven classification tasks measuring the capacity of language models to understand tweets. Other evaluation benchmarks exist [31], [36]; however, existing benchmarks are skewed towards particular social media platforms or particular tasks. We seek to provide a more comprehensive measure with our newly-complied benchmark in [sec:benchmark].

2.0.0.2 Tokenizer setup

Tokenization is the process of segmenting strings of text into sequences of characters known as tokens. The set of possible tokens is determined in advance and constitutes the vocabulary of the language model, the basic units of the model’s language understanding. Our study relies on the SentencePiece model (SPM) [37], which selects frequently observed whitespace-agnostic tokens from a large training corpus. Given the discrepancy between social media language and the standard language of the web, one may conjecture that training a tokenizer specific to social media language could be helpful. However, related results from existing works in learning domain-specific language models are mixed [19], [25]. In 5 we investigate this conjecture further.

2.0.0.3 Model training

Pioneered by [3], masked language modeling (MLM) is an effective procedure of pretraining transformer based language models [38]. The setup asks the model to predict randomly masked tokens in sentences drawn from the pretraining corpus in a Cloze test fashion. Once the model is pretrained, it is further trained (i.e. fine-tuned) on the evaluation benchmark train split. The fine-tuned model is then evaluated on the benchmark test split to measure its performance on a particular task. There are a few existing language models for social media language understanding. For example, [2], [19], [28], [30], [31] reported language models trained on Twitter text. Meanwhile, MentalBERT [29], a langauge model for mental-health-related tasks, is trained on Reddit data. In general language understanding, [35] and [6] use 50% social media content in pretraining LaMDA and PaLM, respectively. As mentioned in 2, this class of models is not in scope for this study.

3 Comparing Social Media Language with Conventional Language↩︎

In this section, we empirically study the difference between conventional and social media language to motivate our subsequent exploration of model training recipes. Our findings suggest that (1) the language used on social media is significantly different from the standard language of the web (3.1) and (2) social media language changes twice as fast as conventional language (3.2).

3.0.0.1 Datasets

To represent social media language, we compile a collection of English-only text posts from publicly-crawlable Twitter (80%) and Reddit (20%) pages (T&R). To represent conventional language we use the English portion of mC4, C4’s multilingual descendant that is also widely used to pretrain language models for general language understanding [32]. We focus on the subset of T&R and English mC4 that overlap in post/document publish time: between April 2020 and August 2020. We then split the two corpora into five segments representing each month, sampling 4M posts/documents from each split.

3.1 Vocabulary Difference between Conventional and Social Media Language↩︎

We quantify the difference in social media language and conventional language by measuring the difference in token distribution between T&R and the English mC4 corpus.

3.1.0.1 Metrics

We compute the symmetric KL divergence (SKL, a.k.a. population stability index or PSI) between token distributions in each five monthly segments of T&R and English mC4 as a measure of their difference. SKL ranges from \(0\) to \(+\infty\). [39] suggests that, as a rule of thumb, an SKL smaller than \(0.1\) implies little difference, an SKL between \(0.1\) and \(0.25\) implies moderate difference, and an SKL higher than \(0.25\) implies a significant difference between two distributions. We use this rough rule of thumb to interpret the SKL divergence results. Following a related study [16], we also report Jaccard distance (JD) between each two vocabularies (token sets). Jaccard distance ranges from \(0\) to \(1\). While various other approaches are proposed in computational linguistics [17], [40]–[42] to compare corpora, we focus on the comparison between token distributions because our downstream workflow takes tokens as input, and hence the changes in token distribution directly influence subsequent language modeling.

3.1.0.2 Protocol

For each of the five month-segments of T&R and English mC4, we train an SPM tokenizer with a vocabulary size of 50,000 to compute the token frequency distribution for that subcorpus. For each segment of T&R and segment of English mC4 corresponding to the same month, we take the tokens that are in either vocabulary along with their frequencies to obtain two categorical distributions over the same set of tokens. We then calculate the SKL divergence between these two distributions. To calculate the Jaccard distance, we count the number of tokens in the intersection \(i\) and union \(u\) of the two segments of the same month from T&R and English mC4. The Jaccard distance is then given as \(1-i/u\). Note that our time-aligned comparison controls for temporal confounding, hence providing a more accurate perspective on the difference between T&R and English mC4.

3.1.0.3 Results

1 reports the token distribution difference between T&R and the English mC4 corpus over five months. In terms of SKL, the divergence between the two is consistently higher than \(2.8\), a significant difference. The consistently high Jaccard distance (\(>0.65\)) between T&R and English mC4 corroborates this finding. Thus we conclude that social media language is significantly different from conventional language. These empirical observations validate our intuition and motivate the need for a social media language modeling recipe distinct from the status quo.

3.2 Temporal Vocabulary Shift in Social Media Language and Conventional Language↩︎

We quantify the temporal vocabulary shifts in social media and in conventional language by measuring the month-to-month token distribution shift in T&R and English mC4 and compare the rate at which these two modes of language change over time.

3.2.0.1 Metrics and Protocol

For both T&R and English mC4, we compute SKL divergence and Jaccard distance between the token distributions of adjacent months to measure. The calculation of these metrics follows the same protocol as in 3.1. The five monthly segments result in four month-to-month SKL divergence and Jaccard distance statistics for both T&R and English mC4.

3.2.0.2 Results

1 presents the month-to-month SKL and Jaccard distance for T&R and English mC4. In terms of SKL, we observe significant month-to-month changes (\(>0.25\)) of the vocabulary in T&R. Meanwhile, the changes in the English mC4 corpus are less significant (\(<0.25\)). In terms of both month-to-month SKL divergence and Jaccard distance, the numbers for T&R are roughly twice as high as for English mC4. These observations suggest that social media language changes faster than conventional language.

4 The Social Media Language Evaluation (SMILE) Benchmark↩︎

Motivated by the distinction between social media language and conventional language observed in 3, we compile the Social MedIa Language Evaluation (SMILE) benchmark that aims to provide a more comprehensive assessment of social media language understanding by improving platform and task coverage. The SMILE  benchmark consists of eleven English-only tasks drawing from five datasets that are derived from four source platforms: Twitter, Yelp, Reddit, and Civil Comments¹ (a now defunct comment hosting service for web publications). The SMILE  benchmark covers both classification and generation tasks in the areas of self-expression, opinion discovery, and online safety. All eleven tasks are publicly available on TensorFlow Datasets² (TFDS) as listed in 2. In what follows, we describe each dataset (4.1) and the summary evaluation metrics (4.2) to be used with SMILE. In 4.3, we investigate whether language in SMILE  tasks is more similar to conventional or social media language.

4.1 Datasets↩︎

1 summarizes key characteristics of the SMILE  datasets. Below we describe each dataset in more detail and discuss the evaluation metrics of each task.

4.1.0.1 TweetEval (TE, [19])

SMILE  includes all seven TweetEval classification tasks (emoji prediction, emotion recognition, hate speech detection, irony detection, offensive language identification, sentiment analysis, and stance detection) and uses the same performance metrics for each task as TweetEval. The seven tasks were derived from SemEval³ challenges between 2016 and 2019.

4.1.0.2 Civil Comments Toxicity (CCT, [23])

This dataset contains news-site user comments collected between 2015 and 2017 labeled for toxicity classification by human raters. A comment is considered toxic if at least one rater labeled it as such. Under this definition, \(30\%\) of the comments in the dataset are toxic. SMILE  uses accuracy and F1 as the performance measures for this task.

4.1.0.3 Yelp Review Polarity (YRP, [20])

Yelp Review Polarity dataset includes Yelp reviews from 2015 labeled positive or negative according to the user star ratings. The dataset is balanced and uses accuracy and F1 as the evaluation metrics. While it is arguable whether online reviews should be considered social media, 4.3 shows that Yelp reviews have more in common with social media language than with conventional language.

4.1.0.4 Reddit TIFU (RTIFU, [22])

RTIFU is a weakly supervised abstractive summarization task derived from the r/tifu subreddit⁴ crawled between 2013 and 2018. The task is to generate a post’s title given that post’s body content. RTIFU is the only SMILE  task suitable for measuring generation capabilities of language models in the social media domain. As the metric for this task, SMILE  uses ROUGE-1 score [43], which ranges between 0 and 1, with higher values indicating better performance.

4.1.0.5 GoEmotions (GE, [21])

The GoEmotions dataset includes snippets from Reddit posts published between 2005 and 2019 labeled with one or more of 27 possible emotional categories. Compared to TweetEval Emotion where each example belongs to only one of four emotional categories, the GoEmotions task involves multiclass classification on a much finer scale. SMILE  uses the macro-averaged (MA) F1 score across all categories as the performance metric for this task.

4.2 Summary Evaluation Metrics↩︎

To measure the overall performance of language models on SMILE we use three summary evaluation metrics that aggregate over the performance score of each the eleven tasks in SMILE . The summary metrics are: the task macro-averaged performance, the platform macro-averaged performance, and the TweetEval macro-averaged performance.

4.2.0.1 Performance Scores

To construct a summary metric, we first compute a performance score for each of the eleven tasks in 1. Note that most tasks only use a single evaluation metric to measure performance. We use these single metrics scaled by \(100x\) as their performance scores. For CCT and YRP where both accuracy and F1 are used for evaluation, we compute the average between accuracy and F1 scaled by \(100x\) as the performance score following the practice of [9], [10]. Note that all the evaluation metrics in 1 range between \(0\) and \(1\). The overall performance score of each task hence ranges between \(0\) and \(100\).

4.2.0.2 Task Macro-Averaged Performance (TMA)

This is the simple average of the performance scores across all the eleven tasks in SMILE. Consequently, this score assumes that each task in SMILE  is equally important in measuring the performance of language models for social media language understanding. We refer to this metric as the SMILEscore.

4.2.0.3 Platform Macro-Averaged Performance (PMA)

Because the majority of the SMILE  tasks are Twitter-based, it is arguable that the task macro-averaged performance overweights one platform. To mitigate confounding effects of different social media platforms, SMILE  also uses a per-platform macro-averaged performance score as a summary evaluation metric. To compute this metric, the performance scores of the tasks from the same platform are first averaged, yielding four per-platform average performance scores. The platform macro-averaged performance is then computed by taking the mean of these four per-platform scores.

4.2.0.4 TweetEval Macro-Averaged Performance (TEMA)

This summary evaluation metric is a simple average of the performance scores across all the TweetEval tasks. This is the same summary metric used in the TweetEval benchmark. We include this metric as several key existing works [2], [31] evaluate performance on the TweetEval benchmark.

4.3 Similarity between SMILE and the Pretraining Corpora↩︎

In this section, we seek to understand whether SMILE  tasks share more resemblance with social media language than with conventional language. To this end, we compute the difference between five subsets of SMILE  and the August 2020 segments of T&R and the English mC4 corpus introduced in 3. Following the protocol established in 3, we use SKL divergence and Jaccard distance between token distributions to measure how the language in SMILE  tasks differs from T&R and English mC4. The results are summarized in 3.

From 3, SKL and JD are high between all five SMILE subsets and both T&R and English mC4. Nonetheless, the five SMILE  subsets are less different from T&R than they are from English mC4 in terms of SKL. This may suggest pretraining on a corpus of social media language as an avenue to improving benchmark performance (as supported by subsequent experiments in 5). Meanwhile, the high SKL (\(>0.25\)) could imply that the capacity of the pretrained language models in domain adaptation may still be a key factor in determining downstream benchmark performance. Finally, the comparison results for the Yelp dataset in 3 also suggest that Yelp reviews are more similar to social media language than to conventional language, despite user reviews not being considered the prototypical example of social media.

5 Social Media Domain Adaptation↩︎

In this section, we explore adaptation of language models to the social media domain. We carry out a large scale empirical study of training regimens for producing language models that work well on social media text as proxied by the SMILE benchmark introduced in [sec:benchmark]. Our winning recipe adapts a general T5 language model [4] by

The resulting SociAl Media langUage modEL (SAMUEL) can outperform the best alternative of similar size and pretraining budget on the SMILE  benchmark by 4.2 points.

In what follows, we first highlight the key results of comparing SAMUEL with alternative language models in terms of the performance on SMILE  (5.1). We then provide a detailed description of the empirical study that informed SAMUEL’s design. Specifically,

• In 5.2, we introduce a social media language corpus of publicly available social media text that includes data from four major social media platforms (Twitter, Reddit, Facebook, and Telegram).

• In 5.3, we show that tokenizer setup and language model pretraining can benefit from the social media language corpus compared to using a corpus of conventional language.

• We consider the standard domain adaptation practice of continual pretraining in 5.4.

• Finally, we explore strategies for mixing social media and conventional language data as alternatives to continual pretraining (5.5), completing the design of SAMUEL’s training recipe.

In addition, to understand whether language models adapted for social media understanding can maintain decent performance in general language understanding, we report results on the GLUE [9] and SuperGLUE [10] benchmark in the Appendix (9.2).

5.1 Performance of SAMUEL and Alternatives on the SMILE Benchmark↩︎

We describe SAMUEL and compare its performance on the SMILE benchmark with multiple representative competing language models.

5.1.0.1 SAMUEL

SAMUEL is a T5-based language model adapted for social media understanding. SAMUEL is pretrained on a corpus of both social media language and conventional language. We discuss the choice of architecture, training corpus, tokenizer setup, and pretraining procedure for SAMUEL below.

• Architecture. SAMUEL is a based on the T5 [4] architecture. It is a transformer-based [38] encoder-decoder model that is capable of tackling both classification and generation tasks as required by SMILE. It does so by processing the input and output of the tasks as free form text. Following the T5 1.1.⁵ implementation, SAMUEL has a parameter size of 220M that matches the size of T5 1.1. base.

• Pretraining Corpus. SAMUEL is pretrained on a corpus that is \(80\%\) social media text and \(20\%\) conventional web text. The social media language portion is drawn from the corpus described in 5.2. The conventional language portion is drawn from the C4 corpus that was used to pretrain the original T5 model.

• Tokenizer Setup. SAMUEL’s tokenizer is an SPM with 32k tokens. It is trained directly on the pretraining corpus. As a result, the tokens selected are drawn from both social media and conventional language.

• Pretraining Procedure and Hyperparamters. SAMUEL is first pretrained on conventional language and then on social media language. For SAMUEL and for other pretraining experiments in this paper, we pretrain all models using the span corruption objective [4] on tensor processing unit (TPU) pods with batch size of 2048, sequence length of 512, and a total of \(2^{18}\) steps with input packing. We use the Adafactor optimizer with an reciprocal square root decay learning rate schedule.

5.1.0.2 Competing Language Models

We compare five representative language models with SAMUEL on SMILE. Because SAMUEL is T5-based, we consider a variety of other T5-based language models in our comparison. We also consider BERTweet, a state-of-the-art language model that specializes in tweet understanding. We describe each of the competing language models below and provide a summary in 10 in the Appendix.

• T5 1.1. base [4] shares the same parameter and tokenizer size as well as the same pretraining hyperparamters as SAMUEL. However, both the model and the tokenizer are trained exclusively on the C4 corpus. It differs from the off-the-shelf version of T5 1.1 base because it was pretrained on 4x as much data to match SAMUEL’s pretraining budget.

• mT5 base [32] is an off-the-shelf multilingual variant of T5. It has 580M parameters and is pretrained on mC4 for 1M steps. This model’s SPM has 250k tokens including 256 byte tokens, which means no string of text is out of vocabulary for mT5. We explore the byte fallback feature of SentencePiece tokenizers in more detail in 6.2.

• byT5 base [33] is an off-the-shelf byte-level model that closely follows the T5 setup. Instead of relying on a SentencePiece model for tokenization, byT5 simply uses bytes as tokens to flexibly represent text. We hypothesize that such flexibility may be beneficial for representing text in the social media domain, given its noisy, informal, and dynamic nature.

• T5 1.1 XXL [4] is an off-the-shelf large language model with 11B parameters, which is more than 40x larger than SAMUEL. While a head-to-head comparison between T5 XXL and SAMUEL may not be fair because of the discrepancy in parameter size [44], we nonetheless include this model to estimate performance headroom.

• BERTweet [28] is a language model for social media understanding that achieves state-of-the-art performance on the TweetEval benchmark⁶. BERTweet is a 110M-parameter encoder only model and hence cannot perform generation tasks. We only report its performance on TweetEval.

5.1.0.3 Protocol

After pretraining SAMUEL, we fine-tune SAMUEL and the five alternative models on each SMILE  task separately for 10k steps and report the performance metrics on the test splits. To get a sense of the uncertainty in performance metrics, we fine-tune SAMUEL on each task 12 times and compute the standard deviation for each metric. Note that we presume the performance of different models to share similar levels of uncertainty and do not compute the standard deviation for each model because of the high computational cost of doing so. We run all experiments using the T5X framework [45].

5.1.0.4 Results

4 summarizes the performance comparison of SAMUEL against other language models on SMILE. With the exception of T5 XXL, SAMUEL outperforms its similar-sized alternatives by at least 4.2 points in terms of task macro-averaged (TMA) score. This suggests the practical utility of the recipe behind SAMUEL in building effective language models for social media language understanding. Moreover, SAMUEL’s score is only 0.2 points below that of T5 1.1 XXL with 50 times more parameters, which suggests that SAMUEL may be nearing the upper limits of performance headroom on this benchmark.

5.2 Social Media Corpus↩︎

In 3, we observe that social media language is different from conventional language. Meanwhile, commonly used pretraining corpora, like C4 in the case of off-the-shelf T5, represent conventional language. We begin our adaptation of T5 to social media language understanding with compiling a large social media (SM) corpus.

5.2.0.1 Protocol

We parse out post text from publicly crawlable Twitter, Reddit, Facebook, and Telegram pages. We filter out posts with images, videos, or URLs because the text of such posts may not contain the entirety of their meaning and therefore may be too difficult to learn from in the span corruption setting. To mitigate the risk of inadvertently training any models on posts from the benchmark test sets, we only include posts published in 2022 to avoid temporal overlap (the content used in SMILE  was published between 2015 and 2020).

5.2.0.2 Results

The resulting corpus has 6B examples. Its platform composition is 51% Twitter, 26% Reddit, 23% Facebook, and 0.1% Telegram. In contrast, the C4 corpus contains only \(0.005\%\) Reddit data and no data from the other three platforms. In addition, while C4 has 16 times fewer examples, C4 examples are on average 10 times longer than the examples in our social media corpus. The presence of short examples makes it important to minimize padding with input packing.

5.3 In-Domain Tokenization and Pretraining↩︎

We consider the effect of using the SM corpus defined in 5.2 for tokenizer setup and LM pretraining on the model’s capacity for social media language understanding.

5.3.0.1 Protocol

Keeping the T5 1.1 backbone fixed, we modify its tokenizer and pretraining regimen separately and then together. We measure the impact on the SMILE  score.

5.3.0.2 Results

5 and 6 show the metric improvement compared to the baseline model—a T5 1.1 model where both the tokenizer and the model are trained on the C4 corpus. Unsurprisingly, including in-domain data helps the performance on most tasks. Swapping out the standard tokenizer with one trained on SM data and then pretraining on C4 leads to a 1.8-point gain on the overall SMILE  score (see 15 in the Appendix for examples), while using SM pretraining data with a standard tokenizer adds 2.8 points. These performance gains are not exactly additive when both the model’s tokenizer and parameters are trained on the in-domain data, but the combination does lead to the biggest improvement—3.7 points to the TMA. It is worth noting the tasks that buck the overall trend. Reddit TIFU, the only summarization task, appears to be hurt by a tokenizer trained on in-domain data, whether or not the rest of the model is pretrained on SM or C4. In-domain pretraining with a C4 tokenizer does improve the ROUGE-1 score, but barely more than a single standard deviation for this task (+0.25 vs 0.22), all despite the fact that the token distribution of this dataset is closer to SM data than it is to clean web text (3). The opposite effect can be observed for the TweetEval Emotion task—there, the in-domain tokenizer alone yields a 6.1 point gain, 1.5 times larger than from in-domain pretraining.

5.4 In-Domain Continual Pretraining↩︎

While in 5.3 we established that using social media data in tokenizer and model training significantly improves performance on the SMILE  benchmark, this recipe requires retraining the model from scratch, which is undesirable in practice because of engineering effort, data, and computation required. A much more popular and practical recipe is continual pretraining, where an existing pretrained model is adapted to a different domain or task via more pretraining steps. In this section, we investigate how well continual pretraining can perform in the social media domain.

5.4.0.1 Protocol

We take a T5 model with a standard C4 tokenizer pretrained on C4 for \(2^{17}\) steps and continue training it on the same span corruption task but on our four-platform SM corpus for another \(2^{17}\) steps, yielding a total of \(2^{18}\) pretraining steps.

5.4.0.2 Results

5 and 6 report the results of our continual pretraining experiment. We see that continual pretraining does not help as much as training a T5 1.1 backbone on SM from scratch (+2.3 vs +2.8). This finding is in line with [25]. However, it is remarkable that continual pretraining achieves \(80\%\) of the gain afforded by from-scratch in-domain pretraining—while using half as much in-domain data, in half as many steps. Part of the reason may be that the model is able to leverage the foundation derived from the clean language of its original pretraining corpus, C4, despite the fact that it is much further from social media language in token distribution.

5.5 Mixed-Domain Pretraining↩︎

Inspired by the relative success of the continual pretraining recipe in 5.4, we drill down into the idea of combining in- and out-of-domain data into mixed-domain data to further improve the performance. In particular, we consider various mixing schedules and ratios.

5.5.1 Mixing Schedules↩︎

5.5.1.1 Protocol

We try three different mixing schedules for C4 and our SM corpus (see 2):

As we learned from 5.3, updating the tokenizer to match the target distribution contributes significantly to the model performance. We train a tokenizer on a \(50/50\) mix of C4 and SM to match the pretraining data and pretrain three models using the schedules described above.

5.5.1.2 Result

7 shows that the model trained on the sequential mixture performs best. Note that though its schedule is identical to continual pretraining, the model is different because it has a custom tokenizer and was pretrained from scratch. We also note that the dynamic mixture is very close in performance to sequential: 66.55 vs 66.38 on the overall score, which has a standard deviation of 0.14. However, pretraining with a dynamic schedule is much more challenging to set up from the infrastructure perspective than the sequence mixture. We hence commit to the use of sequential mixture schedule in our subsequent investigation.

5.5.2 Mixing Rates↩︎

5.5.2.1 Protocol

Using the sequence mixture as the schedule, we then try three different SM/C4 ratios: 20/80, 50/50 and 80/20. We update the tokenizers to match the ratios and pretrain two more models.

5.5.2.2 Results

The best overall model is trained on 20% C4 and 80% SM in sequence, with an overall score of 66.63. (That corresponds to our best model, SAMUEL, in 4). Interestingly, different benchmark tasks react to the rate of in-domain data differently (see 12 in the Appendix). For TweetEval Emotion, the more in-domain data, the better—its performance peaks when the model is trained exclusively on SM data. On the other hand, Yelp polarity classification benefits roughly equally from any proportion of in-domain pretraining, while Reddit TIFU summarization benefits from more C4 pretraining, even though it is out of domain.

6 Ablation Studies↩︎

We conduct ablation studies on the platform composition of our SM corpus and the effect of the byte-level fallback feature of SPM tokenizers to see how these aspects impact social media language understanding.

• In 6.1, we evaluate performance contributions of using data from different SM platform and conclude that a variety of platforms in the training data yields optimal performance on the SMILE benchmark.

• In 6.2, we consider adding byte tokens to the model vocabulary, which should intuitively help parse noisy social media text, and find that while byte fallback does indeed help the model trained on conventional language, it no longer has a positive effect when the model is trained on SM data.

6.1 Cross-Platform Transfer↩︎

Given the multi-platform nature of the SMILE  benchmark, we investigate how using the four-platform mixture compares to training the model on each platform separately. For example, we presume Reddit data will be helpful on the Reddit-based tasks, but how well will it transfer to tasks from other platforms?

6.1.0.1 Protocol

To answer this question, we train three more T5 models using the best performing recipe from 5.3 but on single-platform slices of the same SM corpus — Twitter-only, Reddit-only, and Facebook-only. The amount of Telegram data in the original SM corpus is negligibly low—about \(0.1\%\) and only 7M examples—so we can not train a Telegram-only model. The three single-platform models see the same amount of data despite different average example lengths thanks to input packing.

6.1.0.2 Results

The results for per-platform average performance scores as well as the PMA are shown in 8. Overall, we see that single-platform models perform very similarly to each other as well as the four-platform mixture ("All SM"). Though the four-platform mixture has the highest score on the PMA, it is within one standard deviation from the second place—the Twitter-only model. The Twitter-only model also, unsurprisingly, does best on the Twitter portion of the SMILE  benchmark. On the other hand, the Reddit-only model takes the third place on the Reddit portion of the benchmark, lagging behind the four-platform mixture as well as the Facebook-only model. We conclude that though the scores are very close, having a mix of platforms in the pretraining data results in beneficial transfer across platforms.

6.2 Byte-level Fallback↩︎

We evaluate the impact of including byte tokens in the model vocabulary. Also known as byte-level fallback, this feature allows the tokenizer to segment any string of text without resorting to the <UNK> token. Can it improve performance on social media text with its typos, abbreviations, and emojis?

6.2.0.1 Protocol

To answer this question, we replace the 256 least frequent tokens in the standard C4 SPM with tokens representing the 256 bytes and use the modified tokenizer to pretrain a model on C4. We perform the same vocabulary surgery on a model whose tokenizer and parameters are trained on social media data.

6.2.0.2 Results

The results are shown in 9. Overall, even though the C4 model with byte fallback did not see a single example from the social media corpus, its performance on the SMILE  benchmark is 0.87 points (\(\sigma = 0.14\)) better than the equivalent model without byte fallback. We speculate that allowing the tokenizer to fall back to byte tokens does indeed help the model handle emojis and typos that otherwise would not be represented by the vocabulary. However, for the model trained on social media data, replacing 256 organically selected tokens with bytes no longer has a positive effect (+3.73 vs +3.47).

7 Future Work↩︎

We envision the following directions as future work.

• Internationalization. Our SMILE  benchmark spans multiple platforms and tasks but is currently English only. An important opportunity for future work is the extension to other languages, ideally covering locales where social media usage is most pronounced and locales with multilingual content.

• Multi-modality. Social media content is inherently multi-modal and context dependent. Relevant modalities can be content based (text, image, etc.) but also include creators, communities, and any other aspects of a social graph. Future opportunities lie both in exploring techniques for modeling multi-modal content and in providing evaluation benchmarks for this setting.

• Time Sensitivity and Adaptability. Social media language changes faster than conventional language (3.2). Consequently, it would be valuable to define time-sensitive tasks that measure the ability of models to adapt to these shifts. Byte-level language models [33], [34] may be of particular interest in tackling distribution shifts because they are not constrained by a fixed vocabulary and can be fully fine-tuned (i.e. including the tokenizer). Efficient architectures [46] that hinge on the principle of sparsity [47]–[54], among other principles [55]–[57], can be particularly relevant given the longer sequence length induced by byte representation of the data.

• Domain Adaptation vs. Model Scaling. Given the popularity of very large language models, it would be beneficial to study how scaling up the model size affects the gains from domain adaptation. At what model size, if any, does the performance improvement afforded by domain adaptation cease to be worth the computationally expensive pretraining regimen?

8 Conclusion↩︎

We have shown that the language of social media is significantly different from the standard language of the web and proposed the SMILE benchmark that captures the peculiarities of this distinct domain. Through pretraining experiments with T5, we came up with simple recipes for improving performance on the benchmark. We believe it can open up opportunities for future research in adapting language models to the social media domain.

9 Appendix↩︎

In the appendix, we include additional experiment details (9.1) as well as evaluation results on GLUE and SuperGLUE (9.2). We also provide concrete examples demonstrating the potential benefits of training the tokenizer and the model parameters on social media data compared to their generic counterparts in 9.3.

9.1 Additional Experiment Details↩︎

9.2 GLUE and SuperGLUE Experiments↩︎

To understand performance on standard NLU tasks, we report detailed results on the GLUE ([9] and SuperGLUE ([10]) benchmarks in 13 and 14 respectively. All models were pretrained for \(2^{18}\) steps, including our version of T5 1.1 (which thus is different from the publicly available version). We trained each model on a mixture of all tasks with proportionate sampling (GLUE and SuperGLUE separately) for 50k steps with a batch size of 128 and a dropout rate of 0.1.

9.3 Examples of Social Media Language Tokenization↩︎

In 15 we present three real-world examples below to illustrate the misalignment between conventional and social media language as well as the resulting difference in predictions made by the T5 1.1. baseline and SAMUEL (trained on conventional and social media language, respectively). These examples are drawn from the TweetEval 4-way Emotion classification task.

9.3.0.1 On interpreting tokenization results

Each group of symbols in square brackets represents a single token. There are two special symbols: "▁", which represents a word-separating character like a space, and "<UNK>", which represents an out-of-vocabulary token the tokenizer resorts to when it cannot break down the input string into any of its known 32k tokens.

9.3.0.2 Discussion

As one can see in 15, T5 1.1. struggles with hashtag tokenization and fails to understand semantically important emojis. In example 1, T5’s tokenizer (trained on C4’s clean text) tokenizes properly space-separated text well (e.g. [▁worry]) but when the same text appears inside a hashtag, it gets mangled ([▁#] [w] [or] [ry]). In contrast, SAMUEL’s tokenizer is capable of tokenizing hashtags in a manner that better aligns with human intuition because it learns not to assume that words will be properly space-separated ([▁#] [worry]). SAMUEL’s tokenizer is also capable of identifying semantically important emojis ( and in examples 2 and 3), while T5 does not because they are unknown to its tokenizer. These examples show that the token distribution differences reflect real distinctions in language use between social media and the standard web. In addition, the vocabulary learned from these distinct distributions influences downstream task performance: in all three examples, SAMUEL predicts the correct labels while T5 1.1. fails to do so.

References↩︎