1 Introduction↩︎

In today’s LLM-driven era, BERT-based models [1], [2] remain essential for tasks that demand high precision and low latency, such as span labeling, classification, and information retrieval. Their lightweight nature makes them particularly well-suited for resource-constrained applications. Rather than competing with large language models (LLMs), BERT-based models often serve as core components in hybrid systems, delivering strong performance at a fraction of the computational cost while effectively complementing LLMs [3]. As a result, the BERT family continues to evolve, with recent additions such as ModernBERT [4] and NeoBERT [5]. While English benefits from a rich ecosystem of pre-trained BERT-based models, the development of Vietnamese counterparts remains comparatively limited. The multilingual model XLM-RoBERTa [6] achieves competitive performance on a wide range of Vietnamese NLP tasks by leveraging 138GB of CC100 Vietnamese text in its multilingual pre-training corpus. PhoBERT [7] was the first large-scale monolingual BERT model pre-trained from scratch specifically for Vietnamese, using 20GB of text. Subsequent general-domain monolingual models include viBERT and vELECTRA [8], pre-trained on 60GB of Vietnamese text, and ViDeBERTa [9], which uses the same 138GB CC100 Vietnamese corpus as XLM-RoBERTa. More recently, CafeBERT [10] was continually pre-trained from the XLM-RoBERTa “large” model on 18GB of Vietnamese text collected prior to 2021. Several domain-specific monolingual models have also emerged: VnLawBERT [11] for legal text, ViHealthBERT [12] and ViPubmedDeBERTa [13] for health and biomedical text, and ViSoBERT [14] for social media text.

Among these monolingual models, PhoBERT has become the default choice for many Vietnamese NLP tasks thanks to its strong and consistent performance. Since its release, it has gained widespread adoption, with over 200K monthly downloads on HuggingFace and active use across the NLP community, whereas all other Vietnamese monolingual models receive fewer than 5K monthly downloads each. Despite its popularity, PhoBERT has several limitations: it supports only a short maximum context length of 256 subword tokens, and it requires input text to be word-segmented by an external tool prior to processing. These limitations motivate the development of a new Vietnamese BERT-based model that supports a longer context and operates directly on raw text.

In this paper, we introduce BamiBERT—a new pre-trained language model for Vietnamese—trained from scratch on a large corpus of 129GB of uncompressed text for 20 epochs, with an extended maximum context length of 2048 tokens. Unlike PhoBERT, which requires Vietnamese text to be pre-segmented by an external word segmenter, BamiBERT operates directly on raw input text, making it more flexible and easier to integrate into a wider range of downstream applications. Experimental results on 8 Vietnamese benchmark datasets show that BamiBERT delivers state-of-the-art or near-state-of-the-art performance (ranking #1 on 11/15 metrics, #2 on 3/15 metrics, and #3 on the remaining one), demonstrating strong cross-domain generalization. We release BamiBERT at: https://huggingface.co/Qualcomm-AI-Research/BamiBERT.

2 Pre-trained language model BamiBERT↩︎

This section presents how we pre-train our new BERT-based language model from scratch.

2.0.0.1 Architecture:

We pre-train a text encoder, named "BamiBERT",² from scratch, employing the BERT’s "base" architecture with 12 Transformer block layers [1]. To pre-train BamiBERT, we use the masked language modeling objective [1] and the RoBERTa pre-training approach [2] which optimizes BERT with a dynamic masking strategy and without the next sentence prediction objective. For tokenization, we extend the PhoGPT’s Vietnamese-specific byte-level BPE tokenizer [15] with an additional "<mask>" token, resulting in a final vocabulary of 20481 token types. We set a maximum sequence length of 2048.

2.0.0.2 Pre-training dataset:

We use a clean, 129 GB dataset of uncompressed, general-domain text.

2.0.0.3 Optimization:

The model is optimized using Adam [16]. We use a batch size of 1024 sequence blocks distributed across 8 A100 GPUs (each with 40GB of memory) and a peak learning rate of 0.00015. The pre-training process runs for 20 epochs, with the initial 2 epochs dedicated to warming up the learning rate.

3 Experiments↩︎

3.1 Setup↩︎

0.3em

We conduct experiments to compare our model BamiBERT with the previous strong and public pre-trained "base"-architecture ones for Vietnamese, including: Vietnamese-specific models ViDeBERTa-base [9], ViSoBERT [14] and PhoBERT-base [7] as well as the multilingual XLM-RoBERTa-base [6].³ Here, BamiBERT, ViSoBERT and XLM-RoBERTa take raw texts as input, while ViDeBERTa and PhoBERT are Vietnamese word-level models. That is, a Vietnamese word segmentation tool must be applied to produce word-segmented texts before feeding them to the word-level ViDeBERTa and PhoBERT. For ViDeBERTa and PhoBERT experiments, we utilize the RDRSegmenter component [17] from the VnCoreNLP toolkit [18] for Vietnamese word segmentation.

We employ the following experimental benchmark datasets: ViNLI—a Vietnamese dataset for open-domain natural language inference [19], PhoNER_COVID19—a dataset for recognizing COVID-19 related named entities in Vietnamese [20]; UIT-VSFC (Sentiment) and UIT-VSFC (Topic)—Vietnamese students’ feedback benchmarks for sentiment-based and topic-based classifications [21]; ViSpamReviews—a dataset for spam review detection on Vietnamese e-commerce websites [22]; UIT-ViSFD—a Vietnamese aspect-based sentiment analysis dataset of feedbacks and comments for smartphone e-commerce [23]; and UIT-ABSA (Hotel) and UIT-ABSA (Restaurant)—Vietnamese aspect-based sentiment analysis datasets for hotel and restaurant domains [24]. ViNLI and PhoNER_COVID19 are based on general-domain texts, whereas the remaining benchmarks are derived from social media and forum discussions. See Table 1 for the statistics of these datasets.

For all experimental models, we employ transformers [25] to fine-tune them using the AdamW optimizer [26] and set the batch size to 32. We also perform a grid search on the validation set to select the initial learning rate for AdamW from {1e-5, 2e-5, 5e-5}. We train for 30 epochs on the training set, compute F₁ on the validation set after each training epoch, and select the model checkpoint with the best F₁ to report final metric scores on the test set.

3.2 Main Results↩︎

Table 2 reports the performance of BamiBERT and four baselines—ViDeBERTa, ViSoBERT, XLM-RoBERTa, and PhoBERT—across eight Vietnamese benchmarks. BamiBERT achieves the best performance on 11 of the 15 evaluation metrics, ranks second on three metrics, and ranks third on the remaining metric, establishing a new state of the art for "base"-sized Vietnamese BERT-based language models.

3.2.0.1 ViNLI

BamiBERT achieves the best performance on both metrics (81.01 Accuracy and 81.15 F₁), yielding substantial absolute gains of +3.01 Accuracy and +3.10 F₁ over the second-ranked PhoBERT (78.00/78.05). XLM-RoBERTa follows in third place (76.83/77.01), trailing PhoBERT by roughly one point on both metrics. ViSoBERT (67.70/67.82) and ViDeBERTa (61.08/60.71) lag considerably behind, with gaps of 13–20 points relative to BamiBERT.

3.2.0.2 PhoNER_COVID19

BamiBERT obtains the highest F₁ score (94.90), outperforming ViDeBERTa by 0.40 points and PhoBERT by 0.70 points. Its advantage widens against the remaining baselines, reaching +2.0 F₁ over ViSoBERT (92.90) and +2.4 F₁ over XLM-RoBERTa (92.50).

3.2.0.3 UIT–VSFC (Sentiment)

With 93.86 Accuracy and 83.41 F₁, BamiBERT ranks among the top-performing models. It is essentially on par with PhoBERT, trailing by 0.24 Accuracy but leading by 0.14 F₁. BamiBERT maintains consistent margins over XLM-RoBERTa (+0.30 Accuracy, +1.21 F₁) and ViSoBERT (+0.71 Accuracy, +1.92 F₁), while ViDeBERTa underperforms substantially on both metrics.

3.2.0.4 UIT–VSFC (Topic)

BamiBERT again leads on both metrics (89.34 Accuracy and 79.90 F₁), outperforming PhoBERT by +0.10 on both, XLM-RoBERTa by +0.16/+0.34, and ViSoBERT by +0.54/+0.04. Although the top four models are tightly clustered (within 0.54 Accuracy and 0.34 F₁), BamiBERT remains the most consistent. ViDeBERTa, in contrast, trails markedly (83.94/66.49).

3.2.0.5 ViSpamReviews

BamiBERT ranks second overall (90.76 Accuracy and 78.20 F₁), narrowly behind ViSoBERT (90.99/79.06) by 0.23 Accuracy and 0.86 F₁. Nevertheless, it clearly outperforms XLM-RoBERTa (+0.60 Accuracy, +1.65 F₁) and PhoBERT (+0.93 Accuracy, +2.02 F₁), while ViDeBERTa lags well behind (86.21/67.04). BamiBERT remains highly competitive at the top tier and continues to surpass other established baselines by clear margins.

3.2.0.6 UIT–ViSFD

BamiBERT delivers the strongest performance on both subtasks. For aspect detection, it attains 89.14 F₁, ahead of ViSoBERT (88.63; \(-\)​0.51) and PhoBERT (86.03; \(-\)​3.11), with XLM-RoBERTa (82.73) and ViDeBERTa (75.53) trailing further. For aspect-based sentiment classification, BamiBERT achieves 84.24 F₁, again surpassing ViSoBERT (83.55; \(-\)​0.69) and PhoBERT (78.76; \(-\)​5.48). The consistent gains over PhoBERT (+3.11 and +5.48 points) underscore BamiBERT’s robustness across both subtasks.

3.2.0.7 UIT–ABSA (Hotel)

On aspect detection, BamiBERT obtains the highest F₁ (79.99), outperforming ViSoBERT (79.41; \(-\)​0.58) and PhoBERT (79.16; \(-\)​0.83), while XLM-RoBERTa (77.70) and ViDeBERTa (72.05) remain less competitive. On aspect-based sentiment classification, however, ViSoBERT takes the lead (74.24 F₁), followed by PhoBERT (73.73; \(-\)​0.51) and BamiBERT (72.65; \(-\)​1.59); XLM-RoBERTa (71.23) and ViDeBERTa (62.97) trail by a wide margin.

3.2.0.8 UIT–ABSA (Restaurant)

BamiBERT produces the strongest end-to-end performance, with 88.01 F₁ on aspect detection and 74.89 F₁ on aspect-based sentiment classification. These results exceed those of ViSoBERT (86.86/\(-\)​1.15 and 73.87/\(-\)​1.02) and PhoBERT (86.53/\(-\)​1.48 and 73.52/\(-\)​1.37), while XLM-RoBERTa (82.18/71.58) and ViDeBERTa (73.56/63.78) lag considerably behind.

4 Discussion↩︎

Overall performance Across 8 Vietnamese benchmarks and different subtasks (Table 2), BamiBERT consistently delivers SOTA or near-SOTA results. The most substantial improvement appears on ViNLI, where BamiBERT surpasses the next-best PhoBERT by +3.01 Accuracy and +3.10 F₁, while outpacing ViSoBERT/ViDeBERTa by 13–20 points—demonstrating strong sentence-pair semantics and cue-word sensitivity.

4.0.0.1 Domain effects

Note that PhoNER_COVID19 and ViNLI represent general-domain text, while the remaining benchmarks reflect social media and forum content. BamiBERT exhibits strong cross-domain generalization, outperforming or closely matching the social-media-focused ViSoBERT across both general-domain (e.g., ViNLI, PhoNER_COVID19) and social-domain benchmarks. Its consistent top-tier performance across diverse tasks—NER, span detection, and classification—demonstrates resilience to domain shift and label granularity. This robustness positions BamiBERT as a reliable choice for NLP pipelines operating under domain heterogeneity and distributional uncertainty.

4.0.0.2 Detection vs.classification

In aspect-based sentiment analysis pipelines, BamiBERT often excels in detection (e.g., Hotel: 79.99 F₁; Restaurant: 88.01 F₁), while its classification performance is strongest in the Restaurant domain (74.89 F₁) but trails ViSoBERT/PhoBERT in the Hotel domain (72.65 F₁). This pattern suggests complementary strengths: BamiBERT appears particularly effective at span/target localization and boundary-sensitive cues, whereas domain-specific sentiment nuances in the Hotel domain may benefit more from domain-adapted pretraining (ViSoBERT). Combined with the new SOTA results on UIT-ViSFD, this indicates BamiBERT’s robust end-to-end capability for fine-grained social content analysis.

4.0.0.3 Takeaway

BamiBERT delivers SOTA or near-SOTA performance across diverse Vietnamese benchmarks, excelling in NLI, topic classification, and fine-grained sentiment tasks. Its cross-domain stability and strong results on both detection and classification make it a robust default for real-world Vietnamese NLP applications.

5 Conclusion↩︎

In this paper, we have presented BamiBERT, a new pre-trained language model for Vietnamese designed to address key limitations of existing monolingual encoders. Unlike PhoBERT—the current de facto choice of Vietnamese text encoder—BamiBERT is trained from scratch on a 129 GB corpus of general-domain text for 20 epochs, supports an extended maximum context length of 2048 tokens, and operates directly on raw text, removing the dependency on external word segmentation. Experiments on 8 Vietnamese benchmark datasets show that BamiBERT does better than PhoBERT, establishing a new state of the art among “base”-sized Vietnamese encoders with strong cross-domain generalization.

References↩︎