1 Introduction↩︎

The development of generative artificial intelligence has settled into an era of massive parameter scaling [1]–[4]. However, the operational footprints of multi-billion parameter architectures heavily restrict their utility on decentralized consumer edge devices, mobile operating engines, and IoT infrastructures. This computational bottleneck is deeply exacerbated for low-resource languages that use non-Latin scripts [5], [6].

Bangla (Bengali), despite possessing over 300 million native speakers worldwide, remains structurally disadvantaged in mass multilingual foundation models [7]. Because Bangla tokens account for less than a fraction of a percent in high-register pretraining datasets, standard multilingual models display high token-to-word (fertility) rates on its unique Abugida (alphasyllabary) script [8]. This structural inefficiency creates an unfair "token tax" that results in inflated memory caches, high inference latency, and early context window degradation [8], [9].

To break this scale-dependency, we introduce bangla-smollm-135m, a compact foundational model of only 135 million active parameters engineered exclusively for native language understanding and reasoning. Our evaluation demonstrates that custom script adaptation, precise data filtering, and a novel cross-model tokenizer intersection paradigm allow ultra-compact systems to perform on par with much larger general-purpose multilingual architectures [8], [10], creating a strong benchmark for resource-constrained edge deployments.

2 Tokenizer Training and Matrix Cross-Alignment↩︎

Standard subword tokenization models optimize for the statistical distribution of character configurations in dominant languages [10]. When adapting highly compact architectures like SmolLM2-135M [11] to low-resource domains, training a vocabulary completely from scratch can destabilize early pretrained parameters [9]. Conversely, relying solely on an unadapted base model induces high fertility penalties (\(\Phi\)), which counts the subword slices (\(T\)) generated per standard word unit (\(W\)): \[\Phi = \frac{|T(W)|}{1}\]

To bridge this architectural gap, bangla-smollm-135m uses a structured programmatic merging pipeline to inject the localized vocabulary configurations of TituLLMs [12] (acting as the base vocabulary layer) directly into the token framework of SmolLM2-135M [11].

2.1 Algorithmic Intersect-and-Append Paradigm↩︎

Let \(\mathcal{V}_{\text{base}}\) define the token vocabulary space of the target regional model (TituLLMs), and \(\mathcal{V}_{\text{src}}\) define the source vocabulary space of the compact architecture (SmolLM2). The goal is to construct a unified embedding target vocabulary \(\mathcal{V}_{\text{merged}}\) that preserves all native subword patterns while retaining compatibility with cross-lingual embeddings [9].

The merge routine enforces a deterministic isolation pipeline:

Special Token Harvesting: Isolate control tokens, padding symbols, and chat-template configurations mapping to \(\mathcal{V}_{\text{src}}\). Any structural tokens missing from the target base are added to prevent syntax parsing errors during sequence decoding: \[\mathcal{S}_{\text{missing}} = \{ t \in \text{Special}(\mathcal{V}_{\text{src}}) \mid t \notin \mathcal{V}_{\text{base}} \}\]

Lexical Intersection and Regular Appending: Isolate standard text subwords from \(\mathcal{V}_{\text{src}}\) that are completely absent from the base set. These tokens are sorted alphabetically to maintain strict structural determinism and reproducibility during initialization: \[\mathcal{R}_{\text{candidates}} = \text{Sort}(\{ t \in \mathcal{V}_{\text{src}} \mid t \notin \mathcal{V}_{\text{base}} \cup \mathcal{S}_{\text{missing}} \})\]

Boundary-Constrained Expansion: The final vocabulary matrix expansion is limited by an upper bound parameter \(\mathcal{K}_{\max} = 48,000\) to maintain a highly compact hidden embedding configuration: \[\mathcal{R}_{\text{final}} = \mathcal{R}_{\text{candidates}}[0 : \mathcal{K}_{\max}]\]

The execution of this architecture is described algorithmically in Algorithm 1.

3 Model Training and Infrastructure↩︎

The training engineering for bangla-smollm-135m uses an iterative continual pretraining workflow designed to build linguistic understanding without losing the model’s core multi-lingual alignment capabilities [11].

3.1 Data Purification and Subset Distribution↩︎

The training pipeline ingests a highly diversified, multi-modal regional corpus based on the hishab/titulm-bangla-corpus repository [12]. To ensure empirical stability and downstream performance, text records are partitioned across three separate, stratified thematic slices:

Common Crawl (\(\mathbf{20\%}\) Stratified Weight): Web-derived corpora containing unstructured modern idioms, casual expressions, and vernacular dialogue [13].

Translated Text (\(\mathbf{30\%}\) Stratified Weight): High-register datasets translated from balanced academic and programmatic frameworks to capture formal reasoning structures [14], [15].

Romanized Text (\(\mathbf{30\%}\) Stratified Weight): Phonetically mapped text collections designed to improve performance on *Banglish* code-switched structures, expanding the contextual coverage of edge deployments [16].

3.2 Dynamic Block Packing Optimization↩︎

To prevent wasted computations over empty padding vectors, raw incoming sequences are first limited by string-length depth constraints (\(\text{Chars}_{\max} = 4 \times \text{BlockSize}\)) to prevent map thread choking. After tokenization, sequential lines are flattened into a continuous 1D token matrix and divided into uniform chunks fitting the model’s exact context dimension: \[\mathcal{B}_m = \mathbf{X}\left[ m \cdot d_{\text{block}} : (m+1) \cdot d_{\text{block}} \right]\] Where \(\mathbf{X}\) is the flattened token matrix, and \(d_{\text{block}} = 4096\) defines the target context window boundary. Labels are mapped natively to mirror the input indices (\(\text{Labels} \leftarrow \text{Input\_IDs}\)).

3.3 Hardware Infrastructure and Hyperparameters↩︎

The pretraining workflow runs on a multi-GPU Linux node utilizing an optimized Hugging Face Trainer backend. The hardware distribution relies on three active NVIDIA GPUs integrated via Hugging Face Accelerate. To maximize tensor throughput, training uses native Bfloat16 (BF16) mixed-precision execution. To avoid device Out-of-Memory (OOM) errors over the 4096 context window, gradient checkpointing is activated. Detailed training configurations are listed in Table 1.

4 Evaluation Methodology↩︎

To verify its linguistic capability and context-handling proficiency, bangla-smollm-135m was tested in a strict zero-shot (\(0\)-shot) setup across four established localized benchmarks [12], [17].

The primary comparison metric used is Normalized Accuracy (\(\text{acc\_norm}\)). This factor mathematically normalizes multi-choice scoring distributions by penalizing lucky guesses or selection biases: \[\text{acc\_norm} = \frac{\text{acc} - \frac{1}{k}}{1 - \frac{1}{k}}\] Where \(k\) corresponds to the total number of multiple-choice answer possibilities available per evaluation prompt. The evaluation suite tracks performance across four core domains:

PIQA_bn: Evaluates physical common sense reasoning by prompting text choices regarding everyday physical interactions translated into Bangla [6].

OpenBookQA_bn: Tracks factual knowledge and multi-hop question-answering capabilities using open-book scientific facts [12].

CommonsenseQA_bn: Focuses on complex linguistic reasoning and social abstract common sense [17].

Bangla_MMLU: A rigorous cross-disciplinary academic suite spanning advanced humanities, regional history, geography, and legal structures [18].

5 Comparison with Other Models↩︎

To determine the model’s true parameter efficiency, its scores were analyzed alongside baseline data from larger multilingual foundations [19], [20], as well as heavily adapted architectures in the 1B parameter range [7], [12]. These empirical results are compiled in Table 2.

5.1 Key Cross-Model Insights↩︎

The empirical comparison reveals significant structural patterns in model scaling efficiency:

Outperforming the 270M Tier: Despite operating with half the parameter count of Gemma-3-270m, bangla-smollm-135m scores higher on abstract reasoning fields like CommonsenseQA_bn (0.256 vs 0.249) and Bangla_MMLU (0.237 vs 0.234).

Parity with 1B Parameter Models: The model matches or exceeds the vanilla Llama-3.2-1b on physical reasoning and common sense tasks, despite requiring approximately \(7.4\times\) less memory space. This performance demonstrates the value of deterministic vocabulary intersection over raw unadapted parameter scaling for regional non-Latin scripts [8], [9].

5.2 The Broader Architectural Ecosystem↩︎

To place bangla-smollm-135m within the structural landscape of contemporary Bangla language computing, Table 3 maps its tokenization approach against alternative architectural frameworks [7], [13], [21].

6 Conclusion↩︎

The performance profile of bangla-smollm-135m demonstrates that efficient tokenization, targeted sub-corpora packing, and optimized hardware training loops can overcome the limitations of smaller model scales for low-resource languages. By matching or outperforming models up to seven times its size on core reasoning and knowledge tasks, this architecture provides a viable path for deploying efficient AI tools directly on consumer edge devices.

7 Limitations↩︎

While bangla-smollm-135m exhibits notable parameter efficiency across several core zero-shot reasoning tracks, several fundamental constraints must be carefully highlighted for downstream engineering layouts:

• Contextual Window Horizons: Although configured with an operational block boundary of \(d_{\text{block}} = 4096\), the physical model space limits high-fidelity retrieval or long-form narrative synthesis compared to ultra-large multilingual structures that support extended testing loops.

• Complex Mathematical and Code Derivation Limits: As identified in the taxonomy mappings, compact architectures beneath the 1B parameter layer experience rapid decay when processing multi-hop symbolic calculations or structural programmatic mutations in Bangla, requiring offloading to enterprise configurations like Qwen3-235B for heavy algorithmic reasoning.

• Linguistic Adaptation Hallucinations: Despite mitigating subword script fragmentation using our deterministic vocabulary merging framework, the underlying capacity limits can lead to trivial semantic alignment errors or phrasing degradation under highly stylized regional variations.

8 Appendix↩︎

References↩︎