1 Introduction↩︎

Arabic is spoken by more than 400 million people and is the official language of 25 countries, yet open Arabic language model weights trail English models by an uncomfortable margin. The strongest open Arabic models, Jais [1], AceGPT [2], ALLaM [3], and SILMA, all live in the 7B–70B parameter range. They achieve strong benchmark scores but demand 16–140 GB of memory and are out of reach for phone, laptop-CPU, and embedded deployment. At the other end of the spectrum, the sub-1B open models that can actually run on commodity hardware, Qwen2.5-0.5B-Instruct [4], Falcon-H1-0.5B, treat Arabic as one of a hundred languages and do not allocate Arabic-specialized vocabulary or training data.

The one published effort that targeted this gap, Kuwain-1.5B [5], proposed “language injection” on TinyLlama [6] and reported an 8% average Arabic improvement over the base. Misraj.ai (Khobar, Saudi Arabia) released the paper but did not release the weights. This leaves practical Arabic deployment on edge devices without a real option: the smallest downloadable Arabic-specialized decoder LLM on HuggingFace is nothing.

We present RightNow-Arabic-0.5B-Turbo, a 518M-parameter Arabic-specialized decoder LLM. The model is built by injecting 27,032 new Arabic tokens into Qwen2.5-0.5B [4], continuing pretraining on 504M Arabic tokens, supervised fine-tuning with response-only loss masking on 129,116 instruction pairs, applying direct preference optimization [7], and merging three checkpoints via weight averaging [8]. The result is the smallest open Arabic-specialized decoder LLM released to date. On COPA-ar it matches Falcon-H1-1.5B at one-third the parameter count; quantized to 4 bits it fits in 398 MB and reaches 635 tokens/s at bs=1 on a single H100 via llama.cpp.

1.0.0.1 Key insight.

Sub-1B Arabic models do not need new training techniques, every component of our pipeline exists in prior work. What they need is careful orchestration: vocabulary expansion that actually lowers Arabic token fertility, a data loader that avoids multi-rank HuggingFace Hub stalls, SFT that masks prompt tokens so the loss signal is concentrated on the assistant response, and an export path that produces artifacts small enough for the device class the model is targeting. Every mistake costs the accuracy budget of a model already near its scale ceiling.

1.0.0.2 Contributions.

1. The smallest open Arabic-specialized decoder LLM on HuggingFace as of the submission date, at 518M parameters and 398 MB on disk (q4_k_m).

2. A reproducible vocabulary-injection pipeline that mean-subtoken-initializes 27,032 Arabic tokens into Qwen2.5-0.5B and cuts Arabic tokenizer fertility from 2.18 to 1.80 tokens per word (17.3% reduction).

3. Direct head-to-head benchmarks against 6 competing models on three Arabic lm-evaluation-harness tasks under identical methodology, showing where a 0.5B model wins and where it loses to models 3–18\(\times\) larger.

4. A weight-soup merge ablation over 7 variants (SLERP and LERP across the DPO/SFT/pretrain checkpoints), selecting a configuration that improves mean accuracy by 0.44 absolute points over the DPO endpoint alone.

5. llama.cpp GGUF conversion producing 4 quantization levels (f16, q8_0, q5_k_m, q4_k_m) with measured throughput of 582–646 tokens/s at bs=1 on H100.

6. Full open release: 5,555 lines of Python across 25 scripts and 13 runtime modules, all training configs, all benchmark scripts, all intermediate checkpoints, and the final weights.

2 Related Work↩︎

2.0.0.1 Arabic-specialized LLMs.

Jais [1] from Inception/MBZUAI set the initial standard with a GPT-3-style 13B/30B decoder pretrained on a mixture of Arabic, English, and code. AceGPT [2] from FreedomIntelligence targeted cultural alignment via RLAIF and released 7B/13B chat variants. ALLaM [3] from SDAIA (Saudi Arabia) scaled to 7B/13B/34B/70B using vocabulary expansion and Arabic-English mixed pretraining. SILMA is a 9B Arabic-specialized model released by SILMA AI without an accompanying paper. All four operate in the 7B+ parameter range and target server deployment.

2.0.0.2 Sub-2B Arabic models.

The closest prior work to ours is Kuwain-1.5B [5], published by Misraj.ai in April 2025. Kuwain injects Arabic vocabulary into TinyLlama-1.1B [6] and reports an average 8% improvement on Arabic benchmarks over the base. Mutarjim [9] builds on Kuwain for Arabic-English translation. Critically, neither Kuwain nor Mutarjim weights are published on HuggingFace at the time of this submission, so neither can be directly compared or deployed. RightNow-Arabic-0.5B-Turbo is strictly smaller (518M vs 1.5B) and fully open.

2.0.0.3 Vocabulary expansion for language adaptation.

Extending a pretrained tokenizer with new vocabulary is a standard adaptation technique. ALLaM [3] argues that vocabulary expansion paired with English anchoring prevents catastrophic forgetting. The specific initialization scheme we use, averaging the old embeddings of a new token’s sub-piece decomposition, was formalized by WECHSEL [10] and subsequently adopted by Kuwain and ALLaM.

2.0.0.4 Small-model training stack.

Training on 8\(\times\)H100 SXM5 demands specific infrastructure. FlashAttention [11] and its varlen variant remove the memory and kernel-launch overhead of dense attention. PyTorch FSDP [12] shards optimizer and gradient state; we use the _HYBRID_SHARD_ZERO2 strategy. Liger Kernel [13] replaces RMSNorm, RoPE, SwiGLU, and fused linear cross-entropy with Triton implementations, saving memory and time. Our vocabulary expansion increases the output projection to 178,697 rows, which makes fused linear cross-entropy essential: materialized float32 logits at batch 16\(\times\)​4096\(\times\)​178697 would require 44 GiB.

2.0.0.5 Post-training.

Direct preference optimization [7] reformulates RLHF as a classification loss over preference pairs, eliminating the reward model. Model souping [8] averages weights across multiple fine-tuned checkpoints and has been shown to improve out-of-distribution generalization. Our pipeline applies both: DPO on 6,750 Arabic preference pairs from argilla-dpo-mix-7k-arabic, and a linear-weight soup across the DPO, SFT, and pretrain checkpoints.

2.0.0.6 Data.

Pretraining uses Arabic Wikipedia (504M tokens after merging with the new tokenizer) via the wikimedia/wikipedia corpus. We originally planned to mix in FineWeb-2-ar [14] but encountered persistent HuggingFace Hub 504 stalls during multi-rank streaming; we resolved this by pre-tokenizing the corpus to a flat int32 memmap and sampling windows per rank. Instruction tuning uses a merge of five Arabic instruction datasets; preference tuning uses the argilla Arabic DPO mix. All data is publicly available. Evaluation uses three tasks from lm-evaluation-harness [15]: COPA-ar, Arabic MT HellaSwag, and the Arabic Leaderboard ArabicMMLU [16]. Table 1 summarizes the open Arabic LLM landscape and shows where our model sits.

4pt

3 Method↩︎

Figure 1 shows the full pipeline. The model starts as Qwen2.5-0.5B (base, not Instruct), gains 27,032 new Arabic tokens, sees 504M pretraining tokens, 129,116 SFT examples, and 6,750 DPO pairs, and ends as a merged checkpoint exported in bf16, int8, and four GGUF quantizations.

3.1 Tokenizer surgery↩︎

Qwen2.5-0.5B’s original byte-level BPE tokenizer has 151,665 tokens and encodes Arabic text at 2.18 tokens per word on a held-out sample, roughly \(1.4\times\) the English rate. To lower this, we train a SentencePiece unigram model [17] with 32,000 tokens on a 12.54 GB Arabic corpus composed of 21.5M lines of Arabic Wikipedia (via wikimedia/wikipedia 20231101.ar) and 5.4 GB of filtered Arabic web text. We apply standard Arabic text normalization before training: NFKC, alif-variant normalization (hamza-above, hamza-below, madda \(\to\) bare alif), tatweel stripping, and ya-variant normalization.

The resulting SentencePiece model is merged into the Qwen tokenizer with a deduplication pass: any SentencePiece piece whose surface string already round-trips to a single existing Qwen token is discarded, leaving 27,032 net-new Arabic tokens. We then call model.resize_token_embeddings(178697) and initialize the 27,032 new embedding rows as in Algorithm 2.

After merging, we verify that model.lm_head.weight.data_ptr() still equals model.get_input_embeddings().weight.data_ptr(), Qwen2.5-0.5B ties these, so that both the input embeddings and output projection receive the mean-init rows simultaneously. On a 1000-word Arabic sample, fertility drops from 2.08 to 1.77 tokens per word (14.7% reduction). On a larger held-out 368-word sample used for the final measurement, fertility drops from 2.18 to 1.80 (17.3% reduction).

3.2 Continued pretraining↩︎

The merged model is trained for 2,500 optimizer steps on a pre-tokenized flat int32 memmap containing 504,318,692 tokens of Arabic Wikipedia encoded with the new tokenizer. We use a per-GPU micro-batch of 16 sequences of 4096 tokens with gradient accumulation of 8, giving an effective batch of \(16 \times 4096 \times 8 \times 8 = 4.19\)M tokens per step and 10.5B total training tokens. FSDP [12] wraps each Qwen2DecoderLayer with ShardingStrategy._HYBRID_SHARD_ZERO2 and bf16 mixed precision; FlashAttention varlen [11] takes packed cu_seqlens directly from the memmap iterator so document boundaries are respected without padding; Liger fused kernels [13] cover RMSNorm, RoPE, SwiGLU, and fused linear cross-entropy.

The optimizer is fused AdamW, \(\beta = (0.9, 0.95)\), \(\epsilon = 10^{-8}\), weight decay 0.1, peak learning rate \(2\times10^{-4}\) with 500-step linear warmup and cosine decay to \(2\times10^{-5}\). Training runs at 415,000 tokens/s aggregate on 8\(\times\)H100 SXM5 with peak per-GPU memory of 24 GB.

3.3 Supervised fine-tuning↩︎

We merge five Arabic instruction datasets into a single deduplicated pool: FreedomIntelligence/evol-instruct-arabic (59,022 rows), FreedomIntelligence/alpaca-gpt4-arabic (49,969), FreedomIntelligence/sharegpt-arabic (5,231), arbml/CIDAR (10,000), and the Arabic subset of CohereForAI/aya_dataset (4,947 after filtering on language_code=="arb"). MD5 deduplication on the rendered ChatML string yields 129,116 unique examples.

Each example is rendered into ChatML with the system prompt “you are a smart assistant that answers in formal Arabic” and pre-tokenized into two parallel int32 memmaps: sft_tokens.bin contains the full input id sequence, sft_labels.bin contains \(-100\) on prompt positions and the actual token id on assistant-response positions. Training is 5 epochs at peak learning rate \(2\times10^{-5}\), grad accumulation 2, 30 warmup steps, and computes loss only on response tokens [18]. Of the 43,918,266 total pretokenized tokens, 31,678,374 (72.1%) carry non-negative-100 labels; masking the prompt saves capacity for the distribution we actually care about.

3.4 Direct preference optimization↩︎

The SFT checkpoint undergoes DPO [7] on 6,750 Arabic preference pairs from 2A2I/argilla-dpo-mix-7k-arabic. Each pair provides a user prompt, a chosen assistant response, and a rejected assistant response. Training runs on a single H100 with per-device batch size 2, gradient accumulation 8, \(\beta = 0.1\), peak learning rate \(5\times10^{-7}\), 2 epochs, and an explicit frozen reference model (not LoRA-implicit). Total training time is 34 minutes for 844 optimizer steps.

3.5 Weight soup merging↩︎

The DPO run converges to a loss near \(\ln 2 \approx 0.693\) with reward-accuracy 0.48 and margin \(-6.9\times10^{-4}\), indicating the preference dataset supplies weak signal at this scale. We mitigate this by averaging the DPO checkpoint with the earlier pretrain and SFT checkpoints, all three share the same 178,697-vocab architecture, which makes direct weight averaging safe. We produce 7 merge variants: linear interpolations of DPO and pretrain at \(t \in \{0.3, 0.5, 0.7\}\), SLERP at \(t \in \{0.3, 0.5\}\), linear DPO/SFT at \(t=0.5\), and a 50/25/25 soup of DPO/SFT/pretrain. Each merge is benchmarked on the same 3 Arabic tasks (Section 5.1); the 50/25/25 soup wins and becomes the final checkpoint.

3.6 Edge deployment↩︎

For edge deployment we convert the merged model to llama.cpp GGUF format and produce four quantization levels. The llama.cpp converter does not recognize our tokenizer hash (because we added 27k tokens), so we patched convert_hf_to_gguf.py to map our hash to the existing qwen2 pre-tokenizer type. After conversion, llama-quantize produces q8_0 (525 MB), q5_k_m (419 MB), and q4_k_m (398 MB). Note that q5_k_m and q4_k_m fall back to higher-bit quantization for 144 of the 290 tensors, specifically the expanded Arabic embedding rows, because k-quants require tile sizes that the added vocabulary does not align with. Effective bits-per-weight is therefore 6.79 (q5_k_m) and 6.45 (q4_k_m) rather than the nominal 5 and 4.

4 Implementation↩︎

Table 2 summarizes the codebase. Total Python source is 5,555 lines across 25 scripts and 13 runtime modules. The full pipeline runs on a single Nebius gpu-h100-sxm instance (8\(\times\)H100 80GB SXM5, Ubuntu 24.04, CUDA 13.0, PyTorch 2.11.0+cu130, flash-attn 2.8.3, transformers 5.5.0, trl 1.0.0, liger-kernel 0.7.0).

4.0.0.1 Memmap data loader.

Early runs attempted HuggingFace Hub streaming for the pretraining corpus. Under 8-rank torchrun, one or more ranks would stall for 15+ minutes at the first parquet fetch, causing NCCL collective deadlocks. We resolved this by pre-tokenizing the entire corpus to a flat int32 memmap (arwiki_tokens.bin, 2.0 GB) and sampling fixed-length windows per rank. The loader is 162 lines and serves data at the speed of sequential memory reads, effectively zero overhead compared to the GPU forward pass.

4.0.0.2 Packed attention with document boundaries.

The loader also produces cu_seqlens pointers for every batch: each window is scanned for EOS positions and boundaries are inserted so FlashAttention varlen correctly masks cross-document attention. We monkey-patch Qwen2Attention.forward to read cu_seqlens from a process-global (not thread-local) dictionary, because PyTorch gradient checkpointing recomputes forwards on a different thread during backward, and a thread-local would be empty at that time.

4.0.0.3 FSDP hybrid sharding.

For 0.5B parameters on 8\(\times\)​80 GB H100, full parameter sharding is unnecessary. We use ShardingStrategy._HYBRID_SHARD_ZERO2, which replicates parameters across ranks but shards optimizer state and gradients within each rank group. This keeps the training compute local and all-gather overhead minimal.

5 Experimental Evaluation↩︎

5.0.0.1 Hardware.

All experiments run on a single Nebius gpu-h100-sxm node: 8\(\times\)NVIDIA H100 80 GB SXM5 HBM3, NVLink4, 128 vCPUs, 1.5 TiB RAM, 1.28 TiB SSD, driver 580.126.09, CUDA 13.0.88. NCCL allreduce busbw was measured at 466.9 GB/s on a 1 GiB tensor across 8 ranks at session start.

5.0.0.2 Evaluation methodology.

We use lm-evaluation-harness [15] v0.4.11 with three tasks: copa_ar, arabic_mt_hellaswag, and arabic_leaderboard_arabic_mmlu (our model gets evaluated on the full 14,575-question ArabicMMLU [16]). Every model is scored with apply_chat_template=True (with a fallback to raw prompts for models whose tokenizer has no chat_template attribute, specifically AceGPT-7B-chat), batch size 2, max_length=1536, per-task limit=200, and acc_norm preferred over acc where available. The same methodology is applied to all baselines. Evaluating each model on all three tasks takes 10–25 minutes depending on model size.

5.1 Main results↩︎

Table 3 shows the head-to-head comparison. Figure 3 plots the same data on a Pareto scatter of accuracy versus parameters.

5pt

5.1.0.1 Same-class wins.

Among models in the 0.5B class, we are first on COPA-ar (+4.5 vs Qwen2.5-0.5B-Instruct, +13.5 vs Falcon-H1-0.5B), first on HellaSwag-ar (+3.5 vs Qwen, +3.0 vs Falcon), and first on mean (+1.8 vs Qwen, +5.2 vs Falcon). We lose ArabicMMLU by 2.8 points to Qwen, the one task where the base multilingual model’s broader world knowledge still beats our Arabic-specialized continued pretraining.

5.1.0.2 Scaling gap.

Against the 7–9B Arabic-specialized models, the gap is exactly where theory predicts it should be: knowledge-intensive ArabicMMLU separates by 12–30 points, HellaSwag by 1–12 points, and COPA-ar by 10–11 points. On the mean, the 0.5B model recovers \(35.9 / 53.5 = 67.1\%\) of SILMA-9B’s score at \(518\text{M} / 9\text{B} = 5.8\%\) of the parameters.

5.1.0.3 A specific tie.

COPA-ar is the one task where our 0.5B exactly matches Falcon-H1-1.5B (both at 58.4%) despite a 3\(\times\) parameter gap. COPA is a commonsense reasoning task with short inputs where the Arabic tokenizer efficiency directly helps, fewer tokens per premise means more of the model’s context is real content. Figure 4 breaks down the per-task scores across all models.

5.2 Merge ablation↩︎

Table 4 shows the 7 merge variants and the original DPO checkpoint, each benchmarked on the same 3 tasks with the same methodology.

5pt

The soup gains +0.44 absolute points over the DPO endpoint alone. The gain comes almost entirely from HellaSwag-ar (+1.33). The \((\text{DPO},\text{Pretrain})\) lerp family clusters in a tight 0.13-point band around 35.35%, and the SLERP variants are indistinguishable from linear at these ratios. The \((\text{DPO},\text{SFT})\) lerp is the only merge that underperforms the DPO baseline.

5.3 Tokenizer efficiency↩︎

Table 5 reports the result. The 17.3% fertility reduction translates directly into a 17.3% speedup on Arabic-only workloads at the same parameter count, because inference cost is linear in token count. Combined with the 0.5B parameter count, this compounds: on a single Arabic user query, RightNow-Arabic-0.5B-Turbo emits the same semantic content as Qwen2.5-0.5B-Instruct using 17.3% fewer forward passes, each of which is already small because the model is small.

5.4 Inference speed↩︎

We measure inference throughput via llama.cpp’s llama-bench on H100 SXM5 (CUDA backend) for each GGUF quantization at batch size 1, prompt length 128, generation length 128. Numbers are reported in Table 6.

All four quantizations clear 580 tokens/s at batch size 1. HuggingFace model.generate() on the same hardware tops out at 82 tokens/s at bs=1 due to Python and per-token kernel-launch overhead; llama.cpp’s CUDA graph capture and optimized C++ sampling loop removes that ceiling and delivers an 8\(\times\) speedup at identical model weights.

5.5 Training dynamics↩︎

Table 7 lists the three major training phases, their step count, and their loss trajectory.

Figure 5 visualizes the pretraining loss trajectory. Pretraining reduces perplexity from roughly \(e^{14.21} = 1.48\text{M}\) to \(e^{1.69} = 5.42\), a 270,000\(\times\) improvement.

SFT, which computes loss only on the roughly 72% of tokens that are assistant responses, drops response-only loss from 1.95 at step 70 to 1.81 at step 415. DPO, on weak Arabic preference data, moves essentially nothing: the training loss stays at \(\ln 2\) and reward margin stays near zero, confirming that 6,750 machine-translated preference pairs are insufficient signal for a 0.5B model. The soup merge in Section 5.2 recovers some capability not by reweighting via a learned objective but simply by averaging with the earlier checkpoints that the SFT stage partially forgot.

6 Discussion and Limitations↩︎

6.0.0.1 Knowledge ceiling is a parameter ceiling.

The ArabicMMLU gap to the 7B+ models (29+ points) is the clearest evidence of a fundamental limit: knowledge benchmarks scale with parameter count, period. No amount of tokenizer efficiency, merge tuning, or training-stack optimization can close 29 points at 0.5B. If the target deployment environment can afford 14 GB of model weights, AceGPT-7B or ALLaM-7B are the correct choice. We target the niche where 14 GB is not available, phones, edge CPUs, browsers, and knowledge is a secondary concern.

6.0.0.2 DPO was weak at this scale.

The DPO stage did not move the model. We attribute this to two causes: (1) the 6,750-pair preference dataset was machine-translated from English DPO data rather than written by native Arabic speakers, so the preference signal is noisy, and (2) 0.5B parameters may be too small to benefit from preference tuning beyond what SFT already provides. A better dataset and a stronger reference model might change this; we report what we observed. The soup merge was the more productive post-SFT intervention.

6.0.0.3 Single Arabic variety.

The pretraining corpus is Modern Standard Arabic (via Wikipedia). The model handles MSA well and dialects poorly: a query in Egyptian, Gulf, or Levantine Arabic will receive an MSA response. For dialect coverage, the pretraining corpus would need explicit dialect data, which was outside the scope of the initial release.

6.0.0.4 Tokenizer fertility short of target.

Our 17.3% fertility reduction is real but below the 30% reduction that a larger-vocab, Arabic-only SentencePiece model would achieve. The main constraint is that we merged into the existing Qwen2.5 BPE rather than replacing the tokenizer entirely; full replacement would invalidate the base model’s pre-trained embedding geometry and demand a full retrain. A future version could split the difference by adding a second wave of common multi-word Arabic phrases.

6.0.0.5 Pretraining tokens per parameter.

At 504M pretraining tokens for 518M parameters, the ratio is \(\approx 1\), two orders of magnitude below the Chinchilla-optimal ratio [19]. Our training is continued pretraining on top of an already-trained base, so the absolute ratio is misleading, but additional Arabic pretraining data would benefit the model directly on ArabicMMLU-style knowledge tasks. Gating issues prevented the use of FineWeb-2-ar and CulturaX-ar during this release; we plan to address this in v2.

6.0.0.6 GGUF quantization tile alignment.

q4_k_m and q5_k_m fall back to higher-bit quantization for the 144 tensors that interact with the added vocabulary, because k-quant tile sizes do not align with our 178,697-row embedding matrix. Effective bits-per-weight is 6.45 and 6.79 rather than 4 and 5. A proper fix would require aligning the vocabulary padding to a llama.cpp-friendly block size; we leave this to future work.

7 Conclusion↩︎

RightNow-Arabic-0.5B-Turbo is a 518M-parameter Arabic-specialized decoder LLM built by injecting 27,032 Arabic tokens into Qwen2.5-0.5B, continuing pretraining on 504M Arabic tokens, supervised fine-tuning with response-only loss masking, direct preference optimization, weight-soup merging, and exporting to GGUF for edge deployment. At the same parameter count, the model beats Qwen2.5-0.5B-Instruct and Falcon-H1-0.5B on Arabic benchmarks; at 398 MB quantized it delivers 635 tokens/s at bs=1 on a single H100 via llama.cpp and fits comfortably on a phone. It is the smallest open Arabic-specialized decoder LLM released to date. The full pipeline, weights, training code, benchmark scripts, and seven merge variants, is available at https://huggingface.co/RightNowAI/RightNow-Arabic-0.5B-Turbo.

References↩︎