1 Introduction↩︎

In recent years, Large Language Models (LLMs) have demonstrated remarkable proficiency across a diverse range of natural language tasks [1]–[6]. Their rapid and widespread adoption as virtual assistants [7] has been largely driven by their accessibility through intuitive chat-based interfaces [3], [8]. The training of these assistants involves multiple steps, beginning with a pre-training stage where the model learns from a vast text corpus using a self-supervised objective such as next-word prediction [3], [8], [9]. The subsequent, critical step is instruction tuning (IT) [10] which refines the model’s ability to follow user instructions and engage in a conversational manner [1], [8]. This process involves fine-tuning an LLM on an extensive dataset, which typically consists of instruction-answer pairs. By training on these pairs, the model learns to generate appropriate responses to user instructions, resulting in more anticipated and manageable outputs [1]. A notable benefit of this approach is that it can enhance the model’s ability to generalize and respond to previously unseen instructions [9], [11], [12].

A significant issue at present is the scarcity of open-source, instruction tuning datasets for underrepresented languages. Existing datasets are predominantly in English, marginalizing other languages and leading to inferior model performance and higher deployment costs in these settings [8]. In the context of this work, "low-resource" refers to a scarcity of NLP tooling, open-source datasets, and specialized instruction-tuning resources. Luxembourgish, a West Germanic language spoken by about 600 000 people¹ primarily in Luxembourg, exemplifies this challenge as adequate training data is scarce.

In this paper, we introduce LuxIT, a monolingual instruction tuning dataset in Luxembourgish, synthesized from articles of the news website RTL² and Wikipedia entries. Following the work of [13], we select DeepSeek-R1-0528, as the top-performing model for Luxembourgish generation and comprehension to synthetically produce a final dataset of 227,507 high-quality instruction-answer pairs. To validate the quality of our dataset, we assess the Luxembourgish capabilities of several LLMs before and after they have been fine-tuned on LuxIT.

To our knowledge, no instruction tuning dataset for Luxembourgish currently exists that has been created exclusively from monolingual seed data for the purpose of fine-tuning state-of-the-art LLMs.

Our primary contributions are twofold:

• We introduce LuxIT³, a synthetically generated instruction tuning dataset in Luxembourgish derived from monolingual seed data.

• We evaluate the utility of LuxIT by fine-tuning 14 widely-used small-scale LLMs (\(\leq\) 15B parameters) on standardized Luxembourgish proficiency exams and five downstream NLP tasks, demonstrating improvements in language proficiency for 12 of 14 models and NLP task performance for 9 of 14 models.

To validate the pipeline’s ability to produce high-quality data, we assess the generated outputs against four custom metrics (see Section 3.4), quantifying the proportion of retained samples. We then address the following research questions:

• RQ1: How does fine-tuning on LuxIT affect performance on Luxembourgish language exams?

• i) Does fine-tuning on LuxIT improve overall accuracy across CEFR proficiency levels? We fine-tune 14 LLMs (\(\leq\) 15B parameters) on LuxIT and benchmark their performance in Luxembourgish against their corresponding base instruct models, following the evaluation framework by [13].

• ii) How does the impact vary across individual linguistic categories? We examine the results from RQ1 further and analyze them over the individual categorical exam levels, akin to [13].

• RQ2: Does fine-tuning on LuxIT improve performance on downstream Luxembourgish NLP tasks? We benchmark our fine-tuned models against their respective base instruct models on 5 well-known NLP tasks in Luxembourgish.

2 Related Work↩︎

2.1 Multilingual Instruction Tuning Datasets↩︎

xP3 [14] is a multilingual, human-crafted instruction tuning dataset, where the data is taken from P3 and other multilingual datasets [1]. The dataset is constructed by combining data from various sources into a unified format [1]. [14] further extend xP3 to xP3mt by applying machine translation [8].

Multilingual datasets like Bactrian-X [15] emerged from machine-translated Alpaca instructions and matching GPT-3.5-Turbo [2] generated answers [8]. LIMA [9] is a fine-tuned version of Llama [16], trained on 1000 attentively selected instruction-answer pairs [1], [8]. [8] extend LIMA into Lima-X by translating the instructions from the LIMA dataset into 4 languages.

We refrain from using machine translation and multilingual seed-data and investigate the dataset generation in a low-resource setting from a strictly monolingual approach. While the model we use for data generation is pre-trained on a multilingual corpus, the data we feed to the model for synthesizing the IT dataset is exclusively in Luxembourgish.

2.2 Luxembourgish Language Resources↩︎

Recently, there have been various contributions to the creation of Luxembourgish language resources. [17] developed a pre-training dataset in Luxembourgish through a data-augmentation technique where they partially translated text from German into Luxembourgish, resulting in LuxemBERT. [18] presented LuxGen, a benchmark for evaluating data generation in Luxembourgish and LuxT5, an mT5-based [19] text generation model in Luxembourgish, pre-trained on a German, French and Luxembourgish text corpus, where the latter is obtained through transfer learning from German and French. Most recently, [20] introduced a hand-crafted cross-lingual dataset for training Luxembourgish sentence embedding models, which led to the LuxEmbedder model.

With the increasing ubiquity of LLMs, there is a need for a high-quality instruction tuning dataset in Luxembourgish to guide weaker models in improving their Luxembourgish capabilities. The cross-lingual dataset from [21] demonstrates an effective strategy for generating instruction data, pairing Luxembourgish with English, German, or French content. However, this approach is contingent on the availability of parallel seed data, which is often a bottleneck for low-resource languages. Our work complements this by exploring a monolingual methodology specifically designed for scenarios where such parallel data is unavailable. This focus on a strictly monolingual setting is the primary distinction of our approach and motivates the investigation into creating resources under these more constrained conditions.

3 LuxIT↩︎

This section details the methodology employed for the creation of our Luxembourgish instruction tuning dataset. The data generation process for LuxIT, illustrated in Figure 1 is executed as a five-step pipeline: The initial step (1) involves the extraction of raw data from two distinct sources (see Section 3.1). In the second step (2), we subject the data to a series of heuristic filters (see Section 3.2). We subsequently provide the refined data to DeepSeek-R1-0528, which generates instruction-answer pairs in Luxembourgish (3). Following this, we implement an LLM-as-a-judge approach to evaluate the quality of our synthetic data (4). In parallel, a subset of these pairs undergoes manual human evaluation. In the final step, we apply a post-filtering process to remove all samples that received poor scores (5), resulting in the final LuxIT dataset.

3.1 sec:Data↩︎

Our methodology utilizes two primary data sources: a complete dump of the Luxembourgish Wikipedia and a collection of all news articles and comments from RTL up to May 2024. We obtained the most recent dump of the Luxembourgish Wikipedia⁴ on August 5th, 2025 and subsequently use Wikiextractor⁵ for extraction and cleaning. We convert the processed data from both sources into a JSON format for easier handling. Further details on the data structure are available in Appendix 8.1. The initial collection comprises approximately 1.2 million RTL comments, 303,000 RTL news articles, and 80,000 Wikipedia articles, summing to approximately 1.6 million entries. We choose to exclude RTL user comments from our seed data due to their brevity and the potential for biased content, typographical mistakes, and grammatical inaccuracies.

3.2 Heuristic Filtering↩︎

Before the generation phase, we implement a set of heuristic filters to remove low-quality samples. We retain only articles containing at least 750 characters. For Wikipedia articles, we exclude list articles, disambiguation pages, and pages marked as stubs. We filter out all non-Luxembourgish RTL news articles based on the language tags provided directly by RTL and apply additional filters, including boilerplate removal and semantic format filtering, which identifies keywords indicating a non-prose structure. This filtering process yields 16,558 Wikipedia articles and 100,340 RTL news articles.

3.3 Instruction-Answer pair generation↩︎

We employ DeepSeek-R1-0528⁶ to synthetically create instruction-answer pairs in Luxembourgish from the filtered RTL news and Wikipedia articles. We select all 16,558 Wikipedia articles and randomly sample 66,776 RTL news articles. We then prompt the model to generate three instruction-answer pairs for each seed, yielding a total of 245,624 synthetic pairs⁷.

To ensure the generation of high-quality data, we engineer a comprehensive prompt designed to process information from both data sources effectively. The prompt instructs the model to embed all necessary context within the instruction to facilitate a self-contained answer. For news articles, we direct the model to handle temporal context with care. We also guide the model to produce a diverse set of instruction types, including summarization, question answering, information extraction, and explanation, where the latter task consists of providing a justification or reasoning alongside the response. For summarization tasks specifically, the model is required to include the original source text within the instruction itself. The complete data generation prompt is available in Appendix 8.2.

The final LuxIT dataset is structured with three columns: instruction, response, and messages. The instruction and response columns store the generated instructions and their corresponding answers. The messages column consolidates these two into ChatML format. We show the structure of our prompt template in Appendix 8.4 and provide a representative sample of an instruction-answer pair from LuxIT in Table 4 in Appendix 8.3.

3.4 LLM-as-a-judge scoring↩︎

Following generation, we apply an LLM-as-a-judge approach for quality control using GPT-5-mini [22]. We select GPT-5-mini over GPT-4o-mini based on a comparison reported in Table 5 (Appendix 8.5) which shows that despite a slight underperformance at lower CEFR levels, GPT-5-mini outperforms GPT-4o-mini by up to 24 points at higher levels (+9.7 pp macro average). A small-scale pilot comparing GPT-5 and GPT-5-mini judging found retention rates differing by only a few percent, justifying the cost-performance tradeoff.

For the judging task, we prompt GPT-5-mini to assign scores to each pair based on four criteria: linguistic quality, factual accuracy, instruction adherence (or instruction following), and helpfulness & relevance. We use a three-point scoring system, with 1 representing poor quality and 3 indicating excellent quality. The linguistic quality score aims to check for the overall quality of the Luxembourgish language in the text, such as grammar, spelling and phrasing. Factual accuracy simply checks whether the answer factually answers the instruction without contradictions. Instruction adherence aims to verify whether the response correctly answers the instruction as intended and helpfulness relevance examines the general usefulness of the instruction. The full breakdown of our scoring system and the prompt used for this evaluation are provided in Appendices 8.6 and 8.7. We retain only samples that achieve a score of at least 2 across all four metrics, discarding all others to form the final dataset.

3.4.1 Human evaluation↩︎

To validate our automated quality control, we conduct a manual human evaluation on a random subset of 100 generated samples. This sample size is selected to provide a meaningful preliminary audit of the dataset’s quality while remaining feasible within the standard resource constraints of low-resource language research, where recruiting qualified native-speaker annotators is both challenging and cost-intensive. The evaluation was performed independently by three native Luxembourgish speakers (two male, one female; ages 26, 32, and 35) who hold academic backgrounds in computer science and history.

Each annotator evaluated the samples using the identical 3-point scoring rubric applied by the automated LLM judge. All scoring disagreements are resolved through a strict majority vote to determine the final aggregated label for each sample.

The automated judge retains a higher proportion of samples (92.6%) than the human evaluation (69.0%), indicating a leniency bias in the LLM judge. Inter-annotator agreement (Fleiss’ Kappa) is fair for linguistic quality (\(\kappa=0.243\)) but poor across the remaining metrics, including below-chance agreement for factual accuracy (\(\kappa=-0.019\)). This reflects the inherent difficulty of consistently evaluating synthetic Luxembourgish text and suggests that future annotation efforts would benefit from more granular rubrics and calibration sessions. Despite this low agreement, majority-vote aggregation over retained samples shows 94.2% rated "Excellent" for factual accuracy, pointing to generally high quality while underscoring the subjectivity of the individual judgments. We report the full breakdown in Table 7 in Appendix 8.8.1.

To directly evaluate the alignment between our automated judge and human insight, Table 1 presents a per-item agreement analysis across the 100-sample subset. Exact agreement is low and Cohen’s Kappa reveals poor reliability overall, highlighted by a negative coefficient for Factual Accuracy (\(\kappa = -0.058\)). This below-chance agreement is explained by a mismatch visible in the metric means. For Factual Accuracy, human reviewers heavily favored perfect marks, resulting in a highly skewed mean of 2.84. Conversely, the LLM judge was significantly more conservative, frequently downgrading those same pairs to acceptable, leading to a lower mean of 2.40. Conversely, a reverse trend is visible in Helpfulness Relevance, where the LLM displays a leniency bias, overestimating value compared to humans (2.74 vs. 2.29).

3.5 sec:Post-filtering↩︎

After filtering out all samples with at least one score of 1, the final version of our dataset contains 227,507 instruction-answer pairs in Luxembourgish. We provide a detailed overview in Table 6 and show the score distribution on LuxIT from our post-filtering step. Due to space reasons, we put the table in Appendix 8.8. The original dataset consists of 245,624 instruction-answer pairs. We reject 18,117 samples with low scores, observing a retention rate of 92.62%. Among all samples, 32,215 (13.12%) have perfect scores (score 3) across all score types. We observe that instruction adherence scores best among all metrics, with 223,065 ( 90.8%) obtaining a score of 3. Furthermore, among the rejected samples, factual accuracy has the highest rejection rate, with 11,745 entries having a score of 1. For the rejection rate, we observe 6,924, 11,745, 1,039 and 246 entries of score 1 (lowest) for linguistic quality, factual accuracy, instruction adherence and helpfulness relevance respectively.

3.6 Dataset Composition↩︎

To characterize the task-type distribution of LuxIT and evaluate whether our open-ended generation prompt produces balanced coverage, we conduct a classification audit. While categorizing the entire 227,507-pair dataset is computationally prohibitive, we sample 100 random pairs and classifiy their instruction types using GPT-5.5 [23]. The distribution of the sampled dataset is as follows: Summarization (32%), Information Extraction (30%), Question Answering (23%), and Explanation (15%). These results indicate that the generation prompt yields a relatively balanced mix of task types, though it exhibits a slight skew toward summarization and extraction tasks.

4 Experimental Setup↩︎

In this section, we outline and break down our experiments in detail.

4.1 Fine-tuning on LuxIT↩︎

We fine-tune a variety of small models (\(\leq\)​15B parameters) on LuxIT. We split the dataset into train, evaluation and test split with ratios 0.92, 0.05 and 0.03 respectively, training on 209307 and evaluating on 11375 samples. We select the following models: Llama-3.1-8B-Instruct, Llama-3.2-1B-Instruct [24], Gemma-3-1B-IT, Gemma-3-12B-IT [25], GLM-4-9B-0414 [26], Qwen2.5-0.5B-Instruct, Qwen2.5-1.5B-Instruct, Qwen2.5-7B-Instruct [27], Mistral-7B-Instruct-v0.3 [28], Ministral-3-3B-Instruct-2512 [29], Phi-4 [30], Apertus-8B-Instruct-2509 [31], EuroLLM-1.7B-Instruct [32], Olmo-3-7B-Instruct [33]. The models are fine-tuned with LoRA [34] using the Unsloth⁸ framework. Unless otherwise specified, we apply a standard configuration across models: a LoRA rank (\(r\)) of 16, a LoRA alpha (\(\alpha\)) of 32, a learning rate of \(1 \times 10^{-4}\) with a cosine learning rate scheduler, and a warmup ratio of 0.03. Models are trained for 2 epochs using the AdamW 8-bit optimizer. Due to computational constraints, hardware setups differ slightly between runs. A comprehensive table detailing the hyperparameters, exceptions, and GPU hardware used for each specific model can be found in Appendix 9.1.

4.2 Evaluation on Language Exams↩︎

We investigate the impact of LuxIT on the Luxembourgish language proficiency of small LLMs, following the same approach proposed by [13]. After fine-tuning, we evaluate the LLMs on Luxembourgish language proficiency exams, instructing the models to solve exams consisting of multiple-choice questions, testing for vocabulary, grammar, reading comprehension, and conversational comprehension (done through transcripts). The language exams stem from the Luxembourgish language institute Institut National des Langues Luxembourg (INLL)⁹ and are divided into 6 skill levels defined in the Common European Framework of Reference for Languages [35], ranging from A1 (basic level) to C2 (native level), each exam level consisting of fill-in-the-blank and multiple choice questions [13]. The full dataset consists of 629 such questions, with slightly more than 100 questions per difficulty level and a nearly equal distribution of testing categories (vocabulary, grammar, reading comprehension, conversational comprehension). We measure the models’ performance using accuracy as a metric, akin to [13]. We compare base and instruct models to our fine-tuned models.

The INLL exams are proprietary and not publicly accessible. This effectively eliminates the risk of data contamination within our Wikipedia and RTL news seed corpora, ensuring that the models’ performance reflects genuine language comprehension rather than test-set memorization.

4.3 Evaluation on Luxembourgish downstream tasks↩︎

We evaluate our 14 fine-tuned models on 5 well-known NLP downstream tasks in Luxembourgish, namely Intent Classification (IC) [36], Winograd Natural Language Inference (WNLI) [17], Recognizing Textual Entailment (RTE), Sentiment Analysis and Stanford Sentiment Treebank (SST-2), taken from [37]. According to the authors, the data in these task-benchmarks was not taken from Wikipedia or RTL articles. The dataset consists of 3,823 samples. We report the class distribution in Table 9 in Appendix 9.2.

5 Results↩︎

5.1 RQ1 i): Does fine-tuning on LuxIT improve overall accuracy across CEFR proficiency levels?↩︎

Table 2 shows the comparison between our fine-tuned models and their respective base models evaluated on Luxembourgish language exams, reporting total accuracy on all language exam levels.
Fine-tuning on LuxIT leads to a macro average accuracy increase in 12 out of 14 models, with macro average gains ranging from +2.5 percentage points (pp) (Gemma-3-12B-Instruct) to +21.0 pp (Ministral-3-3B-Instruct). On average, the models see a macro average accuracy increase of 5.37 pp. The effectiveness of instruction tuning on LuxIT is most evident for Ministral-3-3b-Instruct, with its macro average increasing from 25.8% to 46.8% (+21.0 pp). Similarly, for Llama-3.2-1B-Instruct, Phi-4 and Qwen2.5-7b-Instruct, the macro average increases by 14.4 pp, 14.2 pp and 13.6 pp respectively. For 2 of the 14 models tested, Gemma-3-1B and Qwen2.5-0.5B, we observe a degradation. We observe a decrease in macro average accuracy of -22.0 pp and -13.6 pp, respectively. With regard to the individual language exam levels, we observe the most consistent increase for B1/B2 level across all models except our 2 overall decreasing models, Gemma-3-1B-Instruct and Qwen2.5-0.5B-Instruct. C2 level shows the most inconsistent gains, with 6 out of 14 models declining.

Out of all models, Gemma-3-12B-Instruct has the strongest baseline before fine-tuning, with a baseline macro average of 56.9%. We identify GLM-4-9B-0414 as the top performing model in terms of absolute performance, with a final macro average of 60.6%, followed by Gemma-3-12B-Instruct with an average of 59.4%. Appendix 10.1 contains a more detailed visual breakdown.

5.2 RQ1 ii): How does the impact vary across individual linguistic categories?↩︎

Table 3 shows macro-averaged accuracy changes across the four skill categories introduced in section 4.2. The most consistent gains are observed in vocabulary and conversational comprehension, where multiple models improve substantially — notably Llama-3.2-1B-Instruct (+24.4 pp vocabulary) and Phi-4 (+25.8 pp conversational comprehension). Reading comprehension shows more variable results, with large gains for some models (Ministral-3-3B-Instruct: +30.8 pp) but modest improvements elsewhere. Grammar scores show the most modest changes, with gains rarely exceeding 10 pp, suggesting that the synthetic data is less effective at improving structural linguistic knowledge. The two degrading models (Qwen2.5-0.5B-Instruct and Gemma-3-1B-IT) show uniform drops across all four categories, consistent with their overall regression on language exams in Table 2.

5.3 RQ2: Does fine-tuning on LuxIT improve performance on downstream Luxembourgish NLP tasks?↩︎

Figure 2 shows the average change in performance on the Luxembourgish NLP tasks (Section 4.3), comparing baseline and LuxIT fine-tuned models (see Appendix 10.2 for a breakdown of the results). Fine-tuning on LuxIT leads to improvements in Luxembourgish NLP task performance for 9 out of 14 models, as measured by macro-averaged F1 (Figure 2) across intent classification, RTE, sentiment analysis, SST, and WNLI. The most substantial gains are observed for Phi-4 (\(\Delta\)F1-macro = +0.325), Olmo-3-7b (+0.211), and qwen2.5-7b (+0.172), while the largest regressions occur for Apertus-8b (-0.206) and Gemma-3-12b (-0.128). Models that improve on language exams (RQ1) do not systematically improve on NLP tasks, and vice versa with notable divergences for Ministral-3-3b, which shows the largest language exam gain (+21.0pp) alongside near-zero NLP task change (-0.003), and Gemma-3-1b, which shows a modest NLP task gain (+0.064) despite a substantial language exam regression (-22.0pp).

It is to note that there are substantial outliers that influence the average performance change (see Appendix 10.2, Figure 5). Most notably, Apertus-8b performs poorly on the RTE task after fine-tuning on LuxIT, with a change in performance of -0.677 and Gemma3-12b’s performance on IC changes by -0.522. On the other hand, we observe a considerable gain of +0.802 for Phi-4 in the SST task. While those outliers explain the extreme average changes we observe for Phi-4, Gemma3-12b, and Apertus-8b, it is unclear why those outliers occur.

6 Discussion↩︎

LuxIT represents an important step toward addressing the scarcity of instruction-tuning resources for Luxembourgish. Our pipeline effectively synthesizes a large dataset from monolingual seed data. Furthermore, our dataset composition audit demonstrates that the open-ended generation prompt yields a balanced coverage of diverse instruction types (summarization, extraction, Q&A, and explanation). The human evaluation underscores the subjectivity of manually evaluating synthetic Luxembourgish text. We observe low human inter-annotator agreement (Table 7) alongside metric-specific biases between the automated judge and native speakers (Table 1). While the LLM judge shows a leniency bias regarding Helpfulness Relevance, human evaluators are more forgiving regarding Factual Accuracy. This divergence highlights the need for more objective evaluation frameworks for future iterations.

Regarding language proficiency, fine-tuning generally enhanced formal Luxembourgish comprehension. However, the divergent outcomes among the smallest models suggest that fine-tuning receptivity depends on more than model size alone. Models with limited prior task alignment saw the most benefit, whereas regressions in other sub-1.5B models with stronger baselines likely point to catastrophic forgetting of pre-trained representations.

A deeper analysis of the language exam levels reveals that gains from LuxIT are not uniformly distributed across proficiency levels. Improvements were most consistent at intermediate proficiency levels (B1/B2) and in vocabulary and conversational comprehension. Contrarily, structural grammar and advanced native-like nuances (C2) proved difficult to improve. This suggests that the synthetic data, constrained by the generator model’s capabilities and the formal nature of the seed corpora, effectively expands the models’ lexicon but skews toward intermediate proficiency.

Furthermore, the limited correlation between language exam improvements and downstream NLP task performance highlights the specific nature of our dataset. The language exams feature standardized, well-formed Luxembourgish, while the NLP tasks contain noisy, non-standardized text. Because LuxIT is derived from formal Wikipedia and news articles, the fine-tuned models appear to align better with the language style of the formal exams. This divergence suggests that format alignment and linguistic improvement are separable effects that instruction tuning on LuxIT does not uniformly induce across all models.

Despite these nuances, in terms of absolute capability, fine-tuning medium-sized models on LuxIT successfully yielded highly proficient Luxembourgish models.

7 Conclusion↩︎

This work addresses the scarcity of instruction tuning resources for low-resource languages by introducing LuxIT, a high-quality dataset synthesized exclusively from monolingual Luxembourgish seed texts. By fine-tuning fourteen small-scale LLMs, we demonstrate that LuxIT broadly improves formal language comprehension, particularly in expanding vocabulary and enhancing intermediate-level proficiency, even if performance on noisy, structured downstream NLP tasks remains less consistent. Ultimately, our findings validate that leveraging monolingual synthetic data is an effective and feasible strategy for enhancing LLM capabilities in low-resource linguistic settings. Future work will explore open-ended conversational evaluations and multi-turn data integration.

Limitations↩︎

We acknowledge several limitations. First, the source data for LuxIT is originating from Wikipedia and RTL news articles. This domain specificity means the dataset inherently skews toward formal, standardized, and objective language registers. Consequently, LuxIT may lack the natural conversational flow, regional dialectal variations, or informal creative language styles found in everyday communication. Future data synthesis efforts should target these gaps by incorporating conversational transcripts or informal text to broaden the linguistic coverage.

Second, our human evaluation, while crucial for validation, was conducted on a relatively small subset of 100 samples. A larger-scale human evaluation by multiple native speakers would lend greater statistical power to our quality claims and provide deeper insights into the dataset’s strengths and weaknesses.

Third, we acknowledge the observed leniency bias of our single LLM judge (GPT-5-mini) and human evaluators. As shown by the low Exact Agreement and Cohen’s Kappa scores (Table 1), biases are metric-dependent. The LLM overestimates Helpfulness Relevance compared to humans, suggesting the final dataset may contain instructions that are marginally less useful than their automated scores imply. Furthermore, the human audit reveals a negative inter-annotator agreement among the native speakers themselves (Fleiss’ Kappa of -0.019 for Factual Accuracy, Table 7). Combined with their skewed leniency compared to the LLM (human mean of 2.84 vs. the LLM’s 2.40), this indicates our evaluation guidelines allowed for highly subjective interpretations. This emphasizes the difficulty of manually evaluating synthetic text in Luxembourgish. Future iterations should implement more granular rubrics and annotator calibration to improve human consistency, alongside stricter threshold prompting or multi-model LLM ensembles to enhance automated filtering.

Fourth, while our approach is strictly monolingual in its seed data, the generation model, DeepSeek-R1-0528, is a multilingual model. Its internal knowledge is shaped by the vast multilingual corpus it was pre-trained on, which could subtly influence the style and structure of the generated Luxembourgish.

Fifth, our downstream evaluation is restricted to multiple-choice language proficiency exams. While this provides a rigorous and objective measure of grammatical, lexical, and comprehension skills, it does not evaluate the models’ open-ended text generation, multi-turn conversational abilities, or specific instruction-following nuances. Consequently, the true conversational proficiency of the fine-tuned models remains to be evaluated.

Finally, Apertus-8B-Instruct is constrained to a maximum sequence length of 512 tokens due to hardware limitations, versus 2048 tokens for all other models. As some LuxIT pairs may exceed this length, its results may underestimate the model’s true capacity under standard training conditions.

Ethical Considerations and Licensing↩︎

Licensing of Seed Data and Synthetic Outputs↩︎

The LuxIT dataset was synthesized using two primary data sources: The Luxembourgish Wikipedia and Luxembourgish articles from RTL.

• Wikipedia Data: The Luxembourgish Wikipedia data used as seed text is publicly available under the Creative Commons Attribution-ShareAlike 4.0 International (CC BY-SA 4.0) license.

• RTL Data: The use of RTL news articles as seed data is governed by a formal data-sharing agreement established with RTL. While the original articles are publicly accessible content, our agreement explicitly permits their use for academic research and dataset generation.

• Synthetic Dataset (LuxIT): Because the generation model (DeepSeek-R1-0528) does not impose restrictive licensing on its generated outputs, we release the final synthesized LuxIT dataset under the CC BY-SA 4.0 license, aligning with the terms of our Wikipedia seed data and ensuring open access for future research.

Licensing of Evaluation Benchmarks↩︎

The evaluation of our fine-tuned models relies on proprietary and open-source benchmarks:

• INLL Language Exams: The Luxembourgish language proficiency exams (used in Section 4.2) are proprietary materials owned by the Institut National des Langues Luxembourg (INLL). We obtained explicit permission from INLL to use these standardized exams exclusively for the automated evaluation of our language models. These exams are not publicly distributed and are kept strictly confidential.

• Downstream NLP Tasks: The five downstream NLP tasks (Intent Classification, WNLI, RTE, Sentiment Analysis, and SST-2) were sourced from [36], [17] and [37] for the latter three tasks respectively. We utilized this benchmark data under its original terms of use for academic evaluation.

Ethical Considerations in Data Generation↩︎

We acknowledge that utilizing LLMs to synthesize data carries the risk of propagating or amplifying biases present in the seed text or the generation model itself. To mitigate this, we excluded user-generated comments from the RTL corpus, as they are more prone to biased content, toxicity, and grammatical errors. Our heuristic filtering prioritized formal, objective texts (news reports and encyclopedic articles). However, because LuxIT skews heavily toward formal registers, the resulting models may not fully capture the nuances of informal or conversational Luxembourgish. We encourage researchers utilizing LuxIT to be mindful of these domain-specific constraints when deploying models in real-world applications.

Acknowledgments↩︎

This research was supported by LLMs4EU, co- funded by the Digital Europe Programme under GA 101198470 , and by the BESSER project through the Luxembourg National Research Fund (FNR) PEARL programme under the grant agreement 16544475.
Our gratitude goes out to RTL for supplying the Luxemburgish news articles and to the Institut National des Langues Luxembourg (INLL) for providing the Luxembourgish language exams. We thank Aline Fries and Aaron Conrardy for helping us with the human evaluation part. We also thank Bernhard Voggenberger for advising us on the fine-tuning experiments.

8 LuxIT↩︎

8.1 Data structure↩︎

Both data sources are contained in a JSON file. The following shows an example entry of an RTL news article¹⁰ and a Wikipedia article. We only extract public_date, title, header and text from the RTL news article and extract title and text from the Wikipedia article.

{
 'category_name': 'International',
 'category_id': 5,
 'article_id': 2190645,
 'type': 'news',
 'public_date': '2024-04-28 14:04:54',
 'title': '<article title>',
 'header': '<article header>',
 'text': '<article title>',
 'tags': ['international'],
 'text_id': 301779,
 'lang_id': 'lb'
}

{
  "id": '25',
  "revid": '580', 
  "url": "https://lb.wikipedia.org/
    wiki?curid=25",
  "title": "Matthew Perry",
  "text": "De Matthew Langford Perry, gebuer den 19. August 1969 zu Williamstown am Massachusetts, a gestuerwen den 28. Oktober 2023 zu Los Angeles, war en US-amerikanesch-kanadesche Schauspiller, dee virun allem duerch seng Roll als Chandler Muriel Bing an der Televisiounsserie Friends bekannt ginn ass..."
}

8.2 Data generation model prompt↩︎

The data generation is performed by DeepSeek-R1-0528. We instruct the model to return the data in JSON format, to ensure compatible formatting and to make it easily accessible. The prompt is formulated as follows:

8.3 LuxIT Sample↩︎

Here we show a sample of LuxIT taken from the Wikipedia subset.

8.4 Prompt Template↩︎

Here we show the prompt template in more detail that unifies columns from LuxIT (Table 4). The Instruction and Response columns are brought together in the ChatML format:

messages = 
[
    {
        "role": "user",
        "content": <instruction>
    },
    {
        "role": "assistant", 
        "content": <response>
    }
]

8.5 LLM-as-a-judge↩︎

Here we break down the reasoning for using GPT-5-mini as a judge (Section 3.4). Table 5 shows the evaluation results from GPT-5 and GPT-5-mini compared to GPT-4o and GPT-4o-mini, where the accuracy scores for the latter 2 were taken from [13] and we simply added the macro average.

8.6 Custom scoring metric↩︎

We fully break down our custom scoring metric introduced in section 3.4:

Linguistic Quality score

• Score 1: Contains significant grammatical errors, spelling mistakes, or unnatural phrasing in Luxembourgish. Text that is actually German or French instead of proper Luxembourgish should receive this score.

• Score 2: Mostly correct Luxembourgish, but has minor errors or sounds slightly robotic/unnatural. May mix in too many loan words unnecessarily.

• Score 3: Fluent, idiomatic, and grammatically perfect Luxembourgish. Natural-sounding text that a native speaker would produce.

Factual Accuracy score

• Score 1: Contains factual errors that contradict the source text or general knowledge.

• Score 2: Mostly accurate but might have minor inaccuracies or omissions.

• Score 3: Completely accurate according to the source text and factual knowledge.

Instruction Adherence score

• Score 1: Fails to follow the core instruction (e.g., provides a summary when asked for a list).

• Score 2: Follows the main instruction but misses a constraint (e.g., writes 4 bullet points when asked for 3, wrong format, or incorrect tone).

• Score 3: Perfectly follows all parts of the instruction, including constraints like length, format, and tone.

Helpfulness Relevance score

• Score 1: The instruction is nonsensical, irrelevant to any reasonable context, or the response is unhelpful/off-topic.

• Score 2: The instruction is plausible but not very insightful or creative. The response addresses the instruction but in a basic way.

• Score 3: A genuinely useful, interesting, or creative instruction that elicits a helpful, comprehensive response.

8.7 Post-filtering model prompt↩︎

We post filter the synthetic data with GPT-5-mini. Again, we want the scores to be in JSON format. The prompt is as follows:

8.8 Dataset Quality↩︎

In Table 6 we provide detailed statistics on the LuxIT score distribution introduced in Section 3.5.

8.8.1 Human Evaluation↩︎

We provide details for our human evaluation results in Table 7.

9 Experimental Setup↩︎

Here we provide more details accompanying the Experimental Setup Section (Section 4).

9.1 Fine-Tuning on LuxIT↩︎

We provide more details for the hyperparameter settings in Table 8. (Introduced in Section 4). Several hyperparameters and configurations are held constant across all models unless otherwise specified.

All models are fine-tuned for 2 epochs using the 8-bit AdamW optimizer and a cosine learning rate scheduler. We consistently apply a warmup ratio of 0.03, a weight decay of 0.01, and keep the default value for max_grad_norm at 1.0. For the Low-Rank Adaptation (LoRA) setup, the rank (\(r\)) is set to 16, the scaling factor (\(\alpha\)) to 32, and the dropout rate to 0.05. Gradient checkpointing is enabled across all training runs to optimize memory consumption. During training, logging occurs every 200 steps, while evaluation and model checkpoint saving are performed every 500 steps. For the dataset, 5% is strictly held out for validation and 3% for testing across all runs.

Notably, while a maximum sequence length of 2048 tokens is utilized for the vast majority of the models, we constrain the Apertus-8B-Instruct model to a sequence length of 512 tokens due to hardware memory limitation requirements. Furthermore, all models trained via the Unsloth framework utilize a fixed random seed of 3407 to ensure reproducibility, whereas the Apertus-8B-Instruct model, which we train using standard Hugging Face pipelines, is initialized with a random seed of 42.

9.2 Evaluation on Luxembourgish downstream tasks↩︎

We break down the dataset class distribution in Table 9 introduced in Section 4.3.

10 Results↩︎

Here we provide more details accompanying the Results section (Section 5).

10.1 RQ1 i)↩︎

We provide detailed visual comparison between our fine-tuned models and their respective baseline in Figure 3 (Section 5.1).

10.2 RQ2↩︎

We report the results of benchmarking our fine-tuned models against their respective baseline models (Section 5.3) on the Luxembourgish downstream tasks in Table 10 (introduced in Section 4.3) and show the per-task changes and the average change per model in Figure 5 and Figure 2 respectively.

5pt

References↩︎