1 Introduction↩︎

Language Service Providers (LSPs), large and small, are increasingly using large language models (LLMs) in their localization workflows. Freelance translators do it too in sporadic and ad hoc ways [1]. In an earlier paper [2], we demonstrated the value of simple analytic tools—automatic metrics, structured human assessment, and lightweight statistical analysis—that were historically confined to MT research and large-scale industrial QA but are increasingly accessible to individual translators and small LSPs. Using the Christopher & Dana Reeve Foundation Trilingual Corpus (RFTC) derived from a real medical-domain translation project, we showed how freelancers can compute and interpret automatic evaluation metric scores (BLEU, chrF, TER, COMET) with minimal infrastructure and can correlate those scores with structured human judgments to determine which metrics track translator-perceived quality in a specific setting. A key takeaway was pragmatic: evaluation need not be “big tech only.” Even modest, sample-based analyses can be informative and statistically stable for system comparison in a real workflow. Our findings emphasize the importance of proactive engagement with modern technologies and analytic methods to not only adapt but thrive in the rapidly evolving professional environment.

In our second paper, we apply this framework to a niche area of the translation services market. We focus on the base-level translation performance of smaller language models that can be installed and run locally by users with modest resources and no special programming skills. This is important for several reasons outlined below.

1.1 Practical Considerations: Why Offline Translation May Be Required↩︎

In parallel with the spread of cloud MT and frontier LLMs, translation practice has also seen a countervailing demand: strict data-control workflows [3]–[6]. Certain projects in defense, security, finance, and patent-related work require offline processing or air-gapped environments. In such scenarios, API-based commercial systems are not suitable, regardless of quality. This creates a concrete need for translators to evaluate and deploy locally runnable technologies—either local NMT toolchains² or newer open-weight LLMs that can be executed locally through end-user applications.

Local LLM execution has rapidly become more feasible for non-programmers due to desktop inference tools and model packaging ecosystems. The remaining question for freelancers is the empirical one we address below: whether base translation quality from small-to-mid-size local LLMs is good enough to support confidential professional workflows.

1.2 Theoretical Considerations: Scaling Laws and Interpretability↩︎

From a more theoretical perspective, studying the translation performance of smaller language models of different sizes, such as llama3.1-8b, gpt-oss-20b, or gemma3-27b, could cast additional light on “scaling laws.” The earlier literature on neural language modeling shows robust scaling-law behavior for model performance as a function of parameters, data, and compute [9], [10]. Translation provides a concrete capability domain where scaling may or may not behave monotonically across languages and writing systems [11]–[15]. Our multilingual corpus and a broad selection of local LLMs of different sizes and provenance give us an opportunity to study this behavior in a practical setting.

Additionally, compared with many proprietary systems, open-weight releases often provide more transparent model cards, training descriptions, and language coverage [16]–[20]. Observed translation differences can therefore be discussed in relation to architecture and multilingual training choices with fewer unknowns.

1.3 Related Work↩︎

The application of LLMs to translation and other multilingual tasks began essentially as soon as LLMs themselves emerged. Brown et al. demonstrated that GPT-3 could perform zero- and few-shot translation, and subsequent work rapidly explored the boundaries of this capability. Major evaluations by Hendy et al., Jiao et al., and Vilar et al. suggested that frontier LLMs could achieve translation quality competitive with dedicated NMT systems for high-resource language pairs, though gaps remain for low-resource directions and specialized domains. A comprehensive recent survey by Ataman et al. traces the trajectory from early neural MT architectures through the current LLM era, highlighting both the promise and the open problems including domain adaptation, evaluation methodology, and the role of parallel versus monolingual training data. Lyu et al. argue that a paradigm shift is underway in which LLMs will increasingly subsume traditional NMT, particularly for document-level and stylized translation. Prompting, few-shot, and iterative-refinement strategies have continued to push LLM translation quality upward [21]–[32]. Alongside general-purpose families, dedicated translation LLMs such as Tower [12], [13] and, more recently, SalamandraTA [33] and Tower+ [34] have continued to improve open-weight translation quality at modest scale.

The more recent rollout and rapidly increasing accessibility of local language models, facilitated by inference platforms such as Ollama [35] and LM Studio [36], have led MT researchers and translation scholars to explore the translation capabilities of models that can run entirely on consumer hardware. Cui et al. systematically evaluate open LLMs with fewer than ten billion parameters on multilingual MT tasks, finding that models like Gemma2-9B exhibit impressive multilingual translation capabilities. They introduce the GemmaX2-28 model, which achieves competitive performance with Google Translate and GPT-4-turbo across 28 languages. Sandrini investigates the feasibility of locally deployable, free language models as alternatives to proprietary cloud-based solutions from the perspective of practicing translators, evaluating three open-source models (llama3-8b, mixtral-8x7b, and gemma2-27b) with three toolkits (GPT4ALL, Llamafile, and Ollama) to generate translations in tourism/marketing and legal domains from Italian to German. In Sandrini’s setup, MATEO was used to score those translations against outputs from ChatGPT-4.0-mini and Gemini-1.5-flash, which served as reference translations.

Our work falls in this category but differs from both [37] and [6] in important respects. Unlike the former, our main audience comprises users, not developers, of translation technologies. Additionally, while  Sandrini’s user study is pioneering in its category, we take it to be subject to several limitations: the sample size is modest (fewer than twenty-five sentences); inference is performed on a CPU, typically relying on lower-precision quantization and suboptimal backends, which may introduce greater output variance than GPU-based higher-precision inference; and LLM outputs, rather than professional human translations, serve as reference translations.

We study a broader range of more recent local LLMs using a medical corpus originating from a real translation project. While our source language is English, our target languages (DE, RU, JA, ZH) comprise a typologically diverse set with different writing systems, thus allowing us to investigate interesting cross-linguistic phenomena. While we only use Ollama for all our translation calls, our sample is much larger (over a thousand sentences) and the translation behavior is more stable due to our use of a consumer-grade GPU. The larger sample makes our results statistically significant.

1.4 Our Contributions↩︎

This paper contributes:

• An expansion of the Reeve Foundation Trilingual Corpus (RFTC) to German and Simplified Chinese. The resulting Reeve Foundation Multilingual Corpus (RFMC) now includes approximately 3,500 source sentences in the medical domain aligned with their recent professional translations to four typologically different languages: EN\(\to\). With the client’s permission, we make the RFMC available for noncommercial/academic use.

• A freelancer-oriented benchmark of 9 locally runnable LLMs against commercial NMT, a frontier LLM, and local NMT baselines, under “base translation only” constraints.

• A meta-analysis of prompting and decoding strategies.

What makes our approach distinctive is the combination of high-quality data from a recent professional translation project, systematic leverage of accessible small LLMs, and the translation experience of some team members, including an ATA-certified professional translator, which informs our evaluation methodology and interpretation of results. We frame this work explicitly as a user-oriented benchmark, written by users of translation technology, for tech-savvy freelance translators and smaller LSPs who have only modest resources and no or little programming skills.The methods and code are intended to be accessible to this audience.

The remainder of the paper presents corpus expansion (Section 2), experimental design (Section 3), translation generation (Section 4), automatic evaluation (Section 5), linguistic observations (Section 6), and implications for practice-, scaling-, and cost-oriented research (Section 7).

2 Corpus Expansion↩︎

The Reeve Foundation Trilingual Corpus (RFTC), introduced in [2], pairs English source sentences from the International Edition of the Christopher & Dana Reeve Foundation’s Paralysis Resource Guide (PRG) with their recent professional translations to Russian and Japanese. To enable a broader, multilingual study, we expanded the RFTC by adding two additional target languages: German (DE) and Simplified Chinese (Mainland China; ZH).

The expansion required aligning the existing professional translations of PRG International in German and Chinese, published by the Reeve Foundation as PDF documents, with the existing trilingual segments (EN–RU–JA) already contained in the RFTC. This entailed two main steps.First, we extracted text from the PDF files in DE and ZH, performing cleanup and downsizing analogous to those described in Section 2 of [2]. Second, we undertook corpus curation to address a version mismatch. The existing translations of the PRG International to RU and JA, already contained in the RFTC, derive from the most recent English edition (2023). However, the official translations of PRG International to DE and ZH were based on the earlier English edition (2022). As a preliminary step, we identified the segments common to both editions (approximately 70% of the corpus) and retained the existing DE and ZH translations for those segments. For the remaining segments (approximately 30%), which had been added or modified in the 2023 edition, we updated the translations using expert knowledge of DE and ZH.

The resulting Christopher & Dana Reeve Foundation Multilingual Corpus (RFMC) presents approximately 3,500 source sentences (EN) aligned with their original professional translations to RU and JA, as well as their partially curated reference translations to DE and ZH. The RFMC is made available on GitHub for noncommercial/academic use.³

3 Experimental Design↩︎

We selected a continuous portion of RFMC—the entire cleaned and curated Chapter 1 of PRG International-2023 containing over a thousand sentences—for our experiments to explore the translation performance of 9 local language models available on Ollama [35]:

• aya-expanse-32b [19]

• deepseek-r1-32b [38]

• gemma3-27b [16]

• gpt-oss-20b [39]

• llama3.1-8b-instruct-fp16 [40]

• mistral-small3.2-24b [41]

• qwen2.5-32b [17]

• qwen3-32b [18]

• translategemma-27b [42]

and compared their performance with three important baselines:

• The best-performing commercial NMT system: DeepL for EN–DE, EN–RU, and EN–JA; Baidu for EN–ZH⁴

• The best-performing frontier commercial LLM: GPT-5.2 [43]

• The best-performing generally available local NMT system: Promt-23 for EN–RU, EN–JA, and EN–ZH; NeuralDesktop for EN–DE⁵

This section describes our model and prompt selection process.

3.1 Model Selection↩︎

Our model selection was guided by several practical constraints reflecting the resources typically available to tech-savvy freelance translators and smaller LSPs. We sought open-weight models that (i) could run locally via Ollama on a consumer-grade GPU, (ii) covered a range of parameter counts from 8B to 32B to allow investigation of scaling effects, (iii) represented diverse model families and provenance (open-weight releases from Google, Meta, Mistral, Alibaba, Cohere, DeepSeek, and OpenAI), (iv) routinely or historically performed well on translation benchmarks, and (v) included both general-purpose and translation-specialized architectures.

Our inference hardware consisted of NVIDIA GeForce RTX 3090 GPUs (24 GB VRAM), a consumer-grade card widely available on the secondary market at the time of writing. Each translation run was executed on a single GPU, reflecting the setup a typical freelancer or a small LSP might reasonably assemble. Although our workstation housed multiple cards, they were used only to parallelize independent experiments; no model was distributed across GPUs, and all reported results reflect single-card performance.

The inference engine was Ollama v0.14.2, running on Ubuntu 24.04. Ollama provides a straightforward command-line interface for downloading, managing, and serving open-weight models, requiring no programming expertise beyond basic terminal familiarity.We note that comparable results would be expected from other llama.cpp-based inference frontends, such as LM Studio, which offers a graphical interface and may be more approachable for users less comfortable with the command line.

Models at or above 24B parameters (aya-expanse-32b, deepseek-r1-32b, gemma3-27b, mistral-small3.2-24b, qwen2.5-32b, qwen3-32b, translategemma-27b) were served in Ollama’s default 4-bit quantization (Q4_K_M), balancing inference speed and memory footprint against output quality. The smaller models (gpt-oss-20b, llama3.1-8b-instruct) fit comfortably within the 24 GB VRAM budget without quantization: gpt-oss-20b was run at its default precision, while llama3.1-8b-instruct was run at full fp16 to explore whether preserving numerical fidelity at a smaller scale could compensate for fewer parameters. As a general principle, we selected the lowest quantization level (i.e., highest precision) that could reasonably fit into 24 GB of VRAM. All models were loaded entirely into GPU memory for each run. Because our translation pipeline processes each sentence as an independent API call with no carried context (Section 4.1), the effective context window per request was short, and VRAM was not a binding constraint for any model in our selection.

We note that while by industry standards, these are modest resources, most freelancers, even technically-inclined ones, may not have immediate access to comparable hardware. However, the landscape is changing rapidly, with GPUs and bundled inference engines increasingly packaged into new PC and Mac consumer offerings, and cloud GPU rental services becoming more affordable. Further details on hardware specifications and runtime configurations are provided in Appendix 9.

3.2 Prompt Selection↩︎

Recent work on using LLMs for translation suggests that smaller models’ outputs tend to be more sensitive to prompting details than the outputs from larger models [14], [21], [22], [30]. Since our explicit goal is to study the translation performance of smaller language models, we began with pilot experiments on a smaller set of 118 test sentences from other parts of PRG International not overlapping our main source document. These sentences were manually selected to cover linguistic phenomena representative of the entire corpus: sentences containing URLs, named entities (organization, program, and product names; personal proper names), technical terms and acronyms; and longer clauses.

The models were then asked to translate these 118 sentences to a single language from our target set (DE) with 11 different candidate prompts. Those candidate prompts were compiled as follows: our 8 selected local language models and 3 frontier LLMs (Claude Opus 4.5, GPT-5.2, and Gemini Pro 3) were “meta-prompted” to generate their own preferred prompt for translation:

We need a system prompt for a translation task. The requirements are:
- Source language: English
- Target language: German
- Audience: expert/academic
- Output: translation only, no explanations, annotations, or transliterations
Write the prompt you would want to receive if you were performing this task. Output only the prompt text itself, nothing else.

Candidate prompts were elicited from each model: local models via the Ollama API used later for translation, frontier LLMs via context-free manual chat sessions.Two prompt suggestions, from llama3.1-8b-instruct-fp16 and aya-expanse-32b, were judged inadequate and discarded.The prompt from llama3.1-8b-instruct was discarded because it produced an improperly formatted response that would not have functioned as a system prompt, and we chose not to manually correct it to avoid introducing experimenter bias.

Interestingly, aya-expanse-32b attempted to create its preferred translation prompt in German instead of English.This intent is consistent with recent studies that explicitly prompt LLMs in languages other than English, sometimes resulting in improved performance on multilingual tasks [44]–[46]. We decided to let our models explore this opportunity and included a German version of a “standard” prompt from [2] alongside its English version.

Each of the 11 candidate prompts (p1–p11; see Appendix 10) compiled in this way was then used to query each of our eight models to translate 118 held-out test sentences from EN to DE. Thus, every model generated a translation from every candidate prompt. The COMET-22 scores for the resulting 88 translation outputs (Table 4) do not, by themselves, mark a clear winner among and do not appear to correlate with the “native” prompt choice: in many cases the prompt suggested by model \(X\) failed to maximize the COMET scores for \(X\)’s own output.

Accordingly, we decided to look for a prompt that was best “on average.”Two methods were used for this purpose: (i) maximizing the sum of \(z\)-scored COMET improvements over the prompt mean across models; and (ii) rank aggregation across the models. Both methods selected p5 as “the best prompt on average”:

p5: Translate the following English text into {target_lang}. The target audience is academic/expert. Provide *only* the {target_lang} translation, without any explanations, annotations, or transliterations.

During this pilot phase (mid-January 2026), TranslateGemma became available [42]. We decided to add it to our 8 initial LLMs but used a different prompt (p0), which was explicitly recommended by its developers:

p0: You are a professional {source_lang} ({src_lang_code}) to {target_lang} ({tgt_lang_code}) translator.Your goal is to accurately convey the meaning and nuances of the original {source_lang} text while adhering to {target_lang} grammar, vocabulary, and cultural sensitivities. Produce only the {target_lang} translation, without any additional explanations or commentary. Please translate the following {source_lang} text into {target_lang}: \n \n \n {TEXT}

We used p0 only for TranslateGemma.Although this introduces an additional variable, TranslateGemma was the sole translation-specialized model in our panel, and it was trained with p0 (ibid., p.).We judged that respecting the developer-recommended prompt for a single model gave a fairer picture of its base translation capability than forcing a generic prompt onto it.

3.3 Temperature Settings↩︎

We conducted temperature sensitivity experiments for a subset of our models across the range \(T=0.0\) to \(T=1.0\) in increments of 0.05. Consistent with prior findings on LLM-based translation [2], [22], translation quality as measured by COMET did not vary substantially across temperature settings. However, analysis of pairwise Levenshtein ratios across temperature conditions revealed a clear pattern: outputs generated at \(T=0.0\) were maximally similar to outputs at all other temperatures, effectively serving as the centroid of the output distribution. As temperature increased, outputs diverged not only from the low-temperature baseline but increasingly from one another, with the lowest pairwise similarity observed between adjacent high-temperature settings (e.g., \(T=0.95\) vs.\(T=1.0\)). These results suggest that for translation, temperature primarily affects output consistency rather than quality. We therefore set \(T=0.0\) for our main experiments with “non-reasoning” models, to maximize reproducibility. A sample Levenshtein ratio matrix for the temperature sweep is provided in Appendix 11 and Figure 3.

However, we found that running three “reasoning” models with zero decoding temperature caused them to “loop” and time out, which is consistent with other recent studies [47]. To minimize the negative impact of infinite “reasoning loops” we decided to use \(T=1.0\) for gpt-oss-20b and \(T=0.6\) for qwen3-32b (model card overrides), and \(T=0.8\) for deepseek-r1-32b (Ollama default). We provide additional details on this in Section 4.1 and Appendix 12.

3.4 Target Language Specification↩︎

Prompting language models for translation requires explicit specification of a target language.In our case, the prompt recommendation for TranslateGemma also required specifying an ISO code for the target language. This was straightforward for German (DE), Russian (RU), and Japanese (JA). The initial options for Chinese included ‘Simplified Chinese’, ‘Mandarin’, ‘Mandarin Chinese’, ‘Chinese (Mandarin)’, ‘Simplified Chinese (Mandarin)’, and ‘Simplified Chinese (Mainland China)’. Upon consultation with a native Chinese speaker familiar with the subject matter and with ChatGPT-5.2, the set of options was reduced to ‘Simplified Chinese (Mainland China)’, ‘Simplified Chinese’, and ‘Simplified Chinese (Mandarin)’. A quick translation of 24 English sentences selected from our corpus with three versions of the p5 prompt incorporating these options did not result in significant COMET score differences for any of our models. We chose to use ‘Simplified Chinese (Mainland China)’ and ‘zh-CN’ in our main experiments.

4 Generation of Translation Outputs↩︎

As noted earlier (Section 3), we selected a continuous part of RFMC—the entire curated Chapter 1 of PRG International-2023 containing 1,143 sentences—for our main experiments.

4.1 Translation Outputs from Local Language Models↩︎

We generated translations of the selected document to {DE, RU, JA, ZH}, using p5 (our “best prompt an average,” Section 3.2) to interact with all the selected models except TranslateGemma, for which we used p0, as recommended by its developers. Outputs were stored in a normalized, line-aligned format to support automated scoring and later manual review.⁶

Each source sentence was translated through an independent API call to the local Ollama server, with the selected prompt (Section 3.2) provided as the system message and the source sentence as the user message. No conversational context or translation memory was carried between calls. This sentence-by-sentence approach, while less sophisticated than document-level translation methods that leverage surrounding context [48], eliminates confounding factors related to context window management and ensures straightforward reproducibility. It also mirrors the segment-by-segment workflow common in CAT tool environments, where MT suggestions are typically generated for individual translation segments. We note that providing additional context—for example, by constructing a lightweight translation memory from previously translated segments—could plausibly improve output quality for local LLMs, but investigating such strategies is outside the scope of the present study. Our results should therefore be understood as a lower bound on achievable translation quality with these models.

As already noted, six “non-reasoning” models were run at \(T=0.0\), consistent with the findings reported in Section 3.3. Output post-processing was minimal: where a model returned extraneous formatting (e.g., markdown fences, list prefixes, or wrapping quotation marks), the output was automatically stripped to the translation content only, though such artifacts were infrequent with the models in our final selection.

The three reasoning-oriented models in our set—gpt-oss-20b, deepseek-r1-32b, and qwen3-32b—were the only models susceptible to generation failures due to their internal chain-of-reasoning processing. These failures occurred when the model entered repetitive reasoning loops, consuming the token budget without producing a final translation. gpt-oss-20b was by far the most affected, producing 99 failed lines across the four language directions when the decoding temperature was set to zero. Changing the temperature to 1.0 dramatically reduced the number of failed calls to 7, but also appears to have a negative effect on the output quality. Appendix 12 provides further details.

All remaining models completed the full corpus without failure with \(T=0.0\). Wall-clock translation times per language direction on a single RTX 3090 ranged from roughly 19–40 minutes for non-reasoning models to 2.9–5.5 hours for reasoning-oriented models (Table 3, Appendix 9); a fuller per-model runtime breakdown will accompany the code release.

4.2 Additional Translation Outputs↩︎

For each of our translation directions (EN\(\to\)), we generated additional outputs from generally-available commercial NMT systems (Google Translate, DeepL, Baidu), a frontier LLM (GPT-5.2), and generally-available local NMT systems (Promt, NeuralDesktop, OPUS-CAT). These systems represent the spectrum of tools currently accessible to professional translators: cloud-based engines offering state-of-the-art quality, API-accessible frontier LLMs, and offline NMT solutions designed for confidential work. See Section 3 for references to these systems.

As detailed below (Section 5), based on the document-level COMET-22 scores of our outputs, we selected three systems for each language pair: the best-performing commercial NMT system, the best-performing frontier LLM, and the best-performing generally available local NMT system. These served as important baselines for our evaluations.

5 Automatic Evaluation↩︎

5.1 Metrics and Tools↩︎

Since our target users are tech-savvy freelance translators and smaller language service providers (LSPs) who have only modest resources and little or no programming skills, we used MATEO [49]—a free, user-friendly web application—to generate the BLEU [50], chrF [51], TER [52], and COMET [53], [54] scores for all 12 translation outputs (9 local LLMs \(+\) 3 baselines) for each of our language pairs. MATEO accepts parallel text files (source, reference, and up to four system outputs) and computes multiple automatic metrics through an intuitive web interface, making it an ideal tool for our target audience.

The evaluation results are reported in Section  5.2 below and Appendix 13, as well as in the companion materials in our repository. We note that most of the pairwise score differences are statistically significant at the \(p < 0.05\) level, lending confidence to the system rankings we derive.

We observe strong pairwise Pearson correlation between COMET and two string-based metrics, BLEU and chrF2, across all available systems for EN\(\to\)DE and EN\(\to\)RU: \(r = 0.73\)–\(0.92\); and moderate correlation for EN\(\to\)JA and EN\(\to\)ZH: \(r = 0.56\)–\(0.68\).The weakest correlation, for EN\(\to\)ZH, is likely attributable to the character-level properties of Chinese writing, which can introduce discrepancies between string-matching and neural evaluation approaches. Following common practice in MT evaluation, we set the string metrics aside at this point and used document- and sentence-level COMET scores in all subsequent evaluations.

5.2 Summary of Document-Level Results↩︎

Table 1 presents document-level COMET scores for all the available translation outputs, including two different temperature settings for three “reasoning models.” The colored bars in Figure 2 display the best results for each of the 9 local LLMS set against the available baselines (gray bars).

Based on these scores, the best-performing local LLM for EN\(\to\) is translategemma-27b, while mistral-small3.2-24b demonstrates the best performance for EN\(\to\)DE. We note that these local LLMs are ahead of GPT-5.2 for EN\(\to\) and slightly behind it for EN\(\to\)RU. However, all the LLM outputs, including our local models and GPT-5.2, score significantly lower than the best commercial NMT systems: DeepL for EN\(\to\) and Baidu for EN\(\to\)ZH.

Classical scaling laws predict that model performance improves as a smooth function of parameter count [9], [10]. Our document-level COMET scores (Table 1 and Figure 2) offer only partial support for this prediction in the translation domain. The clearest scaling signal appears at the lower end of the parameter range: llama3.1-8b-instruct-fp16 is one of the weakest local LLM across all four language directions, with COMET scores 1.5–2.5 points below the best-performing models. This suggests that 8B parameters, even at full fp16 precision, remain insufficient for consistently competitive medical-domain translation.

Beyond this threshold, however, scaling behavior becomes decidedly non-monotonic. Among models in the 20B–32B range, parameter count alone is a poor predictor of translation quality. The 20B gpt-oss matches or exceeds several 32B models for EN→DE and EN→JA, while the translation-specialized translategemma-27b outperforms all 32B general-purpose models for EN→RU, EN→JA, and EN→ZH. Conversely, two 32B models, deepseek-r1 and qwen2.5, rank near the bottom for EN→DE and EN→RU despite their size advantage. These reversals suggest that at the parameter scales we tested, multilingual training data composition and task-specific optimization exert a stronger influence on translation quality than raw model size.

A language-specific pattern is also visible. Performance variance across models is wider for EN→DE and EN→RU than for EN→ZH, where the scores cluster more tightly.This may reflect asymmetries in the multilingual training corpora of different model families.Notably, the relatively strong EN→ZH performance of Qwen and DeepSeek models, both originating from Chinese organizations, hints at training-data effects that interact with scaling in complex ways.

These results caution against extrapolating general scaling laws to translation: model size alone is an unreliable proxy for translation quality, and empirical benchmarking on relevant language pairs remains essential.

We also tested three locally installable NMT systems used by some translators for offline workflows—Promt, OPUS-CAT, and NeuralDesktop—on the supported language pairs (Table 1). Their document-level COMET scores vary widely. This variability may be due to the uneven quality of the base NMT models they use for different language directions [8] as well as the varying degrees of multilingual coverage and domain support.

We emphasize that all outputs reported here come from base models with no domain adaptation. Many systems, including commercial engines, local NMT, and local LLMs, can be fine-tuned or adapted to terminology and style constraints. Understanding adaptation regimes with modest resources is a natural next step and is the focus of ongoing research [55]–[59], with obvious practical implications. We plan to investigate various adaptation regimes, using our corpus, in the future.

We also acknowledge a limitation of our context-free, sentence-by-sentence evaluation approach: both the automatic and manual evaluation scores we report reflect segment-level quality and may not fully capture document-level coherence, consistency, or discourse-related translation phenomena [48], [60]. This is a deliberate constraint: many MT/LLM pipelines still operate sentence by sentence, and local inference costs make long-context prompting expensive and slow. At the same time, lack of discourse context can hurt pronoun resolution, lexical consistency, and terminology coherence. We discuss these limitations and plans for future work in Section 6 below.

6 Discussion, Limitations, and Future Work↩︎

Human evaluation has long been regarded as the gold standard in machine translation research. A companion paper to the present study therefore places human evaluation at the center of its analysis.

6.0.0.1 Human evaluation in the sequel.

A large-scale, blind, randomized human evaluation of all twelve outputs (nine local LLMs plus three baselines) plus reference translations on 100+ strategically selected sentences for EN→DE, RU, ZH, using a novel two-step direct-assessment protocol and a purpose-built Streamlit interface [61], is the focus of a companion paper currently in preparation. Two preliminary findings are worth flagging here: (i) while COMET is a strong system-level proxy for human judgment, it is a much weaker one at the sentence level, which is consistent with other recent findings; (ii) a non-trivial number of reference translations were judged by human experts to be inferior to the best system outputs. This has potential implications for reference-free quality estimation methods and related approaches [62], which could be particularly valuable for freelancers and smaller LSPs who may not always have access to high-quality reference translations.

6.0.0.2 Prompt and configuration effects as a workflow variable.

Unlike conventional NMT engines, local LLM MT makes prompt design part of the system. Our pilot experiments illustrate that prompt choices can materially shift quality and error profiles, and that “native” prompts suggested by a model do not necessarily optimize that model’s outputs. For small teams, this implies that prompts should be versioned and treated like configuration code: changes should be evaluated on a fixed internal benchmark set.

6.0.0.3 Decoding temperature.

For reasoning models, temperature settings appear to be a double-edged sword: We find that \(T=0.0\) is associated with a substantial number of translation-call failures in reasoning models.Changing the temperature to the model card or inference engine default (\(T=0.6-1.0\)) reduces the number of failed calls, but also appears to degrade the overall quality of the translation outputs.

6.0.0.4 When local LLM translation is viable.

Translation with local LLMs is most attractive when confidentiality constraints disallow cloud services, when marginal API costs are prohibitive, or when translators need predictable offline availability. Our results indicate that the best open models can provide high-quality drafts. However, for high-risk deliverables, such drafts should be used within a workflow that includes systematic validation rather than as a fully autonomous system.

6.0.0.5 Toward a fuller panel.

The selection of nine local LLMs reflected a snapshot of late-2025 availability; specialized recent releases such as SalamandraTA and Tower+ are obvious targets for the future.

7 Concluding Remarks↩︎

Using a real-world medical-domain corpus expanded to four target languages, we compared locally runnable open models against commercial MT, a frontier LLM baseline, and locally installable NMT systems.

Two conclusions stand out. First, translation-specialized open models and those trained on balanced multilingual data substantially narrow the translation quality gap between local models and commercial engines, making translation with local LLMs increasingly viable under confidentiality and cost constraints. Second, prompt and decoding temperature choices can materially affect outcomes, so reproducibility and re-validation are essential.

Author Contributions↩︎

YB conceived the study, led the corpus expansion and curation, conducted automatic evaluation of all the outputs with MATEO, and drafted the manuscript. RVH performed the German reference translation curation and contributed to the research strategy and evaluation design. MX performed the Chinese reference translation curation and contributed linguistic observations. AD implemented all local LLM inference operations, conducted prompt selection experiments, managed the technical infrastructure, and authored the description of these processes in Sections 3 and 4. All authors reviewed and approved the final manuscript.

Acknowledgments↩︎

YB’s work was supported by NSF Grant No. SES-2336713. We reiterate our thanks to the Christopher & Dana Reeve Foundation for permission to use their linguistic resources in our experiments. We are indebted to Julian Hennemann for reviewing the EN–DE outputs and contributing valuable linguistic observations. We thank Chris Schertler for sharing the TMX and XLIFF documents for the German translation of PRG International. We are grateful to Sheila Castilho and to the participants in the graduate seminar on translation technologies taught at UGA in Fall 2025 for their input and advice. Finally, we sincerely thank the reviewers for their comments, most of which we have incorporated into this revised version.

8 The Reeve Foundation Multilingual Corpus↩︎

Our corpus comprises 3,500 English source sentences aligned with their original professional translations into Russian and Japanese, as well as partially curated reference translations into German and Chinese. With the client’s permission, we release it for noncommercial/academic use.

9 Local Inference Setup↩︎

All local LLM experiments were conducted on a workstation running Ubuntu 24.04.4 LTS, equipped with NVIDIA GeForce RTX 3090 GPUs (24 GB VRAM), NVIDIA driver version 580.126.09, and CUDA 13.0. Each translation run used a single GPU. The inference engine was Ollama v0.14.2.

Table 2 reports the quantization settings for each model. Models at or above 24B parameters were served in Ollama’s default 4-bit quantization (Q4_K_M). The two smaller models, gpt-oss-20b and llama3.1-8b-instruct, were run at their default (unquantized) precision, as both fit within the 24 GB VRAM budget without compression. Llama3.1-8b-instruct was run at full fp16.

Table 3 reports wall-clock times for translating the full 1,143-sentence corpus per language direction on a single RTX 3090. Non-reasoning models completed each direction in approximately 19 to 40 minutes. The three reasoning-oriented models were substantially slower: qwen3-32b required 4.4 to 5.5 hours per direction, deepseek-r1-32b approximately 2.9 to 3.8 hours, and gpt-oss-20b approximately 43 to 64 minutes. This overhead reflects the models’ internal multi-step processing, which generates extended reasoning traces before producing the final translation.

10 Prompt Portfolio↩︎

This appendix lists the full text of all candidate prompts used in the prompt selection pilot (Section 3.2). Prompts p1–p4 and p6–p9 were elicited by meta-prompting the indicated model to generate its own preferred translation prompt. Prompts p10 and p11 are “standard” prompts from [2], in German and English respectively. Prompt p5 was generated by gemma3-27b and was selected as the best-performing prompt on average across all models (Section 3.2 and Table 4). Prompt p0 is the developer-recommended prompt for TranslateGemma [42] and uses a different template format with explicit language code placeholders.

3.5pt

All prompts shown below are in their EN\(\to\)DE pilot form. For the main experiments, p5 was adapted to each target language by substituting the appropriate language name.

p0 (TranslateGemma developer-recommended): You are a professional {SOURCE_LANG} ({SOURCE_CODE}) to {TARGET_LANG} ({TARGET_CODE}) translator. Your goal is to accurately convey the meaning and nuances of the original {SOURCE_LANG} text while adhering to {TARGET_LANG} grammar, vocabulary, and cultural sensitivities. Produce only the {TARGET_LANG} translation, without any additional explanations or commentary. Please translate the following {SOURCE_LANG} text into {TARGET_LANG}: \n \n \n {TEXT}

p1 (Claude Opus 4.5): Translate the following English sentence to German. Output only the translation, nothing else.

p2 (GPT-5.2): You are a professional translator. Task: Translate the following sentence from English to German. Requirements: Output ONLY the German translation (no explanations, no quotes, no alternatives). Preserve meaning exactly; do not add or omit information. Keep names, numbers, units, and formatting unchanged. Use natural, idiomatic German. Use neutral formal register (Sie) unless the English clearly implies informal (du). Keep the same tense and voice; preserve emphasis if possible.

p3 (Gemini Pro 3): You are a professional translator proficient in English and German. Your task is to translate the provided English text into German. Constraints: Maintain the original tone and meaning. Output only the German translation. Do not include preambles, explanations, or enclosing quotes.

p4 (deepseek-r1-32b): You are a professional translator specializing in academic texts. Translate the following English text into German, maintaining its formal tone and precise terminology. Provide only the translation without any explanations, annotations, or transliterations.

p5 (gemma3-27b, selected for main experiments): Translate the following {SOURCE_LANG} text into {TARGET_LANG}. The target audience is academic/expert. Provide only the {TARGET_LANG} translation, without any explanations, annotations, or transliterations.

p6 (gpt-oss-20b): You are a professional translator. Translate the following English text into German for an expert/academic audience. Output only the translated text, with no explanations, annotations, or transliterations.

p7 (mistral-small3.2-24b): Translate the following English text into German, maintaining an expert/academic level of language and terminology. Provide only the translation, without any explanations, annotations, or transliterations.

p8 (qwen2.5-32b): Translate the following English text into German for an expert/academic audience. Provide only the translation without any explanations, annotations, or transliterations.

p9 (qwen3-32b): Translate the following English text into German. The translation should be formal, precise, and tailored for an expert/academic audience. Provide only the translated text, with no additional explanations, annotations, or transliterations. Ensure technical terminology and nuanced meanings are accurately preserved.

p10 (standard prompt, German, from [2]): Sie sind ein erfahrener Übersetzer und übersetzen für ein Fachpublikum. Bitte fügen Sie Ihrer Übersetzung keine Anmerkungen, Erklärungen oder Transliterationen hinzu. Bitte übersetzen Sie den folgenden Satz ins Deutsche:

p11 (standard prompt, English, from [2]): You are an expert translator, translating for an expert audience. Please do not provide any annotations, explanations or transliterations in your translation. Please translate the following English sentence to German:

Table 4 presents the COMET scores for the pilot experiments with 118 held-out test sentences (EN\(\to\)DE).

11 Levenshtein Ratio Matrices for Different Temperature Settings↩︎

To assess the effect of decoding temperature on output consistency, we ran a sweep across 21 temperature values from \(T=0.0\) to \(T=1.0\) in increments of 0.05, using the 118-sentence EN\(\to\)DE pilot set (Section 3.2). We generated one full translation of the pilot set at each temperature setting, then computed the mean pairwise Levenshtein ratio between all 21 x 21 combination of outputs. Each cell in the resulting matrix reports the mean Levenshtein ratio (computed over character sequences) across all 118 sentences for the corresponding temperature pair.

Figure 3 shows the matrix for aya-expanse-32b. The row and column corresponding to \(T=0.0\) show the highest similarity values throughout, confirming that zero-temperatures outputs are maximally close to outputs at all other temperatures. Similarity declines as the distance between compared temperatures increases, with the lowest values concentrated in the bottom-right corner of the matrix (high-temperature pairs).

12 Translation Failures From Infinite “Reasoning Loops”↩︎

Table 5 presents the number of failed translation calls resulting from prohibitively long “thinking loops” produced by three “reasoning models” on two temperature settings.

The scores reflected in Table 1 and Figure 2 reflect these failures: i.e. the models are penalized for the empty output lines. We think this is fair when reporting the actual performance of the models. To see how temperature settings affect the sentence-by-sentence quality of the outputs in cases of successful translation generation, we eliminated all outputs with at least one missing translation for each language pair and calculated the average document-level COMET score gains or losses denoted by ‘\(T_0\to T_1\)’. We also calculated the document-level average of the absolute values of such gains and losses: ‘\(|T_0\to T_1|\)’. The former gives a rough measure of the overall change in translation performance associated with the temperature change, while the latter represents the variation of such changes (Table 6).

after excision of outputs with missing translations.

We emphasize that these numbers represent relative differences in the automatic scores and do not by themselves tell us anything about the actual quality of the outputs. In fact, the outputs from qwen3-32b and deepseek-r1-32b appears to be substandard for all language pairs except EN–ZH (Section 5.2).

A typical histogram of the sentence-by-sentence COMET score differences representing a fine-grained breakdown of ‘\(|T_0\to T_1|\)’ from Figure 1 is centered in the negative area thus indicating the overall loss of translation quality. We have reason to believe the long tails of the distributions of such differences may have contributed the most to the average losses reflected in Table 6 and that these tails are associated with notable linguistic issues. We plan to review them in a sequel to this paper which will include human evaluation of a substantial number of our translation outputs.

13 Automatic Evaluation: Charts↩︎

Figure 2 displays COMET-22 scores for translation outputs (from those listed in Table 1). Colored bars represent the best outputs from each of the 9 local LLMs, gray bars the outputs from the baselines for each translation direction.

References↩︎

---

1.   Equal contribution. Corresponding author.↩︎

2. For example, OPUS-CAT [7] built around OPUS-MT models [8] and designed for offline professional use. Other locally installable NMT systems include NeuralDesktop and Promt (introduced as baselines in Section 3 below).↩︎

3. https://github.com/YuriBalashov/reeve-mftc↩︎

4. https://www.deepl.com/en/translator; https://www.baidu.com↩︎

5. https://www.promt.com; https://neuraldesktop.com↩︎

6. Code available at https://github.com/asdownes/translation-analytics-llm-benchmark↩︎