Keywords: legal NLP, benchmark, Ukrainian, court decisions, multilingual evaluation, few-shot learning

1 Introduction↩︎

The rapid adoption of large language models (LLMs) in legal practice has spurred the development of benchmarks to evaluate their reasoning capabilities. LegalBench [1], LexGLUE [2], and CUAD [3] have established rigorous evaluation standards – yet all are English-only. Multilingual efforts such as LEXTREME [4] and MultiLegalPile [5] extend coverage to EU languages but exclude Cyrillic-script jurisdictions and civil-law systems outside Western Europe.

This gap is consequential. Ukrainian, with its Cyrillic script, agglutinative morphology, and seven grammatical cases, presents a fundamentally different tokenization surface. Our prior work [6] shows that subword tokenizers fragment Ukrainian legal text at rates 1.6\(\times\) higher than English equivalents across frontier models, directly inflating inference costs and degrading in-context learning. Moreover, the Ukrainian legal system – a civil-law jurisdiction with codified statutes rather than common-law precedent – requires distinct reasoning patterns that English benchmarks do not capture.

Ukraine’s Unified State Register of Court Decisions (EDRSR) provides an unparalleled empirical foundation: 99.5 million decisions published since 2006, constituting one of the largest open judicial datasets in the world. We leverage this resource to construct UA-Legal-Bench, a benchmark spanning five tasks that probe progressively deeper levels of legal understanding:

1. Case-Type Classification (CTC) – classifying decisions into four jurisdictional categories;

2. Judgment Form Classification (JFC) – identifying the document type (decision, resolution, sentence, ruling);

3. Case-Outcome Prediction (COP) – predicting the judicial ruling from case facts alone;

4. Norm Extraction (NE) – extracting legal norm citations from decision text;

5. Cause Category Prediction (CCP) – classifying the legal subject matter into 22 macro-categories.

Our contributions are:

1. The first comprehensive legal reasoning benchmark for a Cyrillic-script, civil-law jurisdiction, with 2,000 decisions spanning five tasks of increasing difficulty.

2. A standardized evaluation of 11 LLMs (3B–675B, five families) with 158K API calls under both zero-shot and few-shot conditions.

3. The discovery of sharply task-dependent few-shot effects, including few-shot as a scale equalizer (\(+38.6\) pp for a 12B model, approaching 120B performance).

4. Within-family scaling analysis showing that the parameter threshold for Ukrainian legal reasoning varies dramatically across model families.

5. All data, prompts, model predictions, and evaluation code released publicly.

2 Related Work↩︎

2.0.0.1 English legal benchmarks.

LegalBench [1] defines 162 tasks spanning six categories of legal reasoning, evaluated on frontier LLMs. LexGLUE [2] provides a multi-task benchmark for legal language understanding across seven datasets. CUAD [3] focuses on contract review. More recently, [7] demonstrate that GPT-4 passes the Uniform Bar Examination, highlighting the rapid capability growth that benchmarks must track. These benchmarks have driven substantial progress but evaluate exclusively in English and predominantly within common-law systems.

2.0.0.2 Multilingual legal NLP.

LEXTREME [4] aggregates 11 datasets across 24 languages in the EU legal domain, while SCALE [8] adds complexity-scaled tasks. MultiLegalPile [5] provides a 689 GB multilingual legal corpus spanning 24 EU languages. [9] provide foundational pre-trained models for legal text, and [10] examine cross-lingual transfer in legal NLP. Despite broad coverage, these efforts exclude Ukrainian and do not address the distinctive challenges of Cyrillic-script legal text.

2.0.0.3 Few-shot evaluation and metric choice.

[11] establish few-shot prompting as a standard evaluation paradigm, but its effectiveness on non-English, domain-specific tasks remains understudied. [12] show that few-shot performance is sensitive to example selection and ordering, motivating our use of fixed, stratified examples. The problem of metric selection for imbalanced classification is well-studied [13]; we show that this concern is especially acute in legal benchmarks where majority-class prediction can yield high accuracy but low macro-F1.

2.0.0.4 Ukrainian NLP.

No prior benchmark exists for Ukrainian legal reasoning. [14] organize the first shared task on Ukrainian NLP at EACL 2023, covering NER and lemmatization but not legal tasks. General Ukrainian NLP resources remain limited compared to EU official languages: Ukrainian constitutes only 0.5% of the mC4 corpus, 18\(\times\) less than Russian and 2.4\(\times\) less than Polish [15]. This data scarcity compounds with tokenizer inefficiency [16] to create a “double penalty” for Ukrainian legal NLP.

2.0.0.5 Positioning.

UA-Legal-Bench fills this gap as the first legal benchmark for a Cyrillic-script, civil-law jurisdiction (Table 1). Unlike prior benchmarks that typically evaluate a single task type, UA-Legal-Bench spans five tasks of increasing difficulty and explicitly evaluates the interaction between few-shot prompting, model scale, and metric choice.

3 Benchmark Design↩︎

3.1 Data Source↩︎

All tasks draw from the Unified State Register of Court Decisions (EDRSR, Єдиний державний реєстр судових рішень), an official Ukrainian government registry containing 99.5 million court decisions spanning 2006–2026 across all jurisdictional levels and court types. Each record includes structured metadata (court, date, case number, jurisdiction type, judgment form, cause category) and the full decision text.

We sample 2,000 decisions from the 2024 partition: 500 per jurisdiction (civil, criminal, commercial, administrative), stratified across judgment forms. Only substantive decisions with full text between 2,000 and 30,000 characters are included (median length: 12,060 characters). For each decision, we extract: the facts section (for COP), cited legal norms (for NE), and the ruling outcome (for COP), using rule-based parsers validated against EDRSR’s canonical document structure.

3.2 Tasks↩︎

3.2.0.1 Task 1: Case-Type Classification (CTC).

Given a court decision, classify it into one of four jurisdictional categories: civil (цивільне), criminal (кримінальне), commercial (господарське), or administrative (адміністративне). Labels derive from EDRSR metadata. This task evaluates basic legal document understanding.

Metric: Accuracy. Size: \(n{=}2{,}000\).

3.2.0.2 Task 2: Judgment Form Classification (JFC).

Classify the document type: rishennya (decision), postanova (resolution), vyrok (sentence), or ukhvala (ruling). Unlike CTC, this requires understanding the procedural nature of the document – a resolution and a decision may come from the same court but serve different legal functions.

Metric: Accuracy. Size: \(n{=}2{,}000\).

3.2.0.3 Task 3: Case-Outcome Prediction (COP).

Given only the facts section of a court decision (ruling masked), predict the outcome from six classes: granted, partial, denied, closed, guilty, left without consideration. This task requires legal reasoning over factual circumstances and is restricted to decisions where the facts section and outcome label can be reliably extracted.

Metric: Macro-F1 (due to 61% majority class). Size: \(n{=}800\).

3.2.0.4 Task 4: Norm Extraction (NE).

Given a court decision, extract all legal norm references (e.g., “ст. 625 ЦК України”, “ч. 2 ст. 16 ЦПК України”). Ground truth is constructed by parsing canonical citation patterns from decision texts, yielding a mean of 7.5 norms per document.

Metric: Set-level F1. Size: \(n{=}1{,}794\).

3.2.0.5 Task 5: Cause Category Prediction (CCP).

Classify the legal subject matter of the case into one of 22 macro-categories (e.g., contracts, theft, family, pension/social, property, violence), derived from EDRSR’s 4,106-category taxonomy via keyword-based aggregation.

Metric: Accuracy. Size: \(n{=}1{,}871\).

3.3 Evaluation Protocol↩︎

All tasks are evaluated under zero-shot and 3-shot prompting. Prompts are in Ukrainian with standardized instruction templates. All inference uses temperature 0 for reproducibility. Few-shot examples are drawn from a fixed pool (not overlapping with test data) using a deterministic seed, with stratified sampling to cover label diversity. For imbalanced tasks (COP: 61% majority class; CCP: 31% majority class), we report macro-F1 rather than accuracy to prevent majority-class bias. 95% Wilson confidence intervals are \(\pm\)​0.8 pp for CTC/JFC (\(n{=}2{,}000\)), \(\pm\)​3.4 pp for COP (\(n{=}800\)), and \(\pm\)​2.3 pp for CCP (\(n{=}1{,}871\)).

4 Models↩︎

We evaluate eleven LLMs accessed via AWS Bedrock from five model families, with three families (Mistral, Meta, NVIDIA) represented at multiple scales to enable within-family scaling analysis (Table 2).

The seven frontier models span 32B to 675B total parameters with tokenizer fertility ranging from 2.43 (Llama 4 Maverick) to 3.90 (Qwen3 32B) – a 1.6\(\times\) spread meaning identical Ukrainian legal text consumes 60% more tokens on the least efficient tokenizer. The four smaller models (3B–12B) from the Mistral, Meta, and NVIDIA families enable direct measurement of how performance scales within a model family on Ukrainian legal tasks.

5 Results↩︎

We report results from 158,419 API calls across 11 models. Table 3 presents performance across all five tasks, with frontier and smaller models separated.

5.1 Task Difficulty Gradient↩︎

The five tasks form a clear difficulty gradient (Figure 1). CTC is nearly solved: all frontier models exceed 96% zero-shot, with only 1.0 pp separating best from worst. JFC is substantially harder (74–84% ZS), requiring models to distinguish procedurally similar document types. COP is the hardest task: frontier models achieve only 23–41% macro-F1 zero-shot, demanding genuine legal reasoning over masked outcomes. CCP sits between JFC and COP in difficulty (44–51% accuracy ZS), testing domain knowledge of legal subject matter.

5.2 Few-Shot Effects Are Task-Dependent↩︎

The central finding of UA-Legal-Bench is that few-shot prompting effects are sharply task-dependent and cannot be predicted from task difficulty alone (Figure 2). Table 4 summarizes the frontier model deltas.

5.2.0.1 JFC: Few-shot consistently helps.

Six of seven models gain 8.6–17.7 pp from few-shot examples on judgment form classification. The largest gain is Llama 3.3 70B (\(+17.7\) pp, from 74.1% to 91.8%). The sole exception is Qwen3 32B (\(+0.4\) pp), which also shows the weakest few-shot response on COP. This suggests that JFC benefits from format learning: the four judgment forms have distinctive textual signatures that few-shot examples help models recognize.

5.2.0.2 COP: Few-shot effects vary; accuracy is misleading.

COP has a heavily imbalanced label distribution (61% “granted”), making accuracy a poor metric (Figure 3). Nemotron Super 3 achieves the highest accuracy (62.3% ZS) but the lowest macro-F1 among frontier models (22.7%). Per-class analysis reveals why: Nemotron recalls 97% of “granted” cases but 0% of “guilty” – it is a majority-class predictor. In contrast, Nova Pro (macro-F1 39.4% ZS, 43.8% FS) achieves 86% recall on “guilty”, 62% on “granted”, and 43% on “denied”, demonstrating genuine cross-class reasoning. Few-shot prompting generally helps COP on macro-F1 (Maverick: 32.5%\(\to\)​38.9%), with the notable exception of Llama 3.3 which drops from 40.6% to 36.6%. This task demonstrates why imbalanced legal benchmarks require class-aware metrics.

5.2.0.3 CCP: Moderate consistent gains.

Cause category prediction shows modest, consistently positive few-shot effects (\(+2.2\) to \(+5.6\) pp), suggesting that topic classification benefits from exemplar-based calibration without the negative length effects seen in COP.

5.2.0.4 CTC: Ceiling effect.

Few-shot effects on CTC are minimal (\(-0.0\) to \(+1.0\) pp), reflecting a ceiling: when zero-shot accuracy already exceeds 96%, examples add little.

5.3 Model Rankings Vary by Task↩︎

No single model dominates across all tasks. Nemotron Super 3 leads on JFC (84.0% ZS) but is a weak majority-class predictor on COP (mF1 22.7%). Nova Pro leads on COP (mF1 39.4% ZS) and NE (.383) but is mid-pack on JFC (79.5%). Qwen3 235B leads on CCP (51.1%) and ties for first on CTC (97.4%). This task-dependent ranking instability means that evaluating on a single legal task – as most prior benchmarks do – provides an incomplete picture of model capability.

Larger models do not reliably outperform smaller ones: Qwen3 235B scores lower than Qwen3 32B on COP macro-F1 (34.7% vs 31.4%) and Ministral 8B (31.4%) nearly matches its 675B sibling Mistral Large (35.4%).

5.4 Norm Extraction↩︎

Norm extraction is evaluated via normalized (article, law-code) pair matching, yielding F1 scores of 0.318–0.391. Frontier models achieve F1 0.339–0.391 while smaller models are competitive (Llama 3.1 8B: 0.359, Ministral 3B: 0.353). This is the only task where small models match frontier performance, suggesting that norm extraction relies on pattern recognition rather than deep legal reasoning. Few-shot prompting has negligible effect on NE (\(|\Delta\text{F1}| < 0.02\)), contrasting sharply with its large effect on JFC.

5.5 Scaling Analysis↩︎

Table 3 includes four smaller models (3B–12B) from three families, enabling within-family scaling analysis (Figure 4). The results reveal that scaling effects are highly task-dependent:

5.5.0.1 Mistral family (3B \(\to\) 8B \(\to\) 675B).

On CTC, Ministral 8B (97.5%) matches Mistral Large 675B (97.4%) – an 84\(\times\) parameter reduction with no accuracy loss. On COP macro-F1, Ministral 8B (31.4%) approaches Mistral Large (35.4%), a surprisingly small gap given the 84\(\times\) scale difference. Only on JFC and CCP do larger models show clear advantages. Ministral 3B remains competitive on CTC (95.5%) and JFC (74.9%), but collapses on COP with few-shot prompting (mF1 21.5%, \(-5.9\) pp from ZS).

5.5.0.2 NVIDIA family (12B \(\to\) 120B).

Nemotron Nano 12B shows a scaling gap on COP macro-F1: 19.3% vs Super’s 22.7%. Both models are weak on COP (near majority baseline), but the gap widens dramatically on CTC accuracy (84.2% vs 97.4%). However, few-shot prompting narrows the gap on JFC: Nano jumps from 51.8% to 90.4% (\(+38.6\) pp), nearly matching Super’s 92.6%. This is the largest few-shot gain in our benchmark, suggesting that few-shot examples can partially compensate for a 10\(\times\) parameter deficit on format-learnable tasks.

5.5.0.3 Meta family (8B \(\to\) 70B \(\to\) 400B).

Llama 3.1 8B is the weakest model overall: COP mF1 5.3% ZS, JFC 36.5% ZS, CCP 23.6% ZS. The jump to Llama 3.3 70B is dramatic (COP mF1: \(+35.3\) pp, JFC: \(+37.6\) pp), indicating that the Meta family requires substantially more parameters for Ukrainian legal reasoning than the Mistral family.

5.5.0.4 Implications.

These results challenge the assumption that legal AI requires frontier-scale models. For simpler tasks (CTC, NE), 8B models suffice. For harder tasks (COP, CCP), scaling matters – but the scaling curve varies dramatically by family: Mistral’s 8B model is competitive while Meta’s 8B model is not.

6 Discussion↩︎

6.0.0.1 Task-dependent few-shot effects.

Our central finding – that few-shot prompting can simultaneously help and hurt the same model on different tasks – has practical implications. Legal AI practitioners cannot assume that few-shot prompting will improve performance; the effect must be measured per task. We hypothesize that the direction of the effect depends on the ratio of format signal (learnable from examples) to length cost (prompt inflation from tokenizer fertility). JFC has high format signal (distinctive document headers), while COP has low format signal (outcomes depend on factual reasoning, not surface patterns).

6.0.0.2 Few-shot as scale equalizer.

The Nemotron Nano result (JFC: 51.8% \(\to\) 90.4% with few-shot) demonstrates that few-shot prompting can partially substitute for model scale on format-learnable tasks. This has cost implications: achieving 90% JFC accuracy with a 12B model and few-shot examples is far cheaper than deploying a 120B model zero-shot.

6.0.0.3 Comparison with English benchmarks.

On LegalBench’s comparable tasks, frontier LLMs routinely exceed 90% accuracy [1]. Our CTC results (96–98%) are consistent, but COP (23–41% macro-F1) and CCP (44–55%) reveal substantially more variance, suggesting that the difficulty gap is task-dependent rather than uniform across all legal reasoning.

6.0.0.4 Why Ukrainian is hard.

Three factors compound: (1) tokenizer inefficiency inflates prompt length, reducing effective context; (2) morphological richness (7 cases, 3 genders, synthetic verb forms) creates surface variation invisible to English-trained models; (3) civil-law reasoning patterns – statute application rather than case precedent – differ from common-law training data that dominates LLM pretraining corpora.

6.0.0.5 Prompt sensitivity.

To test robustness, we evaluate COP with three prompt variants on two models (Figure 5): the original Ukrainian prompt, an English translation, and a detailed Ukrainian prompt with explicit role-playing. Results are stable across Ukrainian variants (Nova Pro: 41.5% vs 42.2% macro-F1, \(\Delta{=}0.7\) pp; Nemotron: 22.6% vs 24.7%, \(\Delta{=}2.1\) pp). English prompts cause moderate degradation for Nova Pro (\(-5.3\) pp) but a surprising improvement for Nemotron (\(+5.8\) pp), suggesting that prompt language interacts with model architecture in non-trivial ways.

6.0.0.6 Label quality.

We validate ground truth labels by independently annotating 50 COP and 50 CCP decisions using Claude Sonnet 4.6 as an expert judge. COP label agreement is 70% (regex-parsed outcome vs LLM judgment), and CCP agreement is 58% (keyword-mapped category vs LLM classification). The moderate CCP agreement reflects the inherent ambiguity of topic classification – many cases span multiple categories – rather than systematic labeling errors.

6.0.0.7 Cross-year stability.

To verify the benchmark is not specific to 2024 data, we sample 200 decisions from 2020 and evaluate CTC with two models. Results are stable: Nova Pro scores 94.5% (2020) vs 97.2% (2024, \(\Delta{=}{-}2.7\) pp) and Nemotron scores 95.0% vs 97.4% (\(\Delta{=}{-}2.4\) pp), confirming that the benchmark captures stable model capabilities rather than year-specific artifacts.

6.0.0.8 Statistical significance.

McNemar’s test confirms that the COP performance gap between Nova Pro and Nemotron Super 3 is significant (\(\chi^2{=}9.2\), \(p{=}0.002\)), validating that the macro-F1 difference (39.4% vs 22.7%) reflects genuinely different prediction patterns rather than noise.

6.0.0.9 Limitations.

The COP evaluation set (\(n{=}800\)) is smaller than other tasks due to the requirement for reliable facts extraction and outcome labeling, yielding wider confidence intervals (\(\pm 3.4\) pp). COP label agreement of 70% with an independent LLM judge suggests approximately 30% noise in outcome labels, which may attenuate measured model differences. The benchmark currently covers a single jurisdiction; cross-lingual evaluation (e.g., Polish, Czech) is planned as future work.

7 Conclusion↩︎

We present UA-Legal-Bench, the first benchmark for evaluating LLMs on Ukrainian legal reasoning. Across five tasks and eleven models (158K evaluations, 3B–675B), we find that: (1) case-type classification is nearly solved even at 8B scale, (2) judgment form classification benefits dramatically from few-shot examples (up to \(+38.6\) pp), (3) case-outcome prediction reveals a critical metric choice: the highest-accuracy model is a majority-class predictor, while macro-F1 identifies genuinely capable models, (4) norm extraction is the only task where small models match frontier performance, and (5) the parameter threshold for competence varies dramatically by model family.

Two findings are particularly striking. First, accuracy is dangerously misleading on imbalanced legal tasks: Nemotron Super 3 achieves 62% COP accuracy but macro-F1 of only 23% (majority baseline: 13%), while Nova Pro scores 44% macro-F1 by actually distinguishing between outcome classes. Second, few-shot prompting can partially compensate for a 10\(\times\) parameter deficit on format-learnable tasks (Nemotron Nano: \(+38.6\) pp on JFC). These patterns – invisible in English-only, accuracy-only evaluation – underscore the need for multilingual legal benchmarks with appropriate metrics.

7.0.0.1 Data availability.

All benchmark data, prompts, model predictions, and evaluation code are available at https://huggingface.co/datasets/overthelex/ua-legal-bench under CC-BY-4.0.

Ethics and Broader Impact↩︎

UA-Legal-Bench is constructed entirely from publicly available court decisions published by the Ukrainian government under open access. All personal identifiers in EDRSR decisions are anonymized at source (e.g., OSOBA_1, ADRESA_1). The benchmark is intended for evaluating NLP models, not for automated judicial decision-making. We caution against using COP results to build predictive systems for real cases, as the task is designed as a benchmark probe, not a deployment-ready pipeline. The benchmark may contain biases inherent in the Ukrainian judicial system, including regional and temporal variation in case outcomes.

Datasheet for UA-Legal-Bench↩︎

Following [17], we provide key dataset documentation.

7.0.0.2 Motivation.

UA-Legal-Bench was created to address the absence of legal NLP benchmarks for Cyrillic-script, civil-law jurisdictions. It was funded by SecondLayer.

7.0.0.3 Composition.

2,000 Ukrainian court decisions (46 MB raw text) sampled from the 2024 partition of EDRSR, with 500 per jurisdiction (civil, criminal, commercial, administrative). Each decision includes full text, metadata labels, parsed facts section, extracted legal norms, and outcome labels. 110 result files contain 158K model predictions across 11 models.

7.0.0.4 Collection.

Decisions are sourced from EDRSR (https://reyestr.court.gov.ua), the official Ukrainian government registry of court decisions. All personal identifiers are anonymized at source (OSOBA_1, ADRESA_1). No human subjects were involved.

7.0.0.5 Preprocessing.

Facts sections extracted via rule-based parser matching canonical Ukrainian court decision structure (ВСТАНОВИВ…ВИРІШИВ). Legal norms extracted via regex, validated at 70% agreement with Claude Sonnet 4.6. Cause categories mapped from 4,106 EDRSR codes to 22 macro-categories via keyword matching (58% agreement with LLM judge).

7.0.0.6 Splits.

Fixed test set only (no train split). Few-shot examples drawn from a separate pool with deterministic seed (seed=42).

7.0.0.7 Distribution.

CC-BY-4.0 on HuggingFace (https://huggingface.co/datasets/overthelex/ua-legal-bench). Source decisions are public domain under Ukrainian law.

7.0.0.8 Maintenance.

Annual updates planned. Version-controlled with semantic versioning. Contact: vladimir@legal.org.ua.

References↩︎