1 Introduction↩︎

Code generation with large language models (LLMs) has become a widely used paradigm for automating routine programming tasks, assisted development, and rapid prototyping[1]. Despite strong performance from instruction-tuned models, achieving high functional correctness remains challenging because training objectives (e.g., next-token prediction) are only indirectly aligned with execution behavior. A natural alternative is reinforcement learning with verifiable rewards (RLVR) [2], where rewards are computed from automatically checkable criteria such as compilation success, unit tests, or static analysis.

In the code domain, unit tests provide a particularly compelling reward signal: they are objective, reproducible, and directly tied to correctness. However, RLVR is not only an “accuracy improvement” tool; it can also change generation behavior in non-trivial ways. Reward design and optimization choices can steer models toward undesirable equilibria, for example, reward hacking [3].

In this paper we test the following hypothesis using Mostly Basic Python Problems (MBPP) dataset: RLVR can substantially improve functional correctness, but its benefits are reward-sensitive; in particular, static-analysis-only shaping may degenerate toward shorter solutions, while a combined unit-test and static-analysis reward yields a more stable trade-off.

We study RLVR in a controlled small-model setting where full training runs are feasible. We train LoRA adapters for two architectures (Qwen3-0.6B and Llama3.2-1B) and compare two group-based policy optimization variants, GRPO and GSPO, under multiple reward designs: (i) unit-test-only rewards, (ii) static-analysis-only shaping via Ruff, and (iii) combined rewards. By analyzing both correctness and behavioral diagnostics, we characterize when RLVR improves pass@1 and when it instead shifts the model toward policies that primarily optimize proxy signals.

Our contributions are as follows:

• we provide a controlled comparison of GRPO vs.GSPO for RLVR on MBPP using small models;

• we quantify the impact of different reward formulations on pass@1 and on behavioral indicators (length, Ruff severity, and execution error types);

• we document a reward-induced degeneration mode in our static-analysis-only setting and show that combined rewards mitigate this effect under the tested weights/normalization.

2 Related Work↩︎

2.1 Code Generation Benchmarks↩︎

The evaluation of functional correctness in code generation is commonly performed using execution-based benchmarks such as HumanEval [4] and Mostly Basic Python Problems (MBPP) [5]. These benchmarks provide programming tasks paired with unit tests, allowing to track objective metrics such as pass@k. For RLVR, these datasets are a natural fit because the evaluation signal (unit-test outcomes) matches the reward family used during training, enabling a direct assessment of whether reward optimization transfers to held-out tasks. Execution-based evaluation yields structured failure signals (e.g., syntax vs.runtime vs.assertion failures), and these error modes are informative for diagnosing reward-induced shifts.

2.2 RL for Code↩︎

Several works have demonstrated that incorporating execution feedback into reinforcement learning can improve functional correctness compared to purely supervised training. For example, CodeRL [6] combines a pretrained code model with reinforcement learning driven by unit test outcomes and introduces critic-based mechanisms to mitigate reward sparsity. Subsequent studies have extended this line of work by designing richer feedback signals. For example, StepCoder introduces curriculum-style reinforcement learning with compiler/unit-test feedback [7]. CodeRL+ further pushes this direction by introducing execution semantics alignment, adding variable-level execution trajectory signals so that optimization is guided by how the generated program behaves during execution [2]. However, RL with verifiable rewards is sensitive to reward shaping: auxiliary terms can introduce proxy objectives that are only loosely coupled to semantic correctness. Recent work has emphasized reward hacking as a practical concern in code-oriented RL environments and has proposed benchmarks and taxonomies for detecting reward exploits [8].

2.3 Group-Based Policy Optimization↩︎

Group Relative Policy Optimization (GRPO) was introduced in the DeepSeekMath work as a variant of Proximal Policy Optimization (PPO) that eliminates the separate critic using instead the baseline from group scores [9]. It has since been adopted in large-scale LLM reinforcement learning, including DeepSeek-R1, for optimizing reasoning and other verifiable tasks.

A complementary formulation, Group Sequence Policy Optimization (GSPO), applies the group-relative objective at the sequence level rather than at the token level [10]. Both objectives are relevant for reinforcement learning with verifiable rewards (RLVR), where rewards are computed programmatically such as from unit tests.

3 Method Description↩︎

3.1 Problem Setup↩︎

We study Python code generation from natural-language specifications. Given a programming task described in text, the model must produce a Python implementation of the required function such that it satisfies a held-out set of unit tests. We use the Mostly Basic Python Problems dataset, where each instance consists of (i) a short problem statement and (ii) reference tests that specify the desired behavior.

Formally, for each task \(i\) we have a prompt \(x_i\) (problem description, optionally augmented with a signature hint) and a candidate program \(y_i\) generated by the model. Correctness is defined by execution: \(y_i\) is considered correct if it compiles and passes all provided tests under the evaluation harness. A typical MBPP task asks for a small utility function, e.g “Write a python function to find the first repeated character in a given string”

The expected output is a self-contained Python function definition that can be executed by the test runner, for example:

3.2 Base models↩︎

We conduct experiments with two small LLMs: (i) Qwen3-0.6B [11] and (ii) Llama3.2-1B-Instruct [12]. Both models are used in their instruct variants. All training is performed with parameter-efficient adapters (LoRA) [13], while the underlying base model parameters remain frozen.

3.3 Prompting↩︎

Each MBPP task provides a natural language problem statement and reference tests. During generation, the model is prompted with the problem description and the example of test case. It is important to note that we provide only one example of test case and not the entire list to avoid leaking potential solutions to the model. Example of the prompt is shown in Figure 1. We post-process model outputs by extracting Python code blocks and normalizing them into an executable module containing the target function.

3.4 Reward Design↩︎

We consider three reward formulations:

• Static-Analysis-Only Reward

• Unit-Test-Only Reward

• Combined Reward

3.4.1 Static-Analysis-Only Reward↩︎

In the first RLVR configuration, rewards are derived exclusively from static analysis feedback produced by the Ruff linter ¹. No unit test execution signal is used in this setting. The objective is to encourage syntactically correct and stylistically clean code while evaluating how such shaping influences downstream functional correctness. For each generated completion \(y\), we invoke Ruff in JSON-output mode:

The generated source code is passed via standard input. Ruff returns a list of issues \(\mathcal{E}(y)\), where each issue contains a rule identifier and diagnostic message.

Each finding is mapped to a severity level: \[\text{severity}(e) \in \{\texttt{error}, \texttt{high}, \texttt{normal}, \texttt{low}\},\] using a predefined rule-to-severity mapping. If a rule identifier is not directly mapped, we attempt a fallback mapping using tokens extracted from the diagnostic message. Unmapped errors or warnings default to the normal severity level. Each severity level is associated with a fixed scalar penalty: \[w(\texttt{error}) = -10.0, \quad w(\texttt{high}) = -5.0, \quad w(\texttt{normal}) = -1.0, \quad w(\texttt{low}) = -0.2.\]

Let \(N_s(y)\) denote the number of findings of severity \(s\) in completion \(y\). The total Ruff-based reward is defined as \[R_{\text{ruff}}(y) = \sum_{s \in \{\texttt{error}, \texttt{high}, \texttt{normal}, \texttt{low}\}} w(s) \cdot N_s(y).\]

If Ruff produces no issues (i.e., \(\mathcal{E}(y)=\varnothing\)) the reward is set to \(0\).

This severity-to-penalty mapping is heuristic rather than learned or tuned exhaustively. Its purpose is to encode an ordinal preference over Ruff findings, assigning larger penalties to more severe issues while keeping low-severity style violations weakly penalized. In this work, we use it as a simple and interpretable shaping signal rather than claiming it is the uniquely correct weighting scheme.

3.4.2 Length-Normalized Static-Analysis Reward↩︎

During the design of the static-analysis-only reward, we identified a potential failure mode: the model may reduce the total penalty simply by generating shorter and simpler code, without actually improving correctness. Because Ruff penalties are accumulated over the generated program, longer completions can receive larger negative rewards even when they represent reasonable solution attempts.

To mitigate this length bias, we introduce a normalized variant of the static-analysis reward. Let \[\tilde{R}_{\text{ruff}}(y)= \sum_{s \in \{\texttt{error}, \texttt{high}, \texttt{normal}, \texttt{low}\}} w(s)\cdot N_s(y)\] denote the unnormalized Ruff penalty, and let \(L(y)\) denote the number of generated lines in completion \(y\). We define the normalized reward as \[R_{\text{ruff-norm}}(y)= \operatorname{clip}\!\left( \frac{\tilde{R}_{\text{ruff}}(y)}{\max(1,L(y))}, -1,0 \right).\]

Thus, the normalized static-analysis reward remains penalty-only: completions with no Ruff findings receive reward \(0\), while completions with more or more severe findings receive negative reward. Normalization reduces sensitivity to output length and discourages degenerate behavior in which the model minimizes penalties primarily by producing overly short code.

3.4.3 Unit-Test-Only Reward↩︎

In the second RLVR configuration, rewards are derived exclusively from functional correctness measured via unit test execution. No static-analysis penalties are used in this setting. The objective is to directly optimize behavioral correctness under the MBPP evaluation protocol.

For each generated completion \(y\), we extract the Python function implementation and evaluate it using the HumanEval-style execution harness. Each completion is paired with the corresponding list of MBPP tests, including both standard and challenge tests.

Following prior work such as CodeRL [6], we employ a graded reward mapping that differentiates between distinct failure modes. The execution result for each completion is categorized into one of the following outcome classes:

Each outcome category is mapped to a scalar reward:

\[R_{\text{unit}}(y) = \begin{cases} \phantom{-}1.0, & \text{if Passed}, \\ -0.3, & \text{if Assertion Error}, \\ -0.6, & \text{if Runtime Error}, \\ -1.0, & \text{if Interpreter Error or otherwise}. \end{cases}\]

This design introduces graded penalties rather than a purely binary reward. Interpreter-level failures receive the strongest penalty, reflecting complete invalidity of the program. Runtime errors receive intermediate penalties, while assertion failures incur milder penalties since the program structure is valid but functionally incorrect.

3.4.4 Combined Reward: Unit Tests + Normalized Ruff Penalty↩︎

In the third RLVR configuration, we combine execution-based unit test rewards with a normalized, penalty-only static analysis signal from Ruff. The unit test component remains identical to the unit-test-only setting (3.4.3), while the Ruff signal is normalized to reduce sensitivity to output length. In addition, we apply a small anti-shortcut penalty to discourage degenerate “empty” generations.

Let \(\mathcal{E}(y)\) denote the set of Ruff issues for completion \(y\) and let \(N_s(y)\) be the number of issues assigned to severity level \(s \in \{\texttt{error},\texttt{high},\texttt{normal},\texttt{low}\}\) (using the same severity mapping as in the Ruff-only setting). We first compute the raw Ruff penalty as a weighted sum: \[R_{\text{raw}}(y) = \sum_{s} w(s)\,N_s(y),\] If Ruff produces no findings, \(R_{\text{raw}}(y)=0\). If Ruff fails (e.g., invalid JSON output), we assign a mild penalty of \(-0.5\).

Because \(R_{\text{raw}}(y)\) scales with the number of lines and may introduce a length bias, we normalize it by the number of lines: \[R_{\text{line}}(y)=\frac{R_{\text{raw}}(y)}{\max(1, L(y))},\] where \(L(y)\) is the number of lines in \(y\).

Finally, we clip the normalized penalty into a bounded interval to prevent rare bursts of violations from dominating the reward: \[R_{\text{ruff}}(y) = \mathrm{clip}\big(R_{\text{line}}(y),\, [c_{\min}, 0]\big), \qquad c_{\min}=-1.0,\] so that \(R_{\text{ruff}}(y)\in[-1,0]\).

To discourage trivial completions that avoid both unit test failures and static analysis findings by producing almost no code, we apply a length-based penalty: \[R_{\text{short}}(y)= \begin{cases} -0.5, & \text{if } \mathrm{chars}(y) < 30 \;\text{or}\;L(y) < 3,\\ \phantom{-}0, & \text{otherwise}. \end{cases}\]

We combine unit test reward \(R_{\text{unit}}(y)\) (3.4.3) with Ruff and anti-shortcut signals using a pass-gated weighting scheme. The final reward is: \[R_{\text{comb}}(y)= \begin{cases} R_{\text{unit}}(y) + \lambda_{\text{pass}}\,R_{\text{ruff}}(y) + R_{\text{short}}(y), & \text{if } R_{\text{unit}}(y) = 1.0 \;(\text{Passed}), \\ R_{\text{unit}}(y) + \lambda_{\text{fail}}\,R_{\text{ruff}}(y) + R_{\text{short}}(y), & \text{otherwise}, \end{cases}\] where \(\lambda_{\text{pass}}\), \(\lambda_{\text{fail}}\) are specified constants introduced to regulate “ruff” component importance.

This “gate” prevents Ruff penalties from dominating optimization on failing solutions (where functional correctness has not yet been achieved), while still encouraging cleaner code among already-correct solutions.

3.5 Group policy optimization↩︎

Given a prompt \(x\), we sample a group of \(G\) candidate completions \(\{y_i\}_{i=1}^G\) from the current policy and compute scalar rewards \(\{R_i\}\). We form a group-relative advantage (TRL default is a simple group baseline; optionally normalized): \[\label{eq:group-adv} \bar{R}=\frac{1}{G}\sum_{j=1}^G R_j, \qquad A_i = R_i - \bar{R} \quad \text{(optionally }A_i \leftarrow \tfrac{A_i}{\mathrm{std}(R)+\epsilon}\text{).}\tag{1}\] Let \(y_i=(a_{i,1},\dots,a_{i,T_i})\) and denote the token-level likelihood ratio (importance weight) between the new policy \(\pi_\theta\) and the sampling policy \(\pi_{\theta_{\mathrm{old}}}\): \[\label{eq:token-ratio} r_{i,t}(\theta)=\frac{\pi_\theta(a_{i,t}\mid x,a_{i,<t})}{\pi_{\theta_{\mathrm{old}}}(a_{i,t}\mid x,a_{i,<t})}.\tag{2}\]

3.5.0.1 Token-level GRPO (TRL)

GRPO applies a PPO-style clipped policy-gradient objective at the token level, using the same scalar group advantage \(A_i\) for all tokens of a sampled completion: \[\begin{align} \mathcal{L}_{\mathrm{GRPO}}(\theta) &= \frac{1}{G}\sum_{i=1}^G \frac{1}{T_i}\sum_{t=1}^{T_i} \min\!\Big(r_{i,t}A_i,\; \mathrm{clip}(r_{i,t},1-\varepsilon,1+\varepsilon)A_i\Big) \\ &\quad - \beta\,\mathrm{KL}\!\big(\pi_\theta(\cdot\mid x)\,\|\,\pi_{\mathrm{ref}}(\cdot\mid x)\big). \end{align}\] where \(\varepsilon\) is the clip range and \(\beta\) controls KL regularization to a reference policy \(\pi_{\mathrm{ref}}\).

3.5.0.2 Sequence-level GSPO (GRPO variation in TRL):

GSPO uses the same group-relative advantage, but moves the importance weighting to the sequence level by aggregating log-probability ratios across the entire completion: \[\label{eq:seq-ratio} r_i(\theta) = \frac{\pi_\theta(y_i\mid x)}{\pi_{\theta_{\mathrm{old}}}(y_i\mid x)} = \exp\!\Big(\sum_{t=1}^{T_i}\log \pi_\theta(a_{i,t}\mid x,a_{i,<t})-\log \pi_{\theta_{\mathrm{old}}}(a_{i,t}\mid x,a_{i,<t})\Big).\tag{3}\] The clipped objective is then applied once per completion: \[\begin{align} \mathcal{L}_{\mathrm{GSPO}}(\theta) &= \frac{1}{G}\sum_{i=1}^G \min\!\Big(r_i(\theta)A_i,\; \mathrm{clip}(r_i(\theta),1-\varepsilon,1+\varepsilon)A_i\Big) \\ &\quad - \beta\,\mathrm{KL}\!\big(\pi_\theta(\cdot\mid x)\,\|\,\pi_{\mathrm{ref}}(\cdot\mid x)\big), \end{align}\] Compared to token-level GRPO, GSPO assigns a single importance weight to each sampled completion, which changes the credit assignment and can yield different length/entropy dynamics under the same reward.

4 Experiments↩︎

4.1 Dataset↩︎

We use the Mostly Basic Python Problems (MBPP) benchmark [5], which consists of short Python programming tasks paired with reference solutions and unit tests. MBPP provides predefined splits; we use MBPP train for RLVR training and report primary results on the MBPP test split.

In addition, we perform a stricter external evaluation using EvalPlus benchmarks (MBPP-Plus and HumanEval-Plus) [14], described in Section 4.6.

We do not modify tasks or tests; all comparisons use the same evaluation harness. There are 974 tasks in total:

• 374 in train split,

• 500 in test split,

• 90 in validation split,

• 10 in “prompt” split (used for few-shot prompts)

4.2 Metrics↩︎

We report pass@1 on MBPP test, i.e., the fraction of tasks for which the first sampled solution passes all tests [4].

For failing tasks, we record the dominant failure mode from execution traces (e.g., AssertionError, TypeError, NameError, SyntaxError) and report category frequencies.

We report the average number of Ruff findings per task, stratified by severity levels.

We report mean generated tokens per run and study its correlation with pass@1 and reward configurations.

4.3 Training Details↩︎

All RLVR training is performed using LoRA adapters [13]. We keep base weights frozen and optimize only adapter parameters. We use:

For LoRa adapter we target several modules such as:

• Self-Attention projections:

  • q_proj - Query projection \(W_Q\)

  • k_proj - Key projection \(W_K\)

  • v_proj - Value projection \(W_V\)

  • o_proj - Attention output projection \(W_O\)

• Feed-Forward (MLP) projections:

  • up_proj - Expansion projection \(W_{\text{up}}\)

  • gate_proj - Gating projection \(W_{\text{gate}}\)

  • down_proj - Contraction projection \(W_{\text{down}}\)

We evaluate the following methods (matching the labels used in figures):

4.4 Test evaluation↩︎

After fine-tuning with RLVR to evaluate the model’s quality we make several runs. For each run we take MBPP test split, generate solutions for each task, using same methods as in training process we extract code, add test cases (both from basic test list and challenge test list) and evaluate the solution. We report pass@1 for each run and provide \(mean \pm SD\) for each method. We use decoding with sampling with temperature \(T=0.6\) and max token count \(max\_tokens=768\).

Our primary baseline is the corresponding normal pretrained model for each architecture using the same prompt formatting and decoding settings.

4.5 Implementation↩︎

Training and evaluation scripts were done with Python 3.12.12, using Transformers v4.57.6 [15], TRL 0.24 [16], Unsloth 2026.1.4 [17] with Pytorch 2.10. For code and test execution we utilize OpenAI human-eval python module ². Static analysis feedback was provided by using Ruff 0.14.11 ³ with default ruleset (Flake8’s F rules, along with a subset of the E rules).

All training runs were done on rented GPU-server with Nvidia RTX 3090 (24GB VRAM), while test evaluations were performed on PC with Nvidia RTX 5060Ti (16GB VRAM).

4.6 EvalPlus evaluation↩︎

To complement the main MBPP test evaluation, we additionally evaluate selected reward configurations using EvalPlus [14] on MBPP+ and HumanEval+, which provide a stricter hidden-test benchmark for functional correctness. For this evaluation, we retrain each reward configuration with two random seeds (101 and 202) and report per-seed pass@1 results. Inference is performed with an OpenAI-compatible vLLM backend (v0.17.0) [18] using greedy decoding. We evaluate on both the base and the additional hidden tests provided by EvalPlus.

5 Results and Discussion↩︎

5.1 Overall Performance↩︎

For each combination of model and method we ran 10 test evaluations. Table 1 reports mean pass@1 on MBPP test set and p-value. We compute the p-value with a paired randomization (sign-flip) test over tasks: we repeatedly flip the sign of each task’s method baseline score difference at random to simulate the null hypothesis of “no effect” and report the fraction of simulations where the resulting mean difference is at least as extreme as the observed one. We observe consistent gains from RLVR with unit-test only and combined rewards across both model families, with the best combined-reward configuration of Qwen3-0.6B model using Combined Reward improving pass@1 by up to 13 percent points over the baseline. Across the reported settings, neither GRPO nor GSPO consistently dominates the other. A plausible explanation is that MBPP and HumanEval consist of relatively short, self-contained tasks with short prompts and completions, so the benefit of sequence-level optimization is less pronounced than it might be in longer-horizon settings.

5.2 Error Analysis↩︎

Figure 2 shows the distribution of execution outcomes. For combined and unit-test only we can see reduction of AssertionErrors and an increase in the fraction of passed solutions. We also observe method-specific shifts among runtime and interpreter errors (e.g., NameError, TypeError), suggesting that different objectives may prioritize distinct correctness improvements (semantic vs. syntactic robustness).

5.3 Static Analysis Effects↩︎

Figure 3 reports Ruff findings by severity. Static-analysis shaping substantially changes the distribution of findings, but these improvements do not always translate proportionally into higher pass@1. This highlights a key RLVR tradeoff: optimizing a proxy reward (style/quality signals) can steer generation away from test-passing correctness if not calibrated carefully.

5.4 Length Degeneration↩︎

Figure 4 plots mean number of generated tokens vs pass rate across runs. We can observe that Ruff-dominant configuration tend to produce shorter output while having lower pass rate. This static-analysis only configuration skews model into reward-driven degeneration which incentives simple code that reduces style violations without improving functional correctness. While unit-test configuration tend to produce longer outputs with bigger pass rates. These findings motivate reporting behavioral diagnostics alongside pass@1 for RLVR-trained code models.

5.5 External Validation with EvalPlus↩︎

So far, the previous results were based on the MBPP test split and its associated diagnostics. To assess whether these patterns transfer to a stricter hidden-test setting, we additionally evaluate independently retrained adapters on EvalPlus benchmarks (MBPP+ and HumanEval+). Table 2 reports per-seed results for both model families.

The overall pattern remains similar to the main MBPP analysis. Static-analysis-only methods remain unstable and often weak, while the unit-test-only and combined reward families are generally stronger. However, under stricter evaluation, the combined reward is competitive in some settings but does not provide a consistent improvement over the unit-test-only reward across both models and both benchmarks. These results suggest that the benefits of reward shaping are sensitive to evaluation protocol and do not always transfer uniformly to stronger hidden-test settings.

4pt

6 Limitations↩︎

Currently all the experiments were done with test split of MBPP dataset and EvalPlus benchmark. These benchmarks consist of short, mostly self-contained Python programming tasks with relatively small test suites. As a result, conclusions may not transfer to long-horizon code generation settings

Our reward formulations include several heuristic design decisions:

• a graded unit-test reward mapping,

• a Ruff-derived penalty,

• a line-normalized and clipped Ruff reward in the combined setting

These choices were motivated by stability and practical considerations, but they introduce additional hyperparameters (e.g., clipping threshold, weighting (\(\lambda\)) coefficients, length thresholds) that may affect outcomes. We did not perform exhaustive tests over these parameters, and alternative normalizations or weight schedules could yield different trade-offs.

We evaluate two small models (Qwen3-0.6B and Llama3.2-1B), which still leaves open whether the same reward-shaping effects hold for larger models or for models trained with different instruction-tuning schemes.

7 Conclusion↩︎

We presented a study of reinforcement learning with verifiable rewards (RLVR) for Python code generation on MBPP using two small language models, Qwen3-0.6B and Llama3.2-1B. Across both architectures, RLVR improves functional correctness, with the best configuration achieving up to +13 percent points in pass@1 over the corresponding baseline. Regarding the optimization variants, we do not observe a universal winner between token-level GRPO and sequence-level GSPO. Both variants are capable of improving performance under suitable reward design, and neither is uniformly dominant across any tests. A more decisive comparison between GRPO and GSPO may require longer-horizon code-generation benchmarks, where sequence-level credit assignment is expected to matter more strongly than in short benchmark tasks such as MBPP and HumanEval. Our analysis also shows that reward design induce behavioral shifts. First, unit-test-driven RLVR substantially changes the failure profile, we can observe not only reduction in Assertion Errors (failing unit tests), but also with Type and Name Errors. Second, static-analysis rewards affects code quality diagnostics but does not necessarily translate into higher pass@1: Ruff-optimized runs shift severity distributions toward fewer “error”-level severity issues, yet can underperform in functional correctness when the static penalty dominates the learning signal. Third, we find that static-analysis penalties interact strongly with output length. According to our tests, Ruff-heavy objectives tend to produce shorter generations, whereas combined rewards shift the model toward longer completions that correlate with higher pass@1. Overall, the results emphasize that RLVR for code should be evaluated with diagnostics beyond pass@1, including execution error categories, static-analysis profiles, and length statistics.

Future work includes more extensive reward-weight sweeps, training-dynamics analysis (KL/reward curves), systematic sensitivity analysis of the Ruff severity weights, including alternative scaling schemes and severity groupings and extending the approach to additional datasets and languages.

References↩︎