1 Introduction↩︎

Post-training with reinforcement learning has become routine for language and multimodal agents. A supervised model is first taught to follow the task format, and GRPO or a close relative [1]–[3] is then run against a verifiable reward in the hope of pushing capability past what supervision alone reached. For web agents the appeal is direct, since rollouts are cheap to score against task success and the recipe that lifted mathematics and coding models is expected to lift an agent that clicks, types, and navigates [4]. What remains unclear is whether this second stage adds new skill to a small language and vision-language model agent, or whether it mostly sharpens choices the supervised model could already make [5]. This matters in practice. If reinforcement learning adds capability, more of it is the right investment; if it only sharpens what the supervised model already does, the effort belongs in better supervision or distillation.

We study one such agent under tight control: a 4B language and vision-language model [6] driving a web agent on the MiniWoB benchmark [7], [8]. Holding the harness, reward, and task set fixed, we sweep the reinforcement-learning choices a practitioner would tune and score every run with clean greedy decoding and a paired test against the supervised baseline. The result is a null. No configuration in the grid of 18 runs credibly beats supervision on tasks the agent has already largely mastered, and once the learning rate leaves a narrow low band the agent gets worse. The null is also robust. It survives adding evaluation and training seeds, adding the schedule and group-size choices the grid omitted, switching from a text observation to a Set-of-Marks screenshot, and scaling the backbone to 8B. Figure 1 previews the study and its outcome.

One reading of a null is that the pipeline is broken or underpowered and would fail to climb anywhere. A positive control rules this out. Using the identical harness, reward, and recipe, we select tasks whose reward is reachable by sampling, where the model already succeeds under temperature more often than under greedy decoding, and retrain. There the agent climbs by up to 22 points, with a paired interval that excludes zero. GRPO improves the agent when the sampled policy can already beat the greedy one, and it stalls when greedy is the better of the two, which is the regime a competent supervised agent occupies. A practitioner can act on this by measuring the gap between sampled and greedy success first and expecting little from reinforcement learning when that gap is not positive. For a competent small web agent, extra skill comes from supervision or distillation, not from more reinforcement learning.

We then open the model to ask why the harmful regimes fail. A low learning rate leaves an update that is real but too small to change behavior; a middle rate degrades the agent, and a high rate collapses it. The two failures are different lesions. The degrade regime destroys the effective rank [9] of the late layers while leaving earlier layers intact, whereas the collapse regime holds that rank at or above supervision and instead destroys the readout, an inversion we confirm at nearly two thousand fixed states. To test whether this correlation is causal, we graft weights. Restoring the attention or MLP blocks from initialization repairs the degraded agent, while restoring the embedding, whose drift dominates the raw weight change, does nothing, which points to the compute blocks as the site of the failure and clears the embedding despite its large movement [10]. Read in the other direction, the same probe shows late-layer rank rising with capability where the agent climbs and falling where it degrades, so at 4B rank tracks skill both ways. At 8B this coupling breaks down, and we say so.

Our choice of a single benchmark is deliberate and follows from the interpretability goal. A mechanistic account requires holding everything except the intervention fixed, so that a change in effective rank or success rate can be attributed to the learning rate and not to uncontrolled variation in the environment. MiniWoB provides that control. Its tasks are deterministic under a seed, its success check is exact, and the breadth of interface skills it covers, from clicking and typing to selection and navigation, makes it a broad probe of web-agent behavior, not a narrow one. A benchmark such as WebArena [11] is closer to deployment but introduces run-to-run variation that is hard to hold constant, and that variation would confound the attributions this study rests on. We treat MiniWoB as the controlled setting in which the mechanism can be read cleanly, and we are clear about what that costs in generality.

Our contributions are the following.

1. A controlled null: across a grid of 18 runs and several robustness checks, GRPO does not credibly add skill to a competent small language and vision-language model web agent, and higher learning rates credibly remove it.

2. A positive control that identifies the cause of the null as a headroom condition, together with a cheap test of sampled against greedy success that can be applied beforehand.

3. A mechanistic account of the failure as a learning-rate-gated double dissociation, causally localized by weight grafting, with the dominant embedding drift shown to be a correlational red herring.

4. Evidence that late-layer effective rank tracks capability in both directions at 4B and dissociates at 8B, reported as a property that depends on scale.

5. A full battery of paired statistics, equivalence tests, and interpretability measurements, released so others can check the null for themselves.

2 Related Work↩︎

6pt

Reinforcement learning with verifiable rewards has driven recent gains in reasoning models, and GRPO [1], with refinements such as decoupled clipping [2] and less biased advantage estimation [3] on top of the proximal policy optimization objective [12], is now a standard recipe. Whether these gains reflect genuinely new capability or a sharpening of behavior the base model can already produce is contested, with evidence that reinforcement learning often fails to extend a model beyond what sampling already reaches [5]. That question has been argued mostly on reasoning benchmarks through pass@k. We bring it to an interactive agent and ask it as a practitioner would, against the agent’s own supervised baseline.

For web agents, both supervised and reinforcement learning pipelines have raised success rates across interactive benchmarks, from the MiniWoB environment [7], [8] to the larger WebArena [11], and online curriculum methods report further gains [4]. This line of work shows that reinforcement learning can help a web agent. It does not settle whether it helps a small language and vision-language model agent on tasks the agent has already learned to solve, which is the regime a competent deployed system usually occupies and the one we isolate here.

Our mechanistic analysis builds on tools for reading a model’s internal state. Effective rank [9] measures how many directions a representation actually uses, and editing in weight space shows that behavior can be moved by recombining parameter groups [10]. We turn these into a causal test rather than a correlational one: we restore a single component group from its initialization, keep the rest, and re-measure success, which separates the parameters that carry a failure from those that merely move the most.

Two gaps motivate this paper. The sharpening question has not been settled for an agent by a controlled comparison against its own supervised start with a matched positive control, and where a reinforcement learning null is reported it is seldom explained at the level of the weights. We address both. We establish a controlled null for a small language and vision-language model web agent, use a positive control to show the null is a property of the task rather than the training pipeline, and give a causal account of the failure at the level of individual weight groups. To our knowledge this is the first study to pair a controlled reinforcement learning null for a web agent with a mechanism for why it occurs.

3 Experimental Setup↩︎

Our aim is to decide whether GRPO adds skill to a competent small language and vision-language web agent and, when it does not, to explain why. The design follows directly. We fix a single agent, environment, reward, and task set, so that any change in success rate traces to the reinforcement-learning choices and not to the setup, and on that fixed harness we run the three experiments summarized in Table 1. The rest of this section specifies the agent and reward (§3.1), the GRPO recipe and grid (§3.2), the evaluation and statistics that make the null credible (§3.3), and the interpretability probes (§3.4).

3.1 Agent, Environment, and Reward↩︎

The agent is Qwen3-VL at 4B and 8B [6], run on MiniWoB [7], [8]. At each step the policy \(\pi_\theta\) reads an observation and emits one action from a compact vocabulary of clicks, typing, selection, navigation, and an explicit finish, up to a fixed step budget. Two observation tracks share all other logic: a text track that serializes the page, and a Set-of-Marks track [13] that presents the screenshot with numbered marks. The control grid and the mechanism use the text track; the Set-of-Marks track is a modality check. The reward is a sparse binary terminal signal, broadcast to every step of the episode, \[r(\tau)=\mathbf{1}\!\left[\operatorname{success}(\tau)\right]\in\{0,1\}.\]

3.2 GRPO Recipe and Control Grid↩︎

For each training prompt the policy samples a group of \(G\) rollouts. Following [3], the advantage is the group mean-centered reward, with no division by the group standard deviation, \[\label{eq:advantage} A_i = r_i-\frac{1}{G}\sum_{j=1}^{G}r_j.\tag{1}\] The per-token importance ratio between the current policy and the policy that produced the rollout is \[\label{eq:ratio} \rho_c(\theta)=\frac{\pi_\theta(y_c\mid x,y_{<c})}{\pi_{\theta_{\mathrm{old}}}(y_c\mid x,y_{<c})},\tag{2}\] and the clipped surrogate minimized over a segment of \(C\) completion tokens is \[\label{eq:grpo} \mathcal{L}_{\mathrm{GRPO}}=-\frac{1}{C}\sum_{c=1}^{C}\min\!\Big(\rho_c A,\;\operatorname{clip}(\rho_c,1-\epsilon_{\mathrm{lo}},1+\epsilon_{\mathrm{hi}})\,A\Big),\tag{3}\] with an asymmetric clip-higher bound [2], \(\epsilon_{\mathrm{lo}}=0.20<\epsilon_{\mathrm{hi}}=0.28\). An optional KL anchor adds \(\beta\widehat{D}_{\mathrm{KL}}\) to the loss, using the non-negative \(k_3\) estimator against the frozen initialization \(\pi_{\mathrm{ref}}\) with \(u_c=\log(\pi_{\mathrm{ref}}/\pi_\theta)\), \[\label{eq:kl} \widehat{D}_{\mathrm{KL}}=\frac{1}{C}\sum_{c}\big(e^{u_c}-u_c-1\big).\tag{4}\] The learning rate follows a linear warmup over a fraction \(f\) of the \(R\) rounds, then a cosine decay, \[\label{eq:lr} \eta_t=\eta_0\cdot\begin{cases}(t+1)/w,& t<w,\\[3pt]\tfrac12\big(1+\cos\pi\tfrac{t-w}{R-w}\big),& t\ge w,\end{cases}\qquad w=\lceil fR\rceil.\tag{5}\] The controlled null (experiment A) is a grid of 18 runs varying learning rate, KL weight \(\beta\), seed, initialization, and the clip bound, with a recipe-ablation battery that adds the warmup and cosine schedule and the group size \(G\in\{8,16,32\}\).

3.3 Evaluation Protocol and Statistics↩︎

Every run is scored with clean greedy decoding on the set \(\mathcal{E}\) of 11 tasks at 5 seeds, giving 55 matched episodes. The success rate over \(\mathcal{E}\) is \[\label{eq:sr} \mathrm{SR}=\frac{1}{|\mathcal{E}|}\sum_{e\in\mathcal{E}}\mathbf{1}[\operatorname{success}(e)],\tag{6}\] reported at level \(z\) with the Wilson score interval [14] \[\label{eq:wilson} \frac{\widehat p+\tfrac{z^2}{2n}\pm z\sqrt{\tfrac{\widehat p(1-\widehat p)}{n}+\tfrac{z^2}{4n^2}}}{1+z^2/n}.\tag{7}\] Because a MiniWoB reward is deterministic given the task seed, each \((\text{task},\text{seed})\) pair is matched across arms and every arm is compared to the supervised baseline with a paired test. On the discordant pairs \((b,c)\) the exact two-sided McNemar \(p\)-value is [15] \[\label{eq:mcnemar} p=2\!\!\sum_{i=0}^{\min(b,c)}\!\binom{b+c}{i}2^{-(b+c)},\tag{8}\] and a task-clustered bootstrap resamples the 11 task clusters to give a \(95\%\) interval on the success-rate difference \(\Delta=\mathrm{SR}_{\mathrm{arm}}-\mathrm{SR}_{\mathrm{SFT}}\). We call an arm credibly better or credibly worse only when this interval excludes zero; a positive point estimate whose interval touches zero is no credible difference. Equivalence within a margin \(\delta\) is a two one-sided test [16]: the \(90\%\) interval of \(\Delta\) lies inside \([-\delta,+\delta]\).

3.4 Interpretability Measurements↩︎

On a fixed cache of \(N\) hidden states we read a small set of quantities. The effective rank of a layer is the exponential of the entropy of its normalized covariance eigenvalues \(\{\lambda_k\}\) [9], and we also report the participation ratio and the stable rank of the same state matrix \(X\), \[\label{eq:erank} \operatorname{erank}=\exp\!\Big(\!-\!\sum_k p_k\log p_k\Big),\quad p_k=\frac{\lambda_k}{\sum_j\lambda_j},\tag{9}\] \[\label{eq:pr} \mathrm{PR}=\frac{\big(\sum_k\lambda_k\big)^2}{\sum_k\lambda_k^2},\qquad \mathrm{sr}=\frac{\lVert X\rVert_F^2}{\lVert X\rVert_2^2}.\tag{10}\] Three probes read the output distribution \(p_n=\operatorname{softmax}(\ell_n)\). Argmax agreement with the initialization counts how often the top next-token prediction is unchanged, \[\label{eq:agr} \operatorname{agr}=\frac{1}{N}\sum_{n}\mathbf{1}\!\left[\arg\max\ell^{\theta}_n=\arg\max\ell^{\mathrm{ref}}_n\right],\tag{11}\] while the mean entropy and the degeneracy, the fraction of states whose mass collapses onto one token above a threshold \(\tau\), are \[\label{eq:beh} \begin{align} H&=\frac{1}{N}\sum_{n}\Big(\!-\!\sum_v p_{n,v}\log p_{n,v}\Big),\\ \operatorname{deg}&=\frac{1}{N}\sum_{n}\mathbf{1}\!\big[\textstyle\max_v p_{n,v}>\tau\big]. \end{align}\tag{12}\] A causal graft restores one component group \(\mathcal{G}\), attention, MLP, or embedding, from initialization and re-evaluates, \[\label{eq:graft} \theta'=\theta,\quad \theta'_{\mathcal{G}}\leftarrow\theta^{\mathrm{init}}_{\mathcal{G}},\qquad \text{recovery}=\mathrm{SR}(\theta').\tag{13}\] Finally, the headroom of a task is the gap between the sampled and greedy success of the supervised policy, \[\label{eq:head} \Delta_{\mathrm{head}}=\mathrm{SR}_{\mathrm{sample}}-\mathrm{SR}_{\mathrm{greedy}}.\tag{14}\] GRPO can climb only when \(\Delta_{\mathrm{head}}>0\); the positive control (experiment B) selects tasks on this criterion, and the grafts drive the mechanism analysis (experiment C).

4 When GRPO Fails↩︎

We begin with the regime a deployed agent occupies, the tasks it has already learned to solve. The supervised policy clears the 11-task grid at a success rate of 49.1% (27 of 55 matched episodes), an aggregate that mixes tasks it solves almost every time with a harder frontier. The question is whether any reinforcement learning configuration moves that number upward, and none does. The two strongest arms are nominally above supervision by 3.6 points, yet their paired intervals stretch from zero to about ten points and the exact McNemar test returns \(p\) between 0.50 and 0.69, so the nudge is indistinguishable from noise (Table ¿tbl:tab:regimes?). Nor does the result hinge on the learning rate alone. Varying the KL weight, the initialization, and the clip bound each moves the point estimate by at most two points and none of them credibly, so no axis of the 18-run grid turns into a gain. Even taking the best arm as the headline, an upward-biased choice, leaves an estimate that only matches supervision plus a statistically empty 3.6 points.

A flat result invites the reading that the tasks leave no room to improve, but they do. Restricting the score to the frontier subset, defined as the tasks whose supervised success sits between 0.2 and 0.8, the baseline is 37.1% and the best configuration reaches 42.9%, a gain whose paired interval still includes zero, so the headroom that exists is not credibly converted into gain. Stated the other way, two one-sided tests place four of the five strongest configurations within five points of supervision, one of them episode for episode identical, which reads the outcome as an equivalence: on the mastered tasks GRPO matches the supervised policy but does not improve on it.

The one variable that moves success is the learning rate, and it moves it only downward. Holding initialization and the KL weight fixed on the text track, clean success falls as the step size grows, from 52.7% at \(3\times10^{-6}\), through the supervised baseline near \(5\times10^{-6}\), down to 33.3% at \(1\times10^{-5}\) and zero at \(2\times10^{-5}\) (Figure 2). Three regimes fall out. At the low end the update is a functional no-op that leaves behavior at supervision. A middle rate degrades the agent by a credible 15 points, and the highest rate collapses it to zero, both with intervals that exclude the baseline and McNemar \(p\) below 0.04 and 0.001 (Table ¿tbl:tab:regimes?). The only rates that leave the agent unharmed are the ones too small to change behavior, and no rate above that band clears the baseline. This monotone shape is specific to the 4B text track; on the Set-of-Marks track and at 8B the nominal peak sits elsewhere, so we keep the shape scoped to 4B text.

The null does not rest on any single choice in the grid, and the full battery is collected in Appendices 11 and 12. Adding 25 evaluation seeds lowers the baseline to 44.7% and still surfaces no credible winner; the one arm that reaches \(p=0.001\) under an unclustered test loses that status once episodes are clustered by task, and a replication over 6 training seeds averages 49.7%, back on the baseline, so its apparent edge was a lucky training seed. The warmup and cosine schedule the grid omitted remove the collapse pathology, since moderate rates now dip and recover instead of dying, but a recovery is not a gain and the schedule still produces no credible improvement. A larger sampling group of 16 or 32 does not help, and at a high rate it breaks the agent sooner, since a larger group is a larger effective step. The Set-of-Marks screenshot raises the baseline to 63.6% and again yields no credible gain, and the 8B backbone repeats the pattern, a nominal low-rate nudge that never clears the interval and credibly worse success at the higher rates. The null holds across every axis a practitioner would turn.

A null could also be an artifact of reward-hacking or of scoring a checkpoint after it broke, and neither is at work here. If the agent were gaming the sparse reward by finishing early, the flat runs would fill with premature terminations, but the per-episode taxonomy puts reward-hacking near zero in every regime, and the degrade and collapse runs fail because they emit invalid output, not because they stop short. If the null came from scoring the final checkpoint after it had already broken, then selecting the best checkpoint by held-out success would surface a hidden win. It does not. Scoring every saved round and running a paired test on the best one, the strongest flat run still gains only 3.6 points with an interval that touches zero, and the brief early rise of the degrade run falls apart before it is credible. The null comes from the setting itself: it is not an artifact of when we stopped, and it is not the agent gaming the reward. That still leaves the worry that the pipeline cannot climb anywhere, which the next section settles.

4.5pt

5 When GRPO Succeeds↩︎

The previous section ended on the obvious worry that the pipeline simply cannot climb anywhere. We test it by keeping the harness, reward, and recipe exactly as before, still on the 4B text track, and changing only the tasks. We profiled the supervised policy on all 119 MiniWoB tasks, recording both its greedy success and the success it reaches under temperature sampling, and selected ten tasks with reachable headroom, where sampling already succeeds more often than greedy and there is therefore a policy for GRPO to move toward that greedy does not already reach. Two examples fix the idea: drawing a line, where greedy never succeeds but sampling does more than half the time, and filling an autocomplete field, where sampling roughly doubles the greedy rate.

On this set the same recipe climbs. Scored on the ten tasks at 5 seeds, 50 matched episodes, the supervised baseline solves 20.0%, and the strongest configuration reaches 42.0%, a gain of 22 points whose task-clustered interval runs from 8 to 40 and excludes zero, with McNemar \(p=0.007\) (Figure 3). The climb is not one lucky run. Five of the six configurations rise between 12 and 22 points, four of them credibly, and only a single run at a middle rate degrades. The low learning rate that was a pure no-op on the mastered grid gains 12 to 18 points here, so the variable that decides the outcome is the task and not the step size.

Table ¿tbl:tab:contrast? puts the two settings side by side. The recipe that cannot separate from supervision on the mastered grid produces a credible 22-point gain on the headroom set, so the null is not a broken or underpowered pipeline; it is GRPO failing where a working GRPO would also fail, on tasks whose reward the greedy policy already reaches. This explains the null in terms of the objective itself. GRPO optimizes sampled rollouts, so it can only climb toward a policy that sampling reaches. On the mastered grid the gap runs the wrong way: greedy is at 0.49 against a sampled 0.32 to 0.38, so there is nothing above greedy to move toward. On the headroom set the gap is positive by construction, and the agent climbs. The gap of Eq. 14 is therefore a cheap screen a practitioner can run before committing compute, since it predicts both the null and the climb from the supervised policy alone.

5pt

6 Why GRPO Fails↩︎

The headroom condition says when GRPO fails to help, but not how the harmful learning rates damage the model, so we open it up. Reading a small set of quantities on a fixed cache of hidden states shows that the two harmful regimes damage the model in different ways. The middle rate that degrades the agent destroys the effective rank of the late layers, read at layer 35, which falls from about nine at supervision to near one while the earlier layers keep their rank. The high rate that collapses the agent leaves that late-layer rank at or above supervision yet drives the argmax agreement with initialization to zero (Figure 4, left). Both regimes push the readout down, collapse completely and degrade in part, but only degrade also destroys the representation. This rank inversion is stable across the full cache of 1999 states, so it is not an artifact of thin sampling.

Rank and agreement are correlations, and to make them causal we graft. Restoring one component group from initialization into the trained model, keeping the rest, and re-scoring isolates the weights that carry the failure. For the degraded agent, restoring the attention or the MLP alone lifts frontier success from 11% to 37% and 40%, at or just above the supervised level, while restoring the embedding leaves it near 14% (Figure 5; the full recovery table for both regimes is in Appendix 13). Either compute pathway is sufficient, so the failure is localizable, and the direction holds across three degrade checkpoints. This is where a correlational reading would have gone wrong. The embedding drifts more than any other group in the raw weight difference, so an argument from movement alone would blame it, yet the graft shows the embedding is causally inert while the attention and MLP carry the damage. To rule out a generic undoing of drift, we restore a random group of matched parameter count as a null; across two dozen draws it recovers to a mean of 19% and a 95th percentile of 23%, and the real attention and MLP grafts sit well above that band, so the localization is specific and not a side effect of moving weights back toward their start.

The collapsed agent behaves differently under the same test. No single group restores it, attention and MLP alone recover almost nothing, and only restoring every group together returns it to the supervised level, so collapse is a distributed corruption of the readout rather than a lesion in one place. This completes a double dissociation: degrade destroys the late-layer rank and is localizable to a single compute pathway, while collapse spares the rank and cannot be localized at all.

For rank to be a meaningful measure, it should rise when the agent improves, not only fall when it breaks. On the family used for the positive control it does. Reading the same late-layer rank on three checkpoints from that family, the supervised start sits at rank 6.9 with 20% success, the configuration that climbs to 42% keeps its rank healthy at 7.1, and the configuration that degrades to 12% has its rank cut to 3.2 (Figure 4, middle). The loss shows up as a sharp rank collapse and the gain as a rank that stays healthy rather than falling, so within this family rank health and capability move together, with the caveat that these are single checkpoints and the upward move is small.

This coupling is a property of the 4B model and does not survive a change of scale. Reading the same probe across the learning-rate sweep on the 8B backbone, on the text track, we find no setting where rank and argmax agreement fall together (Figure 4, right). At the rates that degrade the 4B model the 8B late-layer rank collapses to about two while the argmax agreement stays between 0.78 and 0.95, so the readout is untouched even though the representation is not, and only at the highest rate does the agreement collapse, with the rank left high. The larger model absorbs a late-layer rank collapse without changing what it predicts. We therefore scope the account: the link between rank and capability holds at 4B, shown in both directions by grafting and by the positive control, and it breaks at 8B. We treat that as a limit of the 4B account.

The mechanism, then, is a pair of learning-rate-gated lesions. A middle rate erases the late-layer rank that the attention and MLP carry, and a high rate erases the readout in a way spread across the whole model. Neither adds anything the supervised agent lacked.

7 Discussion↩︎

The result carries a direct lesson for practice. On a small web agent that already performs its tasks, the next increment of skill does not come from a second stage of reinforcement learning, because the agent’s greedy policy already sits above what its own sampling reaches and GRPO can pursue only the sampled policy. The headroom gap of Eq. 14 is the screen to run first: a value at or below zero on a held-out set is the signal to spend the budget on supervision instead. In a distillation pilot, expert supervision lifted the base model by 30 points on a task set where reinforcement learning on top of any initialization added nothing (Appendix 11).

Two boundaries on the claim should be read alongside it. The study is one benchmark and one model family, chosen so the mechanism can be read without the confounds a noisier environment adds, and the robustness battery covers the recipe choices a reviewer would raise (schedule, group size, seeds, observation track, and model scale), none of which changes the verdict. The credible harm from a large step is established on the 4B text track and is only nominal on the Set-of-Marks track, so we state it as a text-track result and not a universal one. The mechanism has its own scope. The link between late-layer rank and capability that grafting and the positive control establish at 4B does not hold at 8B, where a rank collapse and a change in behavior never coincide, so the rank account is a property of the smaller model and not a law across scale.

The finding brings the sharpening debate, so far settled largely on reasoning tasks by pass@k [5], to an interactive agent with a controlled answer. Where that debate asks whether reinforcement learning extends a model beyond its base, we ask the same of an agent against its own supervised start and find that it does not, then show with the positive control that the boundary is not the method but the headroom of the task. The contested general claim becomes a specific one we can test: reinforcement learning extends the agent only when its sampled policy already beats its greedy policy.

8 Conclusion↩︎

We asked whether GRPO adds skill to a small language and vision-language web agent on tasks it has already learned. Across the grid and the robustness checks it does not, and past a narrow band of learning rates it subtracts skill instead. The same pipeline climbs 22 points once the task has headroom, so the null is about the task, not the method. The failure has a clear mechanism: a middle rate erases the late-layer rank that grafting traces to attention and MLP, a high rate erases the readout that no single group repairs, and the dominant embedding drift is inert. For a competent small agent, more skill comes from better supervision, and GRPO earns its compute only where sampling beats greedy.

Limitations↩︎

This study is deliberately narrow, and several boundaries should be kept in mind. It uses a single benchmark, MiniWoB, and a single model family, Qwen3-VL at 4B and 8B; the null and the mechanism may look different on a more varied environment such as WebArena or on a different backbone. The credible harm from a large learning rate is established on the 4B text track, and on the Set-of-Marks track it is only nominal, so the harm side of the result is text-track evidence and not a universal claim. The mechanism has a scale boundary of its own: the link between late-layer rank and capability holds at 4B but breaks at 8B, so the rank account is specific to the smaller model. The causal grafting and rank-health readings rest on a small number of checkpoints per regime and their point estimates are noisy, so we report them qualitatively. Finally, the null concerns the regime where the agent has already mastered its tasks, and the positive control shows that the same recipe does climb once the task has headroom; we therefore make no claim that reinforcement learning is useless for web agents in general, only that it adds no skill in the competent-agent regime we isolate.

The appendices below collect the full result tables, a gallery of the supporting figures, and the algorithms referenced in the main text. All numbers are transcribed from the same frozen result tables used in the body.

9 The Control Grid in Full↩︎

The controlled null is an 18-run grid on the 4B text track that varies the learning rate, the KL weight, the seed, the initialization (supervised or base), and the clip bound. Every arm is scored by clean greedy decoding on the 11 tasks at 5 seeds, giving 55 matched episodes, and compared to the supervised baseline with the paired test of the main text. Table 2 lists the representative arms. Classified by their training dynamics the 18 runs split into 9 functional no-ops, 6 degrade runs, and 3 collapse runs, and no arm’s paired interval clears the baseline. Reporting the best of eighteen is itself an upward-biased statistic, and even that best matches supervision plus a statistically empty \(3.6\) points. The “no headroom” reading is refuted on the frontier subset, the tasks whose supervised success lies in \([0.2,0.8]\): the supervised frontier baseline is \(37.1\%\) \([23.2,53.7]\) (13 of 35) and the best configuration reaches only \(42.9\%\) \([28.0,59.1]\) (15 of 35), whose interval lower bound of \(28.0\) sits below the baseline point estimate.

6pt

10 Paired Statistics, Equivalence, and Checkpoint Selection↩︎

The verdict uses the exact McNemar test on the discordant pairs and a task-clustered bootstrap on the 11 task clusters, both over episode-matched arms. Beyond the failure to reject, a two one-sided test states a positive equivalence: four of the five strongest configurations lie within \(\pm 5\) points of supervision (Table ¿tbl:tab:tost?), one of them episode-identical. Because the reported success is the final checkpoint, we also retrained the three regime exemplars saving every round and paired-tested the single best-by-eval checkpoint of each; none is credibly superior (Table ¿tbl:tab:bestckpt?), so the null is not an artifact of scoring an already broken checkpoint.

5pt

4pt

11 Robustness Battery↩︎

The null is unchanged by every recipe and evaluation choice a reviewer would raise. Table ¿tbl:tab:robust? collects the axes. Widening evaluation to 25 seeds lowers the baseline to \(44.7\%\) and leaves no credible winner. The one borderline arm (McNemar \(p=0.001\) under an unclustered test) loses credibility once episodes are clustered by task, and a six-seed replication of it averages \(49.7\%\) (Table ¿tbl:tab:trainseed?), so the edge was a lucky training seed. Adding the warmup and cosine schedule removes the collapse pathology without producing a gain, and a larger group breaks the agent sooner at high rates (Table ¿tbl:tab:recipe?). A per-episode taxonomy shows the failures are output degeneration, not reward-hacking: premature finishing is near zero in every regime (Table ¿tbl:tab:taxonomy?). On a separate distillation pilot (5 tasks, 6 seeds, \(n=30\)), expert supervision lifts the base model from \(50\%\) to \(80\%\) while GRPO on top of any initialization adds nothing, which is the evidence that the constructive lever is supervision rather than reinforcement learning.

4pt

4pt

5pt

5pt

12 Observation Track and Model Scale↩︎

The no-gain null generalizes across observation track and model scale, while the credible harm from a large rate is established on the 4B text track and is only nominal elsewhere. On the Set-of-Marks track the supervised baseline is higher, at \(63.6\%\), and no configuration credibly beats it; the high-rate degradation is directional but not credible (Table ¿tbl:tab:som?). At 8B on Set-of-Marks the baseline is also \(63.6\%\), the low-rate nudge is again nominal, and the higher rates are credibly worse (Table ¿tbl:tab:som8b?). The 8B mechanism cache (372 states, text) reproduces the late-layer rank inversion: supervised \(13.6\), flat \(15.1\), degrade \(1.9\), collapse \(19.6\).

5pt

5pt

13 Mechanism in Detail↩︎

The interpretability probes read a fixed cache of hidden states. Movement magnitude does not predict the failure mode: collapse moves less in \(L_2\) than degrade yet is more destructive, and the single largest total movement in the grid is the KL rescue arm, which is behaviorally at the baseline (Table ¿tbl:tab:wmove?). The rank and output-behavior probes give the double dissociation in full, including next-token entropy (Table ¿tbl:tab:rankfull?). Causal grafting localizes degrade to the attention or MLP and refutes the embedding, whose drift is largest; a magnitude-matched random-component null over 24 draws recovers only to a mean of \(19.3\%\) and a 95th percentile of \(22.9\%\), below the real grafts, while for collapse no single group recovers (Table ¿tbl:tab:graft?). The rank-to-capability coupling that holds at 4B does not transfer to 8B, where rank and argmax agreement never fall together across the sweep (Table ¿tbl:tab:decouple?). The KL anchor is a mode-preserving stabilizer, not a teacher: every KL arm holds argmax agreement at \(1.0\) while next-token entropy inflates to \(4.9\)–\(7.7\), and KL suppresses embedding drift (\({\sim}0.006\) with KL against \({\sim}0.011\) without, at \(1\times10^{-5}\)) rather than freezing the weights.

4pt

4pt

7pt

7pt

14 The Positive Control in Detail↩︎

The positive control keeps the harness, reward, and recipe of the 4B text grid and changes only the tasks. We profiled the supervised policy on all 119 MiniWoB tasks and selected ten with a positive sampled-minus-greedy gap (Table ¿tbl:tab:headroom?). The same GRPO recipe, run over a small learning-rate sweep at two seeds, climbs on this set: five of six configurations rise and the strongest gains a credible 22 points (Table ¿tbl:tab:pcconfig?). Reading the late-layer rank on three checkpoints from this family shows rank health accompanying the gain and rank collapse accompanying the loss (Table ¿tbl:tab:rankhealth?), the constructive complement to the degrade lesion of the mastered grid.

6pt

5pt

6pt

15 Figure Gallery↩︎

This section collects the supporting figures that do not appear in the main text, grouped by the role they play in the argument. All success rates are clean greedy decoding on the 11-task text track (11 tasks by 5 seeds, 55 episodes per run) unless a caption states otherwise, and all intervals are Wilson \(95\%\) or task-clustered paired bootstrap intervals as noted.

15.1 Controlled null↩︎

15.2 Robustness↩︎

15.3 Mechanism↩︎

15.4 Generalization↩︎

16 Algorithms and Additional Definitions↩︎

This section states the three procedures the main text refers to by name, and recaps two estimators used in the evaluation. Algorithm 20 is the update we sweep; the only knobs the grid varies are the learning rate \(\eta_t\), the KL weight \(\beta\), the clip pair \((\epsilon_{\mathrm{lo}},\epsilon_{\mathrm{hi}})\), the seed, and the initialization of \(\theta\). Algorithm 21 is the interval behind every “credible” verdict, and Algorithm 22 is the causal test behind the localization claim.

Two estimators complete the evaluation. The headroom criterion that selects the positive-control tasks compares the greedy success to the mean success under temperature sampling of the supervised policy, \[\Delta_{\mathrm{head}} = \tfrac{1}{K}\textstyle\sum_{k} \mathbf{1}[\operatorname{success}(\tau_k)] - \mathbf{1}[\operatorname{success}(\tau_{\mathrm{greedy}})],\] averaged over evaluation seeds, and a task is climbable when \(\Delta_{\mathrm{head}}>0\). The random-component null for grafting draws a set \(\mathcal{R}\) of non-attention parameters with \(|\mathcal{R}|\) equal to the attention parameter count, restores \(\mathcal{R}\) from initialization, and repeats the draw \(24\) times to form the null distribution whose mean and 95th percentile Table ¿tbl:tab:graft? reports; a real graft is a specific cause only when its recovery exceeds that percentile.

References↩︎