Review Residuals: Update-Conditioned
Residual Gating for Transformers
June 30, 2026
Abstract
Residual connections add every sublayer’s proposed update with a fixed coefficient of one; the network never evaluates whether an update is reliable before committing it. Drawing on the human-factors principle of independent verification—check a proposed action before taking it—we introduce Review Residuals, which scale each update by a learned, input-dependent gate conditioned on both the current state and the proposed update: \(h_l = h_{l-1} + r_l \odot u_l\) with \(r_l = \sigma(W[\,\mathrm{RMSNorm}(h_{l-1}), \mathrm{RMSNorm}(u_l)\,])\). Conditioning the gate on the update is the property that distinguishes it from prior gated and scaled residuals. We report two findings. First, a depth-stability result: a convex (Highway-style) form of the gate reintroduces vanishing gradients and fails to train beyond \({\sim}20\) layers, whereas the additive, identity-preserving form trains stably at all depths we tested. Second, and more importantly, an emergence-with-scale result: trained from scratch across five sizes (60M–1B parameters, multi-seed), Review Residuals show no advantage at small scale—at 60M a parameter-matched plain standard residual is in fact slightly better—but at 590M they significantly outperform both a parameter-matched Highway gate and a parameter-matched standard residual (\(p<0.05\)), with a larger (\(+0.016\) nats) advantage at 1B. Crucially, the benefit grows with model size rather than fading—the profile one hopes for in a method meant to matter at scale, and the opposite of how most architectural tweaks behave. In absolute terms the advantage is small at the \(\le 1\)B scales we could afford to train, and the 1B result is a strong trend (\(p\approx0.07\)) rather than fully significant—but it grows across every size we tested. That trajectory, not the present magnitude, is the result: a benefit that compounds with scale is precisely what a method built for large models should show. We outline the experiments that would carry it to frontier scale.
1 Introduction↩︎
The transformer’s residual stream is, by design, trusting. Each sublayer proposes an update and the architecture adds it with a coefficient of one, \(h_l = h_{l-1} + f_l(h_{l-1})\) [1], [2]. Nothing evaluates whether that update is reliable. Human-factors engineering and high-reliability organizations [3], [4] take the opposite stance: capable agents fail predictably, and reliability comes not from greater intelligence but from building verification into the work. The point is sharpened by how bounded any single agent’s attention is—famously, observers asked to count basketball passes routinely fail to notice a person in a gorilla suit walking through the scene [5]—and, more broadly, by the thesis that intelligence is itself bounded, so reliable behaviour must be engineered around the agent rather than demanded of it [6]. High-reliability industries respond with an explicit toolkit of human-performance (HP) tools: self-checking (Stop–Think–Act–Review), peer-checking, three-way communication, a questioning attitude, and—most relevant here—independent verification, in which a proposed action is checked before it is committed [7]. We ask whether that one tool can be expressed inside the network, by learning, per update, how much to trust it.
1.0.0.1 Contributions.
Review Residuals: an identity-preserving residual update scaled by a learned gate conditioned on the proposed update, not only the state (Section 2).
A depth-stability finding: the convex gate form reintroduces vanishing gradients and fails at depth; the additive form fixes it (Section 4).
An emergence-with-scale result from a multi-seed sweep (60M–1B): no benefit at small scale, but a statistically significant advantage over both a parameter-matched Highway gate and a parameter-matched standard residual at 590M, growing at 1B (Section 5).
A parameter-matched, multi-seed protocol with an honest account of effect sizes and significance, and fully reproducible data and code (Sections 6–8).
2 The Review Residual primitive↩︎
Writing the proposed update of sublayer \(l\) as \(u_l = f_l(\mathrm{RMSNorm}(h_{l-1}))\), Review Residuals scale it by a learned, input-dependent gate and add it to the (preserved) residual stream: \[h_l = h_{l-1} + r_l \odot u_l, \qquad r_l = \sigma\!\Big( W_r \cdot \big[\,\mathrm{RMSNorm}(h_{l-1}),\; \mathrm{RMSNorm}(u_l)\,\big]\Big), \label{eq:review}\tag{1}\] where \(r_l \in (0,1)^d\) is a per-channel gate. Two choices matter. Identity preservation: the update is added (coefficient one on \(h_{l-1}\)), so the skip path—and the gradient through it—is preserved at every layer; Section 4 shows this is essential at depth. Update-conditioning: the gate inspects \(u_l\). It does not only ask “where am I?” (the state); it asks “is this proposed change worth making?” (the update). Conditioning the gate on the proposed update in this way is novel: to our knowledge, no prior gated or scaled residual reads the update it is modulating. Figure 1 contrasts the three. We initialise \(W_r=0\) so \(r\approx 0.5\) at the start (the gate begins agnostic), and use standard GPT-2 initialisation [1] with residual-projection scaling and a short learning-rate warmup, which are necessary for stable training at the larger depths.
3 Related work and positioning↩︎
Methods that modulate the residual update fall into two groups. Static methods scale the update by a learned scalar or per-channel factor that does not depend on the input: ReZero [8], Fixup [9], DeepNorm [10], and LayerScale [11]. Input-dependent methods compute the modulation from the activations: Highway [12] gates on the state, and Attention Residuals [13] aggregate across depth with softmax attention. Review Residuals are an input-dependent scaling of the update—closest in form to LayerScale, but learned and conditioned on the update itself (Table 1). A parallel line makes the amount of computation conditional (adaptive computation time [14], PonderNet [15], mixture-of-experts [16], mixture-of-depths [17]); Review Residuals instead keep depth and width fixed and learn how much to trust each update.
| Method | Update modulation | Input-dependent? | Conditioned on the update? |
|---|---|---|---|
| Standard residual | coefficient \(=1\) | no | no |
| ReZero / Fixup / DeepNorm | learned scalar | no | no |
| LayerScale | learned per-channel scale | no | no |
| Highway | gate from the state | yes — on the state | no |
| Attention Residuals | softmax over depth | yes — over outputs | n/a |
| Review Residuals (ours) | gate from state and update | yes — on the state | yes — on the update |
4 Depth stability: why the gate must be additive↩︎
An earlier convex form of the gate, \(h_l = (1-r_l)\,h_{l-1} + r_l\,u_l\) (the Highway form), wins at shallow depth but fails to train beyond roughly twenty layers. The reason is structural: at \(r\approx 0.5\) each layer retains only a fraction of the incoming signal, and the Jacobian \(\partial h_l/\partial h_{l-1}\approx (1-r)\) compounds multiplicatively, so the gradient reaching early layers decays as \({\sim}0.5^{\,L}\). At our 590M (\(40\) sublayers) and 1B (\(48\) sublayers) configurations the loss stalled near \(6.0\) and never descended, across multiple seeds and learning rates. This is the vanishing-gradient pathology that residual connections were introduced to remove [1], reintroduced by the convex gate—the same reason additive ResNets historically supplanted Highway networks for very deep models. The additive form of Equation 1 keeps \(\partial h_l/\partial h_{l-1}\approx I\) and trains stably at every depth we tested. Depth-stability of the residual stream is a recognised concern in recent work as well [13], [18], [19]. We therefore adopt the additive form throughout, and treat the convex form as an instructive failure rather than a competing method.
5 Experiments↩︎
5.0.0.1 Setup.
All models are decoder-only transformers (RMSNorm [20], GELU MLPs, tied embeddings) trained from scratch on next-token prediction on TinyStories [21], fp32 with TF32 matmuls, AdamW [22], warmup-then-cosine schedule, identical token budget per size. We compare Review against two baselines, each parameter-matched up to Review by widening (so any Review advantage is not a capacity advantage): the Highway gate and the plain standard residual. We run five sizes with seeds (3 at 60M/150M, 2 at 320M, 3 at 590M/1B). Lower validation loss is better; significance is a two-sample Welch \(t\)-test on the per-seed losses.
| Size | Review | Highway | Standard | Rev \(-\) Hwy | Rev \(-\) Std | significant? |
|---|---|---|---|---|---|---|
| \({\sim}60\)M | 1.6891 | 1.6923 | 1.6805 | \(+0.003\) | \(-0.009\) | no (Std better) |
| \({\sim}150\)M | 1.5576 | 1.5625 | 1.5548 | \(+0.005\) | \(-0.003\) | no |
| \({\sim}320\)M | 1.5001 | 1.5042 | 1.5037 | \(+0.004\) | \(+0.004\) | no |
| \({\sim}590\)M | 1.4795 | 1.4889 | 1.4901 | \(+0.009\) | \(+0.011\) | yes (\(t{=}3.3,\,3.7\)) |
| \({\sim}1\)B | 1.4876 | 1.5031 | 1.5040 | \(+0.016\) | \(+0.016\) | trend (\(t{=}2.3,\,2.4\)) |
5.0.0.2 Result.
Table 2 and Figure 2 tell one story. At small scale, Review Residuals provide no benefit: through 320M the differences from both baselines are within noise, and at 60M a parameter-matched standard residual is slightly better. At 590M the picture changes—Review significantly outperforms both the Highway gate (\(+0.009\), \(t{=}3.3\)) and the plain standard residual (\(+0.011\), \(t{=}3.7\)), clearing \(p<0.05\) at three seeds. At 1B the advantage roughly doubles to \(+0.016\) nats over both baselines, a strong trend (\(t{\approx}2.3\), \(p{\approx}0.07\)). The effect therefore grows with scale rather than vanishing, and—crucially—at scale Review beats the plain standard residual that frontier models actually use, not merely an alternative gate.
6 Limitations↩︎
We state these plainly. (1) Effect size: in absolute terms the advantages are small at the scales we could train (\(\le 1\%\) of loss)—real and, at 590M, significant. The magnitude is not static, however: it grows across the tested range, from negative at 60M to \(+0.016\) nats at 1B, so the caveat is the absolute size at \(\le 1\)B, not the direction—which points upward. (2) Significance: the 590M advantage over both baselines is significant at \(p<0.05\); the larger 1B advantage, with three seeds, sits just below that threshold (\(p\approx0.07\)) and is consistent in direction and magnitude with it. (3) Two large scale points define the emergence claim; a third higher point would strengthen it. (4) Data and regime: a single, simple dataset (TinyStories), undertrained under a fixed token budget, at \(\le 1\)B parameters—two to three orders of magnitude below frontier. (5) No downstream evaluation: validation loss only. We accordingly claim a parameter-controlled, scale-emergent improvement over both gated and standard residuals at \(\le 1\)B—small in absolute magnitude today, growing with scale, and established with multiple seeds—while being explicit that frontier-scale confirmation remains future work.
7 Discussion↩︎
7.0.0.1 A local operation that scales simply.
Review Residuals read only the current layer’s state and its own proposed update; they add no cross-layer or cross-pipeline-stage communication, so unlike depth-aggregation methods they require no special distributed-training infrastructure. The gate adds \({\sim}7\%\) parameters and a small per-layer compute overhead.
7.0.0.2 A design lens.
The result comes from a generative idea: importing human-performance tools—the verification practices of high-reliability work—into the residual update. Review Residuals realise one such tool, independent verification, and it already pays off in measured loss. The rest of the HP toolkit (peer-checking, escalation under uncertainty, conservative defaults, stop-work authority) maps onto further in-network primitives waiting to be built. We read the scale-emergent result as direct evidence for a broader thesis: reliability in a bounded-intelligence system is engineered through explicit checks, not handed down by scale alone [6]. This is self-review—the model checks its own update—not self-improvement.
8 Conclusion and next steps↩︎
Review Residuals scale each residual update by a learned gate conditioned on the update itself. In an identity-preserving (additive) form they train stably at depth where a convex form fails, and their advantage over both a parameter-matched Highway gate and a parameter-matched standard residual emerges with scale: absent at \(\le 320\)M, significant at 590M, larger at 1B. The natural extensions carry the curve further: larger scale points, compute-optimal training on real text, and downstream-task evaluation. Across the range we could test, the margin over the standard default widens with scale—an unusual profile for a residual rule, and precisely the regime where an architectural change has to earn its place. The trend points past our compute budget, and confirming it at frontier scale is the obvious next step. But the result already stands on its own terms: a parameter-controlled, multi-seed improvement over the residual connection that every frontier model uses today.