1 Introduction↩︎

A central result of mechanistic interpretability is that Transformer feed-forward networks (FFNs) behave like key-value memories: they store associations, enrich residual states with factual information, and expose editable factual content [1]–[5]. This view now has engineering consequences. Recent architectures make memory more explicit, sparse, or modular: Memory Layers add trainable key-value lookup modules [6]; DeepSeek Engram introduces conditional memory lookup as a separate sparsity axis [7]; MemoryLLM decouples FFN memory from self-attention and residual flow [8]; STEM replaces part of FFN computation with embedding modules [9]; and PEER-style routing retrieves from many tiny experts [10]. The operational premise is that FFN memory can be factored out, sparsified, or replaced without losing the relevant capability.

The architectural complication, illustrated in Figure 1, is that a native FFN is not a side table. The point is not that FFNs are the only Transformer modules that operate on the residual stream; attention and normalization also participate in residual computation. The point is narrower and more consequential for the memory view: the memory-like FFN computation is written inline into the same state that later retrieval computation consumes.

A Transformer layer writes \[h_{l+1}=u_l+\Delta F_l,\qquad u_l := h_l+\mathrm{Attn}_l(h_l),\] into a residual stream that every subsequent layer reads. In long-context retrieval, attention first aggregates contextual evidence into the target-token residual state; the FFN then applies the only token-wise nonlinear transformation between consecutive attention layers and writes \(\Delta F_l\) back into the same stream; later attention layers and the output head consume that transformed state. This raises two questions: Q1, does an FFN write affect cross-layer retrieval information flow? Q2, if it does, in which direction, and is that direction task-dependent?

We test this view with native FFN scaling (Figure [fig:concept95write]): \[h_{l+1}\leftarrow u_l+\alpha\,\Delta F_l,\qquad \alpha\in\{0,0.25,\ldots,2\},\] one layer at a time. We use this intervention because it isolates the causal role of the native residual write. It does not edit weights, inject an external steering vector, or replace the attention pattern; it preserves the learned content and direction of \(\Delta F_l\) while varying only its gain at a single layer. The identity point \(\alpha{=}1\) is the unmodified model, \(\alpha{<}1\) asks whether attenuating or removing the same write helps retrieval, and \(\alpha{>}1\) asks whether stronger expression of the same write amplifies its effect. The resulting dose-response therefore distinguishes two accounts. If the FFN acted only as passive memory, reducing its write should mainly remove useful stored information. The signed-write hypothesis predicts a stronger and more specific pattern: the response should be signed. Some layers should help retrieval when skipped, some should hurt when skipped, and the same layer can change role when the retrieval mode changes.

The forward intervention answers whether FFN writes have signed causal effects. The next question is whether the pattern can be explained from architecture-internal signals. Around the native point \(\alpha{=}1\), the retrieval margin changes along the model’s own FFN write direction. At layer \(l\), the forward FFN write \(\Delta F_l\) and the backward retrieval gradient \(\nabla M\) meet at the same residual node (Figure [fig:concept95diagnostic]). Their inner product \[s_l=\Delta F_l^{\!\top}\nabla M\] is the local directional derivative of retrieval margin along the native write. We use this quantity as a mechanism anchor, not as a substitute for the causal intervention: the scaling experiment defines the role, and the directional derivative tests whether the role follows the write direction.

1.0.0.1 Contributions.

1. Signed FFN role taxonomy. Native FFN scaling reveals calibrated, amplifying, and suppressive roles in long-context retrieval. These roles are layer-specific and task-conditioned: the final layer is a suppressor in 7/8 controlled model–suite cases, and 60% of layers switch role between literal and semantic retrieval.

2. Directional mechanism and controls. The local directional derivative anchors the two monotone roles: suppressor writes point against the retrieval objective in 34/35 cases, while amplifier writes point with it in 18/18 cases. Seed stability, threshold sweeps, and residual-size controls show that the taxonomy is not a seed artifact, threshold artifact, or norm-only effect.

3. Naturalistic transfer and diagnostic implication. A safety-filtered LongBench retrieval-QA probe transfers the same diagnostic to natural QA, and a targeted attenuation sanity check shows that suppressor layers are causally useful control points. This gives a concrete diagnostic for FFN replacement or decoupling studies in retrieval settings: evaluate the write direction, not only lookup accuracy.

2 Related Work↩︎

2.1 FFN as memory and its engineering consequences↩︎

[1], [2] establish the FFN-as-memory view by interpreting FFN sublayers as key-value memories. Editing work shows that factual associations are localizable at FFN weights [3], [4], and [5] trace factual enrichment to early MLP sublayers with final extraction mediated by attention. This line explains what FFNs store and how stored associations can be localized or edited.

Recent architecture work turns this memory view into a design axis. Memory Layers add explicit trainable lookup modules [6]; DeepSeek Engram adds conditional memory lookup as a separate sparsity axis [7]; MemoryLLM decouples FFN memory from self-attention and residual flow [8]; STEM replaces part of FFN computation with embedding modules [9]; and PEER-style routing retrieves from many tiny experts [10]. These works decouple memory architecturally. We diagnose a complementary property in conventional Transformers: the memory-like FFN computation is written inline into the residual stream. The decoupling literature addresses what the module retrieves; we ask where and in which direction the retrieved or transformed vector pushes the current retrieval objective. §6 discusses the narrower design consequence for retrieval settings: FFN replacement studies should check whether the replacement preserves retrieval-relevant write direction, not only lookup behavior.

2.2 Layer-wise FFN importance↩︎

[11] show that FFNs can be redundant and replaceable by one widened FFN; [12] rank blocks by redundancy; and [13] show that reallocating FFN capacity to middle layers can improve downstream tasks. These works report FFN importance as a magnitude, redundancy, or relative-benefit score. Our role taxonomy is complementary: it measures a signed, task-dependent per-layer response. A large FFN write is not necessarily helpful, and a small write is not necessarily irrelevant; the direction of the native write relative to the retrieval objective determines the two monotone roles.

2.3 Long-context retrieval↩︎

Long-context benchmarks show that retrieval is behaviorally subtle [14]–[18]. Most relevant is [19], which shows that literal and semantic retrieval are distinct capabilities and attributes the gap primarily to attention. Our experiments localize part of that divergence to the FFN write direction: the same FFN layer can switch its entire causal role when the retrieval mode changes. LongBench is used here not as a leaderboard benchmark but as a natural-language retrieval-QA probe for testing whether the signed surface and directional-derivative diagnostics transfer beyond controlled synthetic retrieval.

2.4 Activation steering↩︎

[20]–[23] steer behavior by injecting residual vectors; [24] report an inverted-U in steering coefficient. We instead scale the model’s native FFN residual write and read the response as a signed role taxonomy. The contribution is not that residual interventions can change behavior; it is that native FFN writes form a task-conditioned signed structure, and that the monotone roles are anchored by the local directional derivative of retrieval margin along the native write.

3 Native FFN Writes and Local Direction↩︎

Figure 1 shows where the native FFN write enters the residual stream before intervention. Figure [fig:concept95write] summarizes the forward residual-write intervention. Figure [fig:concept95diagnostic] summarizes the endpoint-response and directional-derivative diagnostics used to classify and explain the signed roles.

3.1 Architectural fact: native FFN writes are consumed downstream↩︎

A pre-norm Transformer layer evolves the residual stream as \[h_{l+1}=h_l+\mathrm{Attn}_l+\mathrm{FFN}_l=u_l+\Delta F_l .\] In Llama-, Qwen-, and SmolLM2-style sequential models, \(\mathrm{FFN}_l\) reads \(u_l\); in Pythia- and GPT-NeoX-style parallel models, it reads \(h_l\) (§7). In retrieval, attention aggregates contextual evidence into the target-token residual state. The FFN then applies a token-wise nonlinear transformation and writes \(\Delta F_l\) back into the same residual stream. Later attention layers and the output head consume this transformed state (Figure [fig:concept95write]). The residual connection itself is not a new observation. What we test is whether the direction of \(\Delta F_l\) is signed, task-conditioned, and predictable from the local directional derivative.

3.2 Intervention design: native FFN scaling↩︎

We scale the native write \[h_{l+1}\leftarrow u_l+\alpha\Delta F_l,\qquad \alpha\in\{0,0.25,\ldots,2\},\] one layer at a time (Figure [fig:concept95write]). We report \(D_l(\alpha)=M_{\mathrm{base}}-M_l(\alpha)\), so \(D_l(\alpha)>0\) means the intervention hurt retrieval and \(D_l(\alpha)<0\) means it helped. We use endpoints rather than AUC or curve slope because the role taxonomy is meant to summarize two explicit causal probes: removing the native write (\(\alpha=0\)) and over-amplifying it (\(\alpha=2\)). The \(|D_l(\alpha)|<0.1\) weak band is a pre-specified margin tolerance for near-zero endpoint responses; §5.5 and Appendix 9 report the corresponding threshold sweep. A layer’s endpoint role is defined by the signs at \(\alpha=0\) and \(\alpha=2\):

• calibrated: skip hurts and amplify hurts; the pretrained gain is locally calibrated.

• amplifier: skip hurts and amplify helps; more FFN write improves retrieval.

• suppressor: skip helps and amplify hurts; more FFN write damages retrieval.

• weak/mixed: either endpoint lies inside the \(|D_l(\alpha)|<0.1\) weak band or the endpoint pattern is not one of the three roles above.

Native scaling is not intended to preserve residual norm; it is a causal dose intervention on the model’s own FFN write. Norm-preserving directional controls are used separately to test whether the signed effect is reducible to update magnitude (§5.5, Appendix 13). This intervention tests two predictions. P1: native FFN writes produce signed, layer-structured response surfaces rather than uniform degradation. P2: the same layer’s role can switch between literal and semantic retrieval because the task changes what counts as useful evidence at the residual state.

3.3 Local directional derivative and backward mechanism↩︎

The forward intervention identifies the response surface; the directional criterion explains the two monotone roles. Let \[M_l(\alpha)=M(u_l+\alpha\Delta F_l)\] denote the retrieval margin after scaling the native FFN write at layer \(l\). Around the unmodified model, \(\alpha{=}1\), a first-order expansion gives \[M_l(\alpha)-M_l(1)\approx(\alpha-1)\Delta F_l^{\!\top}\nabla M .\] Thus \[D_l(\alpha)=M_l(1)-M_l(\alpha)\approx (1-\alpha)s_l,\qquad s_l:=\Delta F_l^{\!\top}\nabla M .\] We interpret \(s_l\) as the local directional derivative of retrieval margin along the native FFN write. The first-order formula itself is standard; the empirical claim is that this simple local quantity remains informative across layers, models, retrieval modes, non-infinitesimal attenuation gains, and LongBench QA transfer, while magnitude-only controls do not explain the same role structure. This is a diagnostic, not a second causal intervention: the causal object is still the native write \(\Delta F_l\) under scaling. If \(s_l>0\), the write locally points toward higher retrieval quality and should behave as an amplifier under attenuation; if \(s_l<0\), the write points against retrieval and should behave as a suppressor. Calibrated layers are dose-shape roles and are not claimed to be sign-determined.

This local interpretation has a limited scope. It most directly constrains attenuation-side signs and ranks near the native point; it need not exactly fit damage amplitude across all gains, task protocols, or depths. P3: the two monotone roles are predicted by the sign of \(s_l\) (suppressor\(\Rightarrow s_l<0\), amplifier\(\Rightarrow s_l>0\)), and the same quantity gives the zero-parameter local prediction \(D_l^{\mathrm{pred}}(\alpha)=(1-\alpha)s_l\) for attenuation damage.

4 Experimental Setup↩︎

4.0.0.1 Metric: retrieval damage \(D_l(\alpha)\).

Throughout, our dependent variable is the retrieval margin of the gold answer: the mean logit margin (gold logit minus the best non-gold logit) over the retrieval task’s answer tokens. The intervention effect is reported as \(D_l(\alpha):=M_{\mathrm{base}}-M_l(\alpha)\) (raw_damage_margin), so \(D_l(\alpha)>0\) means the intervention hurt retrieval (lowered the gold margin), and \(D_l(\alpha)<0\) means it helped. Identity (\(\alpha{=}1\)) gives \(D_l(1){=}0\) by construction. Per-condition \(D_l(\alpha)\) is averaged over (i) three random seeds for the three main-protocol models and the available SmolLM2 cross-protocol validation run, and (ii) three difficulty levels within each suite: \(L_1\) (single key-value), \(L_2\) (needle at varying positions), \(L_3\) (conflicting distractors). Bootstrap confidence intervals (over seeds for the main-protocol end-layer endpoints; over layers for the role-switch rate) are reported in §5.5.

4.0.0.2 Models.

Three main-protocol models span \(\sim\)​1–9B and both residual-composition families (Table 2): Pythia-1B [25] (parallel), Qwen2.5-7B [26] (sequential), and Qwen3.5-9B-Base (sequential). Qwen3.5-9B-Base denotes a Qwen-family sequential 32-layer checkpoint used as an additional mechanism-validation checkpoint in our runs; we report it separately from the standard public Qwen2.5 release and do not use it as a public benchmark reference. Results are reported per model rather than merged into a leaderboard-style score. SmolLM2-1.7B [27] provides external architecture and data-protocol validation. All four have complete literal and semantic suites. Backward-gradient diagnostics (bf16 full-alpha) are available for the three main-protocol models and for sampled SmolLM2 anchor layers; all four are subjects of §5.3.

4.0.0.3 Cross-protocol validation design.

The model set separates estimation from stress testing. Pythia-1B, Qwen2.5-7B, and Qwen3.5-9B form the unified, three-seed main-protocol set used for seed aggregation, leave-one-seed-out stability, and the primary Qwen response-surface figures. SmolLM2 changes both model family and data protocol relative to those three models, providing a cross-protocol stress test. Its literal suite matches the final-layer suppressor pattern; its semantic suite yields a calibrated role, showing how the same taxonomy records protocol-sensitive role changes. We report SmolLM2 side by side for the taxonomy, task-conditioning, block-depth pattern, and sampled backward-anchor role/sign relation, while keeping seed-LOO summaries restricted to the three main-protocol models.

4.0.0.4 Retrieval suites.

Literal: verbatim needle, question shares vocabulary (NIAH lineage [14]). Semantic: needle and question have minimal lexical overlap, forcing associative retrieval (NoLiMa-style [19]). Both at 8K context, three difficulty levels (\(L_1\)/\(L_2\)/\(L_3\)).

4.0.0.5 Naturalistic LongBench retrieval-QA probe.

To evaluate the signed surface under natural-language long-context QA, we add a safety-filtered LongBench retrieval-QA probe [17] on Qwen2.5-7B and Qwen3.5-9B. We use two candidate-scored LongBench subsets, Qasper and MultiFieldQA-en, with 100 examples each (200 total), seed 13, bf16, and the same native FFN scaling grid \(\alpha\in\{0,0.25,\ldots,2\}\). Qwen2.5 has a complete \(28{\times}9\) layer–gain grid plus baseline (253 summary rows, 50,600 predictions); Qwen3.5 has a complete \(32{\times}9\) grid plus baseline (289 summary rows, 57,800 predictions) with tail-shard merge validation. We keep this probe separate from the three-seed controlled taxonomy: it evaluates transfer of the signed surface and directional-derivative mechanism under a natural QA protocol.

4.0.0.6 LongBench subset selection.

LongBench is used here as natural-language long-context retrieval validation, not as a leaderboard benchmark. We therefore select subsets by mechanism-experiment constraints. A subset must be QA/retrieval-shaped, have answers that can be localized or inferred from the context, support forced-choice logprob scoring with same-subset distractors, and use relatively short answer spans so that the measured margin is not dominated by generation format, output length, or fluency. We also prefer English natural-language contexts, avoid code, summarization, dialogue, classification, and Chinese subsets in this first probe, and avoid strong multi-hop reasoning so that the task primarily probes retrieval/readout rather than complex reasoning. We further apply a conservative subtask-level safety screen and include only QA subtasks with zero strict NSFW hits in the strict audit. Qasper and MultiFieldQA-en satisfy these constraints while providing real long-form natural-language contexts and enough answer-pool diversity for length-matched distractors.

4.0.0.7 Backward diagnostic filtering.

Qwen supplementary backward runs include a short-copy control alongside the semantic tasks. For semantic directional-derivative diagnostics, we average only the semantic tasks (\(S_1\)–\(S_3\)) and exclude that control. This affects the directional-derivative diagnostic but not the endpoint taxonomy, which is computed from the intervention suites.

4.0.0.8 Provenance.

All headline numbers are re-verified from raw per-run summaries or per-sample prediction files. The accompanying project artifacts record the reconstruction scripts, audit notes, LongBench merge checks, and robustness tables used to produce the paper figures and tables.

4.0.0.9 Sign convention.

Throughout, positive \(D_l(\alpha)\) indicates retrieval damage and negative \(D_l(\alpha)\) indicates retrieval improvement.

5 Results↩︎

5.1 Q1: Native FFN scaling reveals signed causal response surfaces↩︎

Figure 2 shows the full-alpha response surface for the two Qwen models: every layer is probed across gains, and the color records signed retrieval damage (\(D_l(\alpha){>}0\) hurts, \(D_l(\alpha){<}0\) helps). The important fact is visible before any summary statistic: FFN response is not a scalar importance curve. Adjacent layers can have opposite signs under the same gain, and Qwen2.5-7B and Qwen3.5-9B have visibly different surfaces. Qwen2.5 literal retrieval contains many calibrated layers; Qwen3.5 literal retrieval has a mixed surface with suppressor, amplifier, and weak/mixed regions. A generic perturbation-sensitivity account would predict degradation under gain changes; the heatmaps instead show structured, signed variation. The final-layer suppressor cases are also matched by negative \(s_l\) in the directional-derivative diagnostic, a property that generic fragility does not predict.

The endpoint taxonomy in Table 3 extends the same conclusion across models and retrieval modes. In the three main-protocol models, all six final FFN layers are suppressors: skipping them helps retrieval, while amplifying them hurts. SmolLM2 then acts as the intended cross-protocol validation: its literal suite matches the final-suppressor pattern, while its semantic suite is calibrated. This is the point of including an external protocol. The result separates the robust taxonomy claim from the final-layer specialization: signed roles persist under protocol shift, and the semantic final layer moves into a calibrated role.

5.2 Q2: Signed FFN roles are task-conditioned↩︎

Literal and semantic retrieval differ in the controlled dimension that matters for this paper: lexical overlap. If an FFN layer were simply a static store for the target fact, its role should not systematically change when the retrieval mode changes. It does. Across the 100 layers with both-suite endpoint data, 60 switch role between literal and semantic retrieval (60%, 95% CI [50%, 69%] by layer bootstrap). At the sharper sign level, using attenuation at \(\alpha{=}0.25\) with \(|D_l(\alpha)|{>}0.1\) in both suites, we find 3 opposite-sign layers in Pythia-1B, 4 in Qwen2.5-7B, 6 in Qwen3.5-9B, and 7 in the SmolLM2 cross-protocol run.

These are not small threshold artifacts around zero. In Qwen2.5-7B layer 15, weakening the FFN hurts literal retrieval but helps semantic retrieval; in Qwen3.5-9B layer 29, the same pattern appears with comparable magnitude in both directions. Under the signed-write view, the same layer’s task gradient can rotate, changing the sign of \(s_l=\Delta F_l^\top\nabla M\) and reassigning the layer’s role. We verify that mechanism anchor in §5.3. Semantic retrieval also exposes a propagation/amplitude component, so the role switch analysis combines the local direction signal with the downstream propagation diagnostics summarized in Appendix 10.

5.3 Q3: Local directional derivative explains monotone roles↩︎

The mechanism test is targeted. We compute \(s_l=\Delta F_l^{\!\top}\nabla M\) from backward-gradient diagnostics, then ask whether the sign of \(s_l\) predicts the behavioral endpoint role. The target of this test is the two monotone roles — suppressor and amplifier — because they are single-direction effects. The calibrated role is a different dose shape: both endpoints hurt, so it is modeled as a calibrated-gain role rather than a sign role.

Figure 5 and Table 4 give the main mechanism anchor. Across the three full-backward main-protocol models plus SmolLM2 backward anchor layers, suppressor layers have \(s_l{<}0\) in 34/35 cases (pooled mean \(-0.81\)), and amplifier layers have \(s_l{>}0\) in 18/18 cases (pooled mean \(+0.75\)). The single suppressor exception is Qwen3.5 semantic layer 2 after excluding the short-copy control from the semantic backward diagnostic. The pattern is otherwise model-local: Pythia suppressor 8/8, amplifier 2/2; Qwen2.5 suppressor 8/8, amplifier 7/7; Qwen3.5 suppressor 16/17, amplifier 8/8; SmolLM2 suppressor 2/2, amplifier 1/1. Calibrated layers are balanced around sign (27/61 positive), matching their interpretation as calibrated-gain layers rather than monotone direction layers.

5.3.0.1 First-order prediction.

If \(s_l\) is the signed contribution per unit \(\alpha{-}1\), then \(D_l(\alpha)\approx -s_l\,(\alpha{-}1)\) at small \(|\alpha{-}1|\). We test this zero-parameter predictor on each model separately at attenuation alphas \(\{0.25,0.5,0.75\}\). In literal retrieval, where attenuation is close to a local write-direction regime, the predictor is quantitatively strong: \(R^2{=}0.888\) (Pythia), \(0.601\) (Qwen2.5), and \(0.867\) (Qwen3.5). Semantic retrieval is more revealing when raw \(R^2\) is read together with directional diagnostics (Table 5): Pythia and Qwen2.5 have low raw \(R^2\) but preserve sign and rank signal, while Qwen3.5 already has higher raw \(R^2\). The LongBench retrieval-QA probe (§5.4) recovers raw \(R^2{=}0.796\) (Qwen2.5) and \(0.791\) (Qwen3.5) under natural QA, showing that the controlled-semantic amplitude gap is protocol-specific rather than a failure of the directional mechanism.

5.3.0.2 Semantic retrieval separates direction from propagation.

The zero-parameter predictor tests two things at once: whether \(s_l\) gets the direction/order of damage right, and whether its uncalibrated scale matches the observed damage amplitude. Semantic retrieval often preserves the first while weakening the second. In Qwen2.5 semantic retrieval, raw \(R^2\) is \(0.074\), but Pearson correlation is \(0.770\), sign accuracy is \(0.821\), balanced sign accuracy is \(0.861\), and MCC is \(0.694\). This turns semantic retrieval into a mechanism probe: it changes the task gradient and often flips the same layer’s direction, while the resulting damage magnitude depends more on how that write propagates through later layers and depth structure. The signal-graph diagnostic in Appendix 11 supports this interpretation: it is used as a propagation check, while \(s_l\) remains the sign anchor for the monotone roles.

5.3.0.3 Magnitude vs direction.

Separately, a 33-dim residual-magnitude feature model is useful for predicting damage magnitude, but it does not explain the taxonomy. In three-class leave-one-model-out role prediction (calibrated/suppressor/amplifier), relative norm alone gives balanced accuracy \(0.20\) against \(0.33\) balanced chance; adding depth improves to \(0.32\), and residual norm/cosine/depth features reach \(0.33\). Magnitude carries partial signal, but it is unsigned and cannot tell suppressor from amplifier. The role is set by the direction (\(s_l\)), which magnitude alone cannot recover.

5.3.0.4 Propagation and depth diagnostics.

Block-level attenuation and the forward-only signal graph support the same qualitative interpretation: attenuation damage also depends on downstream propagation and depth structure. These diagnostics are not used to define endpoint roles or the \(s_l\) mechanism, so we keep the details in Appendices 10 and 11.

5.4 Transfer: LongBench retrieval-QA preserves the signed surface↩︎

The controlled literal/semantic suites isolate retrieval from reasoning and domain knowledge. We next ask how the same signed-surface analysis behaves under a more natural QA protocol, and repeat native FFN scaling on the safety-filtered LongBench retrieval-QA probe for the two Qwen models. The grids are complete and directly auditable from predictions: Qwen2.5-7B has 253 conditions and 50,600 prediction rows; Qwen3.5-9B has 289 conditions and 57,800 prediction rows. Rebuilding the summaries from predictions recovers \(D_l(\alpha)\) to floating-point precision (maximum absolute margin-damage differences \(2.2{\times}10^{-16}\) and \(9.7{\times}10^{-17}\)).

Figure 6 shows that the signed surface persists: gain changes do not produce uniform degradation, and both Qwen models contain multiple endpoint types. Under the same \(|D_l(\alpha)|{>}0.1\) endpoint threshold, Qwen2.5 has 3 calibrated layers, 8 suppressors, 0 amplifiers, 5 both-ends-help layers, and 12 weak/mixed layers; Qwen3.5 has 3 calibrated layers, 3 suppressors, 1 amplifier, 9 both-ends-help layers, and 16 weak/mixed layers (Table 6). Thus the naturalistic probe does not merely reproduce the controlled taxonomy counts; it shows the same measurement recovering a signed layer-role surface under a natural QA protocol.

The directional-derivative mechanism also transfers. Using LongBench backward diagnostics and attenuation gains \(\alpha\in\{0.25,0.5,0.75\}\), the zero-parameter predictor \(D_l^{\mathrm{pred}}(\alpha)=(1-\alpha)s_l\) explains damage with raw \(R^2{=}0.796\) for Qwen2.5 and \(0.791\) for Qwen3.5; Pearson correlations are \(0.903\) and \(0.949\), and balanced sign accuracies are \(0.857\) and \(0.802\). The per-subtask breakdown is similarly strong (Appendix Table 9): Qasper and MultiFieldQA-en each have raw \(R^2{>}0.75\) and Pearson correlation above \(0.89\). The weakest sign row is Qwen3.5/Qasper, where sign accuracy is \(0.708\) against a \(0.594\) majority baseline (balanced accuracy \(0.710\)). Unlike the lower-\(R^2\) controlled semantic cases, this probe does not show the same direction-versus-amplitude split for the Qwen models: the local directional derivative is quantitatively predictive of attenuation damage.

Both-help is reported explicitly here because it is frequent under the natural QA protocol: 5/28 Qwen2.5 layers and 9/32 Qwen3.5 layers in the pooled rows. This is a protocol-shift role, not a replacement for the monotone mechanism claim; the suppressor/amplifier analysis continues to center on the two single-direction roles and their \(s_l\) signs.

The final-layer shift is informative. In the controlled retrieval protocol, the final FFN is usually a readout-adjacent suppressor. In the LongBench retrieval-QA probe, Qwen2.5 retains that aggregate pattern (\(D_l(0){=}{-}2.60\), \(D_l(2){=}+0.90\)), while Qwen3.5 moves into a both-ends-help role (\(D_l(0){=}{-}5.47\), \(D_l(2){=}{-}1.25\)): both skipping and doubling the final FFN improve the margin. This supports the paper’s main framing. FFN roles are task-conditioned signed-write roles; final-layer suppression is a strong controlled retrieval phenomenon, and natural QA can move the same architectural position into a different signed role.

Finally, the signed surface supports a held-out policy sanity check, reported in full in Appendix 12. We select top suppressor layers on a calibration split and evaluate only on a disjoint LongBench split. With \(k{=}3\) layers and \(\alpha{=}0.5\), attenuating the selected suppressors improves held-out margin in both Qwen models (mean \(+3.01\)), exceeding random layer sets (\(+0.13\)) and relative-norm-matched controls (\(-0.14\)). This is not proposed as an optimizer; it is a compact check that the taxonomy identifies causally useful layer directions.

5.5 Minimal robustness and confound controls↩︎

The role taxonomy is defined by signs of \(D_l(0)\) and \(D_l(2)\) with a \(|D_l(\alpha)|{<}0.1\) weak band. We therefore audit its three main degrees of freedom: threshold choice, seed variation, and FFN update size (scripts/robustness_checks_20260621.py).

5.5.0.1 Threshold sensitivity.

The exact counts move when the weak band changes, as they should, but the taxonomy does not collapse into one response type. For Qwen2.5-7B literal, the number of calibrated layers is 18 at thresholds 0.05 and 0.1, and remains 13 at 0.25. For Qwen3.5-9B literal, the exact count shifts more (calibrated 8/6/3 at 0.05/0.1/0.25), but suppressor and amplifier classes persist. Thus the threshold affects borderline labels, not the qualitative finding that calibrated, suppressive, amplifying, and weak/mixed roles coexist.

5.5.0.2 Seed robustness (leave-one-seed-out).

For the three multi-seed models we recompute every layer’s role under each of three leave-one-seed-out folds. \(136/152\) layers (\(89\%\)) retain the same role label across all three folds, and all 6 end layers retain the suppressor label across folds. The taxonomy is therefore not a single-seed artifact.

5.5.0.3 Bootstrap confidence intervals.

Seed-bootstrap CIs (\(B{=}2000\)) on end-layer damage values are narrow (Qwen3.5-9B literal: \(D_l(0)=-3.49\,[-3.70,-3.31]\), \(D_l(2)=+3.10\,[+3.02,+3.15]\)); the weakest endpoint is Qwen2.5-7B semantic \(\alpha{=}0\) (\(-0.17\,[-0.30,-0.08]\), close to the weak band). Accordingly, the end-layer claim is a qualitative monotone-suppressor pattern, not equal endpoint strength in every suite. The role-switch rate is \(60\%\) with layer-bootstrap 95% CI \([50\%,69\%]\).

5.5.0.4 Residual-size controls.

Relative FFN norm, depth, and residual geometry do not explain the taxonomy by themselves. Figure 7 gives the minimal control. Relative FFN norm overlaps across role classes (median calibrated \(0.40\), suppressor \(0.33\), amplifier \(0.36\); IQRs overlap substantially). In three-class leave-one-model-out role prediction, relative norm alone gives balanced accuracy \(0.20\) against \(0.33\) balanced chance; adding depth improves to \(0.32\), and residual norm/cosine/depth features reach \(0.33\). Residual size and depth carry partial signal, but they do not determine the signed role. The direction (\(s_l\)), which magnitude cannot recover, is what predicts the monotone roles. Appendix 13 adds a fixed-norm rotation sanity check on Qwen2.5: rotating writes with or against the local retrieval-gradient direction while preserving write and residual norms gives true-gradient win rates of \(0.997\) (literal/main) and \(1.000\) (semantic/supplementary), while random tangent rotations stay near chance (\(0.513\) and \(0.486\)).

6 Discussion↩︎

6.0.0.1 Design consequence.

Memory-decoupling architectures isolate storage from dense computation; Memory Layers, DeepSeek Engram, MemoryLLM, and STEM are examples. Our results do not show that all such designs are at risk. They identify a diagnostic axis for retrieval settings: when an architecture replaces or factors out FFN memory-like computation, lookup accuracy alone may miss whether the replacement write pushes the residual state in the task-relevant direction. The directional-derivative score \(s_l\) provides a zero-parameter post-hoc check: after replacement or decoupling, measure whether the module’s write still aligns with the retrieval gradient. The LongBench policy sanity check makes the diagnostic actionable within our setting: attenuating top suppressor layers selected on a calibration split improves held-out margin by \(+3.01\) on average at \(k{=}3,\alpha{=}0.5\), compared with \(+0.13\) for random layer sets and \(-0.14\) for relative-norm-matched controls.

6.0.0.2 What the end-layer suppressor means.

In the controlled retrieval protocol, the end layer sits immediately before the output head; there is no later attention for it to shape. Its suppressor signature is therefore best read as a readout-adjacent effect: the end-layer FFN write injects or amplifies information that competes with the retrieval decision at output scoring — e.g. prior-token continuations or majority-class priors — so removing it clarifies the retrieved signal and amplifying it drowns it. This is the same signed-write question at the output boundary: non-final layers can change what later attention selects, while the final layer changes what the output head scores.

6.0.0.3 What LongBench changes.

The safety-filtered LongBench retrieval-QA probe makes this separation concrete. It preserves signed FFN response surfaces and strong \(s_l\) predictivity in a more natural QA setting, while changing the readout-adjacent role of Qwen3.5’s final layer to both-ends-help. This separation is useful: the central claim is a signed, task-conditioned native-write taxonomy, and final-layer suppression is one prominent controlled-retrieval role within that taxonomy. Naturalistic QA can change which direction a readout-adjacent FFN write pushes the retrieval margin, while the signed-surface and directional-derivative structure remains visible.

6.0.0.4 Why the role switches between tasks (Q2).

Literal retrieval needs attention to surface a verbatim needle; semantic retrieval needs it to surface an associated concept. The same FFN write can rotate from aiding one to suppressing the other: in Figure 5, several layers have \(s_l\) of opposite sign across suites. This is the mechanism behind the 60% role switch and the core evidence that FFN’s role is task-conditioned rather than a static property of stored knowledge.

6.0.0.5 Claim calibration.

The evidence supports a strong but specific claim: in long-context retrieval, native FFN writes have signed, task-conditioned effects on retrieval margin. Direction is the decisive signal for the two monotone roles; residual size, depth, and propagation features shape attenuation magnitude and explain why the full response surface is richer than a single sign. The mechanism anchor (Q3) is verified across both residual-composition families, with SmolLM2 providing cross-protocol validation of the Pythia/Qwen protocol. The role landscape is depth-structured and mixed in the middle layers, so the paper reports a signed layer-role distribution rather than a depth-only rule.

7 Limitations↩︎

7.0.0.1 Pythia’s parallel residual.

Pythia computes \(h_{l+1}{=}h_l{+}\mathrm{Attn}{+}\mathrm{FFN}\) with \(\mathrm{FFN}\) reading \(h_l\) (not \(u_l\)). The native-write intervention is valid in both residual-composition families, but the “reshapes the next attention’s input” framing is cleanest on the three sequential models. Pythia is included as a cross-family check.

7.0.0.2 Backward scope.

The mechanism anchor (Q3) uses per-layer backward data (bf16 full-alpha) on the three main-protocol models and sampled backward anchor layers on SmolLM2. Qwen semantic backward diagnostics exclude the short-copy control and average only the semantic tasks. The \(s_l\leftrightarrow\)role mapping is therefore checked across both residual-composition families and a range of model scales (1–9B), but the local directional amplitude model is used as a task-local diagnostic rather than a universal cross-model calibration rule.

7.0.0.3 Training dynamics.

We show that trained FFN writes have task-conditioned signed effects on retrieval. The current experiments characterize the trained mechanism; a checkpoint study would test how these directions form during training.

7.0.0.4 Vector-subspace decomposition.

This paper establishes a signed causal surface with native scaling interventions and robustness controls. Decomposing individual FFN writes into finer vector subspaces is a complementary analysis, not a prerequisite for the claims here.

8 Conclusion↩︎

Taken together, the results show that the role of FFNs in LLMs is more than memory. FFNs do not only store or retrieve associations; in a native Transformer, their outputs are written inline into the residual stream that later attention layers and the output head consume. In long-context retrieval, this gives the native FFN write a signed role: it can be calibrated, amplifying, or suppressive depending on layer, task, and direction.

Our theoretical and empirical analyses support this view. Native FFN scaling isolates the model’s own residual write and reveals signed, layer-specific response surfaces. Across the controlled main protocol and SmolLM2 cross-protocol validation, final-layer writes are suppressors in 7 of 8 model–suite pairs, 60% of layers switch role between literal and semantic retrieval (95% CI [50%, 69%]), and the monotone roles are anchored by the local directional derivative: suppressor \(s_l{<}0\) in 34/35 cases and amplifier \(s_l{>}0\) in 18/18 cases. A safety-filtered LongBench retrieval-QA probe preserves the signed surface on two Qwen models and gives strong first-order prediction (\(R^2{=}0.796\) / \(0.791\)), while also showing that final-layer roles can change under naturalistic protocol shift.

For architectures that decouple, sparsify, or replace FFN parameters, the implication is a diagnostic one rather than a universal design rule. In retrieval settings, lookup accuracy should be complemented by a check on the contextual role of the replacement write: where it enters the residual stream, how later attention and the output head consume it, and whether its direction pushes the retrieval state toward or away from the current task objective. The storage view explains localization and editing of facts; the signed-write view adds a directional question to ask when FFN memory-like computation is factored out of the native Transformer.

9 Taxonomy Threshold Sweep↩︎

The main taxonomy uses the pre-specified weak band \(|D_l(\alpha)|<0.1\). Table 7 reports the complete threshold sweep used in the robustness audit. The threshold changes borderline labels, especially for weak/mixed layers, but does not collapse the signed taxonomy into a single response type.

10 Four-Block Attenuation Diagnostic↩︎

The single-layer taxonomy could be read as isolated per-layer effects. As a coarse propagation check, we split each network into four contiguous FFN blocks by depth and scale all FFN writes in one block at a time. Figure 8 shows that the signed depth structure persists at block scale. This diagnostic is not used to define endpoint roles or to replace the local \(s_l\) sign test.

Weakening the earliest block at \(\alpha{=}0.25\) is the most damaging block intervention in all six main (model,suite) pairs: damage \(+2.80\) (Pythia literal), \(+8.68\) (Pythia semantic), \(+4.02\) (Qwen2.5 literal), \(+11.26\) (Qwen2.5 semantic), \(+5.51\) (Qwen3.5 literal), and \(+6.09\) (Qwen3.5 semantic). Weakening the last block has the opposite sign in the same six pairs: \(-0.92\), \(-0.46\), \(-1.54\), \(-2.22\), \(-0.79\), and \(-3.47\). Because a block contains several FFNs (including the final suppressor in the last quarter), this is not a per-layer attribution; it is a coarse confirmation that signed depth structure survives when contiguous depth segments are perturbed together. SmolLM2 supplies the cross-protocol validation: weakening its earliest block at \(\alpha{=}0.25\) hurts literal and semantic retrieval (\(+3.83\) and \(+7.57\)), while weakening its last block helps literal retrieval (\(-1.46\)) and hurts semantic retrieval (\(+1.26\)), again showing signed depth structure rather than one uniform FFN effect.

11 Signal-Graph Propagation Diagnostic↩︎

The signal-graph diagnostic is a forward-only propagation check. It predicts attenuation responses from FFN update magnitude and residual-stream propagation features (norms, ratios, cosines crossed with dose), fit with seed leave-one-out over the three main-protocol models. The source predictions use the delta-margin sign convention, but \(R^2\) is invariant under the global sign convention used for \(D_l(\alpha)\) in the paper.

12 LongBench Policy and Gradient-Path Sanity↩︎

13 Fixed-Norm Directional Rotation Sanity↩︎

The residual-size controls in the main text use scalar diagnostics. As a more direct sanity check, we also run a Qwen2.5 fixed-norm rotation pilot: for selected layers, we rotate the native write in the positive or negative local gradient direction while renormalizing the write and resulting residual state to preserve their original norms. This is not a replacement for native scaling, because it constructs a tangent perturbation rather than varying the model’s own gain. It is a targeted test of whether direction alone can move retrieval margin when norm is held fixed.

The true-gradient direction wins almost every paired comparison, while random tangent directions stay near chance. Because the write and residual norms are effectively fixed, this supports the main claim that the sign of the FFN effect is directional rather than a simple consequence of update size.

13.0.0.1 Directional-derivative path sanity.

We validate the LongBench directional-derivative path with finite differences on five anchor layers per Qwen model. The finite-difference direction and the autograd alpha-gradient direction agree in sign on all sampled anchors (Table 11). The direct dot product and the recorded alpha-gradient are also close, so the sanity check supports the implementation of \(s_l\) rather than adding a separate mechanistic claim.

13.0.0.2 Held-out policy controls.

Table 12 gives the full LongBench held-out policy sanity table. Top suppressor layers are selected from the single-layer margin-damage screen on a calibration split; random and relative-norm-matched controls use the same \(k\) and policy alpha. All values are evaluated on a disjoint 100-example evaluation split. The main text reports the conservative attenuation rows (\(k{=}3,\alpha{=}0.5\)). The ablation rows (\(\alpha{=}0\)) preserve the same ordering but are not used as the main policy claim because they are a stronger intervention.

References↩︎