1 Introduction↩︎

1.0.0.1 Scope.

This paper is a controlled mechanism study—a model organism for edit propagation, not an architecture proposal. The synthetic KG-QA setting and small five-seed design are chosen so that the fact-bearing site is causally localizable and the propagation protocol can be conditioned on direct edit success; we do not claim, and the design does not test, generalization to natural-language pretraining or to coarse-grained memories. Within that scope we adjudicate one question: when two architectures both cross the OOD grokking barrier, what makes a successful single-fact edit propagate through composition?

Dense transformers can learn the in-distribution surface of a compositional task while failing at the compositions that matter. In the synthetic knowledge-graph QA setting of [1], a standard dense transformer memorizes training and in-distribution (ID) facts but remains near chance on held-out out-of-distribution (OOD) 2-hop compositions. Two modifications fix this failure: a looped transformer can recompute with the same backbone across recurrent depth, and an LMC-style model can reread an explicit memory store between iterations. The central question of this paper is what these two successful routes have in common, and where they differ. Throughout, we use grokking for the delayed onset of OOD generalization. [1] treat it as a binary event (a model either does or does not eventually generalize OOD); we adopt that binary view but make the onset measurable by operationalizing grokking as the first training step at which OOD accuracy crosses \(0.1\) (\(\approx25\times\) the chance level), treating \(0.1\) as a conservative above-chance event marker rather than a saturation criterion. We use grokking only as an onset event—whether and when a configuration crosses this threshold; the eventual OOD value enters only to confirm that grokked runs clear chance by a wide margin, and speed claims below refer to time-to-onset.

The common enabling ingredient is repeated shared access. A looped transformer obtains multiple chances to use the same fact-bearing computation by applying tied backbone weights at several iterations; a memory-augmented loop obtains multiple chances to query the same fact-bearing store.

This view distinguishes two levels. At the learning level, loop and memory are unified at the level of crossing the OOD grokking barrier where a non-sharing dense baseline does not (we do not claim identical training dynamics or representations). At the editing level, however, the relevant axis is not loop-versus-memory but addressability: a substrate that exposes an addressable store (LMC, and also a dense backbone with shared memory) lets a localized edit propagate, whereas a substrate that shares only computation does so at best partially. Loop is therefore an intermediate, partial case on the editing axis—it shares a fact-bearing computation but provides no addressable site to edit—rather than a co-equal route.

This distinction defines the paper’s main fork. If edit propagation follows only the repeated-access structure, then editing a reused loop site should behave like editing a reused memory site: loop and memory would be the same all the way down. If, instead, edit propagation depends on explicit addressability and on when edited content is injected, then memory should afford stronger and cleaner propagation than tied looped weights. We adjudicate between shared-access and addressability accounts rather than assuming the memory answer in advance.

Our results support a refined version of the addressability account. The final pattern is not a binary “memory propagates, loop does not.” Rather, edit propagation splits the substrates along whether they expose an addressable site: the two memory-bearing configurations propagate strongly with no significant difference at our sample size, the dense no-memory model does not propagate, and the looped no-memory model is a partial, intermediate case. As a coarse substrate-level reading—not a calibrated seed-level cause—we describe this ordering with two operational notions: the injection point is the layer or iteration at which an edited fact first enters the computation in a causally usable form, and the reuse window is the amount of computation that remains after that point for downstream steps to read it. The three-way ordering (LMC\(\,>\,\)Loop\(\,>\,\)Dense) is consistent with this reading: LMC injects early into an addressable site and leaves a wide reuse window; Loop’s bridge-injection window appears late and the remaining reuse window is narrow; Dense exposes no reliable bridge-injection window. We stress that the injection-window account tracks the coarse substrate ordering only—an interchange “ruler” that supports it saturates within LMC and is anti-correlated with propagation within Loop at the seed level (§5.3)—so we treat it as an interpretive lens consistent with the substrate ordering, not as a seed-level mechanism claim.

1.0.0.2 Contributions.

1. Repeated shared access as an enabling principle. We show that a single-shared-FFN memory control and a partial-sharing loop control each grok, while dense no-memory models remain flat. This supports the view that repeated access to a shared substrate, rather than sparse MoE routing or full recurrence alone, is the common ingredient behind the loop and memory routes.

2. Localization from grokking to editable addressability. In the fine-grained LMC memory, causal patching and interchange interventions show that one dominant memory site stores each fact and is reused in 2-hop reasoning: the single-hop step0 dominant expert equals the 2-hop step1 dominant expert for 199/199 tested atoms, and bridge substitution flips the 2-hop answer in 92–99.5% of cases across four seeds.²

3. Local LMC editing. Editing only the largest activated value row at the knowledge-injection step gives 100% direct success (LMC-internal probe set) and mean 2-hop bridge propagation \(0.989\) on that set; on the cross-substrate shared ID measurement set (Contribution 4), the same edit propagates at \(0.78\)–\(0.92\) across seeds. Random unrelated-fact damage is about \(0.1\%\). Locality is not absolute: co-resident facts can move when they share the same expert, and hidden-state overlap predicts this leakage.

4. A 2\(\times\)​2 edit-propagation adjudication. Using a shared ID measurement set, pre-edit-correct probes, and conditioning on direct edit success, we compare Dense, Loop, Dense+Mem (a dense backbone with the same memory store but no loop), and LMC. The result ties high semantic propagation to memory-bearing cells (LMC \(0.78\)–\(0.92\), Dense+Mem \(0.71\)–\(0.96\)), while Loop remains intermediate (\(0.04\)–\(0.30\)) and Dense remains near zero (\(0.00\)–\(0.03\)).

The rest of the paper follows this arc: we first define the KG-QA setup and architecture grid, then present Study 1 on learning, a localization bridge that establishes editable memory reuse, and Study 2 on edit propagation. We return to related work after the main evidence, where the contrasts are easier to evaluate.

2 Experimental Setup and Protocols↩︎

This section defines the task, the architectural variants, and the intervention protocol used in the edit-propagation adjudication. The central design discipline is that learning and editing are evaluated with different controls: Study 1 asks which repeated-access structures cross the OOD grokking barrier, whereas Study 2 asks whether a successful direct edit propagates through a composition.

2.1 Task: synthetic knowledge-graph QA↩︎

We use the synthetic knowledge-graph QA setting of [1]. The data are generated from a set of atomic facts \((e,r)\mapsto e'\) and composed 2-hop queries of the form \((e_0,r_1,r_2)\mapsto e_2\), where \(e_0 \xrightarrow{r_1} e_1 \xrightarrow{r_2} e_2\) and the intermediate entity \(e_1\) is the bridge. Models are trained on single-hop and in-distribution examples and evaluated on held-out OOD compositions. Throughout the paper, we define grokking as the first step at which OOD accuracy exceeds \(0.1\), and we use only two quantities derived from it: whether a configuration groks (the OOD phase transition occurs at all) and how fast it does so (steps to first cross \(0.1\)). We treat \(0.1\) as a conservative above-chance event marker rather than a saturation criterion: grokked runs clear chance by a wide margin and do not sit near \(0.1\), so the threshold marks a genuine transition rather than a transient bump. The eventual OOD value enters only to confirm this departure from chance; we do not rank substrates by it.

The task is useful for the present question because it separates memorizing atomic or ID facts from systematically composing facts. Dense no-memory transformers can fit training and ID data while remaining near random on held-out OOD 2-hop compositions; repeated shared access through a loop or a memory can cross that barrier.

2.1.0.1 Notation and terminology.

We collect the recurring terms in one place so that later sections can use them without repeated re-introduction. The looped backbone is iterated \(R{=}4\) times; we index these iterations as step0 \(\ldots\) step3 and read memory at step0/1/2 (the final step is memory-free, see §2.2). A hop is a reasoning edge in the query and is not the same as a step: in the 2-hop query \((e_0,r_1,r_2)\mapsto e_2\), step0 computes the bridge \(e_1\), step1 rereads \(e_1\) and performs the second single-hop lookup, and step2 emits \(e_2\). We refer to facts that share the same dominant memory expert as co-resident facts; a fact’s site is the dominant value row at its single-hop read step. An edit is the targeted modification of a single fact at its site. Direct edit success means that the edited single-hop fact’s prediction has flipped to the new target \(e_1'\). Conditioned on direct success, we distinguish strong propagation (the 2-hop prediction equals the answer implied by the new bridge, \((e_0,r_1,r_2)\mapsto e_2'\)) from weak propagation (any change away from the original 2-hop answer); the main text uses strong unless noted. We say a substrate is memory-bearing if it has a shared memory store—LMC and the dense-with-memory control—and memory-free otherwise (Loop and Dense). The adjudication asks whether edit propagation tracks the memory-bearing/memory-free split or the backbone (looped/dense) split.

2.2 Architectural variants↩︎

We organize the main KG-QA architectures as a 2\(\times\)​2 control grid crossing backbone type with shared-memory access:

This grid separates two factors: repeated computation through a looped backbone and repeated access to a shared memory. The edit-propagation comparison uses all four cells: Dense tests dense no-memory editing, Loop tests loop-only editing, Dense+Mem tests memory without loop recurrence, and LMC combines loop and memory.

Model scale is reported as an architectural covariate rather than force-matched, because the editing protocol aligns on the fact-bearing site and on direct edit success rather than on parameter count. Scale differences do not explain the propagation gap: every substrate reaches a successful direct edit (Loop and Dense at 100% direct success on all five seeds), so the differences arise after the edit takes, in whether it propagates, not in editability or in raw capacity.

2.2.0.1 Loop-memory coupling.

LMC consists of a shared transformer block \(f_\theta\) of \(L\) layers together with a shared memory \(\mathrm{Mem}_\phi\) (Figure 2). Given token embeddings \(x^{(0)}\), the model iterates the shared block \(R\) times and queries memory between iterations: \[\begin{align} h^{(r)} &= f_\theta(x^{(r)}), \\ x^{(r+1)} &= \begin{cases} h^{(r)} + \mathrm{Mem}_\phi(\mathrm{LN}(h^{(r)})), & r < R-1, \\ h^{(r)}, & r = R-1 . \end{cases} \end{align}\] The memory is implemented as a sparsely routed expert store: a router selects top-\(k\) expert MLPs and mixes their outputs. The paper’s claims do not depend on sparse routing alone; Study 1 explicitly tests a single shared-FFN memory variant.

2.2.0.2 Memory placement across the variants.

The same parameters \(\theta,\phi\) are reused across all \(R\) iterations; the only per-step variation comes from the evolving latent \(h^{(r)}\) and its memory retrieval, and we train end-to-end with the next-token loss using no step-level supervision and no curriculum. For the dense-with-memory control (Dense+Mem), the backbone is a single-pass dense transformer rather than a loop, but the memory principle is kept identical: one shared memory is reread at each configured access point. In the \(12\)-layer Dense+Mem cell the shared memory is queried after layers \(3\), \(6\), and \(9\), matching the three non-final reads of LMC, while the terminal layer stays memory-free so the final retrieval is processed by later layers before the LM head. Removing the memory from LMC recovers a pure looped transformer (Loop), and removing both recurrence and memory recovers the dense baseline (Dense); together with full LMC (LMC) these form the \(2\times2\) grid above.

2.3 Study 1 protocol: repeated shared access↩︎

Study 1 tests whether OOD grokking tracks repeated access to a shared substrate rather than an architecture label. We use two diagnostics.

2.3.0.1 Single-shared-FFN memory control.

The MoE memory is replaced by one dense shared FFN inserted at the same repeated access points. If this model groks, sparse MoE routing is not necessary for the existence of grokking under the recipe.

2.3.0.2 Partial-sharing loop control.

We train a partially shared looped model with \(4\) unique layers applied \(3\) times (4L\(\times\)​3, still \(12\) effective layers), so its degree of weight tying lies between the dense \(12\)-layer baseline (no sharing) and the full 3L\(\times\)​4 loop (maximal sharing). This tests whether the full recurrence pattern is necessary, or whether repeated shared access at an intermediate degree of sharing also crosses the OOD barrier.

Both diagnostics are reported across five seeds. Auxiliary controls support the same conclusion: dense no-memory stays flat, one memory access is insufficient, and at least two memory accesses can cross the OOD barrier.

2.4 Localization measurements: addressability and iterative single-hop reuse↩︎

The localization step from grokking to editing is the claim that LMC facts are both localized and reused by composition. We use three intervention measurements.

Throughout, we index the model’s repeated memory reads by step: step0 is the first memory read (iteration \(r{=}0\)), step1 the second (\(r{=}1\)), and so on. A hop is a reasoning edge in the query and is not the same as a step: in our 2-hop setting the model computes the bridge entity \(e_1\) at step0, rereads that bridge at step1 to perform the second single-hop lookup, and emits \(e_2\) at step2. Editing “at step0” therefore means writing to the memory site read on the first iteration, where the bridge fact is first stored.

2.4.0.1 Operational single-store localization.

For each atomic fact, we identify the dominant memory expert at the direct single-hop read step. Fine-grained memory yields an operational single-store regime: within the measured memory read, one dominant expert/value site causally controls the measured fact and is reused in 2-hop. We do not claim that no other parameter carries correlated information. The dominant expert carries weight \(0.93\)–\(0.98\) across seeds, and wrong-expert interventions leave recall intact.

2.4.0.2 Same-location reuse.

We compare the expert used by the direct single-hop fact with the expert used when the same fact appears as the second hop / bridge fact in a 2-hop query. The key measurement is whether the single-hop step0 dominant expert equals the 2-hop step1 dominant expert. This is the precondition for interpreting a single memory edit as an edit to the location composition will later read.

2.4.0.3 Interchange connectivity.

We also use bridge substitution: replace the answer-position memory input at the 2-hop step with the corresponding vector from a donor query sharing \(r_2\) but having a different bridge. If the answer flips to the donor answer, the bridge information in the residual/memory input is causally controlling the 2-hop answer. This distinguishes iterative single-hop composition from a direct memory lookup of a whole 2-hop composition.

2.5 Study 2 protocol: edit propagation↩︎

Study 2 uses one common edit-propagation protocol across LMC, Loop, Dense+Mem, and Dense. For each substrate, we first localize the smallest causally effective site for the direct bridge fact, apply a ROME-style value write [2] toward a replacement bridge \(e_1'\), and sweep the edit budget until the direct atomic answer changes. The exact site definitions and sparse updates are in Appendix 11.

The edit comparison is restricted to the \(2\times2\) grid because that grid is what isolates the two editing-relevant factors—backbone recurrence and shared-memory access. The Study 1 auxiliary controls (the single-shared-FFN memory and the partial-sharing loop) vary only the internal implementation of an already-present factor (memory granularity and degree of weight tying); they were designed to test the necessity of sparse routing and full recurrence for learning, and add no new addressability axis for editing. In particular, the single-shared-FFN control is, for editing purposes, on the same memory-bearing side as LMC and Dense+Mem, so we do not treat it as a separate edit-propagation condition.

Propagation is measured only on compositions the model answered correctly before the edit and only after direct edit success. A strong propagation event requires the edited 2-hop answer to equal the answer implied by the new bridge, \[\hat{y}_{\text{2-hop after edit}} = \mathrm{fact}(e_1', r_2),\] whereas a weak event records any change away from the original answer. The main text uses the strong metric; weak propagation is reported as a secondary diagnostic in Appendix 14.

To interpret timing, we use an interchange diagnostic that swaps bridge information while holding the final-answer direction fixed. A flip under this answer-subspace-preserving interchange is consistent with usable bridge information being present, not with direct answer copying. The formal definition and calibration are in Appendix 12.1.

3 Study 1: Repeated Shared Access Enables OOD Grokking↩︎

Study 1 shows that OOD grokking follows repeated shared access rather than a specific implementation.

The controls separate repeated access from implementation details. The shared-FFN memory control removes sparse MoE routing but keeps repeated memory reads, and it groks on every seed. The partial-sharing loop control reduces the degree of weight tying, and it also groks. The dense-backbone memory-only cell (Dense+Mem, a dense backbone with the same memory store but no loop recurrence) also groks (60k–85k steps), confirming that repeated memory reads cross the barrier even without loop recurrence; this is the same cell used as the memory-only control in the Study 2 edit comparison. Dense no-memory remains flat. These results support the repeated-shared-access account; per-seed grok steps are in Table [tab:app-perseed-grok].

Memory also changes the quantitative regime. The fine-grained \(N=128\) LMC run reaches the grok onset (OOD\(>0.1\)) in 45k–65k steps, about 4.4\(\times\) fewer steps-to-onset than the loop baseline—a statement about how quickly the OOD transition begins.³ Routing granularity is not required for crossing the barrier, but it sharpens atom-to-site addressability for the editing study; all editing results below are at this fine granularity, and we return to the resulting granularity confound as a scope limit in Section 7.

4 Mechanism Bridge: LMC Reuses a Dominant Causal Memory Site↩︎

The transition from Study 1 to Study 2 is the localization of atomic facts in the memory and their reuse during composition. Three measurements support this bridge.⁴

4.0.0.1 Dominant causal site.

At the direct single-hop step, each tested fact is dominated by one expert site. Across four seeds, the step0 dominant routing weight lies in the 0.93–0.98 range; other experts behave like near-tie routing alternatives. Causal patching confirms that this is not merely correlational: suppressing the detected primary expert collapses atomic recall, while suppressing wrong or random experts leaves recall intact.

4.0.0.2 Same-location reuse.

For 199/199 tested atoms, the expert used by the direct single-hop fact at step0 is the same expert used when that fact is read as the bridge at step1 in a 2-hop query. Thus composition rereads the same stored location rather than consulting a separate 2-hop copy.

4.0.0.3 Iterative single-hop connectivity.

Bridge-substitution interchange interventions flip the 2-hop answer to the donor answer in 92–99.5% of cases across four seeds, with any-change rate 100% (i.e.every intervention changes the answer away from the original, even when it does not land exactly on the donor answer). The mechanism is iterative single-hop composition: step0 computes a bridge; step1 memory reads the bridge and performs an ordinary single-hop lookup; step2 outputs the answer. This is precisely the setting in which editing the single stored bridge fact should propagate if the memory site is indeed reused.

5 Study 2: Addressable Memory Affords Edit Propagation↩︎

Study 2 asks whether a successful direct edit propagates through the composition that normally rereads the edited fact. We first give the cross-substrate result, because it is the paper’s main adjudication, and then show that the LMC memory edit used in that comparison is precise and local.

5.1 Cross-substrate propagation ladder↩︎

All four cells are edited under the common protocol from §2.5: rates are measured on a shared pre-edit-correct ID set and conditioned on direct atomic edit success.

5.1.0.1 Edit-site fairness across substrates.

The four configurations are not edited at the same physical weights, because they do not have the same fact-bearing site: LMC writes to a single value row of the dominant memory expert (the site identified in §4), whereas Dense, Loop, and Dense+Mem write to value columns of the corresponding MLP c_proj matrix—the standard targeted-edit site for dense transformer MLPs in the ROME/MEMIT line. We use each architecture’s most natural edit site rather than imposing a single physical site, because the comparison of interest is whether after a successful localized edit the substrate’s downstream computation can reread it, not whether one shared physical recipe fits all backbones. Two facts limit how much this could distort the comparison. First, every substrate reaches \(100\%\) direct edit success on every seed (Loop and Dense; LMC’s conditioned direct success stays in \(184\)–\(229\) of \(213\)–\(247\) probes; the LMC seed with the lowest direct count is s3 at \(177/219\) on the shared cross-substrate set, see Table 10), so the chosen sites are causally effective at flipping the single-hop fact. Second, the propagation gap separates configurations along the memory axis (LMC and Dense+Mem both propagate strongly; Loop and Dense do not), including the dense-backbone memory cell that uses an MLP-style memory and is edited at its expert value row—so the gap is not explained by “loop+expert-row vs. dense+c_proj” alone. We return to the residual concern (whether a coarser-grained memory would still afford propagation) in §7.

Crucially, the low-propagation substrates are not an artifact of failed edits: Loop and Dense reach 100% direct edit success on every seed, and conditioned LMC direct success stays high (\(177\)–\(229\) of \(213\)–\(247\) probes; per-seed values in Table 10). The separation therefore reflects whether a successful direct edit propagates, not whether the edit takes. Table 2 shows the strong-propagation ranges. The two memory-bearing configurations propagate strongly (LMC \(0.778\)–\(0.919\), mean \(0.863 \pm 0.053\); Dense+Mem \(0.713\)–\(0.955\), mean \(0.810 \pm 0.116\)), Loop remains intermediate (\(0.039\)–\(0.297\), mean \(0.129 \pm 0.098\)), and Dense stays near zero (\(0.000\)–\(0.033\), mean \(0.014 \pm 0.015\)). The ranges do not overlap between memory-bearing and non-memory cells: even Loop’s highest seed (\(0.297\)) stays below LMC’s lowest (\(0.778\)), so the ladder is not driven by a single outlier. The separation is statistically clean despite the small sample: each memory-bearing cell beats Loop and Dense with maximal rank separation (exact Mann–Whitney \(U{=}25/25\), two-sided \(p{=}0.008\), every memory seed above every non-memory seed), while the two memory-bearing cells are statistically indistinguishable from each other (LMC vs Dense+Mem, \(U{=}16/25\), \(p{=}0.55\); bootstrap difference of means \(0.05\), \(95\%\) CI \([-0.05, 0.15]\), straddling zero). The data thus support a two-level addressability split, not a four-way ranking.⁵

This ladder rules out a flat shared-access account in which LMC and Loop both propagate strongly, and refines the simpler addressability-only claim that only memory matters: Loop is a real intermediate case. Dense weak changes mostly reflect nonspecific disruption of memorized ID answers rather than propagation to the edited bridge (Appendix 14); a matched held-out unrelated-fact specificity column for Loop and Dense is a direct follow-up that we flag in Appendix 14 but do not run here. Full per-seed values and the 2\(\times\)​2 table are in Appendix 13.

5.2 LMC one-row edits are precise↩︎

The LMC result is not obtained by a broad destructive write. At the step0 knowledge-injection site for the target atomic fact, we edit only the single largest activated value row (\(m{=}1\)). This edit changes the direct fact with 100% success and propagates to bridge 2-hop queries with mean propagation \(0.989\).

\(^{\dagger}\) Evaluated on the LMC-internal probe set; the cross-substrate strong-propagation comparison uses a different shared ID set (Table [tbl:tab:app-sot-prop]

).

Additional edit-budget sweeps clarify what the \(m{=}1\) edit actually buys. Direct success (\(1.000\)) and intended bridge propagation (\(\approx0.99\)) are already saturated at \(m{=}1\) and do not improve as \(m\) grows to the full \(320\) rows; what changes is collateral movement of co-resident facts—other facts routed to the same dominant expert as the edited fact—which rises roughly an order of magnitude (from \(0.06\)–\(0.30\) at \(m{=}1\) to \(0.78\)–\(0.92\) at \(m{=}320\); Appendix 15.1). The \(m{=}1\) edit is therefore not required for the fact to write or propagate; its value is a nearly free order-of-magnitude reduction in collateral, i.e.locality bought at essentially no cost in direct success or intended propagation. The locality result has one caveat: these co-resident facts, sharing the same expert, can move. Leakage increases with hidden-state cosine, from about 5% in the \([0.0,0.2)\) bin to 21% in \([0.2,0.4)\) and 50% in \([0.4,0.6)\), while random unrelated facts are nearly unchanged.

5.3 Injection-window analysis (coarse substrate-level diagnostic, not a seed-level predictor)↩︎

Propagation rates separate the substrates, and the interchange ruler is consistent with the coarse cross-substrate ordering. We stress the relative status of the two measurements up front: the five-seed propagation comparison (§5, conditioned on direct edit success and therefore stable on every seed) is our primary evidence, whereas the HOLDANS interchange below is an auxiliary mechanism interpretation that we use only at the coarse three-way (LMC\(>\)Loop\(>\)Dense) level. The two are not co-equal: they even point in opposite directions within Loop at the seed level (below), so we never lean on HOLDANS to rank seeds or to carry a claim that propagation does not already establish. We use a diagnostic that swaps bridge information while holding the final-answer direction fixed. A flip is therefore consistent with usable bridge information being present, not with direct copying of an answer-like residual. The intervention is defined in Appendix 12.1, and full per-seed values are reported in Table 7.

The timing separation holds at the substrate level:

early/wide (LMC) \(\rightarrow\) late/narrow (Loop) \(\rightarrow\) no reliable window (Dense).

We read this as a coarse three-way ordering, not a fine-grained per-seed predictor. We refer to the flip rate under this answer-subspace-preserving interchange as the HOLDANS score (HOLD-ANswer-Subspace), and report it per seed in Table 7. The ruler does not track within-substrate variation: LMC HOLDANS saturates at \(1.000\) on all five seeds and so cannot follow LMC’s own \(0.778\)–\(0.919\) propagation spread, and within Loop the seed-level HOLDANS signal is actually decoupled from propagation—the highest-HOLDANS seed (s0, \(0.683\)) has only middling propagation (\(0.112\)), while the near-zero-HOLDANS seed (s2, \(0.075\)) propagates most (\(0.297\)). We therefore treat injection timing and downstream reuse window as consistent with the LMC\(>\)Loop\(>\)Dense ordering rather than as a calibrated seed-level cause; the discriminating factor at the substrate level is whether edited content is injected early enough, and at an addressable site, for later computation to reuse it.

6 Related Work and Positioning↩︎

6.0.0.1 Looped and recurrent-depth transformers.

A growing family of models obtains additional computation by iterating a shared set of layers in the depth direction (“vertical recurrence”) rather than by adding parameters. The Universal Transformer [3] first proposed weight-tied recurrence; Huginn [4] scales recurrent depth to \(3.5\)B parameters with a \(\textit{prelude}\)–\(\textit{recurrent}\)–\(\textit{coda}\) structure and randomly sampled iteration counts, while Ouro [5] loops its entire layer stack (\(24\) layers for the \(1.4\)B model, \(48\) for \(2.6\)B) \(R{=}4\) times and argues that recurrence improves knowledge manipulation rather than capacity. On the theoretical side, looped transformers have been shown to length-generalize on algorithmic tasks by emulating iterative computation [6], and [7] show that looped models carry an inductive bias toward reasoning—matching far deeper iso-FLOP baselines on reasoning primitives despite worse perplexity than comparable-size dense models, a gap rooted in the long-known capacity cost of cross-layer parameter sharing [8]–[11]. LMC sits at the extreme shallow end of this spectrum: only \(3\) unique layers, looped \(R{=}4\), with no prelude/coda. Unlike these models, LMC adds an explicit external memory queried between iterations, and we use the looped-vs-dense contrast as a controlled variable rather than as the headline result.

6.0.0.2 Grokking of implicit reasoning.

[12] first reported grokking—delayed generalization long after the training loss saturates—on small algorithmic tasks, where generalization is essentially binary: validation accuracy eventually climbs from chance to near-perfect. [13] and [14] analyze the transition mechanistically, explaining it as a slow, efficient generalizing circuit overtaking a fast memorizing one. [1] carry grokking into knowledge-graph reasoning: a vanilla transformer learns the in-distribution task but never generalizes OOD to held-out two-hop compositions, and their analysis predicts that cross-layer memory-sharing—via either memory augmentation or explicit recurrence—is needed for OOD systematicity. That prediction is not itself a controlled separation; concurrent work [15] confirms the loop-only half on the same task. Our contribution is orthogonal: building on Wang’s task design and recipe, we separate two implementations of repeated shared access—loop recomputation and memory rereading—and then ask whether they remain equivalent under edit-propagation interventions.

6.0.0.3 Memory-augmented models.

Separating a controller from an external memory is a long-standing goal: Neural Turing Machines and Differentiable Neural Computers couple controllers to addressable memories [16], [17], and Memory Networks retrieve supporting facts for question answering [18], [19]. Modern transformer memories take several forms—segment-level memory tokens [20], non-parametric caches [21], trainable key–value stores [22], [23], and auxiliary neural memories [24], [25]—but most attach storage to a fixed, non-recurrent backbone. LMC differs by coupling: it co-trains a shallow loop and its memory from scratch, interleaving a memory retrieval into every iteration rather than doing long-context retrieval. This coupling matters: [26] show that bolt-on \(k\)NN-LM memory helps memory-intensive tasks yet hurts multi-hop reasoning even with perfect retrieval, so that our in-loop memory instead accelerating the reasoning grokking transition is a non-trivial consequence of how it is coupled, not of adding a store per se.

6.0.0.4 Model editing and edit propagation.

ROME and related causal-tracing work localize factual associations in dense transformers and edit them with rank-one updates [2]; MEND and MEMIT scale this direction to faster or batched factual edits [27], [28]. Multi-hop editing benchmarks such as MQuAKE show that a successful direct edit often fails to propagate through downstream compositions [29]. Our goal is a controlled model organism for edit propagation: the fact-bearing site is causally localized, the normal computation either rereads it or does not, and propagation is measured after conditioning on direct edit success.

6.0.0.5 Concurrent loop-plus-memory studies.

Closest to us are recent works pairing recurrence with memory. [15], on the same knowledge-graph task as ours, show that a looped transformer groks where a vanilla one fails—but study only the loop route (varying the recurrence count \(R\)), with no external memory and no loop-vs-memory separation; their looped comparison also lets effective depth grow with \(R\) (\(4\!\to\!32\)), entangling the recurrence mechanism with added depth, whereas we hold effective depth fixed. [30] augment an adaptively looped language model with gated local/global memory banks, and [31] add learned memory tokens to a single-block Universal Transformer on Sudoku; both separate knowledge manipulation (looping) from knowledge capacity (memory), whereas we separate the two routes to cross-layer sharing themselves (loop-recompute vs.memory-reread) under a shared task and intervention protocol. Our memory is also a sparse mixture-of-experts knowledge store consulted between iterations—holding retrievable knowledge, not the transient computation state of a scratchpad/register [31] or per-layer gated key–value bank [30].⁶

7 Discussion and Limitations↩︎

7.0.0.1 Decisive evidence from the dense-backbone memory cell.

The decisive evidence is the dense-backbone memory cell (Dense+Mem): it has no loop recurrence at all, yet it propagates edits as strongly as LMC (\(0.713\)–\(0.955\) vs \(0.778\)–\(0.919\); statistically indistinguishable at our sample size, Mann–Whitney \(U{=}16/25\), \(p{=}0.55\) (failure to reject under \(n{=}5\) vs \(n{=}5\) is underpowered: the bootstrap difference is \(0.05\) with \(95\%\) CI \([-0.05, 0.15]\), so we cannot rule out a moderate gap of order \(0.15\)), while both cleanly separate from the non-memory cells at \(p{=}0.008\)). The propagating factor is therefore an editable site that an edit can be written to and that later computation rereads, not loop recomputation per se. Loop is an intermediate, partial case rather than a co-equal route: it shares computation but exposes no addressable edit site, so the interchange ruler identifies a real but late bridge window and edits sometimes propagate, broadly in line with the injection-window reading. We are careful not to overstate this ruler. It tracks only the coarse three-way ordering (LMC\(>\)Loop\(>\)Dense); it does not predict propagation at the seed level—LMC’s flip rate saturates and Loop’s seed-level flip rate is in fact decoupled from, even anti-correlated with, its propagation (Appendix 12). The injection-timing account should therefore be read as a substrate-level interpretation consistent with the ordering, not as a calibrated per-seed cause.

7.0.0.2 Measurement and scope.

Because Dense does not answer OOD compositions correctly, OOD propagation would conflate edit propagation with basic compositional failure; we therefore use a shared pre-edit-correct ID set for the edit-propagation comparison while retaining OOD as the Study 1 learning metric. More broadly, our experiments are deliberately controlled and do not by themselves establish that the same affordance appears in pretrained large language models or retrieval-augmented systems. They establish a mechanism precondition: if a model stores facts in an addressable in-loop memory that the normal forward computation rereads, a localized edit can propagate through composition in a way that dense and looped weight edits do not match.

7.0.0.3 Locality and future tests.

The LMC value-row edit is highly local for random unrelated facts, while co-resident facts can move in proportion to hidden-state similarity—a superposition view of locality in which unrelated facts stay stable, intended bridge updates propagate, and nearby co-resident facts reveal the remaining interference structure.

7.0.0.4 Falsifiable \(N\)-hop predictions (future work).

The injection-window account suggests falsifiable predictions for longer compositions that we leave to future work. If propagation depends on the downstream reuse window, fixed-depth Loop should decay faster with hop length than LMC, and adding loop iterations after the injection point should recover some propagation. We do not run a 3-hop point in this paper: the present scope deliberately stops at 2-hop, where the bridge site and the reread step are both causally identified, and a clean \(N\)-hop replication would require redesigning the data-generation protocol, re-establishing grokking at the new hop length, and re-localizing the bridge site—a separate study rather than a single supplementary point.

7.0.0.5 Granularity confound, and what an \(N\) sweep tells us.

Our editing claim is demonstrated at one memory granularity (\(N{=}128\) experts, \({\approx}312\) atoms/expert), which we deliberately chose because fine-grained routing sharpens atom-to-site addressability. This raises a fair confound: part of LMC’s editing advantage may be injected by this design knob rather than by “memory” in general, so that “memory affords editing” risks collapsing into “we tuned memory to be finely addressable.” We are explicit that the comparison is not granularity-controlled. The mechanism analysis already suggests why granularity should matter: at coarse routing (\(N{=}13\), \({\approx}3077\) atoms/expert) a single atomic fact cannot occupy its own expert slot and spills onto roughly two neighbors, whereas at \(N{=}128\) a fact collapses to essentially one expert (effective fan-out \({\approx}1.04\), \(95\%\) single-expert; Appendix 15 and the single-store analysis). A localized edit is only as clean as the site it can target, so we expect the editing advantage to weaken as \(N\) shrinks and the target site smears over co-resident facts—the same superposition mechanism that already governs our co-resident leakage result.

To check this empirically we re-ran the full three-substrate edit-propagation protocol with the LMC configuration replaced by its \(N{=}13\) counterpart (D11; same backbone, same training recipe, same five seeds), holding Loop and Dense fixed. The result is consistent with the prediction. The LMC editing advantage weakens but does not collapse: direct edit success drops from \(1.00\) at \(N{=}128\) to \(0.84{\pm}0.15\) at \(N{=}13\), and strong propagation drops from \({\approx}0.99\) to \(0.78{\pm}0.09\), while Loop (\(0.14{\pm}0.10\)) and Dense (\(0.02{\pm}0.02\)) at \(N{=}13\) are statistically indistinguishable from their \(N{=}128\) values. So even at coarse \(N\) the LMC strong-propagation rate is roughly \(5{\times}\) the looped no-memory model and \(50{\times}\) the dense no-memory model. The claim this paper supports is therefore the addressability claim, in a form refined rather than retracted by the sweep: an addressable, in-loop memory affords edit propagation that a dense or loop-only substrate does not, and that advantage scales with how finely the store is addressed—larger at \(N{=}128\), attenuated but still substantial at \(N{=}13\). The narrow form, “a fine-grained, addressable memory affords edit propagation,” is the form supported with highest precision; the broader form, “addressable memory affords edit propagation,” is supported in direction at \(N{=}13\) but with a measured granularity cost.

8 Conclusion↩︎

Looped transformers and memory-augmented looped transformers both solve an OOD compositional grokking problem that dense no-memory transformers fail. The common enabling ingredient is repeated shared access: recomputing with a tied backbone and rereading a shared store are two ways to reuse a fact-bearing substrate across a computation. But intervention reveals a difference. Fine-grained LMC memory localizes atomic facts to addressable sites, rereads the same site during iterative single-hop composition, and supports one-row edits that propagate through 2-hop reasoning with low unrelated collateral. In the edit grid, both memory-bearing cells propagate strongly (Dense+Mem and LMC), Loop remains intermediate, and Dense remains near zero. Loop and memory are thus the same at the level of crossing the OOD grokking barrier; for editing, what matters is addressability—whether the substrate exposes an addressable store that an edit can target and later computation rereads, with the precision of edit propagation scaling with how finely that store is addressed—with loop a partial, intermediate case under the value-column edit protocol natural to it rather than a co-equal route.

9 Model, configuration, and training details↩︎

This appendix starts with the information a reviewer needs to reproduce the controlled comparisons: the primary architecture grid, the auxiliary controls, and the shared optimizer/data recipe.

10 Full Study 1 per-seed grokking results↩︎

Table [tab:app-perseed-grok] gives the complete five-seed results behind the Study 1 summary in the main text. The entries are intentionally reported as first-crossing steps rather than only averages, so the reader can check both the binary grokking outcome and the across-seed spread for every shared-access control.

11 Site localization and edit protocol↩︎

This appendix specifies how facts are localized before editing, then gives the exact sparse edit update used in Study 2.

11.1 Localization measurements↩︎

For each atomic fact, we identify the dominant memory expert at the direct single-hop read step. The same-location reuse test compares this expert with the expert used when the same fact appears as the bridge at the 2-hop step. The interchange-connectivity test replaces the answer-position memory input at the 2-hop step with the corresponding memory input from another query that shares \(r_2\) but has a different bridge.

11.2 Edit formula↩︎

For a direct bridge fact \((e,r_1)\mapsto e_1\), the edit target is a replacement bridge \(e_1'\) such that \((e_1',r_2)\) is a known fact for each measured 2-hop probe. The write direction is \[d = \frac{u(e_1') - u(e_1)}{\|u(e_1') - u(e_1)\|},\] where \(u(\cdot)\) is the output-embedding row for the answer token.

11.2.0.1 LMC memory write.

For LMC, we identify the step0 dominant memory expert \(a\) for the direct single-hop query and capture the selected-expert hidden key vector \(h\in\mathbb{R}^{H}\) at the answer position. The expert’s output is the key-weighted sum of its value rows, \(\mathrm{out}=\sum_{j} h_j\, W_2^{(a)}[j,:]\), so we apply a sparse masked rank-one update restricted to the \(m\) largest-magnitude rows. Let \(S=\operatorname{top}\text{-}m_j |h_j|\) be the edited row set. The update is \[W_2^{(a)}[j,:] \leftarrow W_2^{(a)}[j,:] + \alpha\,\frac{h_j}{\|h_S\|}\, d^\top \quad (j\in S),\qquad \text{unchanged for } j\notin S,\] where \(h_S=(h_j)_{j\in S}\) and \(\alpha\) is chosen by the direct-success sweep. This realizes \(\Delta\mathrm{out}=\alpha\, d\) exactly when the edit is unmasked (\(S\) spans all rows); the paper uses the sparsest setting \(m{=}1\), editing only the single largest activated value row (so the update reduces to \(\alpha\,\mathrm{sign}(h_{j^\star})\,d^\top\) at the peak row \(j^\star\)). This is an edit to the same memory site later reread by the 2-hop computation; \(m{=}H\) recovers the full rank-one ROME write.

11.2.0.2 Loop and Dense value-column write.

For Loop and Dense, the comparable value sites are columns of mlp.c_proj. Candidate columns are ranked by activation magnitude at the localized edit site. For selected columns \(S\), the sparse write uses the same answer direction \(d\): \[\Delta W_{\ell,:,j} = \frac{\alpha}{\sum_{i\in S} h_i^2}\, h_{\ell,j}\, d \qquad (j\in S).\] The edit-propagation run uses the first budget that reaches direct success; the default sparse write uses top-\(k{=}4\) columns for Loop/Dense and sweeps scale.

11.3 Direct-success gate and measurement↩︎

The protocol has three gates:

1. choose \(e_1'\) only when every measured \((e_1',r_2)\) fact exists;

2. keep only compositions the model answers correctly before the edit;

3. score strong propagation only after the direct single-hop prediction changes to \(e_1'\).

Strong propagation then requires the edited 2-hop answer to equal \(\mathrm{fact}(e_1',r_2)\), not merely to change away from the original answer.

12 Answer-subspace-preserving interchange calibration↩︎

12.1 Definition↩︎

This answer-subspace-preserving interchange is used as the injection-window ruler. The intervention compares two residual substitutions for a recipient 2-hop query and a donor 2-hop query that share the second relation \(r_2\) but have different bridges and answers. We define two modes:

FULL replacement injects the whole donor residual. This can be tautological late in the computation because the donor residual may already point directly toward the donor answer. The HOLDANS interchange removes this confound. For recipient residual \(h_r\), donor residual \(h_d\), recipient answer direction \(u_r\), and donor answer direction \(u_d\), let \(U=\operatorname{span}\{u_r,u_d\}\) and \(P_U\) be the orthogonal projection onto \(U\). The injected vector is \[h_{\mathrm{HOLDANS}} = P_U h_r + (I-P_U) h_d .\] Thus the recipient answer-subspace component is held fixed while the donor’s non-answer residual content is transferred. The measured event is a flip to the donor answer under this answer-preserving interchange. Such a flip indicates that non-answer bridge information was available early enough for downstream computation to recompute the donor answer, rather than being copied directly through the answer direction.

12.1.0.1 The ruler is a coarse, not a seed-level, predictor.

The HOLDANS flip rate orders the three substrates in the same direction as propagation (LMC\(\approx\)​1 \(>\) Loop\(\approx\)​0.26 \(>\) Dense\(\approx\)​0.035), but it does not explain within-substrate variation. LMC saturates at \(1.000\) on every seed and therefore cannot follow LMC’s own \(0.778\)–\(0.919\) propagation spread. Within Loop the seed-level ruler is in fact decoupled from, even anti-correlated with, propagation: the highest-HOLDANS seed (s0, \(0.683\)) propagates only \(0.112\), while the near-zero-HOLDANS seed (s2, \(0.075\)) propagates the most (\(0.297\)). We therefore use the ruler only as evidence for the coarse three-way ordering and the early/late/none reading of injection timing, and we do not claim it as a calibrated seed-level cause of edit propagation.

13 Full Study 2 propagation tables↩︎

This appendix collects the full Study 2 edit-propagation evidence: the substrate-by-window summary, the Dense+Mem dense memory-only control, and the full-precision per-seed values that the main text rounds and summarizes in Table 2.

14 Weak propagation explanation↩︎

The main text reports strong propagation: the edited 2-hop answer must equal the answer implied by the new bridge. Weak propagation, defined as any change away from the original answer, is secondary.

We note a current asymmetry in how edit specificity is reported. For LMC we measure damage to held-out random unrelated facts directly (\(\approx0.1\%\) moved, Appendix 15.2), but we do not yet report the matched unrelated-fact damage for Loop and Dense. The weak-propagation column above is only a partial proxy: it is measured on the edited query (any change from the original answer), so the high Dense weak rate (\(0.27\)–\(0.33\) on s2–s4) shows that the Dense edit perturbs its own target non-specifically without reaching the correct bridge, but it does not quantify collateral movement of other, unrelated facts. A clean comparison requires re-running the existing edit protocol on each substrate against a shared held-out unrelated-fact set and reporting one specificity number per cell (e.g.\(1-\text{unrelated-moved}\), matched to the \(m{=}1\) LMC locality result). We flag this missing column as a direct, low-cost follow-up.

15 Sparse edit locality and co-resident leakage↩︎

15.1 Edit-budget sweep↩︎

The sparse value-row edit used in the main text writes only \(m{=}1\) row at the dominant expert. To check that this \(m{=}1\) choice is not driving the precision result, Table 12 sweeps \(m \in \{1,4,22,320\}\) at the same step0 site. Direct success and intended bridge propagation are already saturated at \(m{=}1\) (both \({\approx}1.0\)) and do not improve as \(m\) grows. The single quantity that grows with \(m\) is collateral movement of co-resident facts (other facts routed to the same dominant expert), which rises from \(0.06\)–\(0.30\) at \(m{=}1\) to \(0.78\)–\(0.92\) at \(m{=}320\). The \(m{=}1\) choice therefore buys locality at essentially no cost in direct success or intended propagation.

15.2 Co-resident leakage↩︎

The LMC memory edit is local with respect to random unrelated facts, but not absolutely isolated among facts that share an expert. The pooled leakage diagnostic bins co-resident facts by hidden-state cosine with the edited fact:

This supports the superposition interpretation used in the discussion: unrelated facts are stable, but nearby co-resident facts can leak in proportion to representation overlap.

References↩︎

---

1. Correspondence: niuyanan@kuaishou.com.↩︎

2. The per-atom routing and interchange diagnostics were originally run on s0–s3 and not extended to s4 because by that point the cross-substrate edit-propagation comparison (Contribution 4, all five seeds) had already become the load-bearing evidence and the interchange diagnostic was only used for coarse substrate-level interpretation; per-seed propagation results in Section 5 include all five seeds.↩︎

3. Per-step compute differs across cells (Loop reuses 3 unique layers \(R{=}4\) times; LMC additionally queries a 128-expert memory store; Dense uses 12 unique layers), so this is a step-count comparison, not an iso-FLOP efficiency claim.↩︎

4. The localization measurements in this section are run on the four LMC seeds (s0–s3) for which the full per-atom routing and interchange diagnostics were computed; the five-seed convention elsewhere in the paper refers to the learning (Study 1) and edit-propagation (Study 2) experiments, which use seeds s0–s4.↩︎

5. Tests are over the five per-seed strong-propagation values per cell (Table 10). We use the exact (enumerated) two-sided Mann–Whitney \(U\) test; for \(n{=}5\) vs \(n{=}5\), complete rank separation gives \(U{=}25/25\) at the minimum attainable two-sided \(p{=}0.0079\). Bootstrap CIs resample seeds with replacement (\(2\times10^{5}\) draws).↩︎

6. [31] find memory and ponder depth substitutable, while [30] find them complementary; our controls show that either route alone can trigger the transition, but the degree of substitutability depends on the scale and allocation being compared.↩︎