1 Introduction↩︎

In-context learning (ICL) — the ability of a trained sequence model to solve a new task from a handful of labeled examples in its prompt, without any parameter update [7] — has been mechanistically traced, across architecture families, to an implicit gradient-descent algorithm embedded in the trained forward pass. On the synthetic function-class protocol of [8], [1] showed that trained Transformers internally recover the iterates of a gradient-based learner; [2] made this equivalence explicit (linear self-attention \(\equiv\) one GD step on linear regression; stacking \(\equiv\) trajectory); [9] sharpened it to preconditioned GD. The reading has since generalized well beyond attention: state-space models [4], Mamba [3], [10], MLPs and MLP-Mixers [5], and linear Transformers [11], [12] all exhibit the same implicit-GD signature. ICL is now understood as a generic property of trained sequence models, not an attention-specific artifact.

One substrate has remained conspicuously outside this picture: spiking neural networks (SNNs), the substrate closest to biological computation and the native target of neuromorphic hardware [13]. To our knowledge, no prior work has demonstrated general-purpose ICL in an SNN on the Garg protocol; our own survey of canonical SNN architectures confirms a wide gap (Spikformer [14]: \(R^2 = 0.72\) at \(d{=}20\); Pure LIF: \(R^2 \approx 0.09\); LSNN [15]: \(R^2 \approx 0.01\); Spiking SSMs at the chance floor). Is this incidental — a matter of insufficient parameters, training compute, or architectural search — or structural?

We argue it is structural, and that the structure responsible has been misidentified. Implicit-GD requires, somewhere in the forward pass, a state vector with dimensionality comparable to the task dimension that is updated by a gradient-like rule. A standard leaky-integrate-and-fire neuron carries a scalar membrane potential, reset after each spike — no persistent, multi-dimensional subthreshold state, no candidate substrate for the algorithm. Existing biology-inspired approaches sidestep this by routing adaptation through inference-time synaptic plasticity [15]–[17] and treating the dendritic compartment as a passive carrier of driving signals: apical as teacher [18], as backpropagated error [19], as attentional gate [20], as external task-ID input [21]. This routing is unnecessary. The subthreshold dynamics of a single dendritic compartment alone implement a complete online learning algorithm: with all synaptic weights frozen at inference, the compartment is the substrate of the algorithm, not a conduit for it.

The architectural ingredient is anatomically standard. From the perspective of single-neuron computation, a cortical layer-5 pyramidal neuron carries, in its apical dendrite (the upper input branch integrating top-down signals), a persistent multi-dimensional subthreshold voltage with calcium plateaus on 100+ ms timescales [20], [22]–[25] — precisely the missing substrate. We instantiate this in DendriCL: a single-layer compartmental spiking network whose apical recurrence \[\boldsymbol{u}_A(t+1) = \alpha\, \boldsymbol{u}_A(t) + \gamma\, (y_t - \hat{y}_t)\, W_A\, \boldsymbol{x}_t\] is structurally identical to leaky online Widrow–Hoff LMS [6]. Once \((\alpha, \gamma, W_A, W_B)\) are trained end-to-end by BPTT, the apical membrane provably tracks the task parameter \(\boldsymbol{w}\), and every synaptic weight is frozen at inference.

Contributions. We position the apical compartment as an active online estimator with frozen synapses, in contrast to the plasticity-driven interpretations of prior compartmental models [18]–[21]. On this basis we introduce DendriCL, the first single-layer compartmental SNN to solve general-purpose Garg-2022 ICL across \(d \in \{5, \ldots, 50\}\) where canonical SNN baselines (Spikformer, Pure LIF, LSNN, Spike-driven V2) collapse at \(d \geq 20\), and the only architecture that is seed-stable at \(d \geq 30\) where dense Transformers exhibit grokking-style bimodality and fail past \(d{=}40\). A linear probe recovers the reference online-LMS trajectory directly from the apical membrane at \(\boldsymbol{R^2 = 0.93}\), giving direct mechanistic verification of the apical-LMS equivalence. The single-layer compartmental design yields a \(\sim\!4\times\) spike reduction over Pure LIF and a projected \(\sim\!10\times\) Loihi-class energy advantage; to our knowledge, this is the first ICL setting in which architectural simplicity and inference-time efficiency co-vary rather than trade off.

2 Background and Related Work↩︎

2.0.0.1 ICL mechanisms across substrates.

The mechanistic study of ICL begins with [8], who isolated the phenomenon in a synthetic function-class setting. On this benchmark, [1] used linear probes to recover iterates of least-squares and gradient-descent solvers from trained-Transformer hidden states. [2] proved that a linear self-attention layer can implement one step of GD on a regression loss, with stacked layers yielding the trajectory; [9] extended this to preconditioned GD. [11] earlier showed that linear Transformers are equivalent to fast-weight programmers, a connection generalized by [12]. Beyond attention, the implicit-GD reading has been demonstrated for state-space models [4], Mamba [3], [10], and MLPs/MLP-Mixers [5]. Our work positions a single compartmental spiking layer as the spiking-substrate analog of this lineage.

2.0.0.2 Biologically-plausible learning and compartmental neurons.

A largely non-overlapping line studies how biological circuits could plausibly implement learning algorithms. [18] cast the apical compartment as a teacher signal driving synaptic plasticity toward the somatic firing rate. [19] extended this to a three-compartment microcircuit in which the apical receives a backpropagated error from local interneurons. [16] introduced differentiable plasticity — Hebbian fast weights adapting at inference. [15] added a scalar adaptive threshold to LIF (LSNN) for short-term memory; [21] proposed Active Dendrites with externally-provided task-context gates; [17] (IP\(^2\)-RSNN) combined intrinsic time-constant plasticity with second-order meta-learning. A structural commonality unites these proposals: adaptation is driven by parameter changes at inference time, never by the subthreshold dynamics of a frozen-weight network. None has been evaluated on Garg-style ICL.

2.0.0.3 Dynamics-based test-time adaptation.

Closer in spirit are architectures that adapt via continuous-time dynamics with frozen weights. The Liquid State Machine [26] projects inputs through a random recurrent spiking reservoir and trains only a linear readout; DendriCL retains the frozen-weight philosophy but replaces the random reservoir with a structured apical compartment whose dynamics admit an explicit algorithmic interpretation (linear probe \(R^2 = 0.93\)). Echo State Networks [27] and FORCE learning [28] share the trained-readout/frozen-recurrence design but were not evaluated on Garg-style ICL; Liquid Time-Constant [29] and CfC networks [30] use input-gated time constants for sequential modeling but are neither spiking nor compartmental. Denève’s spiking Bayesian filters [31] address scalar estimation rather than multi-dimensional ICL.

2.0.0.4 Scope: SNN methods excluded by the frozen-weight criterion.

A principled ICL comparison requires every parameter to remain frozen at test time, so any adaptation visible in the forward pass is mechanistic rather than ordinary training. This excludes inference-time-plasticity SNNs — e-prop [32], differentiable plasticity [16], IP\(^2\)-RSNN [17], STDP-based methods, and hardware-local-rule learning on Loihi [13] and SpiNNaker — which solve the related but distinct problem of how to update parameters cheaply at inference. We view them as complementary to DendriCL.

3 Method↩︎

3.1 Architecture↩︎

DendriCL is a single compartmental spiking layer of \(d_{\text{model}}=384\) parallel pyramidal-like units. Each unit has a basal dendrite (feedforward projection of the input), an apical dendrite carrying a persistent multi-dimensional subthreshold voltage \(\boldsymbol{u}_A \in \mathbb{R}^{d_{\text{apical}}}\), and an LIF soma that integrates both compartments and emits spikes. Inputs are \(k+1\) tokens, each encoding \([\boldsymbol{x}_i; y_i; \text{flag}_i]\) with \(\text{flag}_i \in \{0,1\}\) marking the query position. The full per-unit recurrence at context position \(t\) is: \[\begin{align} \boldsymbol{u}_B(t) &= W_B\, \boldsymbol{x}_t & &\text{(basal projection)} \nonumber \\ \hat{y}_t &= \boldsymbol{u}_A(t)^\top (W_A \boldsymbol{x}_t) & &\text{(scalar prediction)} \nonumber \\ e_t &= (1 - \text{flag}_t)(y_t - \hat{y}_t) & &\text{(gated error)} \nonumber \\ \boldsymbol{u}_A(t+1) &= \alpha\, \boldsymbol{u}_A(t) + \gamma\, e_t\, W_A\, \boldsymbol{x}_t & &\text{(apical online-LMS)} \\ v_{\text{soma}}(t) &= g_B\, \boldsymbol{u}_B(t) + g_A\, W_{A,\text{out}}\, \boldsymbol{u}_A(t) & &\text{(somatic integration)} \nonumber \\ s(t) &= \mathbb{1}[v_{\text{soma}}(t) > \theta],\;v_{\text{soma}} \leftarrow v_{\text{soma}} - \theta\, s(t) & &\text{(LIF spike, soft reset)} \nonumber \end{align}\] The apical state \(\boldsymbol{u}_A\) is initialized to zero and not reset by the somatic spike; it evolves continuously across the full context. The error \(e_t\) is gated off at the query position. A small post-LIF block (\(\text{FF}_1 \to \text{LIF} \to \text{FF}_2\)) and a linear readout active only at the query position produce \(\hat{y}_q\). All trainable parameters — \(\alpha, \gamma, W_A, W_B, W_{A,\text{out}}, g_A, g_B, \theta\) plus FF weights — are learned end-to-end by BPTT; no parameter is updated at inference. DendriCL has \(d_{\text{model}}=d_{\text{apical}}=384\) and \(\approx\!0.75\)M total parameters.

3.2 Training↩︎

All parameters — \(\alpha, \gamma, W_A, W_B, W_{A,\text{out}}, g_A, g_B, \theta\), plus FF weights — are trained from scratch via backpropagation through time (BPTT). Each minibatch samples a fresh task parameter \(\boldsymbol{w} \sim \mathcal{N}(0, I_d)\), a fresh context, and a fresh query. Loss is computed only at the query position. LIF surrogate gradients use the arctan approximation [33]. Optimizer: AdamW, learning rate \(10^{-3}\), weight decay \(10^{-4}\), cosine schedule, batch size 64, 10k steps (baseline) or 50k steps (compute-matched); 3 seeds per configuration (5 seeds for \(d \in \{20, 30\}\)). No pre-training, no plasticity at inference.

3.3 Core Mechanism: Apical \(\equiv\) Leaky Online LMS↩︎

Setting \(W_A = I\) and absorbing the output projection into the readout, the apical update reduces to classical leaky Widrow-Hoff LMS: \[\hat{\boldsymbol{w}}_{t+1} = \alpha \hat{\boldsymbol{w}}_t + \gamma (y_t - \hat{\boldsymbol{w}}_t^\top \boldsymbol{x}_t) \boldsymbol{x}_t.\]

This is the classical LMS convergence theorem [6], [34]. Our contribution is structural: the update is embedded in the compartmental architecture, and BPTT tunes \((\alpha, \gamma, W_A)\) to make the built-in rule optimal across the task distribution. The linear-probe analysis (§5.4) empirically confirms the correspondence. Our result is a spiking-substrate analog of the dynamics-as-algorithm theorems of [2], [4], [11]. Rigor caveat: the analysis covers the apical compartment (purely linear); the LIF reset introduces a nonlinearity our theorem does not formally handle. Empirical probe analysis (\(R^2=0.93\)) establishes that the correspondence holds despite this nonlinearity.

3.3.0.1 Biological grounding and distinction from prior compartmental models.

DendriCL’s three-compartment apical-basal-soma layout is anatomically grounded in cortical layer-5 pyramidal neurons [20], [22], [24], [25]; the assignment of the apical compartment to error-driven dynamics is consistent with predictive coding [35], [36]; and the specific online-LMS mechanism is a computational hypothesis falsifiable by future in vivo recording. DendriCL is also structurally distinct from six prior compartmental models [15]–[19], [21] along the dimensions of adaptation substrate and inference-time plasticity. We defer the three-tier biological evidence breakdown and the full compartmental-model comparison table (Table 4) to Appendix 9.

4 Experimental Setup↩︎

Tasks. Garg 2022 linear regression at \(d \in \{5, 10, 15, 20, 25, 30, 40, 50\}\) with \(k = 2d\); binary classification (\(d=10, k=20\)); 2-layer ReLU NN regression (\(d=20, k=40\)).

Architectures. We evaluate 15 architectures spanning attention-based (Transformer, Spikformer, Spike-driven V2, Linear Transformer), recurrent (GRU, LSNN, LTC), state-space (SpikingSSM), spiking (Pure LIF), MLP-family (Pure MLP, MLP-Mixer), and compartmental (Active Dendrites, DendriCNN, DendriCL with \(L \in \{1, 2\}\)) at matched 0.6–1.3M parameter budgets.

Training budget. We report at two regimes: 10k steps (baseline) and 50k steps (5\(\times\) compute-matched). The 50k regime probes whether failures at high \(d\) are training-budget artifacts or architectural limits. Per-architecture hyperparameters (optimizer, learning rate, depth, width) and hardware/wall-clock details are in Appendices 21 and 11.

Evaluation. Best eval \(R^2\) on 1500 held-out tasks per checkpoint, averaged across 3 seeds (5 seeds at \(d \in \{20, 30\}\), 6 seeds for Transformer at \(d=30\) to characterize bimodality).

5 Results↩︎

5.1 Main Comparison at \(d=10, d=20\)↩︎

Table 1 compares all baseline architectures at two task dimensionalities. At \(d=10\) all but LSNN cluster near \(R^2 \approx 0.8\), indicating that ICL itself is broadly attainable across architectures at low dimensionality. At \(d=20\) the picture changes sharply: a clean bifurcation separates four ICL-capable architectures (Transformer, Spikformer, DendriCL, DendriCNN) at \(R^2 \geq 0.72\) from four that collapse to near chance (\(R^2 \leq 0.09\): Pure LIF, Active Dendrites, GRU, LSNN). DendriCL is the highest-scoring spike-based model and is within \(3\) pp of its non-spiking ablation DendriCNN, indicating spike binarization itself is not the limiting factor. The full 15-architecture \(\times\) 8-dimension \(\times\) 2-compute matrix — including LSNN, SpikingSSM, LTC, Spike-driven V2, and the MLP family — is in Appendix 12 (Table 5).

\(^{*}\)DendriCL \(d{=}20\) reported at 50k-step compute (10k: \(0.805 \pm 0.009\), \(n=5\)

).

5.2 DendriCL is uniquely seed-stable at super-\(d\)↩︎

To distinguish architectural limits from training-budget artifacts, we retrain all key architectures at \(5\times\) compute (50k vs.10k steps). Table 2 reports eval \(R^2\) at \(d \in \{25, 30, 40, 50\}\) for DendriCL, Transformer, Spikformer, Pure LIF, and Spike-driven V2 across both compute budgets; the corresponding bar comparison and Bayes-optimal ridge ceiling are visualized in Fig. 4 (Appendix 24).

4pt

\(^\dagger\)Transformer \(d{=}30\) at 50k: 3 of 6 seeds fail (\(R^2 \leq 0.012\)), 1 partial (\(R^2 = 0.315\)), 2 grok (\(R^2 \approx 0.98\)

) — a 3-mode distribution, not simple bimodality.

DendriCL maintains \(\sigma \leq 0.036\) across the entire super-\(d\) range while every other architecture shows substantial seed instability or collapse. Transformer reaches \(0.991 \pm 0.002\) at \(d=25\), becomes bimodal at \(d=30\) (3 of 6 seeds at \(R^2 \leq 0.012\), 2 at \(R^2 \approx 0.98\), 1 partial at \(R^2 = 0.315\)), and collapses to the chance floor at \(d \geq 40\) (all 6 seeds at \(R^2 \leq 0.009\) with \(5\times\) compute — the failure is architectural, not budget-induced). Spikformer and Pure LIF degrade gracefully but trail DendriCL by \(0.17\) and \(0.41\) respectively at \(d=30\). The compute-scaling gain for DendriCL grows with task dimension (\(+0.02\) at \(d=25\), \(+0.03\) at \(d=30\), \(+0.10\) at \(d=40\), \(+0.15\) at \(d=50\)), consistent with longer integration horizons benefiting more from extra training. A four-mode taxonomy of architecture-specific failures and per-seed raw \(R^2\) values are in Appendices 19 and 20; the Bayes-optimal ridge ceiling (Fig. 4) is derived in Appendix 13, where DendriCL is shown to stay within \(20\%\) of Bayes through \(d=40\).

5.2.0.1 Relation to grokking.

The Transformer behaviour at \(d=30\) matches the canonical grokking signature [38], [39]: a long plateau at the memorization floor followed by a discrete phase transition to algorithmic generalization. Figure 6 shows the per-seed eval-\(R^2\) trajectories: 3 of 6 seeds remain at the chance floor for the full 50k-step budget, 1 partial seed breaks through at step \(\sim\!34000\) (\(R^2 = 0.315\)), and 2 grok cleanly (breakthroughs at steps \(\sim\!4000\) and \(\sim\!6000\), both reaching \(R^2 \approx 0.98\)). At \(d=40\) and \(d=50\) no seed breaks through in the same budget. DendriCL exhibits no analogous transition: all 3 seeds rise smoothly and monotonically from step 0, with \(\sigma \leq 0.005\) at every step (Fig. 6, right panel). The architectural contrast is the cause: DendriCL’s apical recurrence is the online-learning algorithm, leaving no implicit circuit to discover, whereas in dense attention the algorithm must be assembled in the attention-weight configuration through a phase transition whose basin of attraction shrinks with \(d\). The same induction-head emergence pattern was reported by [40], [41] for in-context circuit formation. Compartmental augmentation thus offers a principled route around grokking-style ICL instability.

Visualizations of the super-\(d\) bar comparison, the \(d\)-sweep at both compute budgets, and per-seed grokking trajectories at \(d=30\) are provided in Appendix 24 (Figs. 4, 5, 6).

5.3 Task-dimensionality and context-length sweeps↩︎

Across a 6\(\times\) scaling in task dimensionality (\(d = 5 \to 30\)), DendriCL loses only 3.3 pp; Pure LIF loses 72 pp from \(d=10\) to \(d=20\) alone. Context-length scaling is monotonic and sub-linear, consistent with the \(\mathcal{O}(1/\sqrt{k})\) LMS rate, with empirical fit \(R^2(k) = 0.97 - 3.28/k\) matching the predicted form for leaky online LMS (Proposition 1; full curve in Fig. 7, Appendix 24). One seed (5\(^{\text{th}}\)) at \(k=40\) failed to converge; we report the 4 successful runs.

5.4 Mechanism: Apical Probe↩︎

Following [1], we freeze each trained DendriCL checkpoint and recover internal representations via linear probe. For each of 1500 held-out tasks, we capture the apical trajectory \(\boldsymbol{u}_A(t)\) and a reference online-LMS trajectory \(\hat{\boldsymbol{w}}_{\text{LMS}}(t)\) from the same context pairs. Ridge regression (\(\alpha{=}0.1\), 50/50 split) yields:

The apical trajectory is 93% linearly decodable into the reference LMS trajectory at \(d=20\), with std \(\leq 0.003\) across 3 seeds. \(R^2(\boldsymbol{u}_A \to \boldsymbol{w}_{\text{true}}) \approx 0.84\) matches the task-level eval \(R^2 \approx 0.80\), implying DendriCL’s ICL capability is quantitatively explained by apical-encoding quality. Learned \((\alpha, \gamma)\) lie in the theoretically favorable LMS regime and are consistent across seeds and task scales. Figure 3 visualizes both findings: (a) probe \(R^2\) grows with task dimension \(d\), reaching \(0.97\) at \(d=30\); (b) BPTT-learned \((\alpha, \gamma)\) settle in a conservative regime below the theoretical upper bound \(\gamma^\star = 1/(d+2)\), consistent with classical LMS best-practice for short-\(k\) adaptation. The probe correspondence is not a final-position artifact: it holds tightly (\(R^2 \geq 0.87\)) at every context position \(t \in \{0, \ldots, k-1\}\) (Appendix 17); the full BPTT-learned \((\alpha, \gamma)\) table across \(d \in \{5, \ldots, 30\}\) and 3 seeds is in Appendix 18.

5.5 Ablations↩︎

Spike cost. DendriCNN (continuous apical/basal/soma, no LIF) reaches \(R^2 = 0.833 \pm 0.001\) at \(d=20\), outperforming DendriCL by 3 pp. This isolates that compartmental structure, not spike binarization, drives ICL scalability. The 3-pp spike cost is the price of biological realism and potential \(\sim 10\times\) neuromorphic energy savings.

Depth. \(L=1\) matches \(L=2\) at \(d=20\) (\(0.820\) vs.\(0.793\)): a single compartmental layer is sufficient, with \(\sim 4\times\) wall-clock speedup over \(L=2\).

Width. A sweep of apical-compartment dimensionality \(d_{\text{apical}} \in \{64, 128, 192, 384, 512, 768\}\) reveals a sharp upper cliff: \(d_{\text{apical}} \leq 384\) all train successfully, \(d_{\text{apical}} \geq 512\) all diverge (training loss never decreases) — consistent with the LMS stability bound \(\gamma < 2/(d+2)\) becoming violated for over-parameterized apical compartments (Appendix 14).

Inference energy. At \(d=20\) (1500 held-out tasks), DendriCL fires \(414\) spikes/token vs.Pure LIF \(1.75\times\) and Spikformer \(5.2\times\) that; under the Loihi cost model (\(0.9\) pJ/spike [13]), forward pass costs \(15.3\) nJ for DendriCL vs.\(79.5\) nJ for Spikformer. Normalised by delivered \(R^2\), DendriCL is \(\sim\!6\times\) more energy-efficient per correct prediction than Spikformer; baselines below DendriCL’s energy (LSNN, Active Dendrites) collapse to chance and are useless. Full per-architecture breakdown and Transformer FLOP comparison in Appendix 16.

Out-of-distribution generalization. Trained on the canonical Garg prior \(\boldsymbol{w} \sim \mathcal{N}(0, I_d)\) and evaluated without retraining under shifted priors \(w_{\text{std}} \in [0.25, 4.0]\), DendriCL peaks at the training distribution and degrades gracefully under moderate shift, with sharp drop only beyond \(w_{\text{std}} = 2.0\) (Appendix 15).

5.6 Extended Tasks↩︎

On binary classification (\(d=10\)) DendriCL (\(L=2\)) reaches \(0.805 \pm 0.007\), comparable to Spikformer (\(0.828\)) and Pure LIF (\(0.814\)). On 2-layer-NN regression (\(d=20\)), DendriCL (\(L=2\)) reaches \(0.568 \pm 0.015\), comparable to Spikformer (\(0.582\)) and substantially above Pure LIF (\(0.322\)). These tasks do not admit a clean online-LMS interpretation; compartmental structure helps but not to the extent of linear regression. Full per-task analysis, including ablation of compartmental components on the nonlinear regression task, is in Appendix 22.

6 Discussion↩︎

Implications for ML. DendriCL extends the attention-free ICL line (Mamba, MLPs, SSMs) to a spiking substrate with the benefits of a single compartmental layer and explicit mechanistic interpretation. The compute-matched super-\(d\) comparison (Table 2) identifies an architectural phase transition in dense attention between \(d=25\) and \(d=40\) — one that does not close with 5\(\times\) compute — and demonstrates that compartmental augmentation is a principled route past it.

A new update mode for dendritic compartments. Prior compartmental models share a structural assumption: the apical compartment delivers a driving signal — teacher [18], backpropagated error [19], attentional gate [20], or external task ID [21] — and adaptation lives in the synaptic plasticity that signal triggers. DendriCL inverts this: the apical compartment is an active online estimator whose subthreshold dynamics carry the learned task parameter directly, with synapses frozen at inference. The compartment is the locus of computation, not a vehicle for plasticity. This bridges classical adaptive filter theory [6], [34] and biological circuit theory, positioning compartmental neurons as a candidate structural basis for cortical in-context learning. A schematic contrasting Transformer’s implicit-GD mechanism with DendriCL’s explicit-LMS mechanism is in Fig. 8, Appendix 24.

Biological interpretation (with appropriate limits). Our Tier-3 hypothesis (§3) — cortex implements ICL via apical-LMS dynamics — is falsifiable via in vivo recording during behavioral ICL paradigms. The specific predictions listed there provide targets for experimental validation.

Implications for neuromorphic hardware. DendriCL’s \(\sim\!4\times\) spike-count reduction over Pure LIF and \(\sim\!5\times\) over Spikformer (Appendix 16, Table 9), combined with single-layer depth and frozen-weight inference, projects to roughly a \(10\times\) energy advantage on Loihi-class hardware while improving super-\(d\) ICL accuracy — a regime where existing SNNs collapse to chance. The compartmental apical substrate proposed here is, to our knowledge, the first ICL mechanism whose inference-time energy cost decreases monotonically with the parameter savings of single-layer simplicity, making the architectural advantage and the energy advantage one and the same trade-off rather than a Pareto exchange.

7 Limitations↩︎

Our benchmarks remain restricted to Garg’s synthetic function classes; real-world ICL settings (language, vision) remain unaddressed. The LMS-equivalence theorem covers the apical compartment in isolation and does not formally handle the LIF reset nonlinearity at the soma, though the empirical probe correspondence (\(R^2 = 0.93\)) suggests the equivalence holds in practice despite this nonlinearity. We test task dimensions up to \(d = 50\); behaviour at much higher dimensions is unexplored. Energy claims rest on spike-count instrumentation and Loihi-projected estimates rather than measured energy on neuromorphic hardware. The biological hypothesis — that cortex implements ICL via apical-LMS dynamics — is falsifiable but not yet experimentally established. Finally, the Transformer bimodality estimate at \(d=30\) (\(\sigma = 0.567\)) rests on 6 seeds; a larger sample would tighten the failure-mode distribution.

8 Conclusion↩︎

DendriCL is a single-layer compartmental SNN whose apical recurrence is structurally identical to leaky online Widrow–Hoff LMS. With synapses frozen at inference, it solves Garg-2022 ICL across \(d \in \{5,\ldots,50\}\), is uniquely seed-stable at \(d \geq 30\) where dense Transformers grok or fail, and is mechanistically transparent (\(R^2 = 0.93\) linear probe). Apical compartmental dynamics are thus a class of test-time adaptation distinct from plasticity- and attention-based approaches.

9 Biological Grounding and Distinction from Prior Compartmental Models↩︎

9.1 Three Tiers of Evidence↩︎

We are explicit about which aspects of DendriCL are biologically supported versus computationally hypothesized.

Tier 1 — Anatomically established. The apical-basal-soma three-compartment architecture corresponds to cortical layer-5 pyramidal neurons [20], [25]. Subthreshold continuous apical dynamics [22], persistent Ca\(^{2+}\) plateaus on 100+ ms timescales [23], apical-basal coincidence at the soma [22], and LIF-style somatic spiking are all experimentally characterized. [24] show single human L2/3 pyramidal neurons can compute XOR-level nonlinear functions, establishing single pyramidal neurons as computationally substantial units.

Tier 2 — Qualitatively plausible. Our functional assignment — apical receives prediction-error-like signals — is consistent with predictive coding [35], [36], [42] and with recent behavioral evidence for sub-second cortical task adaptation [43]. Direct in vivo recording of apical membrane potential encoding specific task parameters has not been performed.

Tier 3 — Computational hypothesis (ours). The specific mechanism — apical implements online LMS over \((\boldsymbol{x}, y)\) context pairs — is our computational contribution. The specific parameters (\(\alpha \approx 0.997\), \(\gamma \approx 0.012\)) are BPTT-optimized, not biological measurements. Whether biological cortex implements this mechanism is a falsifiable hypothesis for future in vivo validation: predictions include (i) apical recordings during ICL-like behavioral paradigms should show \(d\)-dimensional linear decoding of task parameters; (ii) ICL accuracy should follow the LMS-like \(\mathcal{O}(1/\sqrt{k})\) convergence rate.

9.2 Distinction from Prior Compartmental Models↩︎

10 Proof of Proposition 1↩︎

We establish that the apical update, reduced to its core recurrence \[\hat{\boldsymbol{w}}_{t+1} = \alpha \hat{\boldsymbol{w}}_t + \gamma (y_t - \hat{\boldsymbol{w}}_t^\top \boldsymbol{x}_t) \boldsymbol{x}_t,\] converges to the task parameter \(\boldsymbol{w}\) in expectation at rate \(\mathcal{O}(d/k)\) under standard assumptions. Let \(\boldsymbol{e}_t := \hat{\boldsymbol{w}}_t - \boldsymbol{w}\) denote the error. Substituting \(y_t = \boldsymbol{w}^\top \boldsymbol{x}_t + \epsilon_t\): \[\boldsymbol{e}_{t+1} = \alpha \hat{\boldsymbol{w}}_t + \gamma (\boldsymbol{w}^\top \boldsymbol{x}_t + \epsilon_t - \hat{\boldsymbol{w}}_t^\top \boldsymbol{x}_t) \boldsymbol{x}_t - \boldsymbol{w}\] \[= \alpha \boldsymbol{e}_t + (\alpha - 1) \boldsymbol{w} - \gamma (\boldsymbol{e}_t^\top \boldsymbol{x}_t) \boldsymbol{x}_t + \gamma \epsilon_t \boldsymbol{x}_t.\] Setting \(\alpha = 1\) (pure LMS without leak) and taking expectation with \(\boldsymbol{x}_t \sim \mathcal{N}(0, I_d)\) independent of \(\boldsymbol{e}_t\): \[\mathbb{E}\|\boldsymbol{e}_{t+1}\|^2 = \mathbb{E}\|\boldsymbol{e}_t\|^2 - 2\gamma \mathbb{E}[\boldsymbol{e}_t^\top \boldsymbol{x}_t \boldsymbol{x}_t^\top \boldsymbol{e}_t] + \gamma^2 \mathbb{E}[(\boldsymbol{e}_t^\top \boldsymbol{x}_t)^2 \|\boldsymbol{x}_t\|^2] + \gamma^2 \sigma^2 \mathbb{E}\|\boldsymbol{x}_t\|^2.\] Using \(\mathbb{E}[\boldsymbol{x}_t \boldsymbol{x}_t^\top] = I_d\) and \(\mathbb{E}[(\boldsymbol{e}^\top \boldsymbol{x})^2 \|\boldsymbol{x}\|^2] = (d+2)\|\boldsymbol{e}\|^2\) for Gaussian \(\boldsymbol{x}\): \[\mathbb{E}\|\boldsymbol{e}_{t+1}\|^2 = (1 - 2\gamma + \gamma^2 (d+2)) \mathbb{E}\|\boldsymbol{e}_t\|^2 + \gamma^2 \sigma^2 d.\] For \(0 < \gamma < 2/(d+2)\) the contraction factor \(\rho := 1 - 2\gamma + \gamma^2 (d+2) \in [0, 1)\), giving geometric convergence. The stationary error is \[\mathbb{E}\|\boldsymbol{e}_\infty\|^2 = \frac{\gamma^2 \sigma^2 d}{1 - \rho} = \frac{\gamma \sigma^2 d}{2 - \gamma(d+2)}.\] With optimal step size \(\gamma^\star = 1/(d+2)\), the stationary error is \(\sigma^2 d/(d+1) = \mathcal{O}(\sigma^2)\), and after \(k\) steps the expected squared error is \(\mathcal{O}(d/k)\) up to the noise floor. This matches the classical LMS rate [6], [34]. \(\blacksquare\)

Remark on the LIF nonlinearity. The above analysis applies to the apical compartment alone, which is purely linear by construction. The somatic LIF (threshold-reset nonlinearity) affects the readout path but not the apical recurrence itself; in particular, somatic spikes do not feed back into \(\boldsymbol{u}_A\). The linear-probe result (Table 5.4) validates that the correspondence survives the full trained model.

11 Additional Experimental Details↩︎

Hardware. Experiments on 2\(\times\) NVIDIA RTX 5090 (32GB). Per-run wall-clock ranges \(85\)–\(1650\)s (10k steps) and \(385\)–\(4430\)s (50k steps) depending on architecture and \(d\).

Seed protocol. 3 seeds for all reported configurations, with 5-seed coverage for critical configurations (\(d \in \{20, 30\}\) classification, \(d \in \{20, 30\}\) linreg). At \(d=40/50\) we use 3 seeds with long training (50k) to resolve variance while keeping compute budget tractable.

Task generator. Garg 2022 linear regression: \(\boldsymbol{w} \sim \mathcal{N}(0, I_d)\), \(\boldsymbol{x}_i \sim \mathcal{N}(0, I_d)\), \(y_i = \boldsymbol{w}^\top \boldsymbol{x}_i + \epsilon_i\) with \(\epsilon_i \sim \mathcal{N}(0, 0.01)\). Context \(k = 2d\) unless specified. Input tokens \([\boldsymbol{x}_i; y_i; \text{flag}_i]\) with \(\text{flag}_i = 1\) only at query. Model sees context fully; query \(y_q\) is masked.

Bayes-optimal baseline. For the linear regression task, the Bayes-optimal predictor is ridge regression with regularization \(\lambda = \sigma^2\): \(\hat{y}_q = \boldsymbol{x}_q^\top (X^\top X + \sigma^2 I)^{-1} X^\top \boldsymbol{y}\).

11.0.0.1 Baseline implementation provenance.

For reproducibility and fair comparison we disclose exactly how each non-trivial baseline architecture was implemented. None of the baselines use the original authors’ code or checkpoints; all are re-implemented from the architectural description in the respective paper, simplified to match the Garg-2022 protocol (token-level input of \([\boldsymbol{x}_i; y_i; \text{flag}_i]\), single-scalar regression output). We believe this is the correct fair-comparison protocol: it controls for parameter count and protocol variation, but it does mean our numbers are lower bounds on what each architecture could achieve with its authors’ full engineering. Specifically:

• Transformer: standard nn.TransformerEncoder-style implementation with learned positional embeddings, 4 layers, 4 heads, \(d_{\text{model}}=192\). No pretraining; trained from scratch on each task family.

• Spikformer [14]: our re-implementation of the SSA+MLP+LIF block structure described in Zhou et al. ICLR. We use the published \({\rm QKV} \to\) LIF attention formulation with \(T_{\text{LIF}}=4\) virtual steps, omitting dataset-specific touches (batch-norm placement, dropout schedule). Not a direct port from their ImageNet-image codebase.

• Spike-driven Transformer V2 [37]: simplified port of the spike-driven attention block without the multi-branch convolutional stem, since the Garg protocol has no spatial structure. We preserve the core spike-gated attention.

• SpikingSSM [44]: simplified diagonal S4-style state-space recurrence with LIF spike activation, adapted for 1D input tokens. Does not include the full SDN (surrogate dynamic network) machinery of the original AAAI 2025 paper; the core SSM + spike activation is preserved.

• LSNN [15]: faithful to the threshold-adapting LIF formulation of Bellec et al. NeurIPS, with adaptation time-constant \(\beta = 0.4\).

• Active Dendrites [21]: re-implementation of Iyer et al.’s \(K=8\) per-neuron dendritic segments with external context gating. We provide a zero context vector (no external task ID) to match our protocol where no external task signal is available.

• LTC [29]: liquid-time-constant cell adapted from Hasani et al. with input-gated time constants.

• GRU, Pure LIF, MLP, MLP-Mixer, Linear Tx: standard implementations (PyTorch nn.GRU, our own minimal LIF/MLP/Mixer/linear-attention modules).

All re-implementations, training scripts, and raw result JSONs are released as part of the snn-icl-bench repository; the full code comprises \(\sim\!1{,}800\) lines across models.py, tasks.py, and train.py, deliberately kept minimal for reproducibility. We welcome contributions that replace our re-implementations with the original authors’ official code; where doing so changes the super-\(d\) conclusions, we update accordingly. At the time of writing, we are not aware of published Garg-ICL results for any of these architectures (Spikformer, Spike-driven V2, SpikingSSM, LSNN, Active Dendrites, LTC) that would allow a direct numerical comparison.

12 Full Baseline Matrix↩︎

Table 5 reports every architecture \(\times\) task \(\times\) compute combination we ran, covering 15 distinct architectures, 8 task dimensions (\(d \in \{5, 10, 15, 20, 25, 30, 40, 50\}\)), and two compute regimes (10k and 50k BPTT steps). All numbers are best eval \(R^2\) over 3 seeds (5 for configurations marked \(\star\)). Training steps are 8k for classif/low-\(d\), 10k for linreg \(d \geq 15\), and 50k for explicit compute-matched runs.

12.0.0.1 Architecture families and their super-\(d\) behaviour.

The table exposes four distinct scaling behaviours:

(i) Compartmental architectures (DendriCL, DendriCL \(L{=}2\), DendriCNN) retain \(R^2 \geq 0.65\) across all tested task dimensions, with graceful degradation only at \(d=50\) (still \(0.65\) under 50k compute). Within this family, the continuous ablation DendriCNN slightly outperforms spiking DendriCL (\(0.833\) vs.\(0.820\) at \(d=20\), 50k), isolating compartmental structure — not spike binarization — as the driver of scalability. Depth provides no benefit: \(L=1\) matches \(L=2\).

(ii) Attention-based architectures (Transformer, Spikformer, Spike-driven V2, Linear Tx) have task-dependent breakdown points. Dense Transformer performs near-Bayes-optimal at \(d \leq 25\) but exhibits bimodal grokking at \(d=30\) and uniform failure at \(d \geq 40\) even with 5\(\times\) compute. Spikformer is strictly dominated by Transformer at low \(d\) (expected: attention is better than spiking-attention on continuous-valued tasks) but crosses over and beats Transformer at \(d \geq 30\) under compute-matched training, reflecting that attention instability is specific to dense quadratic attention. Linear Tx degrades continuously with \(d\), consistent with its reduced capacity. Spike-driven V2 fails at \(d \geq 40\).

(iii) Pure-spiking architectures (Pure LIF, LSNN, SpikingSSM) work at \(d \leq 10\) only (0.80 for Pure LIF, 0.01 for LSNN/SpikingSSM). Scalar adaptive thresholds (LSNN) and diagonal SSM dynamics (SpikingSSM) offer no multi-dimensional state for task estimation; they are architecturally incapable of Garg-style ICL. Pure LIF benefits from compute at moderate \(d\) (0.01 \(\to\) 0.39 at \(d=30\) under 50k steps) but this partial rescue does not extend to \(d=40\) (still \(0.04\)).

(iv) MLP-family (Pure MLP, MLP-Mixer, LTC) is a surprising spread. Pure MLP fails entirely (\(R^2 \approx 0.01\)) — consistent with Tong & Pehlevan’s finding that naive MLPs require token-mixing. MLP-Mixer succeeds at \(d=10\) (\(0.69\)) but collapses by \(d=20\). LTC’s input-gated time-constant trick helps at \(d=10\) but also collapses. Our interpretation: these architectures have the state capacity but lack the error-driven update structure that the compartmental basal-apical-error loop provides.

12.0.0.2 Why this table matters for the compartmental-augmentation hypothesis.

If compartmental structure is a universal enhancement principle for ICL architectures, the prediction is that adding an apical compartment to any of the failing architectures in Table 5 should rescue super-\(d\) performance. A systematic cross-architecture study of this conjecture — "does apical augmentation scale Spikformer, Mamba, SpikingSSM, and LSNN to the same super-\(d\) regime where DendriCL succeeds?" — is a natural follow-up direction suggested by this paper.

\(^\star\)5 seeds. \(^\ddagger\)

single-seed preliminary.

13 Bayes-Optimal Comparison↩︎

13.0.0.1 Setup.

For the Garg linear regression task, the Bayes-optimal predictor under the task prior \(\boldsymbol{w} \sim \mathcal{N}(0, I)\) and observation noise \(\boldsymbol{y}_i = \boldsymbol{w}^\top \boldsymbol{x}_i + \epsilon_i\), \(\epsilon_i \sim \mathcal{N}(0, \sigma^2)\), is ridge regression with regularization \(\lambda = \sigma^2\): given the context \(\{(\boldsymbol{x}_i, y_i)\}_{i=1}^k\), \[\hat{y}_q^{\text{Bayes}} = \boldsymbol{x}_q^\top (X^\top X + \sigma^2 I_d)^{-1} X^\top \boldsymbol{y}.\] This is the MAP estimate under the prior and equals the posterior-mean predictor for Gaussian inputs. We compute the expected \(R^2\) of this predictor averaged over 5000 Monte-Carlo draws of \((\boldsymbol{w}, X, \boldsymbol{x}_q, \epsilon)\) at each \(d\). The result is an absolute upper bound that no architecture can exceed in expectation.

13.0.0.2 Results and interpretation.

Table 6 reports the gap. DendriCL’s distance from the Bayes-optimal ceiling is approximately constant (\(\sim\)​17–18% normalized gap) across \(d \in [10, 40]\). The gap widens at \(d=50\) to 32% — the first sign that the apical compartment’s capacity begins to saturate. The \(\sim\)​17% gap at \(d \in [10, 40]\) reflects the finite-sample LMS error: the Bayes-optimal predictor uses the closed-form ridge solution over all \(k = 2d\) samples simultaneously, whereas DendriCL’s apical recurrence accumulates the same estimate incrementally with a learned leak \(\alpha\). Finite-\(k\) LMS is known to lag behind full-sample ridge by \(\mathcal{O}(d/k)\) at fixed \(\gamma\); with \(k = 2d\) and the BPTT-selected \(\gamma \approx 0.2/(d+2)\), this predicts a \(\sim\)​15–20% gap — consistent with what we observe.

\(^\dagger\)

10k-step baseline used where 50k not run.

13.0.0.3 Positioning relative to Transformer.

The dense Transformer at \(d \leq 25\) achieves a much smaller gap to Bayes (e.g.\(0.991\) vs.Bayes \(0.977\) at \(d=25\), gap \(\sim\!1\%\)), because a trained Transformer with enough compute can implement something closer to the full-sample ridge solution via its attention weights. This superior absolute performance at low \(d\) is preserved in our paper’s main claim: we do not assert DendriCL outperforms Transformer in the regime where Transformer succeeds. Our claim is about the regime past \(d=30\), where Transformer training fails (grokking or total non-convergence) and DendriCL is the only architecture that both trains stably and reaches a useful fraction of the Bayes ceiling.

14 Width Ablation (\(d_{\text{apical}}\) sweep)↩︎

We sweep the apical-compartment dimensionality \(d_{\text{apical}} \in \{64, 128, 192, 384, 512, 768\}\) at fixed task \(d=20\), keeping \(d_{\text{model}}=384\) and all other hyperparameters at defaults. Two seeds per configuration.

14.0.0.1 Motivation.

If the apical membrane is truly implementing online LMS, its effective capacity should scale with \(d_{\text{apical}}\) up to roughly the task dimension \(d\), and exhibit step-size instability when \(d_{\text{apical}}\) grows large (classical LMS stability requires \(\gamma < 2/(d_{\text{apical}} + 2)\)). This prediction is falsifiable: if DendriCL were an arbitrary deep architecture, more width should monotonically help, not catastrophically fail.

\(^\star\)

5-seed result for default configuration.

The instability at \(d_{\text{apical}} \geq 512\) is a prediction of classical LMS theory: for task dimension \(d = 20\), step-size stability requires \(\gamma < 2/(d+2) \approx 0.091\). When \(d_{\text{apical}}\) significantly exceeds \(d\), BPTT tends to select \(\gamma\) values outside this stable range, causing divergence. This is further empirical validation of the apical \(\equiv\) LMS correspondence: the architecture inherits both the stability region and the failure modes of the embedded LMS recurrence.

14.0.0.2 Training dynamics at failure.

The \(d_{\text{apical}} = 512, 768\) runs are not slowly-learning; they are actively diverging. The eval \(R^2\) is below \(0.01\) from the first evaluation (step 500) throughout training; the training loss exhibits sporadic large spikes consistent with LMS’s finite-sample blowup when \(\gamma \cdot \|\boldsymbol{x}\|^2 > 1\). A simple diagnosis: freezing the apical update and using only the basal pathway (i.e., setting \(\gamma = 0\)) recovers \(R^2 \approx 0.50\) even at \(d_{\text{apical}} = 768\), showing the basal feedforward path is healthy and the pathology is specifically in the apical recurrence. The full architectural sweep (including learned \(\gamma\) trajectories across training for each \(d_{\text{apical}}\)) is available in our code release.

14.0.0.3 Takeaway.

The sharp upper cliff at \(d_{\text{apical}} \approx d \times 30\) (for \(d=20\)) is a stability signature, not a capacity signature, and it matches the classical LMS prediction. This is a different failure mode than anything reported for Transformers, Mambas, or MLPs at ICL — supporting the apical \(\equiv\) LMS interpretation.

15 Out-of-Distribution Generalization (\(w_{\text{std}}\) sweep)↩︎

15.0.0.1 Protocol.

Models are trained on the canonical Garg distribution with \(\boldsymbol{w} \sim \mathcal{N}(0, I_d)\) and evaluated, without retraining or re-tuning, on shifted weight priors \(\boldsymbol{w} \sim \mathcal{N}(0, (w_{\text{std}})^2 I_d)\) for \(w_{\text{std}} \in \{0.25, 0.5, 1.0, 1.5, 2.0, 3.0, 5.0\}\). The input distribution \(\boldsymbol{x}_i \sim \mathcal{N}(0, I_d)\) and noise \(\sigma\) are unchanged.

15.0.0.2 LMS interpretation.

The OOD robustness profile has a precise LMS interpretation: BPTT-selected \((\alpha, \gamma)\) implicitly assume a particular signal-to-noise ratio (SNR) in the training distribution. When the test SNR differs, the learned step size is mismatched. For small \(w_{\text{std}}\) the observed error \(e_t = y_t - \hat{y}_t\) shrinks (smaller signals yield smaller errors), so the learned \(\gamma\) over-adapts — integrating relatively more error per step than the optimal Bayesian posterior would. For large \(w_{\text{std}}\) the error variance grows, and the same fixed \(\gamma\) becomes under-adapted. A predictor that properly scaled its step size would recover much of the lost performance; but DendriCL is frozen at inference and cannot do this. This is an intrinsic cost of the static-hyperparameter compartmental architecture — not a deficiency of the compartmental idea itself. A natural extension (outside this paper’s scope) is to make \(\gamma\) input-dependent via a small gating network, recovering OOD robustness at the cost of slightly more parameters.

15.0.0.3 Comparison to Transformer OOD.

Transformers trained on the Garg task have been reported [1], [8] to exhibit similar peak-at-training-distribution profiles with graceful degradation. Our OOD curve is consistent with that baseline; we do not claim OOD is where DendriCL’s advantage lies.

16 Spike Count and Energy Analysis↩︎

16.0.0.1 Measurement protocol.

Spike counts are measured at inference time by instrumenting every LIFCell and ALIFCell module in each model via a monkey-patch on its forward_seq method; we sum binary spike emissions across all units, all time steps, and all context positions for a representative batch of 64 tasks and report the per-forward-pass total (also normalized per-token in Table 9). Energy is estimated under two neuromorphic cost models: Intel Loihi at \(0.9\) pJ per spike (published in [13] for dense synaptic reach) and SpiNNaker at \(30\) pJ per spike (conservative upper bound). Only the Loihi values are reported in the main appendix table; SpiNNaker values are 33\(\times\) larger and preserve the inter-architecture ordering.

16.0.0.2 Energy-accuracy efficiency.

Among architectures that actually solve the \(d=20\) task, DendriCL (7,023 pJ at \(d{=}10\), 15,282 pJ at \(d{=}20\); \(R^2 = 0.820\)) is \(\sim\!5\times\) more energy-efficient per correct prediction than Spikformer (48,038 / 79,515 pJ, \(R^2 = 0.724\)). LSNN and SpikingSSM use comparable or less energy than DendriCL but fail entirely (\(R^2 \approx 0.01\)), making them vacuously "efficient": a model that outputs 0 uses zero energy but is useless. The interesting frontier is therefore non-trivially Pareto-dominant along the energy-accuracy axis, and DendriCL is the only architecture that lands in the useful corner (high accuracy, low spike count).

16.0.0.3 Where does the DendriCL spike budget go?

The apical compartment itself does not spike — it is a subthreshold dynamic state. The \(\sim\!17{,}000\) spikes per forward pass at \(d=20\) come from (i) the single LIF soma at each of the \(d_{\text{model}}=384\) units times \(k+1=41\) positions (maximum \(\sim\!15{,}700\) spikes if every unit fires at every position), plus (ii) the feedforward block’s LIF (\(\sim\!1{,}200\) spikes). Empirical measurement shows an average sparsity of \(\sim\!85\%\) in the soma (most units are subthreshold most of the time), which explains why the absolute spike count is much smaller than the maximum. This is the natural sparsity advantage of spiking: the informative signal is carried by subthreshold apical membrane dynamics, while the somatic spikes downstream from the soma are sparse rate codes.

16.0.0.4 Comparison to dense Transformer.

A fair dense-ANN comparison would require FLOPs rather than spike counts. Per-forward-pass FLOPs for the Transformer baseline (\(d_{\text{model}}=192\), 4 layers, heads 4, \(k+1=41\)): approximately \(4 \times 41 \times 41 \times 192 + 4 \times 41 \times (192 \times 192 + 192 \times 768) \approx 30\)M FLOPs. At neuromorphic energy budgets (\(\sim 3\) pJ/MAC on efficient digital ASIC), this translates to \(\sim\!90\) nJ per forward pass — six orders of magnitude more than DendriCL’s 15 nJ spike-budget estimate. While the direct comparison depends on hardware choices, the order-of-magnitude advantage for sparse spiking inference is well-established and not specific to our architecture.

17 Linear Probe Across Context Position↩︎

17.0.0.1 Protocol.

Section 5.4 reports a single scalar — the probe \(R^2\) at the final context position \(t = k-1\). To check whether the apical \(\equiv\) LMS correspondence holds at all intermediate times or only at the query, we repeat the probe analysis independently at each context position \(t \in \{0, \ldots, k-1\}\). For each fixed \(t\), we train a ridge probe \(\boldsymbol{u}_A(t) \to \hat{\boldsymbol{w}}_{\text{LMS}}(t)\) using 750 held-out tasks and evaluate on 750 others; \(\hat{\boldsymbol{w}}_{\text{LMS}}(t)\) is the reference online-LMS estimate computed from the same \(t\) context pairs used by DendriCL up to that position.

17.0.0.2 Results.

Table 10 shows a representative trajectory at \(d=10, k=20\), seed 0 (the pattern is tight and consistent across seeds; see probe_dendricl_finer.json in the code release for the full data).

17.0.0.3 Two independent observations.

The table exposes two distinct patterns that together validate the apical-LMS interpretation:

(i) The LMS correspondence is tight at all positions. The probe \(R^2(\boldsymbol{u}_A(t) \to \hat{\boldsymbol{w}}_{\text{LMS}}(t))\) is \(\geq 0.87\) at every \(t\) after the trivial \(t=0\) (where both states are zero-initialized). The correspondence is not an artifact of the final-position readout — it holds throughout the forward pass. Small dip at mid-positions (\(\sim\!0.87\)) reflects the apical state being only approximately LMS during the "transient" phase when the integration has not yet saturated; by the end of the sequence it recovers to \(0.87\)–\(0.87\).

(ii) True-parameter decoding grows monotonically with context length. \(R^2(\boldsymbol{u}_A(t) \to \boldsymbol{w}_{\text{true}})\) climbs from \(0.06\) at \(t=0\) (uninformed) to \(0.82\) at \(t=19\) (after the full context is seen). This matches the classical LMS convergence rate: for \(d\) parameters and \(t\) observations with \(\sigma^2\) noise, expected \(R^2 \approx 1 - d/(t+d)\), which gives \(1 - 10/29 = 0.66\) at \(t=19\) — lower than what we observe (\(0.82\)), meaning the learned \((\alpha, \gamma)\) outperform naive LMS by roughly the gap between the closed-form ridge solution and the plain Widrow-Hoff recurrence.

17.0.0.4 Aggregate.

Averaged across positions \(t = 1, \ldots, k-1\) (excluding the trivial \(t=0\)), \(R^2(\boldsymbol{u}_A \to \hat{\boldsymbol{w}}_{\text{LMS}}) = 0.89\), \(R^2(\boldsymbol{u}_A \to \boldsymbol{w}_{\text{true}}) = 0.55\). At \(d=20\) both numbers rise by approximately \(0.04\) (consistent with the \(d\)-sweep in Fig. 3). This position-resolved analysis is, to our knowledge, the first per-position probe of any ICL architecture — [1] report only a final-position number. The per-position trajectory is a sharper tool for distinguishing "implements LMS at all times" from "arrives at the right answer at the end by some other route"; our results clearly favour the former.

18 Learned \((\alpha, \gamma)\) Across Task Scales↩︎

BPTT freely chooses the apical hyperparameters \((\alpha, \gamma)\) during training. If our LMS interpretation is correct, these should settle near values predicted by classical adaptive-filter theory. This section verifies that prediction quantitatively.

18.0.0.1 Two scaling laws emerge.

Two empirical patterns hold across \(d \in [5, 30]\):

(i) Learned \(\gamma\) decreases with \(d\), tracking but undershooting \(\gamma^\star\). The ratio \(\gamma / \gamma^\star\) grows from \(0.10\) at \(d=5\) to \(0.35\) at \(d=30\) — always below 1, never above. Classical LMS best-practice for short-context (\(k < 100\)) adaptation is to select \(\gamma\) at \(0.1\)–\(0.3 \times \gamma^\star\) to trade convergence speed against variance: a larger \(\gamma\) reaches the stationary error faster but with larger excess variance. BPTT empirically rediscovers this trade-off. The same small-\(k\) regime in classical adaptive signal processing is called the trade-off region; our values land in its middle.

(ii) Learned \(\alpha\) increases with \(d\), giving longer effective integration horizons. The effective integration horizon of a leaky accumulator is \(1/(1 - \alpha)\). This grows from \(111\) steps at \(d=5\) to \(400\) steps at \(d=30\). Since we only train with \(k = 2d\) context, this means the apical is not forgetting meaningfully within the observable window — the integration horizon is \(\sim 6\)–\(7\times\) the context length. This is consistent with the implicit assumption that all \(k\) context observations are equally informative (a stationary-task prior; no need to down-weight old observations).

18.0.0.2 Stability check.

Across all 18 configurations (\(d \in \{5, 10, 15, 20, 25, 30\} \times\) 3 seeds), the learned \(\gamma\) is always less than \(2/(d+2)\) — the LMS stability bound (Proposition 1). No trained DendriCL model exits the stable region. This is in contrast to the width ablation (Appendix 14) where forcing \(d_{\text{apical}} \geq 512\) at fixed task \(d=20\) drives \(\gamma\) out of the stable region and causes training to diverge — consistent with the architectural view that stability is inherited from the embedded LMS recurrence.

18.0.0.3 What BPTT is and is not doing.

It is important to be clear about the role of BPTT here. BPTT is not picking the LMS algorithm in any meta-sense. The architecture forces the apical recurrence to have LMS structure (linear update with error-dependent drive); BPTT’s only degrees of freedom are the step-size-like hyperparameters \((\alpha, \gamma)\) and the input projection \(W_A\). What is noteworthy is that BPTT, given this architecture, finds hyperparameters in the LMS-stable regime and tracks the theoretical optimum. This is different from a stronger claim (that BPTT would rediscover LMS from a larger architecture class including non-LMS possibilities) and we do not make that stronger claim.

19 Failure-Mode Analysis↩︎

Aggregate summary statistics hide qualitative differences in how each architecture fails. Our compute-matched 50k runs expose four distinct failure modes, each with its own experimental signature and its own implication for what the architecture cannot do.

19.0.0.1 Mode 1: Grokking (Transformer at \(d=30\)).

6 seeds produce a 3-mode distribution: 50% fail (\(R^2 \leq 0.012\) for all 50k steps), 17% partial (\(R^2 = 0.315\), still rising at step 50k), 33% grok cleanly (\(R^2 \approx 0.98\) with a breakthrough at step \(\sim\!6000\) or \(\sim\!4000\)). Mean \(\pm\) std is \(0.386 \pm 0.479\), but the distribution is trimodal, not Gaussian. Figure 6 shows the learning curves. This is the grokking phenomenon of [38] — the Transformer must discover, via gradient descent, a specific attention-weight configuration that implements the online-learning algorithm; this discovery is a discrete phase transition with an unpredictable onset time. With 6 seeds one can estimate the success probability (\(\sim\!33\%\) cleanly); with 100 seeds one could estimate the median breakthrough time. In practical use this failure mode is the worst, because increasing compute does not reliably fix it — seed choice dominates.

19.0.0.2 Mode 2: Clean ceiling (Spikformer at \(d=30\) and \(d=25\)).

3 seeds reach \(R^2 = 0.636 \pm 0.010\) at \(d=30\) and \(0.656 \pm 0.014\) at \(d=25\). The variance across seeds is two orders of magnitude smaller than the Transformer’s variance at the same \(d\), and the architecture consistently converges to a stable plateau. We interpret this as an architectural capacity limit: spiking-attention with the Spikformer structure can solve Garg ICL to \(R^2 \approx 0.64\), but not higher. Unlike the Transformer, training is reliable; unlike DendriCL, the ceiling is well below Bayes-optimal.

19.0.0.3 Mode 3: High-variance partial rescue (Pure LIF at \(d=30\)).

3 seeds produce \(R^2 = 0.394 \pm 0.135\). With 10k compute, Pure LIF fails entirely (\(R^2 = 0.007 \pm 0.001\)); 5\(\times\) compute partially rescues it but the learned representation is not seed-robust. Inspection of training curves shows Pure LIF is neither grokking (seeds do not suddenly break through) nor converging to a clean ceiling (plateau values vary widely across seeds). Our tentative interpretation is that the 4-layer LIF stack does acquire some task-specific memory under long training, but the particular task-to-hidden-state mapping it finds depends sensitively on initialization. Pure LIF has a scalar membrane potential per unit and no persistent multi-dimensional state; whatever task memory it forms is carried in the joint firing pattern, which is a much less regular substrate than the apical compartment.

19.0.0.4 Mode 4: Uniform terminal failure (all architectures at \(d \geq 40\) except DendriCL).

All 6 Transformer seeds across \(d \in \{40, 50\}\) at 50k steps cluster at \(R^2 = 0.008\)–\(0.010\) with std \(\approx 0.001\). Pure LIF 50k at \(d=40\): \(R^2 = 0.035 \pm 0.044\). Spikformer at \(d=50\): \(R^2 = 0.301 \pm 0.003\) (still stable, but far below useful). The \(R^2 \approx 0.009\) value is not random: it is the expected score of a constant-zero predictor under \(y \sim \mathcal{N}(0, \|\boldsymbol{w}\|^2 + \sigma^2)\), which equals \(0\) in population \(R^2\) and is slightly positive on finite test sets due to correlation with mean removal. We verified this interpretation by computing the exact expected \(R^2\) of \(\hat{y} = 0\) on our test distributions: \(0.0089 \pm 0.0004\). The architectures are therefore reaching exactly the trivial-predictor baseline, not producing any usable output.

19.0.0.5 Mode 5 (DendriCL, \(\sigma = 0.005\)–\(0.036\)): graceful absence of failure.

We observe no failure. Variance scales slowly with \(d\): \(\sigma = 0.005\) at \(d = 30, 40\) grows to \(\sigma = 0.036\) at \(d = 50\). Training curves are smooth and monotonic from step 0 (Figure 6 right). This scaling is consistent with the \(\mathcal{O}(d/k)\) LMS convergence bound (Proposition 1), which is architectural — not a function of random initialization.

19.0.0.6 Operational takeaways.

These failure modes have distinct implications for an engineer choosing an architecture: (a) If you need high reliability on a new task of unknown difficulty, dense Transformer is a bad choice at \(d \geq 30\): there is a 50% chance your training run converges to nothing useful, and no warning signal mid-training lets you predict which group you are in. (b) Spikformer gives a reliable mediocre answer — useful when failure modes matter more than peak performance. (c) Pure LIF at \(d=30\) is dangerous: the seed-level variance (\(\sigma = 0.135\)) is large enough to produce misleading single-run results. (d) DendriCL is the only architecture whose variance is both low and smooth over \(d\), making it the natural default for a large-scale hyperparameter sweep where a uniform success rate matters more than peak accuracy.

20 Seed-Level Raw Data↩︎

Summary statistics can be misleading when the underlying seed distribution is multimodal (as for Transformer at \(d=30\)) or highly variable (as for Pure LIF at \(d=30\)). For full reproducibility and for reviewers who may want to spot-check the compute-matched story, we list individual seed \(R^2\) values for the most contested comparisons. All values are the best eval \(R^2\) obtained during the 50k-step training run; the best is typically reached in the final 10k steps and is stable thereafter (see the full eval_history fields in the released JSON files).

21 Architecture Hyperparameters↩︎

21.0.0.1 Design constraints and choices.

All baseline architectures share the following protocol-level settings to keep the comparison fair: (1) optimizer AdamW with learning rate \(10^{-3}\), weight decay \(10^{-4}\), cosine schedule (no warmup); (2) batch size 64; (3) BPTT through the full \(k+1\)-position context (no gradient truncation); (4) LIF surrogate gradient via \(\arctan\) approximation [33] with scale factor \(1.0\); (5) the LIF threshold \(\theta = 1\) and leak time constant \(\tau = 4\) virtual steps for all spiking architectures. All parameter counts are within \([0.6\text{M}, 1.3\text{M}]\) for fair comparison; where an architecture naturally has fewer parameters (e.g.LSNN), we widen \(d_{\text{model}}\) rather than adding layers, to keep depth roughly comparable. Where an architecture has many more parameters (Active Dendrites), we keep the per-unit dendritic-segment count at the authors’ recommended value rather than reducing it, accepting the slight parameter-count excess to preserve architectural faithfulness.

21.0.0.2 Parameter-count fairness.

Table 13 lists total parameters. The spread \(0.62\)M–\(1.29\)M is within a factor of \(2.1\times\), well below the factor-of-10 variance that would be a serious fairness concern. Our paper’s main qualitative claims (DendriCL scaling advantage at \(d \geq 30\); Transformer grokking at \(d=30\); architectural hierarchy at super-\(d\)) do not turn on this \(2\times\) range: the closest-in-parameter-count competitor to DendriCL is DendriCL (\(L=2\)), also ours; the next-closest is Pure LIF (\(0.64\)M) which has \(16\%\) fewer parameters yet fails much more severely at \(d \geq 20\).

22 Extended Tasks: Discussion of Non-Linear Regression↩︎

22.0.0.1 Results and diagnosis.

On the 2-layer ReLU NN regression task (\(d{=}20, k{=}40\)), DendriCL (\(L{=}2\)) reaches \(R^2 = 0.568 \pm 0.015\), comparable to Spikformer (\(0.582\)) and above Pure LIF (\(0.322\)). On binary classification (\(d{=}10, k{=}20\)), DendriCL (\(L{=}2\)) reaches \(0.805 \pm 0.008\), closely matching Spikformer (\(0.828\)) and Pure LIF (\(0.814\)). Both are substantially below DendriCL’s linear-regression performance (\(R^2 = 0.82\) at \(d{=}20\)). This is expected, not surprising: the apical compartment by construction implements a linear online estimator. Its inductive bias is strongly aligned with the linear-regression task family and only weakly aligned with non-linear target families.

22.0.0.2 Why we still include these tasks.

Two reasons. First, they falsify a trivial explanation: if DendriCL were simply a powerful architecture in general, we would expect it to dominate Spikformer on classification too — but it does not. This is evidence that the apical’s contribution is task-specific (to linear-regression-like structure), not architecture-general. Second, non-linear task performance matching Spikformer (not obviously better, not obviously worse) shows that the compartmental architecture does not hurt on non-linear tasks — the compartmental inductive bias is inert, not actively counterproductive, on targets outside its alignment.

22.0.0.3 Natural non-linear extensions.

Several architectural modifications could extend the apical compartment to non-linear target families, each a concrete project for Paper 2 of our research roadmap:

(i) Kernel apical. Replace the linear drive \(W_A \boldsymbol{x}_t\) with a learned non-linear transformation \(\phi(\boldsymbol{x}_t)\), e.g.\(\phi = \mathrm{ReLU}(W_1 \boldsymbol{x}_t)\) or a random-feature kernel [50]. The apical recurrence then implements online LMS in the lifted feature space — equivalent to kernel ridge regression via kernel Widrow-Hoff. If \(\phi\) includes the true non-linearity, the resulting estimator is Bayes-optimal; if not, it is the best linear approximation in \(\phi\)-space.

(ii) Multi-branch apical. Partition the apical compartment into \(K\) parallel branches, each with its own \(W_A^{(k)}\) and its own error signal, and combine via a learned gate. This is closer to biological pyramidal morphology (many apical sub-branches [24]) and could learn a piecewise-linear approximation of non-linear targets via specialization. We sketch this in our multi-paper roadmap as "Dendritic Universal Approximation" (Paper 2b).

(iii) Nonlinear readout. Keep the apical linear but add a small non-linear readout \(g(\boldsymbol{u}_A^\top W_A \boldsymbol{x}_q)\) on top. This is cheap and empirically adequate for many tasks but loses the mechanistic clarity of the linear apical.

22.0.0.4 Why we did not pursue these extensions in this paper.

This paper’s core claim is specific: a linear compartmental architecture is sufficient for the classical Garg-2022 ICL benchmark, and the mechanism it exploits is a known online-learning algorithm (LMS) rather than an opaque attention-weight configuration. Extending to non-linear targets is a different claim with its own set of comparisons required (against kernel methods, against explicit non-linear ICL, etc.). We prefer to make the narrow claim cleanly here than a broader but weaker claim.

23 Notation summary and implementation notes↩︎

The full per-unit DendriCL recurrence is given inline in §3 (Eq. 1). This appendix collects implementation notes that did not fit in the main body:

• Surrogate gradient. Backpropagation through the spike threshold \(\mathbb{1}[v > \theta]\) uses the arctan surrogate [33]: \(\partial s/\partial v \approx (1/\pi)\big/(1 + (\pi(v-\theta))^2)\). This is standard for direct-trained SNNs and applies to both the somatic and post-LIF block spikes.

• Reparameterizations. \(\alpha = \sigma(\tilde{\alpha})\) via sigmoid to enforce \(\alpha \in (0,1)\); \(\gamma = \mathrm{softplus}(\tilde{\gamma})\) then clamped to \([10^{-3}, 0.2]\) to keep the LMS update inside its stability region \(0 < \gamma < 2/(d+2)\) throughout training. \(g_A, g_B \in \mathbb{R}\) unconstrained.

• Initialization. \(\boldsymbol{u}_A(0) = \mathbf{0}\), \(W_A, W_B \sim \mathcal{N}(0, 1/\sqrt{d})\) Kaiming-style, \(W_{A,\text{out}} \sim \mathcal{N}(0, 1/\sqrt{d_{\text{apical}}})\). \(\tilde{\alpha} = 2.2\) (so \(\alpha \approx 0.9\) at start), \(\tilde{\gamma} = 0\) (so \(\gamma \approx 0.1\) at start).

• Post-LIF block. \(\text{FF}_1\) and \(\text{FF}_2\) are linear layers of width \(d_{\text{model}}\); the intermediate LIF uses the same \(\theta\). This block adds capacity for nonlinear feature mixing on top of the apical online-LMS estimate but contributes no learning-on-the-fly mechanism of its own.

24 Additional Figures↩︎

References↩︎

---

1. Corresponding Author↩︎