1 Introduction↩︎

1.0.0.1 Motivation.

This project started with a musical observation. When producing a track, you hear “something is off” before you can say what. That perception works across timescales at once: the note, the phrase, the harmonic arc of the piece. Standard language models have no equivalent. A Transformer processes all token pairs with equal weight regardless of distance, and produces no internal signal when its output is temporally inconsistent. The question behind this work: does making a model explicitly aware of multiple timescales improve its language modeling, and by how much?

1.0.0.2 The computational problem.

Self-attention requires \(O(L^2)\) computation per layer. At \(L = 32{,}768\), attention costs \(1{,}024\times\) more per token than at \(L = 1{,}024\). State space models (SSMs) compute in \(O(L)\), but prior work has not consistently shown quality advantages over Transformers under fair equal-budget comparisons at long context.

1.0.0.3 This paper.

We show that a three-level SSM hierarchy with multi-timescale recurrence outperforms both Transformer and Mamba baselines at every tested sequence length from 1K to 32K (Figure 2). The H–TF gap grows from \(+1.4\%\) to \(+11.4\%\) with context; the H–Mamba gap is smaller (0.7–1.8%) but consistent. At 64K tokens, both baselines run out of memory on an H100 80GB; Harmonic trains successfully. The long-context quality advantage holds across 7M–100M parameters and replicates on WikiText-103 (H–TF gap: \(+1.7\%\) at 1K to \(+7.2\%\) at 32K). At short context (1K, \(\geq\)​100M params) the Transformer wins; we report these results without qualification.

At 1B parameter scale (Section 5), we transplant the HarmonicBlock SSM module into TinyLlama 1.1B by replacing all 22 attention layers. The resulting model (Hallamonic) has no positional encoding limit and maintains stable loss at seq=1K, 4K, and 8K on independent benchmarks, while the unmodified TinyLlama degrades by \(+9\)–\(10\) bpt past its 2K RoPE limit.

2 Related Work↩︎

2.0.0.1 Linear and sub-quadratic sequence models.

S4 [1] introduced structured state space models for long-range sequence modeling. Mamba [2] added input-selective state transitions. RWKV [3] recasts attention in RNN form. Griffin [4] mixes gated linear recurrences with local attention. H3 [5] combines SSMs with a small attention component for associative recall. These models achieve competitive perplexity but most published comparisons against Transformers use unequal token budgets or different tuning, making it hard to isolate architectural effects.

2.0.0.2 Hierarchical recurrence.

Clockwork RNNs [6] and hierarchical multiscale RNNs [7] showed that multiple temporal scales can be handled by separate recurrent modules. Harmonic applies this idea to SSMs with a structured timescale hierarchy and prediction-error inter-level signals.

2.0.0.3 Predictive coding.

In predictive coding [8] each level predicts the activity of the level below; only errors propagate upward. Neural network applications include video prediction [9] and contrastive representation learning [10]. Harmonic uses prediction-error signals for inter-level communication; the timescale hierarchy rather than the prediction errors accounts for the performance advantage (Section 4.5).

2.0.0.4 Attention-free large language models.

Recent work has explored replacing attention in pretrained LLMs. RWKV-7 achieves competitive performance by converting attention to linear recurrence post-training. Mamba-based models at 1B–7B scale have been trained from scratch but require significantly more compute budget than fine-tuning. We take a direct approach: warm-start from TinyLlama’s pretrained FFN and embedding weights, train only the new SSM layers for a small fraction of the original budget, and evaluate whether the resulting model removes the positional encoding constraint without retraining from scratch.

2.0.0.5 Comparison protocol.

We use a strict equal-budget protocol: same dataset, tokenizer, optimizer, schedule, gradient clipping, and total training tokens across all models and sequence lengths. This is the minimum condition for attributing performance differences to architecture.

3 Harmonic Architecture↩︎

Harmonic consists of three stacked SSM levels with a shared embedding and a single linear head. Figure 1 shows the overall structure.

3.0.0.1 Timescale hierarchy.

Each level \(\ell \in \{1, 2, 3\}\) maintains a hidden state \(h_\ell \in \mathbb{R}^d\) updated by a learned recurrence: \[h_\ell(t) = A_\ell(t) \odot h_\ell(t-1) + b_\ell(t),\] where \(A_\ell(t) \in (0,1)^d\) is the data-dependent decay gate and \(b_\ell(t)\) is the input contribution. \(A_\ell(t)\) is computed from the current input token, making the decay rate input-selective rather than fixed — analogous to the selective mechanism in Mamba [2], but constrained by initialization to a level-specific timescale range \([\tau_\ell^{\min}, \tau_\ell^{\max}]\) with \(\tau_1 \ll \tau_2 \ll \tau_3\): level 1 decays fast (local context), level 3 decays slowly (long-range context). Input-selectivity allows the model to gate out irrelevant context without relying on hard-coded decay, which addresses the associative recall limitation of fixed-timescale recurrences.

3.0.0.2 Prediction-error inter-level signals.

Each level \(\ell\) produces a prediction of the level below via a learned linear map \(P_\ell : \mathbb{R}^d \to \mathbb{R}^d\). The prediction error \(e_\ell = h_{\ell-1} - P_\ell(h_\ell)\) is normalized and passed to level \(\ell{+}1\) as part of its input, instead of \(h_{\ell-1}\) directly. Higher levels therefore receive residual signals — what lower levels failed to predict — rather than raw activations. The projectors \(P_\ell\) are trained end-to-end through the top-level language modeling loss; no local reconstruction losses are used at intermediate levels. This design is structurally consistent with predictive coding [8], though as the ablation in Section 4.5 shows, passing raw states instead of errors produces indistinguishable results; the timescale hierarchy is the load-bearing component.

3.0.0.3 Output and training.

The outputs of all three levels are summed (\(h_1 + h_2 + h_3\)) and passed through a single linear language model head. The model is trained end-to-end with cross-entropy next-token prediction. No auxiliary losses are required; the inter-level signal structure emerges from gradient descent on the language modeling objective alone.

3.0.0.4 Gradient stability.

Unrolled recurrence over 32K–64K steps creates a deep computational graph. The normalization applied to error signals \(e_\ell\) before they enter the next level prevents gradient explosion and vanishing across the hierarchy. In practice, training is stable at all tested sequence lengths (1K–64K) without gradient clipping beyond the standard value of 1.0 shared with all baselines.

3.0.0.5 Complexity.

Because all three levels are recurrent SSMs with no attention, total computation is \(O(L \cdot d)\) per layer — linear in sequence length. The parallel scan algorithm [11] allows efficient GPU execution with full parallelism over the sequence dimension during training. Figure 4 shows the theoretical scaling comparison.

3.0.0.6 Implementation.

Experimental logs are available at https://github.com/Omibranch/harmonic-logs. All experiments use a custom Triton kernel for the SSM scan and torch.compile for the Transformer baseline, ensuring neither model is disadvantaged by implementation quality.

4 Experiments↩︎

4.1 Experimental Setup↩︎

4.1.0.1 Fair comparison protocol.

All comparisons use identical training configurations: same dataset, same tokenizer (GPT-2 BPE), same optimizer (AdamW, \(\beta=(0.9, 0.95)\), weight decay 0.1), same cosine learning rate schedule (\(3\times10^{-4}\) peak, \(3\times10^{-5}\) minimum), same gradient clipping (1.0), and the same total number of training tokens. For the crossover study, each (model, sequence length) pair receives exactly \(65.5\)M training tokens; for \(L=32{,}768\) the budget is \(131\)M tokens (largest feasible single-batch configuration). All runs use the enwiki8 byte-level dataset for the primary crossover and scaling experiments, WikiText-103 [12] for cross-dataset validation, and WikiText-2 for the ablation study.

4.1.0.2 Models.

Our primary comparison is between Harmonic and a standard Transformer with FlashAttention [13] (causal, 4 layers, 4 heads). Both models have approximately 28M parameters at the default hidden dimension \(d=256\). For the scaling study we vary \(d \in \{128, 256, 512\}\), corresponding to approximately 7M, 28M, and 112M parameters.

4.1.0.3 Hardware.

All experiments run on NVIDIA H100 80GB GPUs via Modal cloud.

4.2 Crossover Study: Quality vs.Sequence Length↩︎

Figure 2 shows the main result. At every tested sequence length, Harmonic achieves lower validation loss than the Transformer. More importantly, the advantage grows consistently with sequence length:

OOM: CUDA out of memory on H100 80GB (Mamba and Transformer both exceed 80GB at seq=65,536 during training; Harmonic succeeds due to \(O(L)\)

memory).

Harmonic outperforms both Mamba and Transformer at every tested length, with the H–TF gap growing from \(+1.4\%\) at 1K to \(+11.4\%\) at 32K tokens. The H–Mamba gap is smaller and roughly stable (0.7–1.8%), while the Mamba–TF gap also grows from \(+0.7\%\) to \(+10.0\%\). This separates two effects: the advantage of \(O(L)\) recurrence over \(O(L^2)\) attention (shared by both SSMs), and the additional advantage of hierarchical timescales over flat-state SSMs (Harmonic only).

4.2.0.1 Statistical robustness.

To confirm that the headline advantage at seq=8,192 is not a single-seed artifact, we ran 5 independent seeds (\(\{42, 7, 11, 99, 123\}\)) at this length. Harmonic achieves \(6.515 \pm 0.163\) bpt; Transformer achieves \(7.009 \pm 0.159\) bpt; Mamba achieves \(6.575 \pm 0.155\) bpt (all mean \(\pm\) std, \(n{=}5\)). The \(7.1\%\) H–TF gap and the ordering Harmonic \(<\) Mamba \(<\) Transformer hold across all five seeds. The \(0.9\%\) H–Mamba gap is smaller relative to the within-seed variance and should be treated with corresponding caution.

Harmonic’s absolute loss improves with context length (6.571 at 1K \(\to\) 6.196 at 16K), while the Transformer degrades (6.662 \(\to\) 6.873). Mamba improves similarly to Harmonic (6.616 \(\to\) 6.286 at 16K), suggesting that \(O(L)\) recurrence in general benefits from longer context, and hierarchical timescales provide a further consistent gain on top.

4.3 Cross-Dataset Validation: WikiText-103↩︎

To verify that the results are not an artifact of the enwiki8 byte-level encoding, we replicate the crossover study on WikiText-103 [12], the standard benchmark used by Mamba [2], S4 [1], and H3 [5]. All training settings are identical to the enwiki8 protocol (equal token budgets, same hyperparameters, same H100 hardware).

The results replicate the enwiki8 findings. Harmonic outperforms both baselines at every tested length, and the H–TF gap grows from \(+1.7\%\) at 1K to \(+7.2\%\) at 32K. Absolute bpt values are higher on WT103 (harder distribution; word-level tokenization), but the relative advantage pattern is consistent across both datasets. Mamba again falls between Harmonic and Transformer at all lengths, with an H–Mamba gap of 0.9–2.1%.

4.4 Scaling Study: Quality vs.Model Size↩︎

At seq=8,192, Harmonic wins by 8.0%, 8.0%, 7.0%, and 6.6% at 7M, 28M, 112M, and 100M parameters respectively. The long-context advantage holds across all tested scales. At seq=32,768 and 100M parameters, Harmonic wins by 11.8%.

At seq=1,024 the picture depends on scale: Harmonic wins at 7M and 28M parameters, while the Transformer wins at \(\sim\)​100M parameters (\(-3.2\%\)) and \(\sim\)​112M parameters (\(-1.5\%\)). At smaller scales, full self-attention over 1K tokens is insufficient to close the recurrence gap. At larger scales, the associative recall properties of attention become useful enough to flip the outcome at short context. The long-context advantage reverses this: at 8K tokens and every tested scale, Harmonic wins by 6.6–8.0%.

4.5 Ablation Study: Sources of Gain↩︎

To identify which components of Harmonic contribute most to its performance, we run ablations on WikiText-2 with equal token budgets:

• Harmonic: full model with hierarchical timescales and predictive coding between levels.

• Mamba: selective SSM baseline [2] (our implementation, matched parameter count).

• Flat timescales: same architecture as Harmonic but all three levels use identical timescale ranges, removing the hierarchy.

Removing the timescale hierarchy (flat variant) costs 0.501 bpt: 8.178 \(\to\) 8.679. Stacking three SSM levels at the same timescale does not replicate the full model’s behavior. The Mamba baseline (8.290) falls between the two Harmonic variants; selective state transitions alone do not account for the full gap.

We also tested a variant without prediction-error inter-level signals (NoPred), passing raw hidden states \(h_\ell\) between levels instead of errors \(e_\ell\). The difference was \(\leq\)​0.022 bpt across all tested sequence lengths (1K–32K), indistinguishable from noise. The timescale hierarchy is the source of Harmonic’s advantage; the prediction-error signal is an architectural choice with no measurable effect on validation loss.

4.6 Stateful Inference: Streaming Long Documents↩︎

A recurrent SSM can process arbitrarily long sequences by carrying its hidden state across chunk boundaries — no recomputation of prior context is needed. A Transformer cannot do this: attention requires access to all prior key-value pairs, and processing a document in chunks necessarily discards cross-chunk dependencies.

We test whether Harmonic benefits from state carry in practice. We train a stateful variant using sequential batching: the corpus is divided into \(B\) parallel streams, and the raw SSM state \(h^{\text{raw}}[T-1]\) (pre-LayerNorm, not the normalized output) is carried from the end of each chunk to the start of the next. The model is warm-started from a normally-trained Harmonic checkpoint and fine-tuned for half the original step budget.

Three findings emerge. First, stateful fine-tuning consistently improves both architectures: \(+4.6\%\) for Harmonic and \(+4.4\%\) for Mamba at seq=1,024; \(+1.4\%\) and \(+1.8\%\) at seq=8,192. The smaller gain at seq=8,192 is expected: with 8K-token chunks, within-chunk context already captures most long-range dependencies, leaving less room for cross-chunk state carry to help.

Second, inference-only stateful — loading a normally-trained checkpoint into stateful mode without fine-tuning — yields only marginal gain at seq=1,024 and actually hurts at seq=8,192 (\(+0.17\) bpt). The model must be explicitly trained to use carried state; untrained state carry is noise.

Third, the relative advantage between architectures reverses with sequence length. At seq=1,024, Harmonic stateful (6.582) outperforms Mamba stateful (6.646) by 0.064 bpt. At seq=8,192, Mamba stateful (6.838) outperforms Harmonic stateful (6.745) by 0.093 bpt. We attribute this to the timescale hierarchy: Harmonic’s \(\tau = [4, 32, 128]\) levels already provide structured long-range integration within each 8K chunk, reducing marginal benefit from cross-chunk state carry. Mamba’s flat single-level state has less within-chunk long-range capacity and therefore benefits more from explicit state persistence.

4.7 Throughput and Efficiency↩︎

Figure 4 illustrates the theoretical compute gap. At \(L=8{,}192\), the Transformer requires \(8\times\) more compute per token for attention than Harmonic requires for recurrence. At \(L=32{,}768\) this grows to \(32\times\). In practice, FlashAttention reduces memory bandwidth cost but does not change the asymptotic complexity; the measured throughput results in Figure 4 (a) reflect real H100 performance.

5 Hallamonic: HarmonicBlock at 1B Scale↩︎

The small-scale experiments above compare equally-sized models trained from scratch. A complementary question is whether the HarmonicBlock module — the single-layer SSM component used in Harmonic — can replace attention in an existing pretrained LLM without retraining from scratch. This tests a different property: architectural compatibility and the practical utility of removing the positional encoding constraint at inference scale.

5.0.0.1 Setup.

We take TinyLlama 1.1B [14] as the base model. TinyLlama uses Rotary Position Embeddings (RoPE) with max_position_embeddings=2048, which causes catastrophic performance degradation for sequences longer than 2K tokens. We replace all 22 LlamaAttention layers with HarmonicBlock (the same SSM module used throughout this paper, with \(d_{\text{state}}=128\), compress ratio \(K=4\)), keeping FFN, embeddings, and layer norms frozen. The resulting model has 1,033M total parameters: 892M pretrained (FFN + embeddings) and 141M newly initialized (HarmonicBlock weights).

Training proceeds in two phases on fineweb-edu [15] (sample-10BT, educational web content):

• Phase 1 (SSM warmup): 10K steps, seq=512, batch=4. FFN frozen; only HarmonicBlock weights trained. Learning rate \(3\times10^{-4}\), cosine decay.

• Phase 2 (full finetune): 5K steps, seq=1,024, batch=8, gradient accumulation=4 (effective batch \(\approx\)​33K tokens/step). All parameters trained. Learning rate \(3\times10^{-5}\), cosine decay to \(3\times10^{-6}\).

Total compute: approximately $15 on Modal H100.

5.0.0.2 Results.

Table 6 and Figures 5–6 show the results. TinyLlama’s loss degrades sharply past seq=2K on every dataset, consistent with RoPE’s hard positional limit. Hallamonic shows no such degradation: loss at seq=8K is within 0.02–0.04 bpt of its seq=1K value on all three datasets.

5.0.0.3 Verification.

To rule out implementation artifacts, we performed three independent checks: (1) manual cross-entropy computation from raw logits matched model.loss to \(10^{-6}\) nats; (2) the Triton parallel scan and the sequential PyTorch fallback produced identical results (diff \(= 0.0002\) bpt); (3) on a sequence of uniformly random tokens, Hallamonic outputs \(3.83\) bpt while TinyLlama outputs \(17.8\) bpt (expected: \(\log_2 32000 \approx 15\) bpt), confirming that the model is context-sensitive and not producing uniformly low loss regardless of input.

5.0.0.4 Why is Hallamonic’s absolute bpt low?

The absolute values (\(0.36\)–\(0.48\) bpt) are lower than typical 1B LM benchmarks. Two factors contribute. First, Hallamonic inherits TinyLlama’s pretrained FFN and embedding weights (892M parameters, trained on 3T tokens), which encode strong language priors that persist through fine-tuning. Second, SSMs evaluated on coherent single documents benefit from the hidden state accumulating document-level context progressively, whereas training used random cross-document slices starting with a cold SSM state. This eval–train discrepancy inflates eval performance relative to training loss but is not an artifact of the measurement. The key result — growing TinyLlama degradation at long context versus Hallamonic stability — is architectural and robust.

5.0.0.5 Cost and implications.

The full experiment (two training phases plus evaluation) cost approximately $15 on a single H100 and took under 3 hours. This suggests that HarmonicBlock is a practical drop-in replacement for attention in pretrained Transformer-based LLMs when the primary concern is removing positional encoding limits for long-context inference, without incurring the cost of full retraining from scratch.

6 Discussion↩︎

6.0.0.1 Why does the gap grow with \(L\)?

As \(L\) doubles, the Transformer spends twice as many FLOPs on attention per unit of useful signal. Under equal token budgets, this means fewer effective training steps or shallower layers at long context. Harmonic’s compute stays \(O(L)\), so longer sequences cost proportionally more without this attention overhead. The \(\sim\)​2 pp gap per doubling is consistent with this explanation.

6.0.0.2 Short-context exception.

At 112M parameters and 1K tokens, the Transformer wins by 1.5%. Self-attention can compute all pairwise relationships at this length at reasonable cost, and its associative recall properties are useful for dense short sequences. We do not claim Harmonic is universally better; the data says it is better at long context.

6.0.0.3 Memory wall at 64K.

At seq=65,536 both Mamba and the Transformer exceed the 80GB H100 memory limit during training and fail with CUDA out-of-memory errors. Harmonic completes successfully and reaches 6.169 bpt — better than its 32K result (6.433 bpt), consistent with the general trend that longer context helps \(O(L)\) recurrence models. This is not a benchmark result in isolation; it is a direct consequence of architecture. Mamba’s state is \(O(1)\) per position in inference, but during training the unrolled computation and optimizer states push its memory footprint above 80GB at this length. Harmonic has the same \(O(L)\) forward pass structure but a smaller per-position footprint, and remains inside the budget.

6.0.0.4 Stateful inference.

During inference, Harmonic maintains a fixed-size state vector regardless of sequence length: \(O(1)\) memory per generated token. Transformers need a KV cache that grows linearly. For very long contexts this is a practical difference, though we did not benchmark inference latency in this work.

6.0.0.5 1B scale takeaway.

The Hallamonic result adds a dimension not present in the small-scale crossover study: it shows that HarmonicBlock is compatible with a pretrained 1B-parameter LLM backbone, can be integrated for $15 in compute, and removes the positional encoding bottleneck that limits TinyLlama to 2K-token contexts. This is not a claim about overall quality relative to other 1B models; it is a demonstration that the \(O(L)\) long-context property scales to production-scale architectures via fine-tuning.

6.0.0.6 Limitations.

Small-scale experiments cover up to 100M–112M parameters and 64K tokens on English text (enwiki8, WikiText-103, WikiText-2). The long-context advantage is consistent across both enwiki8 and WikiText-103; whether it holds at larger parameter scales is unknown. The 1B Hallamonic experiment uses a single training run without multi-seed replication. Inference throughput and behavior on non-English or non-text modalities are not evaluated.

7 Conclusion↩︎

Harmonic is a three-level SSM with hierarchical timescales. On enwiki8 with equal token budgets, it outperforms a comparable Transformer (28M params) at every tested sequence length from 1K to 32K tokens, with the quality gap growing by \(\sim\)​2 percentage points per doubling of \(L\). The advantage is consistent across 7M–100M parameters at long context and replicates on WikiText-103, the standard benchmark used by Mamba and S4. At 64K tokens, Harmonic is the only tested model that fits in 80GB H100 memory, reaching 6.169 bpt. Compute scales as \(O(L)\).

At 1B parameter scale, replacing all attention layers in TinyLlama 1.1B with HarmonicBlock produces a model that maintains stable loss from 1K to 8K tokens across three independent evaluation datasets, while the unmodified TinyLlama degrades by \(+9\)–\(10\) bpt past its 2K RoPE limit. The transplant costs $15 in compute.

The cases where the Transformer wins are at short context with larger models: 1K tokens at 100M params (\(-3.2\%\)) and at 112M params (\(-1.5\%\)). Both are reported without qualification in the main results table.

The intuition behind this design — that useful context structure spans multiple timescales simultaneously — has a measurable effect at both 28M and 1B parameter scales. Whether it holds further is a question for future work.

8 HarmonicSNN: Spiking Neural Network Variant↩︎

We investigate whether the Harmonic architecture is compatible with spiking compute primitives. In the SNN variant (HarmonicSNN), each level’s hidden state update is replaced by a Leaky Integrate-and-Fire (LIF) neuron: \[V[t] = \beta V[t-1] + I[t] - \theta \cdot s[t], \qquad s[t] = \mathbf{1}[V[t] \geq \theta],\] where \(\beta = 0.992\) is the membrane decay, \(\theta = 1.0\) is the firing threshold, and \(I[t]\) is the input current from the SSM scan. Gradients flow through the binary spike \(s[t]\) via a soft sigmoid surrogate with slope \(k{=}3\): \(\hat{s} = \sigma(3(V{-}\theta))\). The slope choice is derived from the BPTT stability condition: the recurrence multiplier \(\beta(1 - \theta k/4) < 1\) requires \(k < 4/\theta = 4\); we use \(k{=}3\) for a safety margin.

The scan loop is compiled with torch.jit.script (TorchScript), which gives a 3\(\times\) speedup over an interpreted Python loop (0.03 vs. M tok/s). torch.compile hangs during inductor cycle-detection on the \(T{=}1024\) loop and is not viable for this architecture.

8.0.0.1 Results.

Training on enwiki8, seq=1024, 5000 steps, hidden=256 (27M parameters):

HarmonicSNN matches Harmonic SSM within noise on an equal token budget, with average spike rates of \(\sim\)​10% across layers — biologically plausible values consistent with cortical recordings. The result demonstrates that the hierarchical SSM architecture is not tightly coupled to continuous-valued hidden states; binary spikes are a drop-in replacement with no loss in perplexity.

This is relevant for neuromorphic hardware deployment: on sparse-activation accelerators (Intel Loihi, IBM TrueNorth), spike-based computation has \(O(\text{spikes})\) cost rather than \(O(n)\), yielding a theoretical 10\(\times\) reduction in multiply-accumulate operations at the observed 10% firing rate. We do not evaluate on neuromorphic hardware in this work; we report the result as evidence of architectural modularity.

References↩︎