1 Introduction↩︎

Most interpretability reconstructs meaning after training: a dense activation is probed, decomposed with a sparse autoencoder, or decoded through the vocabulary, and a human names what was found. An alternative is to build the meaning in — to make a model’s operators legible by construction, so that a coordinate carries a fixed, stated reading rather than one recovered post hoc. Two prior constructions take this route: the feed-forward layer becomes fuzzy set operations on bounded operands [1], and the attention value becomes bounded, named detectors [2], each at language-model parity (Figure 1). But a legible-by-construction substrate is not yet a legible model. Its operators read cleanly only if training keeps them so; the two constructions have never been trained together; and reading such a model, and editing it, are problems of their own. This paper is about making that line usable end to end — how to train the model so its units are actually legible, how to read them, how to edit them, and how a single further pressure reshapes the circuits they form.

The obstacle is the very pressure legibility depends on. A bounded operator reads cleanly only when it fires decisively — near \(0\) or near \(1\), “absent” or “present” — so both constructions add a crispness penalty that pushes each value to a rail. Pushed too hard, that penalty does the opposite of its purpose: it drives a unit to a fixed value that ignores its input, a dead constant, perfectly crisp and carrying no information, and quality collapses with it. The cause is exact. For a bounded value the crispness penalty equals \(\mu(1-\mu)-\mathrm{var}\), a variance-minimizer that cannot tell a live contextual detector from a dead constant, so gradient descent takes the cheaper constant. Naming the cause gives the cure: a per-channel variance floor — the property that separates a live detector from a dead one, written directly as a loss. It adds one term and no parameters, and turns a pressure that ruins the model into one that recovers both legibility and quality. The same discipline retires a second hand-set dial. Prior work reserved a fixed fraction of feed-forward units as ordinary \(\ifmmode\operatorname{GELU}\else\mathrm{\small gelu}\fi\) for trainability; a per-unit learned gate turns that reservation into a decision the model makes, and given the choice it keeps no unit as pure \(\ifmmode\operatorname{GELU}\else\mathrm{\small gelu}\fi\) and routes \(87\%\) of its load-bearing computation through the legible operators. The result is the most legible transformer we have built, and the first legible on both sublayers at once: \(78\%\) of its feed-forward operands and \(50\%\) of its attention value channels are crisp, contextual detectors.

A legible model can be read, but reading it well takes care. A unit’s write is illegible through the raw vocabulary projection in the middle of the network, where the residual sits in a rotated frame; a per-layer tuned lens rotates it back, and the abstract computation behind a completion becomes visible several layers earlier than the raw lens shows. Reading in that frame separates two properties the construction lets us measure apart: a unit’s detection — the clean, input-dependent bit it responds to — from its naming — whether what it writes decodes to a word. The model detects far more cleanly than it names: most feed-forward operands are selective detectors, but only about half decode to a vocabulary category, and the concepts it cannot name are the abstract ones it carries non-lexically through mid-stack. Those are also the concepts it distributes: token-level features live on single units, while abstract concepts are spread across many and grow more distributed with depth — the superposition of ordinary networks, now visible against a legible backdrop.

Because the objective makes each unit crisp and sparse, the standard tools for editing a transformer — difference-of-means writes, activation steering — apply to it better. A crisp, sparse unit is a clean site: an edit placed there stays local, \(184\times\) more local in the deep layers where the editable sites concentrate, and needs no search to find. And the named set operations make possible an edit no single neuron can express — a conjunction \(A\cap B\) written at one addressable, two-operand unit. Editing borrows its mechanism from prior work; what the objective supplies is the quality of the site.

Reading the model’s circuits shows how it wires a concept: redundantly, a few operators broadcast to many readers — a fan-out circuit, efficient but hard to touch, since editing one concept means altering every copy and disturbing every reader. A single further pressure reshapes it. A between-unit decorrelation penalty — forbidding units to co-fire, and so forbidding the shared building blocks that reuse depends on — trades the fan-out circuit for a fan-in one: many independent units, each aggregating a wide, disjoint input, at no cost to quality. The trade buys editability directly — a concept that lived in a \(562\)-unit tangle becomes one addressable unit — and legibility of a whole prediction: a token’s attribution, spread across a thousand small pieces in an ordinary transformer, collapses onto a short list of named operations one can read as a sentence (“than” because a comparative unit fired and a head read the comparative adjective). What it spends is the compression reuse provides, a cost on out-of-distribution composition we do not yet measure.

Through all of this the through-line is that the contribution is legibility, not a different machine or a better score. The structure these tools read off — where a concept lives, how redundantly it is built, how widely it is read — is structure a conventional transformer plausibly shares, and every measurement is defined on activations and weights a \(\ifmmode\operatorname{GELU}\else\mathrm{\small gelu}\fi\) model has too; what the construction adds is that all of it can be read directly, without a post-hoc dictionary that must be trained, chosen, and re-validated. Quality holds at parity with a conventional baseline throughout.

Contributions.

• A legibility objective. A per-channel variance floor — the legibility metric written as a loss — repairs the collapse the crispness penalty otherwise causes, recovering both legibility and quality; the \(\mu(1-\mu)-\mathrm{var}\) identity names why the naive penalty fails (Section 4).

• An end-to-end legible transformer. The feed-forward and attention constructions are trained together for the first time, and a learned per-unit gate retires the hand-set reserved-\(\ifmmode\operatorname{GELU}\else\mathrm{\small gelu}\fi\) partition, yielding a model legible on both sublayers at parity (Sections 3–4).

• Reading the units. Read in each layer’s rotated frame, the units separate a clean detection from a harder naming; the model detects far more than it names, and distributes exactly the concepts it cannot name (Section 5).

• Editing the units. Crisp, sparse units are surgical edit sites — far more local than a conventional neuron — and support a single-unit conjunctive \(A\cap B\) edit no neuron can express (Section 6).

• A legibility dial. A between-unit decorrelation pressure trades a circuit’s reuse (fan-out) for independence (fan-in) at no quality cost, making concepts surgically editable and a whole prediction explainable as a short list of named operations (Section 7).

2 Background and Related Work↩︎

Our work sits at the intersection of five lines: interpretable-by-construction model components, training objectives for interpretability, vocabulary-space readout of hidden states, superposition and its consequences, and model editing. We build directly on prior results in each.

2.1 Interpretable-by-construction components↩︎

A body of work replaces a standard component with one whose computation is legible by design, rather than recovering structure after training. Softmax Linear Units raise the fraction of first-glance-interpretable MLP neurons at little cost, while noting that such changes can also hide features [3]. Bilinear MLPs remove the elementwise nonlinearity so that interaction structure is recoverable from the weights alone, at language-model parity [4]. Codebook Features quantize the residual stream into a small set of discrete codes [5], and Backpack models express each token as a nonnegative combination of linearly decodable sense vectors, enabling predictable intervention at GPT-2 parity [6]. Kolmogorov–Arnold Networks expose learned univariate functions on edges [7]. Our multiplicative operators are, mechanically, gated bilinear units in the GLU family [8]; what is specific is that both operands are sigmoid-bounded to \([0,1]\) and read as fuzzy set memberships, with an explicit set-difference term as a first-class unit.

A parallel neuro-symbolic literature realizes logical connectives as differentiable layers: Logic Tensor Networks [9], Logical Neural Networks [10], the analysis of fuzzy t-norms under gradient descent [11], logic-structured language representations [12], tensor logic [13], and interpretable logic classifiers [14]. These ground externally specified formulae; our units instead carry emergent structure read out of a model trained only on next-token prediction. Post-training Boolean-operator calculi over frozen embeddings [15] share the vocabulary but are not architectural. The fuzzy-Boolean feed-forward units and bounded value heads themselves are introduced in our prior work [1], [2]; Section 3 recaps that construction so the present paper is self-contained. This paper’s contribution is not the operators but the objective that makes them crisp and alive, and the resulting read-and-edit story.

2.2 Training for interpretability, and objective-induced collapse↩︎

Inducing interpretable structure during training, rather than post hoc, has precedent: Concept Bottleneck Models align an intermediate layer to labeled concepts [16], Self-Explaining Neural Networks regularize explanations for stability [17], and end-to-end sparse dictionary learning folds a task term into an auxiliary dictionary [18]. Unlike concept bottlenecks we require no concept labels, and unlike dictionary methods the structure lives in the primary computation rather than a separate decoder. Our crispness pressure is related to soft binarization and straight-through quantization [19]–[21], but targets legible Boolean operators rather than compression.

The failure we diagnose and repair is a collapse of variance, connecting to the anti-collapse literature in self-supervised learning. VICReg prevents representational collapse with a per-dimension variance hinge \(\mathop{\mathrm{ReLU}}(\gamma-\mathrm{std})\) paired with covariance decorrelation [22]; related mechanisms include redundancy reduction [23], feature whitening [24], and the predictor/stop-gradient schemes that motivated explicit variance terms [25], [26]. Our variance floor shares VICReg’s hinge shape. The difference is what it repairs: VICReg substitutes for contrastive negatives against an invariance loss, whereas our floor counteracts a collapse induced by our own crispness term, which the identity \(\mathrm{mean}\,v(1-v)=\mu(1-\mu)-\mathrm{var}\) reveals to be a variance-minimizer that cannot distinguish a live contextual operator from a dead constant. The resulting dead unit is the crispness-induced analogue of the classical dying-ReLU state [27]. Optimizing a crispness proxy that destroys the target it stands for is a specification-gaming pattern [28], [29], and the identity makes precise why a variance floor is the minimal correction.

2.3 Reading hidden states in vocabulary space↩︎

Projecting intermediate states through the unembedding — the logit lens [30] — is a standard readout, with a well-documented failure at middle layers. The tuned lens attributes this to representations living in a rotated basis and recovers predictions with a learned per-layer affine map, framed as iterative inference [31]; the iterative-refinement view of residual networks is older still [32], [33]. Feed-forward writes decode to vocabulary as key-value memories that promote concepts [34], [35], and per-component logit attribution follows from the residual stream being a linear channel with no privileged basis [36], a basis that is nonetheless weakly privileged in practice [37]. Learned lenses have been applied at the granularity of individual attention heads [38], whole parameter matrices [39], and future tokens [40]. We use these established lenses; our reading results are their application to legible-by-construction units and the detection-versus-naming distinction of Section 5, not a new readout.

2.4 Superposition, polysemanticity, and non-lexical features↩︎

Superposition explains why a crisp unit can fire on an incoherent set of tokens: networks pack more features than neurons as near-orthogonal directions, and polysemanticity is the direct consequence [41]. The polytope lens reframes this geometrically, arguing that meaning is monosemantic over activation regions even where directions are polysemantic [42] — the local-coherence view our per-input reads inherit. Sparse autoencoders recover largely monosemantic features from polysemantic neurons [43], [44], at scale surfacing abstract, multilingual, non-lexical concepts [45]–[47]. That concepts occupy directions not aligned to any token is predicted by the linear representation hypothesis, which separates the context and unembedding geometries [48], and is documented directly for function vectors that steer yet do not decode [49] and for the unexplained residual of dictionary methods [50], [51]. Whether a unit is nameable is operationalized by automated interpretability [52]–[55]. Against this backdrop our contribution is specific: because our units are crisp by construction, their per-input meaning is recoverable cheaply, and we report the layerwise trajectory by which a concept that is non-lexical in mid-stack becomes token-decodable only in the final layers.

2.5 Model editing and activation steering↩︎

The mechanism we use to edit is standard. Feed-forward layers are editable key-value memories [34], [35]; knowledge is localizable to individual neurons [56] and to mid-layer feed-forward sites whose layer-native value vector ROME and MEMIT solve for — not the token embedding [57], [58]. Activation steering adds a direction obtained as a difference of means over contrast contexts [59]–[61], sufficient to move behavior with little collateral [62]; behaviors also compose as task and function vectors [63]–[65], and causal mediation localizes them [66]–[68]. Our deep edits are difference-of-means writes in this native frame. Two results bound the contribution: interpretability is only a weak proxy for steering utility [69], and more monosemantic units incur less collateral [45], [70]. We differ in that our units are made crisp and sparse by construction rather than discovered, are edited in-weights rather than clamped at inference, and support a single-unit conjunctive \(A\cap B\) edit that a single-preactivation neuron cannot represent.

3 The Legible Substrate↩︎

The models we train are built from components that are legible by construction: their operators are bounded and named, so a coordinate carries a fixed meaning a human can state, rather than a dense activation whose meaning must be reconstructed after the fact. The feed-forward units are those of [1] and the attention value heads are those of [2]; we recap both here so the paper is self-contained, and refer to those works for the fuller development. Both share one principle: constrain what a unit detects, and leave what it writes free (Figure 1). This section states the operators, the pressures that shape them, and the metric by which we call a unit legible; Section 4 is about the objective that makes the whole thing hold together.

None

3.1 Legible feed-forward units↩︎

A conventional feed-forward layer computes \(\mathrm{FFN}(x)=W_o\,\ifmmode\operatorname{GELU}\else\mathrm{\small gelu}\fi(W_{\mathrm{in}}x)\), whose hidden units have no reason to be individually meaningful. The legible layer replaces the hidden activation with fuzzy set operations on two banks of sigmoid-bounded operands. From the input \(x\) it forms \[A=\sigma(W_a x)\in[0,1]^{m},\qquad B=\sigma(W_b x)\in[0,1]^{m},\] and constructs the hidden vector as the concatenation of a fuzzy intersection and a negation-capable set-difference, \[h \;=\; \big[\,\underbrace{A\odot B}_{\text{A \mathrm{\small and} B}}\;;\;\underbrace{A\odot(1-B)}_{\text{A \mathrm{\small and-not} B}}\,\big], \qquad \mathrm{FFN}(x)=W_o\, h . \label{eq:ffn}\tag{1}\] A first-block coordinate reads as “both \(A\) and \(B\) present,” a second-block coordinate as “\(A\) present, \(B\) absent”; the blocks are asymmetric since \(A\setminus B\neq B\setminus A\). Because \(W_a,W_b\) together match the size of the projection they replace, the layer is parameter-neutral. For trainability a fraction \(\rho\) of hidden units are kept as conventional \(\ifmmode\operatorname{GELU}\else\mathrm{\small gelu}\fi\) and only the remainder are set operators [1]; we write the named fraction as \(1-\rho\). A set-operator unit can become a \(\ifmmode\operatorname{GELU}\else\mathrm{\small gelu}\fi\) — pass its first operand’s pre-activation \(W_a x\) through the nonlinearity — but only at a \(50\%\) capacity cost: the second projection \(W_b\) then idles, so the resulting \(\ifmmode\operatorname{GELU}\else\mathrm{\small gelu}\fi\) carries twice the input-projection weight of a native \(\ifmmode\operatorname{GELU}\else\mathrm{\small gelu}\fi\) neuron. The reserved fraction \(\rho\) is thus a parameter tax as much as a dilution of the legible substrate, a cost Section 4.5 removes by learning the fraction per unit. The write \(W_o\) is left unconstrained: a unit is read on the detection side — which operands fired — not by decoding what it writes.

3.2 Legible attention value heads↩︎

The same principle applies to the attention value, the opaque half of a head [36]: the attention map already shows where a head reads, but the value it moves is a dense unconstrained vector. We constrain that value before the output projection \(W_O\) and leave \(W_O\) free. In the membership form a head’s value is simply bounded, \[V_h=\sigma(X W_v^h)\in[0,1]^{T\times d_h},\qquad O_h=A_h V_h, \label{eq:membership}\tag{2}\] so each coordinate is a fuzzy membership — the degree to which a named feature is present — and attention forms a fuzzy-weighted average of memberships over the attended context. In the Boolean form the value projection is split into two half-width banks \(A=\sigma(XW_a^h)\), \(B=\sigma(XW_b^h)\) and the value is the same set-operator construction as the FFN, \[V_h=\big[\,A\odot B\;;\;A\odot(1-B)\,\big]\in[0,1]^{T\times d_h}. \label{eq:boolean}\tag{3}\] Both forms are parameter-neutral. A layer may mix bounded heads with conventional ones; we use the fully-bounded (\(100\%\)) setting throughout, coupling Boolean attention (Eq. 3 ) to the legible FFN (Eq. 1 ) to make an end-to-end model in which both what a layer moves and what it combines are named operations. Constraining the value rather than the write is essential: pinning the post-\(W_O\) write to the vocabulary instead hands the head a gradient path to the unigram prior and collapses it to a context-independent bias [2].

3.3 Selectivity pressures↩︎

A bounded operator is not automatically legible. A readable detector should fire on a sparse, specific set of contexts, and it should fire decisively — near \(0\) or near \(1\) rather than in the ambiguous middle. These are two properties, imposed by two small penalties on the bounded operands (the atoms we want to name — \(A\) and \(B\) — not the combined output), added to the language-model loss. A sparsity pressure \[\mathcal{L}_{\mathrm{s}}=\lambda_{\mathrm{s}}\,\textstyle\sum_{t,c}|v_{t,c}|\] pushes each value toward \(0\); it has teeth only because the value is bounded (with a free \(W_O\), a penalty on an unbounded value is rescaled away). A crispness pressure \[\mathcal{L}_{\mathrm{c}}=\lambda_{\mathrm{c}}\,\textstyle\sum_{t,c} v_{t,c}(1-v_{t,c}) \label{eq:crisp}\tag{4}\] is maximal at \(\tfrac12\) and zero at the rails, driving each value decisively toward the nearer rail without preferring off. We use \(\lambda_{\mathrm{s}}=10^{-3}\), \(\lambda_{\mathrm{c}}=3\times10^{-3}\). Which combination is right is not universal — for a lone membership value sparsity is what carves out a selective detector, whereas a conjunction supplies its own sparsity and wants crispness alone [2]. The failure mode of the crispness pressure, and its repair, is the subject of Section 4.

3.4 When is a unit legible?↩︎

We call a bounded operand channel legible when it is both crisp and contextual, and we measure this on held-out text. Over a sample of contexts let a channel have per-token values with mean \(\mu_c\) and variance \(\mathrm{var}_c\). The channel is crisp when its values concentrate at the rails (the fraction within \(0.1\) of \(\{0,1\}\) exceeds one half), and contextual when it varies across inputs, \(\mathrm{var}_c\ge\tau\) with \(\tau=0.003\). This gives three states (Figure 2):

• selective — crisp and contextual: a decisive, input-dependent detector, the readable case;

• constant — crisp but \(\mathrm{var}_c<\tau\): decisive but input-independent, a dead constant;

• fuzzy — not crisp: a soft gate that never commits.

selective is the target state. It is a property of the detection side and is independent of any choice of readout basis. The distinction between selective and constant — both perfectly crisp, separated only by whether the unit still responds to its input — is exactly what a crispness pressure alone cannot see, as Section 4 shows.

One caveat travels with the count wherever we report it. The pressures of Section 3.3 settle the crisp population just above the variance floor — a typical channel’s across-context variance is only a small multiple of \(\tau\) — so the selective count is a hard threshold applied to mass sitting right at the cutoff. A small, transient drift in variance can carry many channels across the line, and the count can swing by tens of points between training checkpoints while the crispness underneath barely moves. We therefore report selective fractions only at convergence (end of epoch), and pair the count with the mean crisp fraction — the average share of a channel’s values within \(0.1\) of a rail — which carries the same crispness information without the threshold fragility and is stable across checkpoints (\({\approx}0.90\) on the annealed model). The crisp fraction is the continuous form of the crisp criterion; the selective count adds only the variance test that a single mid-training snapshot can misread. Section 7 takes this up directly.

3.5 Setup↩︎

All models are \(125\)M-parameter, \(12\)-layer, \(768\)-wide transformers with learned positional embeddings, trained for one epoch on the same open-web corpus and tokenizer, with identical optimizer and schedule; the conventional gelu model is architecturally identical except for the value transform and FFN. We report language-model quality with dev perplexity, LAMBADA [71], and BLiMP [72] via the standard harness [73], and legibility with the selective/constant statistics above. All results are single seed at this scale.

4 Training Legible Models↩︎

The crispness pressure of Eq. 4 is what turns a bounded operand into a decisive, readable detector. It also has a failure mode. Past a point it collapses the operators it was meant to sharpen into dead constants: units that are perfectly crisp and completely uninformative. We show why this happens with a two-line identity, repair it with a variance floor that turns the selective metric into a training objective, and give a learned-fraction variant that lets a unit revert to \(\ifmmode\operatorname{GELU}\else\mathrm{\small gelu}\fi\) where logic is not needed. The repair recovers both legibility and quality.

4.1 The collapse↩︎

Apply the crispness pressure to the feed-forward operands of the fully-Boolean model and, past a point, the operands go crisp but dead. On held-out text only \(2\%\) of feed-forward operands are selective while most are constant — crisp, but pinned to a rail regardless of input — and quality craters: LAMBADA perplexity rises to \(240\) against \(153\) for the same model with the feed-forward operands left unpressured. The damage does not stay local. Because the collapsed units still write into the shared residual stream, the attention operands degrade with them, their selective fraction falling from \(27\%\) to \(20\%\). A pressure meant to sharpen the operators instead switches them off.

4.2 The problem↩︎

For a bounded value \(v\in[0,1]\) with mean \(\mu\) and variance \(\mathrm{var}\) over a batch, the crispness penalty of Eq. 4 decomposes exactly: \[\mathbb{E}\big[v(1-v)\big] \;=\; \mu(1-\mu)-\mathrm{var}. \label{eq:identity}\tag{5}\] Since \(\mathrm{var}\le\mu(1-\mu)\) the quantity is nonnegative, and it reaches zero on the entire crisp manifold (Figure 3): every operator whose mass sits at the rails drives it to zero, whether it is a live contextual detector (\(\mu\) interior, \(\mathrm{var}>0\), mass split between the rails across inputs) or a dead constant (\(\mu\) at a rail, \(\mathrm{var}=0\)). The penalty cannot tell the two apart. Worse, the readout \(W_o\) is free, so crispness carries no obligation to be useful; it is minimized as a pure tax, and the cheapest way to pay it is a constant. Once there, the saturated sigmoid gives almost no gradient, and the dead state is absorbing. The crispness pressure is, by Eq. 5 , a variance-minimizer in disguise — and minimizing the variance of a detector is precisely how you kill it.

None

4.3 The fix: a variance floor↩︎

The identity also says how to fix it: forbid the zero-variance solution. We add a per-channel hinge that is positive when a channel’s variance is below the selective threshold and zero once it clears it, \[\mathcal{L}_{\mathrm{ctx}} \;=\; \lambda_{\mathrm{ctx}}\;\mathrm{mean}_c\;\mathop{\mathrm{ReLU}}\!\Big(1-\frac{\mathrm{var}_c}{\tau}\Big),\qquad \tau=0.003. \label{eq:ctx}\tag{6}\] This is the selective metric of Section 3.4 turned into a loss: paired with the crispness term, the unique zero-loss state is crisp and contextual, and a dead constant now pays \(\lambda_{\mathrm{ctx}}\). Three properties matter. It adds no parameters. It floors variance, not mean or firing rate, which is what protects a rare-but-real detector — a channel active on \(1\%\) of tokens has \(\mathrm{var}\approx0.0099>\tau\) and is left alone, whereas a mean-based floor would force it on. And because a saturated dead unit revives only slowly, the floor must be applied from the first step, where it prevents deaths rather than reversing them. With \(\lambda_{\mathrm{ctx}}=3\times10^{-3}\) on both sublayers — the model we call selective — the collapse reverses on every axis (Figure 4): feed-forward selective rises \(2\%\!\to\!25\%\) and constant falls sharply; attention selective rises \(20\%\!\to\!45\%\); and quality recovers with legibility — dev perplexity \(23.2\!\to\!20.1\), LAMBADA perplexity \(240\!\to\!144\), BLiMP \(0.782\!\to\!0.819\).

The floor shares its hinge shape with the variance term of VICReg [22]. What is specific here is the problem it solves: not the informational collapse of self-supervised learning, but a collapse induced by another term in our own objective, which the identity of Eq. 5 identifies as a variance-minimizer, and which the floor counteracts to select the crisp-and-contextual corner that neither term reaches alone.

4.4 Where the floor is needed: a sublayer dissociation↩︎

Placing the floor on one sublayer at a time separates its two effects (Figure 5). Quality recovery is global: flooring either sublayer recovers most of the collapse (LAMBADA accuracy \(0.20\!\to\!0.24\), dev perplexity \(23.2\!\to\!20.4\)), consistent with the collapse spreading through the shared residual — healing one end helps the whole stream — though BLiMP climbs only when both are floored. Legibility is local: each floor drives selective chiefly on its own sublayer (the feed-forward floor lifts feed-forward selective \(2\%\!\to\!23\%\) but attention only to \(34\%\); the attention floor lifts attention \(20\%\!\to\!49\%\) but feed-forward only to \(8\%\)). The selective count fluctuates by a few points near the variance threshold (Section 3.4), so what the dissociation rests on is each floor’s lift on its own sublayer, not the small wobble in the attention count across configurations (here \(49\%\), and \(45\%\) with both floored). Only flooring both is legible on both sublayers while reaching the top BLiMP. Quality is a coupled property of the residual stream; legibility is a local property of each sublayer’s operators.

4.5 Learning the operator fraction instead of reserving it↩︎

The legible feed-forward layer carries a hand-set hyperparameter: a fraction of its hidden units is held back as conventional \(\ifmmode\operatorname{GELU}\else\mathrm{\small gelu}\fi\) for trainability [1], a fraction as large as one half. We replace that fixed reservation with a per-unit learned choice. Each unit interpolates, by a gate \(\alpha=\sigma(\theta)\), between its set operator and a \(\ifmmode\operatorname{GELU}\else\mathrm{\small gelu}\fi\) computed from the same pre-activation, so the choice costs one scalar per unit and nothing else: \(\alpha\!\to\!1\) is a crisp operator, \(\alpha\!\to\!0\) a reverted nonlinearity. A crisp-first initialization \(\alpha\approx0.9\) starts every unit in the operator basin while leaving a small \(\ifmmode\operatorname{GELU}\else\mathrm{\small gelu}\fi\) path open for trainability, and an output-norm-weighted usage tax \(\lambda(1-\alpha)\lVert W_o[:,u]\rVert\) prices reverting, so a dead operator cannot escape the tax by writing nothing.

Given the choice, the model reserves no unit for pure \(\ifmmode\operatorname{GELU}\else\mathrm{\small gelu}\fi\). The fully-reverted bucket (\(\alpha<0.1\)) is empty in every run — including the null run with the tax switched off — and across all runs the smallest gate anywhere is \(\alpha\approx0.2\) (Table 1). The reversion to an all-\(\ifmmode\operatorname{GELU}\else\mathrm{\small gelu}\fi\) layer never occurs, and the hand-set partition is unnecessary: the crisp-first initialization holds the operator basin without a budget floor, and the per-unit gate supplies the trainability cushion the fixed fraction used to, distributed as a thin per-unit blend rather than a reserved block. The tax then sets how far the population crispens. Firmly-crisp units (\(\alpha>0.9\)) grow from \(0\%\) to \(20\%\) under an annealed schedule (\(10^{-2}\!\to\!10^{-1}\)) and the mean gate rises to \(0.81\) (Figure 6); weighted by output norm — the computation that actually drives predictions — crisp operators carry \(0.87\) of the load. The tax is not only a legibility knob: its lightest setting already improves quality over the untaxed null (Table 1). The annealed model, our showcase, reaches the highest legibility of any configuration — \(78\%\) of feed-forward operands selective at only \(11\%\) constant, and \(50\%\) of attention operands selective — at quality on par with the variance-floor model (Figure 4). The reservation the architecture required is gone; what remains is a small learned residual, and it does not fall to zero — the layer settles as a blend rather than a fully-Boolean one, consistent with the trainability ceiling of Appendix 10.

4.6 Pressures that do not work↩︎

Several natural alternatives fail, and the failures locate the mechanism. Crispness alone is the collapse of Section 4.1. A single-sided floor recovers quality but leaves the unfloored sublayer’s operators dead (Section 4.4), so a model that looks healthy on perplexity can be half-collapsed on legibility. Too strong a floor (\(\lambda_{\mathrm{ctx}}=10^{-2}\)) over-pushes variance and is slightly worse than \(3\times10^{-3}\) on quality. No pressure at all on the learned fraction (the null tax) leaves units at a soft \(\alpha\approx0.5\) with no crisp population — stable and adequate on quality, but producing nothing to read. And the named fraction is not free without limit: past roughly half-Boolean the feed-forward layer meets a late-training instability [1], so legibility is free at parity only up to a partition, not across a fully Boolean layer. The variance floor and the learned crisp fraction are the two pressures that hold a live, crisp, contextual population without either collapse.

5 Reading the Units↩︎

A selective unit detects a clean, input-dependent bit; naming what that bit is means reading what the unit writes. The two questions come apart. This section reads the units of the annealed model in the correct per-layer frame, separates detection from naming as a measured population split, and isolates the part of that split that survives a learned readout. The model is legible on both sublayers, and many of its deep units are nameable by the token category they fire on (Figure 7). The attention half of Figure 7 follows the per-head legibility map of [2], but is new here in three ways: it is measured on the model our training objective produces rather than the attention-only model of that work, it is joined to the feed-forward sublayer on the same scale (the two were never shown together), and its catalog interleaves attention heads with feed-forward units. It reports the legibility our objective induces across the whole layer, not a recomputation of that result.

5.1 The reading problem↩︎

Project a unit’s write through the raw unembedding — the logit lens [30] — and it is legible at the boundary layers but reads as noise mid-stack. This is the known failure of the logit lens, and its known cause: the residual stream has no privileged basis [36], [37], so a mid-stack representation lives in a rotated frame that the fixed vocabulary projection does not align with. A learned per-layer affine map — the tuned lens [31] — recovers the legibility by rotating each layer back into the vocabulary frame before decoding. Across the stack, raw mid-layer decoding accuracy sits near the floor (\(\sim\!0.05\)–\(0.08\)) while a single learned linear lens lifts it four- to sevenfold (\(0.30\)–\(0.36\)). This is the standard logit-lens and tuned-lens picture, used here as a tool.

5.2 A worked example: best of times↩︎

The prompt “It was the best of times, it was the ___” makes the frame concrete (Figure 8). The model completes worst. Tracing where a completion is assembled across a model’s layers in this way follows the direct per-layer reading of [1]; what is new here is reading it through the tuned lens rather than the raw logit lens, which moves the visible onset of the circuit six layers earlier and resolves which source position each layer draws from. Under the raw lens the best\(\to\)worst machinery appears only late, at layers \(9\)–\(11\); everything earlier reads as noise. Under the tuned lens the same machinery is legible roughly six layers earlier, from L3: an L3 feed-forward unit already promotes worst; an L5 feed-forward unit decodes to opposite, a literal antonym operation; and the L5 attention head reads the superlative-completion frame ever, imaginable, possible at the best of times positions. The middle stack computes the abstract structure — a superlative frame and an antonym operation — and the deep layers do the lexical commit, with worst crystallizing to rank one only in the final layers (L9–L11). The deep layers are already on task under the raw lens; the tuned lens helps only the rotated middle, which is exactly where the abstract computation lives.

5.3 Detection is not naming↩︎

The distinction the construction lets us draw is between two legibility axes. selective is a detection-side property: the unit responds to a clean, input-dependent bit, measured on its bounded operands and independent of any readout basis. decodable is a promote-side property: what the unit writes projects onto a nameable region of the vocabulary, in the sense of [35]. On the annealed model these do not coincide (Figure 9). Of the feed-forward operands, \(78\%\) are selective (at a stable crisp fraction of \(0.90\); the count is reported at convergence, Section 3.4), but only \(38.7\%\) (\(7131\) of \(18432\)) decode through the logit lens to a nameable content category. On the attention side \(50\%\) of value operands are selective, but only \(18\) of \(72\) deep heads read cleanly (all in L6–L11); of these, \(15\) are nameable under the seven-way category scheme catalogued in Figure 7. decodable is a proper subset of selective: selectivity is necessary but not sufficient for a name. The model detects cleanly but names only about half of what it detects, which quantifies the earlier observation that naming an operand’s meaning is the harder frontier [2].

5.4 Rotation, and the residue that survives it↩︎

Half of the detected bits going unnamed by the raw lens invites the obvious reading: this is just logit-lens weakness, and a learned lens would name them. Re-reading each unit in its own tuned frame tests exactly that. Decodability rises from \(38.7\%\) to \(65.3\%\) when a unit is allowed to decode in either the raw or its own learned frame, and the recovery is concentrated where the rotation account predicts — in the middle stack, where individual layers jump sharply (for example one layer moving from \(15\%\) to \(73\%\)). Rotation therefore explains a large share of the gap: much of what looked undecodable was decodable in the wrong basis.

The remaining part does not close. About \(35\%\) of the units resist even the learned lens. The residue is not uniform. Layers within the stack carry heterogeneous per-unit frames rather than one shared rotation, so a lens fit to the residual bulk helps bulk-frame units and can hurt units that were already canonical — consistent with the absence of a privileged basis rather than a single recoverable rotation. And the final layer’s low decodability (\(\sim\!10\%\)) is genuine, not rotation: its lens is near the identity, and the units there write fine-grained, non-token-peaked logit adjustments. Two properties are what remain after the known recovery is subtracted: this supra-lexical residue, and the layerwise lexicalization trajectory — a concept is non-lexical mid-stack and becomes token-decodable only in the final one or two layers. The closest prior framings are function directions that are steerable without being decodable [49] and the polytope reading of features as globally distributed but locally aligned [42]; neither states the trajectory on units that are legible by construction.

The tuned-lens recovery is a general property of transformers [31] and applies as well to a gelu model. What is specific here is the split between detection and naming, and the trajectory along which naming appears, measured on units whose detection side is legible by construction.

5.5 Where a concept lives: redundant tokens, distributed concepts↩︎

The supra-lexical residue raises a question the construction is unusually well placed to answer: when the model computes a fuzzy conjunction, does the computation live on a unit? Because each intersection unit is a named operator \(A\cap B\), we can ablate one and re-run. Zeroing any single intersection unit’s write moves the loss by essentially nothing (median \(\Delta \approx 0\)), yet zeroing the whole intersection block across the stack costs \(0.65\) nats. The load is real but no single unit carries it. That is consistent with two very different pictures — the same conjunction computed redundantly across many copies, or one concept computed as a distributed code with no single-unit home — and single-unit ablation cannot tell them apart. Rank can.

We separate them causally (Figure 10a). For a layer’s intersection block we take its per-token residual contribution, ablate the block, and add back only a rank-\(r\) reconstruction of that contribution, sweeping \(r\). If the block computes a few operators replicated many times, a low-rank reconstruction restores the loss; if it is a distributed code, the loss returns only as \(r\) grows. The two regimes are stark and depth-ordered. At layer \(0\), rank \(4\) restores \(95\%\) of the block’s effect: the block computes only a handful of distinct operators, replicated across the layer (\(\sim\!180\) units per operator), each reproducible from a single unit whose activation correlates above \(0.9\) with the operator. At layers \(9\)–\(10\), the same reconstruction needs rank \({\sim}256\): the computation is genuinely distributed, and the used operators are derived works of \(15\)–\(40\) units that no single unit reproduces (best single-unit correlation \(0.4\)–\(0.7\)). The effective causal rank of the block climbs monotonically with depth while the redundancy factor falls from \(183\times\) to \(14\times\) (Figure 10b). A single-unit read, which succeeds at the input, gives way to a distributed one at the output.

A second dissociation cuts across the first: the loudest structure is not the load-bearing one. At every layer a single dominant mode accounts for \(74\)–\(90\%\) of the block’s activation variance and is perfectly single-unit reproducible — but at the deep layers it is causally almost inert (at L10 the rank-\(1\) reconstruction restores only \(3.7\%\) of the loss). The computation the model predicts with lives in the quiet, high-rank tail; a variance-ranked search for the important units finds exactly the wrong ones, which is why our first attempts to read these units by activation strength surfaced only a loud, meaningless mode.

What is distributed turns out to be a specific kind of thing. We labelled tokens by five surface categories — digits, punctuation, capitalized words, word-initial pieces, and newlines — and asked, per category, whether a single intersection unit detects it as well as the whole block’s best linear combination does. It does: across the deep layers the best single unit matches the full-block direction to within a few points of held-out auc, and a newline is detected by one unit at auc \(1.00\) (Table 2). Token-level features are localized. The distributed operators are the other kind: reading the deep block’s directions, the derived-work operators fire on abstract, cross-token structure and decode to no vocabulary token at all (write entropy at the vocabulary ceiling). The clearest one fires on clause and sentence boundaries — But, If, When, and reported-speech punctuation — is spread over \(21\) units with no single-unit reproducer, and carries a real if small causal load. The residue of Section 5.4 and the distribution here are the same phenomenon seen twice: the units the model keeps legible and single-homed are the token-shaped ones, and the concepts it distributes and leaves nameless are the supra-lexical ones.

This connects the construction to the superposition account of ordinary networks. Conventional transformers are argued to compute logical functions of features — conjunctions included — delocalized in superposition, spread across neurons for efficiency [41], [74], and such feature interactions are recoverable only post hoc [75]. Making the operator legible by construction does not remove that pressure: the model still builds each conjunction redundantly, and reserves genuine distribution for the abstract concepts it has no token for. Legibility of the operator is not localization of the concept. The gap this leaves — a distributed, nameless concept that we can point to only as a rank-\(r\) direction — invites a between-unit training pressure aimed at it. Section 7 adds one and finds it reshapes the circuit rather than compacting it: a decorrelation penalty trades this distribution for surgically editable units without concentrating the concept. Whether a concept can instead be compacted onto fewer units at no quality cost is a different question, which we leave open (Section 8).

6 Editing the Units↩︎

A legible unit should be a better place to make a change. If a coordinate carries a fixed, stated meaning, an edit aimed at that meaning should land there and nowhere else. We test this directly. The edit mechanism is standard: a value-side promotion that adds \(\beta\,E[T]\) to a unit’s down-projection column, \(W_o[:,u]\mathrel{+}=\beta\,E[T]\), the same operation used to inject a fact by writing into a feed-forward value [35], [56], [57]. What is under test is not the mechanism but the site: whether a legible-by-construction unit is a cleaner edit location than a conventional one. We measure a site by two quantities. Efficacy is whether the promoted concept lands in the model’s output where the unit fires; collateral is \(\mathrm{kl}_{\mathrm{off}}\), the KL divergence between the original and edited next-token distributions on the positions where the unit does not fire — the ROME notion of locality [57]. To compare sites fairly across bounded and unbounded units, we scale each unit’s \(\beta\) to a matched efficacy and read off the collateral it incurs to get there.

6.1 Crisp firing predicts locality↩︎

Within the showcase (anneal) model, how crisply a unit fires predicts how local its edit stays (Figure 11). Across feed-forward units the rank correlation between firing crispness and \(\log\mathrm{kl}_{\mathrm{off}}\) is \(-0.56\): the crisper the detector, the less it leaks. The extremes are far apart — units with firing crispness \(\ge0.9\) carry a median off-target collateral of \(\mathrm{kl}_{\mathrm{off}}=4.5\times10^{-5}\), against \(1.7\times10^{-2}\) for units below \(0.5\), a factor of roughly \(370\). Firing density predicts collateral in the same direction (\(+0.43\)): a unit active on more positions leaks on more. A crisp, sparse detector is a small, well-insulated switch; a diffuse one spreads its edit across everything it touches.

6.2 Boolean units are more targeted than gelu units↩︎

The advantage holds between models, but it emerges with depth. Against the conventional gelu baseline of Section 3.5, at matched efficacy the Boolean feed-forward units are no better targeted than gelu in the earliest layers, where their outputs barely fire (L0–L2, \(1.4\times\); L3–L5, \(2.9\times\)), and become far more local through the middle and deep stack, where the editable units concentrate (L6–L8, \(184\times\); L9–L11, \(52\times\); win probability \(0.98\)–\(1.00\); Figure 12). The reason is the firing property of Section 6.1: an edit stays local when its unit fires crisply and sparsely, and that firing is itself a deep-layer property, so the advantage appears where the firing does. Layers L1, L2, and L4 hold no editable Boolean output unit at all — their set-operator products almost never rise above the firing threshold.

6.3 A conjunctive edit↩︎

Legibility also buys an edit a conventional unit cannot express. This edit differs from the promotions above: rather than editing an existing unit, it installs a new detector into one, so the site is free and any set-operator coordinate serves; we use a single unit (\(j{=}17\)) at layer \(6\). Into its two operand banks we install a selective direction for each of two concepts — for a concept \(C\), the difference of means \(\mathrm{mean}(x\mid C\text{ in context})-\mathrm{mean}(x\mid\text{not})\) — so that \(A=\sigma(W_a x)\) fires when “king” is present and \(B=\sigma(W_b x)\) when “France” is. The unit’s own fuzzy-AND \(A\odot B\) then computes the conjunction, unchanged. It gates cleanly (Figure 13): \(A\) fires only on king contexts, \(B\) only on France, and the intersection is nonzero only when both are present — exactly zero for either concept alone and for neither. A conventional gelu unit cannot be edited this way: it has a single pre-activation, so it can add two directions but cannot gate on their conjunction. The distinction is exactly the one the architecture makes available — a unit that combines two named operands rather than one summed input — and it separates our edit from additive compositional steering, which co-activates two concepts in parallel rather than gating on their conjunction [59], [63].

Two caveats bound the demonstration. The installed direction is a lightweight data fit (a difference of means), not the raw token embedding: an embedding is shaped for the output geometry and is not selective read from a mid-layer residual — the same read-frame effect of Section 5. And the intersection’s peak magnitude (\(0.35\)) is capped by single-position superposition, since two concepts rarely co-encode maximally on one token; the separation from the other classes (exactly \(0\)) is clean.

6.4 The predictor is firing, not meaning↩︎

What makes a good edit site is not what the interpretability reflex would suggest. The predictor of low collateral is bounded, crisp, sparse firing. It is not decodability: whether a unit’s write decodes to a nameable token under a logit lens is, if anything, anti-correlated with edit locality (rank \(+0.48\) in the wrong direction), and it is not the learned crisp gate \(\alpha\) either (\(+0.15\)). Locality is a structural consequence of the firing pattern, and that firing pattern is exactly what the selectivity objective of Section 4 induces.

Knowing where to edit is separate from knowing how, and only one of the two is open. Where follows from the firing pattern: a unit fires sparsely and crisply, so an edit at it stays confined. How depends on the target. For a target that is a vocabulary token, we know how as well — write its direction in the layer-native frame (Section 6.5). What is open is the target that is a logical-operation output: a set-operator unit’s write \(A\cap B\) or \(A\setminus B\) combines the features its two operands detect, and those features are most likely concepts the model minted for its own prediction during training, not human categories with lexical names. There may be no word to look up for “\(A\) and-not \(B\)” because there is no human concept there at all — so steering a unit toward such an output means deriving its direction from data, as the conjunctive edit fit king and France, rather than reading off a token. This is the detection-versus-naming gap of Section 5 on the editing side: the site is in hand, and for a named concept the write is too; naming the model’s own invented logical outputs is the part with no dictionary.

The editability is the downstream payoff of the training objective: the mechanism remains ROME’s, and what the objective supplies is the quality of the site.

This also decouples the result from the standard critique that interpretability is a weak proxy for steering utility [69]. That critique targets a pipeline in which features are discovered post hoc, clamped at inference, and judged by how decodable they are. Our units differ on all three axes: they are constructed rather than discovered, edited in the weights rather than clamped in activations, and scored by off-target KL rather than by decodability — which, as above, does not predict site quality at all.

6.5 Deep edits need the layer-native frame↩︎

Promoting a token bumps its logit; it does not make the model reason as though the concept were present. The two are different edits and want different frames. A surface promotion writes the token embedding \(E[T]\) and rides the additive residual highway: the write persists to the output essentially unchanged and raises \(T\)’s own logit, but injected late it has no machinery left to condition and diverges from any natural continuation. A deep behavioral edit — one that makes the downstream computation behave as if \(T\) were in the context — instead needs the write the layer itself would have received, a layer-native vector \(d_L(T)\). We obtain \(d_L(T)\) empirically as a difference of means between contexts that contain the concept and matched contexts that do not; this construction is the diff-of-means of contrastive activation addition and activation steering, localized as in ROME [57], [59], [60]. The mechanism is theirs; only the frame the write is made in is new.

The two frames behave very differently against the distribution a genuine mention of \(T\) would produce (Table 3). The native write \(d_L(T)\) reproduces that genuine-\(T\)-present next-token distribution, with a KL at or below the no-edit baseline: for “France” it reaches \(0.45\) against a baseline of \(0.77\), for “ocean” \(0.31\) against \(0.60\). The raw embedding write \(E[T]\) only lifts \(T\)’s own logit, and injected late it diverges catastrophically, with KLs of \(13\)–\(30\). Qualitatively the native write promotes the genuinely associated tokens — Paris, French, Atlantic — into a natural distribution, where the raw write spikes the concept word itself. Reading a unit and writing to it are dual operations with opposite correct frames: reading forward projects through the rotated per-layer frame, while a persistent additive write is made in the raw embedding frame, and a behavioral write is made in the layer-native frame recovered by contrast.

6.6 What legibility does and does not buy↩︎

Legibility buys locality, not magnitude. Under a literal all-positions KL the Boolean and gelu edits are within \(1.4\times\) (\(0.019\) against \(0.026\)): the legible edit concentrates its change on the concept rather than shrinking the total shift it makes, so its advantage is the off-target locality of Section 6.2, not a smaller overall divergence. The edit operation is ROME’s and the deep write is contrastive activation addition’s; what is ours is the quality of the site, quantified by off-target KL on non-firing positions. These results are on single target tokens, and the conjunctive edit’s magnitude is capped by lexical superposition at a single position. The nearest prior context is feature steering on sparse-autoencoder features, where more monosemantic units incur less collateral [45], [70]; we differ by constructing the units rather than discovering them, editing them in the weights rather than clamping them at inference, and in a quantified off-target locality.

7 A Legibility Dial: Trading Fan-Out for Fan-In↩︎

Section 5.5 left the model computing each concept redundantly: a fuzzy conjunction is produced across many copies, and read — as we show below — by many downstream units. That structure is legible in one sense and awkward in another. It is a fan-out circuit: a few concepts, each broadcast to a large set of readers. Broadcasting is efficient (one concept, amortized), but it makes the concept hard to touch. To edit what a baseline concept means, one must alter every redundant copy and accept that the change propagates to all of its readers. This section asks whether the circuit can be reshaped into something more editable, and what that costs. The intervention is a single training pressure, and it exposes a dial: at one end the reused, fan-out circuit of the preceding sections; at the other, a fan-in circuit of independent units that are individually editable — at no cost to quality.

7.1 Reuse is correlation↩︎

The reason a concept is reused is also the reason it is hard to isolate. A building block that serves many computations — a “royalty” feature feeding both king and queen contexts — is, by definition, active whenever any of those computations is. Reuse across contexts is co-occurrence, and co-occurrence is correlation. So a pressure that forbids units from correlating is, implicitly, a pressure against shared building blocks. We add one. Alongside the per-unit crispness and sparsity pressures of Section 3.3, we introduce a between-unit term that penalizes the mean squared off-diagonal correlation of the operand activations, \[\mathcal{L}_{\mathrm{d}}=\lambda_{\mathrm{d}}\,\frac{1}{C(C-1)}\sum_{i\neq j} R_{ij}^{2},\qquad R_{ij}=\frac{\mathrm{cov}(v_i,v_j)}{\sqrt{\mathrm{var}(v_i)\,\mathrm{var}(v_j)}}, \label{eq:diff}\tag{7}\] computed over each batch’s tokens on the bounded operands \(v=[A;B]\). The penalty is scale-free (the covariance normalization cancels the token count) and cheap (one \(C\times C\) Gram matrix per block, no eigendecomposition). We train the annealed model of Section 4 unchanged except for this one added term, sweeping \(\lambda_{\mathrm{d}}\) over two orders of magnitude.

7.2 The dial↩︎

The penalty does exactly what it targets: mean off-diagonal operand correlation falls monotonically with \(\lambda_{\mathrm{d}}\), from \(0.51\) in the baseline to \(0.016\) at the strongest setting (Table 4). It costs nothing on quality. Development perplexity and lambada are flat across the sweep — at the strongest setting the model is if anything marginally ahead (\(0.239\) vs.\(0.232\) lambada accuracy, \(133\) vs.\(143\) perplexity) — so the intervention moves the circuit without moving the task. One thing does not happen: the decorrelated model is not more legible per unit. Its fraction of selective detectors is lower than the baseline’s at convergence (\(62\%\) vs.\(78\%\)); the operands stay crisp but a larger share drift below the contextuality floor. Per-unit legibility is not the axis the dial moves.¹

What the dial moves is the shape of the circuit, and it moves it from fan-out to fan-in. We measure both directions of wiring on the converged models. Fan-out — how many downstream units read a given concept direction — collapses from a mean of \(21\) (with hubs read by up to \(1200\) units, most of a layer) to a mean of \(1.4\) (largest hub \(46\)). Fan-in — how many atoms are needed to reconstruct a unit’s input, its description length under a sparse decomposition over tokens, earlier concepts, and attention directions (the apparatus of Section 5)² — rises from \(2\) atoms to \(15\). The two numbers are the two faces of one change: without the pressure, a few concepts broadcast widely and each unit reads almost nothing (short description, high fan-out); with it, no concept is shared, so every unit must assemble its input from a wide, near-disjoint set (long description, low fan-out). The depth of the circuit does not change — causal chains are one to two layers in both models (Section 5.5). It is a change in width, and in its direction. A single log-ratio captures it: \(F=\log_{10}(\text{fan-in}/\text{fan-out})\) moves from \(-0.98\) (broadcast) to \(+1.02\) (aggregate) as the correlation falls, crossing zero — equal parts fan-out and fan-in — at \(\lambda_{\mathrm{d}}\approx3\times10^{-2}\) (Table 4), so the reshaping is graded by the dial, not a switch between two fixed regimes.

7.3 Fan-in units are surgically editable↩︎

The point of the reshaping is that fan-in units can be edited where fan-out concepts cannot. Editability is governed by fan-out: an edit to a unit propagates to everything that reads it, so a hub read by a thousand units cannot be changed without collateral. We make the comparison directly (Figure 14). In the baseline, moving a concept requires editing a co-firing group of a median \(562\) units — the redundant copies — and the edit is not local: at matched on-target effect, off-target kl grows with edit strength (\(0.002\to0.005\to0.014\) across efficacy bins), the signature of the hub rippling outward to its readers. In the decorrelated model the same concept lives on a single unit, and the edit stays put: off-target kl is flat and roughly two orders of magnitude smaller (\(\sim\!10^{-4}\), independent of strength). A concept that was a \(562\)-unit rippling tangle becomes one addressable, surgically editable unit. This is the sense in which decorrelation buys editability: not by making units more readable, but by removing the reuse that made them entangled.

The reshaping also homogenizes the edit sites. Section 6.1 found, within the baseline, that how crisply a unit fires predicts how local its edit stays (Figure 11): edit sites span the crispness range, and the high-collateral, leaky corner is a group of shallow, diffuse units. Running the identical analysis on the decorrelated model, that spread collapses (Figure 15). Its editable Boolean units are almost all crisp (\(95\%\) fire above \(0.9\), against a broader baseline spread) and confined to the deep layers, and their off-target collateral falls in a narrow band around \(5\times10^{-4}\): the leaky diffuse corner is essentially empty. The decorrelated model does not reach the very lowest per-unit collateral of the baseline’s best crisp units, but it has no leaky sites at all — nothing above \({\sim}3\times10^{-3}\), against a baseline tail that reaches \(10^{-1}\). Because each concept now lives on a single unit rather than a redundant group, the population of edit sites is uniformly of the crisp, well-insulated kind, rather than a mixture that includes diffuse leakers.

7.4 A named explanation of a prediction↩︎

The end point of legibility is to explain a prediction. Because every component writes into the residual stream linearly, a predicted token’s logit can be attributed to individual components by direct logit attribution. The point of the end-to-end construction is that both halves that attribution lands on are legible: the feed-forward contributors are named set-operator units, and the attention contributors are the bounded value heads of [2], each read as a detector with a stated target. Every large contributor is therefore a named operation, and — the property that makes it a sentence — there are few of them. This is the by-construction advantage, and it is one of sparsity, not of nameability. Running the identical attribution on a conventional gelu transformer of the same size, the neurons are still labelable by their fired tokens post hoc, and some are recognizably comparative (more, longer, larger); what changes is that the prediction is diffuse. The same than is spread over \({\sim}1200\) effective contributors — a thousand small named pieces no short list captures — against \({\sim}15\) for the bounded model. The construction does not make units nameable where a conventional neuron is not; it makes the attribution sparse enough to state. This is the superposition that motivates the whole line [41], now measured on a single prediction: the standard model distributes the computation across a cloud of polysemantic units — in both its feed-forward and its attention halves — and what the bounded construction supplies is not names but concentration onto a legible few.

For the comparative continuation “The cat is much bigger” the model predicts than, and the attribution reads directly: than is promoted by a feed-forward unit that fires on longer, more \((+2.9)\), an attention head reading the comparative adjective bigger \((+2.9)\), and a second feed-forward unit that fires on more, less, weaker \((+1.9)\). The same three components carry than across every comparative prompt we tried — “taller than his brother but shorter,” “the results were more significant” — with the head re-reading whichever comparative word is present (shorter, more). One can state, in words the reader already understands, why the token was produced.

Two things bound the result, and both echo the earlier findings. First, the explanation concentrates only for feed-forward-driven predictions: on the comparative tokens the effective number of contributors is \({\sim}15\), whereas on attention-driven or diffuse continuations (predicting not, or be after “They will never”) the attribution spreads over hundreds of components with no short named list — the ceiling of confident, feed-forward-driven tokens already noted for these models [2]. Second, the decorrelation sharpens the account rather than making it legible: the baseline explanation is already named (its top contributors are the same legible value heads, reading more), but it is carried by heads alone with no feed-forward operator surfacing. The decorrelated model instead carries it on named comparative operators directly, and few of them (Figure 16). Legibility of both sublayers supplies the named nodes; the dial makes the named explanation short and mechanistic.

7.5 What flatness is worth↩︎

The dial is a Pareto choice, not a free win. The fan-in model is the more editable and plausibly the more robust: with no load-bearing hubs, damage — from pruning, quantization, or an edit — stays local rather than propagating through a thousand dependents, and unlearning a capability need not disturb a shared abstraction. These are exactly the properties an inspection- or editing-first deployment wants, and the construction reaches them natively, without a post-hoc dictionary. What flatness spends is compression: reuse amortizes a concept across its readers, and the fan-in model rebuilds each one disjointly, at a longer description length — a cost we did not see at \(125\)M parameters and one epoch but that should surface under tighter parameter budgets. The deeper open question is compositional: shared abstractions are plausibly what let a model recombine known concepts on novel inputs, and a circuit that reuses nothing may generalize worse out of distribution even at equal in-distribution perplexity. Our quality metrics are in-distribution and do not stress this; a compositional or out-of-distribution probe is the test that would tell us whether the fan-in end pays a generalization cost, and it is the natural next measurement.

This also sharpens where the construction sits relative to the decorrelation literature. Feature decorrelation and orthogonality regularizers have been proposed as routes to monosemanticity — more single-meaning, individually interpretable units [41]. Our result is consistent with the letter of that and complicates its spirit: decorrelation does yield independent, individually addressable units, but at the cost of the shared structure that composition is built from, and not at a gain in the per-unit selective count. What the pressure buys is not more legible atoms but a more editable arrangement of them — fewer readers per concept, at the price of a longer description per unit. Legibility, on this evidence, is not one axis but several, and they do not move together.

None of the structure in this section is special to the legible construction. A conventional transformer plausibly wires its concepts the same way — reuse is efficient wherever a feature serves many computations — and a decorrelation penalty on ordinary hidden activations, the objectives from which ours descends, would reshape fan-out into fan-in the same way. Every measurement here — fan-out, description length, edit locality — is defined on residual-stream activations and weights and runs unchanged on a \(\ifmmode\operatorname{GELU}\else\mathrm{\small gelu}\fi\) transformer. What the legible construction adds is not the structure but the ability to read it: to name the concept a hub broadcasts (data, school, work, research) or the atoms a fan-in unit collects, and to read it off a few named components rather than the diffuse cloud of polysemantic neurons a conventional model spreads the same content across. The dial is a general fact about transformer circuits; legibility is what lets us watch it turn.

8 Discussion↩︎

The result that ties the three parts together is the objective. A legible-by-construction architecture is a substrate, not a guarantee: its operators are legible only if training keeps them crisp and alive, and the natural pressure toward crispness, left alone, destroys exactly the property it was meant to create. Naming why — the \(\mu(1-\mu)-\mathrm{var}\) identity — turns a tuning failure into a structural one and points at the minimal fix, a variance floor that is the legibility metric written as a loss. With that fix the substrate becomes usable: the units are legible enough to read in the rotated frame, and crisp enough to edit surgically. This is the sense in which the paper is a capstone of the legible feed-forward [1] and legible attention [2] line rather than a third architecture — it is the objective that makes both work. The same move retires a hyperparameter. Prior work reserved a fixed fraction of feed-forward units as conventional \(\ifmmode\operatorname{GELU}\else\mathrm{\small gelu}\fi\) for trainability; the learned per-unit gate turns that reservation into a decision the model makes, and given the freedom it keeps no unit as pure \(\ifmmode\operatorname{GELU}\else\mathrm{\small gelu}\fi\) and drives the load-bearing computation crisp. What was a hand-set partition becomes an outcome, and the outcome is that the model wants the legible operators.

On quality the legible model is at parity. Against a conventional model the legible one is even on perplexity and LAMBADA and slightly ahead on grammar, and no result here is a benchmark improvement. The win is internal: the variance floor converts a pressure that collapses a model and wrecks its quality into one that recovers quality to parity while reaching the highest legibility in the family. A contribution of this kind is an objective and a set of measurements, not a leaderboard.

The construction does not make the model compute differently from a conventional transformer: the structure the later sections read off — where concepts live, how redundantly they are built, how they are wired and how a decorrelation pressure reshapes them — is structure an ordinary network plausibly shares, and every measurement is defined on activations and weights a \(\ifmmode\operatorname{GELU}\else\mathrm{\small gelu}\fi\) model has too. What the construction offers is legibility — that this structure can be read, its units named, its edits placed, without a post-hoc dictionary. The objective is what earns it.

A single further knob makes that payoff concrete and exposes its price (Section 7). Adding a between-unit decorrelation pressure, on top of the per-unit crispness and variance-floor pressures, reshapes the circuit along one axis at no cost to quality: from fan-out — a few concepts each reused by up to a thousand downstream units — toward fan-in — many independent units, each aggregating a wide, disjoint input. It is a graded Pareto trade. Fan-in units are individually, surgically editable: a concept that lived in a \(562\)-unit redundant tangle becomes one addressable unit, edited with two orders of magnitude less off-target collateral. And a prediction’s direct-logit attribution collapses from the diffuse cloud a conventional model spreads it across (\({\sim}1200\) components) onto a short list of named operations (\({\sim}15\)) one can read as a sentence — the by-construction advantage being sparsity of the attribution, not nameability, since both sublayers are already legible. What the trade spends is the compression reuse provides, whose cost on compositional generalization we do not yet measure.

The conjunctive edit points at a distinct payoff. Transformers form conjunctive and negated concepts — “this and that,” “this but not that” — as a matter of course, but conventionally in superposition, with nowhere single to address. Building the set operations in changes where such a concept can live: the architecture exposes a two-operand unit whose slots are addressable, so a conjunction can be written at a specific selective site — which is what the conjunctive edit does, installing a clean gate from two derived operand directions. Pointing to where a logical concept forms and writing to it is something a post-hoc method does not readily offer. Reading the model’s own conjunctions back out is a separate question, and the answer depends on how it builds them (Section 5.5): the concepts it computes redundantly, a few operators replicated across a layer as at the input, are recoverable from any single copy, whereas the ones it distributes across many units it leaves nameless — recoverable only as a rank-\(r\) direction, not from any single unit — and it is these abstract, tokenless concepts that it spreads ever wider with depth. The localize-and-recover story is real but bounded: it holds where the model builds redundantly, not where it spreads.

Several limitations bound the contribution. All numbers are single seed at this scale; the direction is consistent across the ablations, but a headline of the form “method A beats method B” would need seeds we do not report. Legibility is free only up to a partition: past roughly half-Boolean the feed-forward layer meets a late-training instability, so the fully legible layer is a target the pressures approach rather than reach. And legibility is not the same as nameability. The units detect cleanly far more often than they can be named — a concept is carried non-lexically through the middle of the network, becomes token-decodable only in the last layers, and may in the end be one the model minted for its own prediction rather than a human category with a name — and closing that naming gap, rather than the detection gap the objective already handles, is where the interpretability work remains. On the editing side the mechanism is borrowed entirely; what the objective contributes is the quality of the site, and the gain there is locality rather than magnitude.

The naming gap also marks the next question. That a concept is legible on the detection side but non-lexical in mid-stack suggests the information a layer passes forward is neither token-shaped nor confined to a single unit, and reading it directly — rather than through a detection metric or a per-unit lens — would require decoding the residual stream in its own rotated, distributed basis. Section 5.5 sharpens what stands in the way: token-level features are already single-homed, but the model distributes its abstract, nameless concepts across many units, and that distribution grows with depth. A concrete step is a pressure that concentrates such a concept onto fewer units. Section 7 tried the between-unit pressure closest to hand — decorrelation — and found it does the reverse, spreading each concept wider in exchange for editability; the pressure this calls for is its dual, one that makes a concept compact rather than a unit decisive, aimed squarely at those distributed operators. Whether a concept can be compacted without costing quality is the natural test, and we leave it to future work.

9 Conclusion↩︎

A transformer built from bounded, named operators is a legible substrate, but a legible model takes an objective. The crispness pressure legibility requires collapses its own operators into dead constants; the \(\mu(1-\mu)-\mathrm{var}\) identity explains why, and a variance floor — the legibility metric written as a loss — repairs it, recovering quality while a learned per-unit gate retires the reserved-\(\ifmmode\operatorname{GELU}\else\mathrm{\small gelu}\fi\) partition and leaves the whole layer legible, on both sublayers at once. That model can then be read and acted on directly. In each layer’s rotated frame its units separate a clean detection from a harder naming, detecting far more than they name and distributing exactly the concepts they cannot; being crisp and sparse, they are surgical edit sites and admit conjunctive edits no single neuron can express; and a between-unit decorrelation pressure exposes a dial from reused, fan-out circuits to independent, fan-in ones, trading compositional reuse for units editable one at a time and predictions readable as a short list of named operations. Quality holds at parity with a conventional model throughout. The objective is the contribution, and readability is what it buys — a transformer whose computation can be read, and changed, not after the fact but from its own weights.

10 Trainability ceiling↩︎

The named fraction of the feed-forward layer is not a free hyperparameter. Below roughly half-Boolean the models train cleanly for the full epoch; above it they diverge late in training, and sooner the larger the Boolean fraction — a fully-Boolean layer destabilizes early, a quarter-\(\ifmmode\operatorname{GELU}\else\mathrm{\small gelu}\fi\) layer much later, each with a sudden spike in instantaneous perplexity rather than a gradual drift. A linear-bypass path and an output normalization each delay the onset but neither prevents it, which points to a gradient pathology of the saturating operators rather than an unbounded-write problem [1]. The consequence for this paper is stated in the main text: legibility is free at parity only up to a partition of the layer, and the pressures of Section 4 operate below that ceiling.

11 Hyperparameters↩︎

All models are \(125\)M-parameter, \(12\)-layer, \(768\)-wide transformers with learned positional embeddings, trained for one epoch on the same open-web corpus and tokenizer with an identical optimizer and schedule, and with torch.compile enabled. The selectivity pressures use \(\lambda_{\mathrm{s}}=10^{-3}\) (sparsity) and \(\lambda_{\mathrm{c}}=3\times10^{-3}\) (crispness). The variance floor uses \(\lambda_{\mathrm{ctx}}=3\times10^{-3}\) and \(\tau=0.003\), applied from the first step on both sublayers for the selective model. The learned crisp fraction initializes \(\alpha=\sigma(\theta)\) at \(\alpha\approx0.9\) (crisp-first), with an output-norm-weighted \(\ifmmode\operatorname{GELU}\else\mathrm{\small gelu}\fi\) tax \(\lambda(1-\alpha)\lVert W_o[:,u]\rVert\); the anneal schedule ramps \(\lambda\) from \(10^{-2}\) to \(10^{-1}\) over training. The end-to-end models place every attention value under the Boolean form (Eq. 3 ).

12 Probe methodology↩︎

Legibility statistics are computed on held-out text. For a bounded operand channel with per-token values of mean \(\mu_c\) and variance \(\mathrm{var}_c\) over the sample, the channel is crisp when the fraction of values within \(0.1\) of a rail \(\{0,1\}\) exceeds one half and contextual when \(\mathrm{var}_c\ge\tau=0.003\); selective is the conjunction, constant is crisp with \(\mathrm{var}_c<\tau\). decodable is measured by projecting a unit’s write through the token unembedding and testing whether the top of the resulting distribution forms a nameable content category, following [35]. The rotated-frame re-read fits a per-layer affine tuned lens [31] mapping each layer’s residual to the pre-unembedding frame (ridge regression, with the final-layer map verified near the identity), and reports a unit as recovered when it decodes in the raw frame or its own tuned frame. The best-of-times attribution reads each layer’s feed-forward and attention writes for the single prompt through both the raw and tuned lenses.

References↩︎

---

1. The selective count is a knife-edge statistic here: the crisp population sits just above the variance floor \(\tau\) (mean across-context variance \({\sim}3\)–\(5\tau\)), so a small global drift in variance moves a large count across the threshold, and intermediate checkpoints can swing by tens of points. We therefore report converged (end-of-epoch) values and the stable continuous quantity (crisp fraction \({\approx}0.9\)) alongside the count.↩︎

2. This decomposition is orthogonal matching pursuit over a fixed interpretable dictionary, in the spirit of the sparse-coding interpretability line [5], applied to a unit’s input; it is apparatus, not a method contribution.↩︎