1 Introduction↩︎

Standard neural network training optimizes parameters in an additive fashion with linear update steps. Yet many computations within a network are naturally relative. Doubling or halving a normalization gain, a feature amplitude, or the singular value of a matrix reflects analogous changes in logarithmic space. However, they have unequal optimization distances in linear weight space. More specifically, moving from \(1\) to \(0.5\) has half the linear distance of moving from \(1\) to \(2\). This is not to say we should necessarily think of all aspects of the network in a multiplicative fashion. Activation functions alongside biases consider absolute scale to be important. Nevertheless, the optimization employed in training has mostly played into the linear/absolute scale nature of networks. When relative changes weights are to be achieved through additive updates, we should expect training to push a subset of parameters far from initialization whenever the task demands large multiplicative factors, while leaving many others near their starting scale.

As shown in Figure 1, we may observe this effect in pretrained language models, which develop heavy-tailed weight distributions. Many parameters remain near initialization scale, while a small fraction move far into the tails. In our Qwen3-4B-Instruct analysis (Figure 1), the maximum weight is roughly 30\(\times\) the Xavier initialization scale even though the standard deviation grows only 1.7\(\times\). This creates a heterogeneous optimization problem in which many coordinates may travel little from their original positions, while some coordinates must traverse much larger distances in weight space.

Adaptive optimizers such as Adam [1] normalize update magnitudes per coordinate inducing similar magnitude across dimensions, rendering the updates invariant to parameter magnitude. Preconditioning decouples step size from gradient scale to keep optimization well-conditioned, but a parameter at \(0.004\) and one at \(0.4\) can receive similar absolute updates while resulting in very different relative perturbations: the same step may be significant for a small parameter yet barely move a large one in relative terms. Global learning rates may be chosen to respect the most sensitive small-scale coordinates, potentially underserving larger magnitude directions such as feature gains, matrix modes, or other coordinates that may benefit from relative movement.

One solution may be to try to correct this at the optimizer level through parameter-magnitude scaling into the update rule, but this may fight against the preconditioning benefit. We then propose to change the parameterization (and thus geometry) of the weights themselves. The natural method for incorporating multiplicative relativity is to make use of an exponential mapping, which converts additive additive updates in the raw space into multiplicative updates in the output space. However, the standard exponential function only outputs positive values when our weights are roughly centered and symmetrical around 0. To address this issue, we utilize the symexp function, which is sign-preserving, odd, monotonic, and offers the exponential behavior. As to not overcommit to the exponential structure, we introduce a linear pathway along with learnable weights allowing the network to scale the effect of expoential and linear components. We hypothesize as well that the linear pathway may provide a direct gradient route which may aid optimization.

Each effective weight is decomposed into two complementary pathways: a symmetric-exponential pathway and a linear pathway. In our recommended configuration, the effective weight is: \[\label{eq:overview} w_{\mathrm{eff}} = \operatorname{sign}(w)\,[E_\theta(\abs{w}) + L_\theta(w)] + \Delta_\phi\tag{1}\] where \(E_\theta(\abs{w})\) is the learnably controlled symexp amplitude, \(L_\theta(w)\) is an identity-like linear pathway, and \(\Delta_\phi\) is an optional additive residual, typically implemented as a low-rank LoRA-style correction. The sign wrap applies the curved geometry to \(\abs{w}\) while keeping sign as a separable degree of freedom. We further find that a deliberate asymmetry between how weights are initialized and how the forward pass combines these pathways yields additional gains, which we analyze in Section 4 and attribute to early symmetry-breaking effects.

Contributions:

• We propose SymExpLin(SEL), a nonlinear weight reparameterization applicable to any learnable weight matrix or bias vector, decomposing effective weights into symmetric-exponential and linear components with independently learnable structured scales.

• We identify mismatched initialization, using a different combination rule for weight inversion than for the forward pass, as a source of beneficial asymmetric bias in our small-scale ablations.

• We introduce structured scale parameters with various factorization patterns and a targeted learning rate annealing schedule for managing redundant exponential controls.

• SEL is implementation-friendly: the training transform is elementwise and incurs modest overhead in compiled implementations, while effective weights are precomputed after training, reducing to standard linear layers with zero inference cost.

• We note that SEL can be folded into standard weights after training and, conversely, standard weights can be inverted back into the SEL parameter space for future continued-training or finetuning experiments.

• We evaluate across a grid of model scales (\(\{1024, 2048, 3072\} \times \{12, 24, 36\}\)) on autoregressive language modeling, with targeted ablations for initialization asymmetry, the linear pathway, and scale learning-rate annealing.

2 Related Work↩︎

2.0.0.1 Weight Reparameterization.

Weight normalization [2] separates weight magnitude from direction, improving optimization conditioning. Spectral normalization [3] constrains the spectral norm for training stability. Both apply relatively simple transformations. SEL introduces a richer nonlinear reparameterization that mixes symmetric-exponential and linear components with learnable mixture coefficients, operating on the weight matrix itself rather than its norms.

2.0.0.2 Symlog Transforms.

The symmetric logarithm \(\operatorname{sign}(x)\ln(\abs{x}+1)\) and its inverse have been used for prediction targets in reinforcement learning [4], handling values spanning many orders of magnitude. SEL applies a similar operation, a parameterized symmetric exponential, to weight space rather than prediction targets, with learnable scale parameters controlling the symmetric-exponential and linear components.

2.0.0.3 Initialization Schemes.

Xavier [5] and Kaiming [6] initialization maintain variance across layers. Data-dependent initialization [7] matches target statistics layer by layer. Our initialization via Newton’s method inversion ensures that despite the nonlinear reparameterization, effective weights at initialization match standard Xavier statistics. Our mismatched initialization is different from these schemes: it preserves the intended Xavier scale under one transform rule, then deliberately evaluates the same raw weights under a slightly asymmetric rule.

2.0.0.4 Learning Rate Parameterization.

Maximal Update Parameterization (\(\mu\)P) [8] provides principled per-tensor learning rate scaling across model widths, addressing one axis of the init/optimizer mismatch we discuss in Section 3. SEL targets a complementary axis, adapting effective step size per-parameter over the course of training rather than per-tensor at initialization; the two are not mutually exclusive.

2.0.0.5 Multiplicative Interactions.

[9] study where multiplicative interactions arise in neural networks and their benefits. Our structured scale parameters with row_col_mult mode implement rank-1 multiplicative interactions on the weight scaling, a lightweight form of structured multiplicative modulation.

The Multiplicative Weights Update (MWU) method [10] makes relative-scale updates explicit: weights are scaled by \((1 \pm \epsilon)^{\text{payoff}}\) rather than shifted additively, so adjustments are naturally proportional to current magnitude. This principle underlies algorithms from boosting to online convex optimization, but has seen limited adoption in neural network training, where additive SGD-style updates remain standard. SEL does not update multiplicatively. The optimizer still takes additive steps in raw parameter space, but the symmetric-exponential pathway induces a related effect at the effective-weight level: raw-space sensitivity scales as \(\exp(\beta\abs{w})\), so identical raw steps produce larger effective-weight displacements at larger magnitudes. The linear pathway preserves additive behavior near the origin, yielding a hybrid regime rather than a fully multiplicative one. This positions SEL as a way to obtain some of MWU’s relative-scale behavior while remaining compatible with standard gradient-based optimizers and preconditioning.

3 Motivation↩︎

Training moves parameters from a sampled simple starting distribution to a much broader final one. At initialization, variance-preserving schemes initalize parameters near a typical magnitude determined by its matrix’s width. By the end of training, we observe that the distribution of weights moves towards a heavy tailed shape in which a small fraction of parameters live far outside the initial typical range (Figure 1). Different families of methods respond to different points on this trajectory: \(\mu\)P and related learning-rate parameterizations set step size from the init-side typical magnitude, a per-tensor summary statistic downstream of model dimension, while SEL adapts effective step size dynamically from each parameter’s current magnitude, so as the distribution spreads during training, parameters that have travelled further receive proportionally larger effective steps. This section discusses preconditioning, learning rate tuning, and update magnitude with respect to parameter magnitude. We pose SEL as a complementary counterpart to \(\mu\)P along a second, dynamic axis.

3.0.0.1 Adam normalizes away scale.

Consider Adam’s update rule. The first and second moment estimates are \(m_t = \beta_1 m_{t-1} + (1-\beta_1) g_t\) and \(v_t = \beta_2 v_{t-1} + (1-\beta_2) g_t^2\), and the update is \(\Delta w \propto m_t / \sqrt{v_t}\). Under steady-state conditions with approximately constant gradient magnitudes, \(v_t \approx \E[g^2]\), so the normalized update \(m_t / \sqrt{v_t}\) has expected absolute value near 1 given the division by the estimated gradient’s standard deviation. In the special case \(\beta_1 = \beta_2\) where the two moving averages track on identical timescales, Cauchy-Schwarz tightens this to a hard bound: \(\abs{m_t / \sqrt{v_t}} \leq 1\) at steady state. Beyond steady state conditions, momentum suffers destructive interference under gradient sign flips, decreasing the numerator. Meanwhile \(v_t\) is sensitive to larger values potentially increasing denominator. The essential point is invariance: uniformly scaling every gradient by a constant leaves \(\abs{m_t / \sqrt{v_t}}\) unchanged, the normalization by \(\sqrt{v_t}\) cancels it, and no term in the update depends on \(\abs{w}\). Per-dimension variation reflects the shape of the gradient distribution (noise pattern, outlier frequency) rather than its overall scale, so updates concentrate around a common magnitude set by the learning rate, invariant to both expected gradient magnitude and parameter magnitude. A parameter at \(0.004\) and one at \(0.4\) therefore receive updates of similar absolute size, corresponding to a \(100\times\) difference in relative perturbation; global learning rate tuning must accommodate the most sensitive small-scale coordinates, potentially under-serving larger-magnitude ones.

3.0.0.2 Variance-Preserving Initialization

Variance-preserving initialization offer a typical parameter magnitude at the start of training, and may also guide common magnitudes at the end state of training. Xavier initialization uses \(\sigma \propto 1/\sqrt{(\text{fan\_in} + \text{fan\_out})/2}\), so smaller matrices receive larger initialization values and larger matrices smaller ones. The effect is pronounced for low-rank adapters [11], where the rank \(r \ll d\) yields much larger initialization scales than full weight matrices. This is consistent with the observation that LoRA requires higher learning rates than full model weight finetuning. Initialization thus writes scale information into the parameters that Adam then normalizes away.

3.0.0.3 \(\mu\)P: per-tensor, width-driven, static.

Maximal Update Parameterization [8] formalizes one axis of this mismatch: typical init magnitude varies systematically with matrix width, and per-tensor learning rates must be rescaled with width to preserve feature-learning behavior across scales. \(\mu\)P’s rescaling is theoretically derived, applied once at the tensor level, and driven by static properties of the architecture. In effect, it utilizes either the standard deviation or uniform bound implied by the width and adjusts step size accordingly.

3.0.0.4 A second, dynamic axis.

\(\mu\)P addresses variation across tensors of different widths but not variation within a tensor over training. Xavier init specifies a typical starting scale for each parameter, but pretrained networks develop heavy-tailed weight distributions in which a small fraction of parameters end up an order of magnitude beyond that typical scale. We verified this on Qwen3-4B-Instruct [12]: while Xavier initialization produces weights with \(\max \approx 0.04\) and \(\sigma \approx 0.014\), the pretrained weights have \(\max \approx 1.23\) (a \(30\times\) increase) while \(\sigma\) increases only \(1.7\times\). The distribution is heavy-tailed: only \(0.05\%\) of weights exceed \(0.1\) in magnitude, yet these outliers are functionally critical [13] (Figure 1). Under Adam updates, or other optimizer with normalization, some parameters fundamentally have greater distance to travel requiring more optimization steps.

3.0.0.5 SEL as a per-parameter, adaptive analog.

SEL takes a similar underlying observation as \(\mu\)P and pushes it along this second axis. Instead of adjusting learning rates at a tensor or model-level from static init statistics, we change the parameterization sucht that the local sensitivity \(\partial w_{\mathrm{eff}} / \partial w_{\mathrm{raw}}\) grows with \(\abs{w}\). Identical raw-space steps produce larger effective-weight movements for larger parameters. In this sense SEL is a per-parameter, dynamic, empirically motivated counterpart to \(\mu\)P’s per-tensor, static, theoretically motivated rescaling. \(\mu\)P derives scale from typical parameter value obtained from width, SEL obtain scale from individual dimensions throughout training. The two operate on distinct axes, are not mutually exclusive, and can be composed where both apply.

4 Method↩︎

4.1 Background: SymLog and SymExp↩︎

The symmetric logarithm (symlog) and its inverse, the symmetric exponential (symexp), are odd functions that handle signed values spanning many orders of magnitude [4]: \[\begin{align} \operatorname{symlog}(x) &= \operatorname{sign}(x) \ln(\abs{x} + 1) \\ \operatorname{symexp}(x) &= \operatorname{sign}(x) \bigl(\exp(\abs{x}) - 1\bigr) \end{align}\] The symexp function is particularly relevant. For small \(\abs{x}\), it behaves approximately linearly (\(\operatorname{symexp}(x) \approx x\)), while for large \(\abs{x}\), it grows exponentially. It is sign-preserving and differentiable everywhere except at the origin (Figure 2, left).

A natural idea is to use symexp as a weight reparameterization. Store raw parameters \(w\) and use \(\operatorname{symexp}(w)\) as the effective weight. This creates a curved optimization landscape where large weights are amplified. However, bare symexp has a fixed curvature and no mechanism to independently control the balance between its near-linear small-weight regime and its exponential large-weight regime.

SymExpLin generalizes this idea by (1) introducing a curvature parameter \(\beta\) that controls the transition between linear and exponential regimes, (2) separating the exponential and linear contributions into independently scalable pathways, and (3) adding learnable modulation parameters that control curvature, thresholding, and pathway balance.

4.2 The SymExpLin Transform↩︎

Given a raw weight parameter \(w \in \R\), SymExpLin generalizes the symexp function in two key ways: (1) introducing a curvature parameter \(\beta\) that controls the transition between linear and exponential regimes, and (2) adding a linear pathway that we hypothesize may aid in optimization. In its simplest form, the transform is: \[\label{eq:base95transform} w_{\mathrm{eff}} = \frac{\operatorname{sign}(w) \cdot \bigl(\exp(\beta\abs{w}) - 1\bigr)}{\beta} + \frac{w}{\beta}\tag{2}\] The first term is a scaled symexp. Placing \(\beta\) inside the exponent sets the exponential growth rate, while the outer \(1/\beta\) normalizes the leading-order Taylor expansion so the symexp itself contributes unit slope at the origin regardless of \(\beta\) (since \(\exp(\beta \abs{w}) - 1 \approx \beta \abs{w}\) near zero). The linear pathway is likewise divided by \(\beta\), so its contribution is proportional to \(1/\beta\): it adds a small \(1/\beta\) slope near the origin that fades as \(\beta\) grows and the exponential regime dominates. The net effect: \(\beta\) controls the transform’s curvature and growth rate rather than its overall scale.

The choice of \(\beta\) should be calibrated to the typical initialization scale. Recall that Xavier initialization uses \(\sigma \propto 1/\sqrt{d}\), so larger matrices have smaller initial weight magnitudes. To achieve meaningful curvature within this range, \(\beta\) must be correspondingly larger, otherwise the transform remains extensively in its near-linear regime. Figure 2 (right) illustrates this relationship.

4.2.0.1 Adding learnable control.

SymExpLin augments the base form with learnable parameters that balance the effect of the linear and exponential pathways, as well as aspects of the exponential pathway such as its growth rate and an offset for the slope at the origin. The symmetric-exponential pathway is: \[\label{eq:exp95path} \begin{align} E_\theta(\abs{w}) &= e_w\Bigl(\exp\bigl(\beta \abs{w} \cdot m - n\bigr) - \exp(-n)\Bigr) \\ S_\theta(w) &= \operatorname{sign}(w) \cdot E_\theta(\abs{w}) \end{align}\tag{3}\] where \(E_\theta(\abs{w})\) is the exponential amplitude, \(S_\theta(w)\) is the signed symmetric-exponential pathway, \(m\) is a learnable inner multiplier, \(n\) is a learnable offset, and \(e_w\) is a learnable output scale. We initialize \(e_w \approx 1/\beta\), so the base transform recovers the normalized symexp form in Eq. 2 .

The offset \(n\) sets the slope of \(E_\theta\) at the origin: differentiating gives \(E'_\theta(0) = e_w \cdot \beta \cdot m \cdot \exp(-n)\), so with defaults \(e_w = 1/\beta\) and \(m = 1\) the initial slope is exactly \(\exp(-n)\). The default \(n = 0\) recovers unit slope; positive \(n\) shrinks the initial slope, making the transform flatter near zero and delaying when the exponential regime takes over; negative \(n\) raises the initial slope so amplification begins earlier. The two appearances of \(n\) in Equation 3 , as \(-n\) inside the exponent and as \(-\exp(-n)\) outside, are what preserve \(E_\theta(0) = 0\) while allowing \(n\) to modulate this slope; the effect is the single-parameter reshaping just described. Figure 2 (middle) visualizes this effect.

The linear pathway is: \[\label{eq:lin95path} L_\theta(w) = w \cdot l_w\tag{4}\] where \(l_w\) is a learnable scale initialized near \(1/\beta\).

A learnable residual \(\Delta_\phi\) optionally provides an additive correction. In our experiments, this residual is typically implemented as a low-rank LoRA-style update and can be omitted without changing the core SEL parameterization. The full transform in mismatch mode (our recommended configuration) is: \[\label{eq:full95transform} w_{\mathrm{eff}} = \operatorname{sign}(w) \cdot \bigl(E_\theta(\abs{w}) + L_\theta(w)\bigr) + \Delta_\phi\tag{5}\]

4.2.0.2 Role of each component.

The static curvature \(\beta\) controls the transition between linear and exponential regimes while normalizing output magnitude. The inner multiplier \(m\) modifies the growth rate as a multiplier of the static choice of \(\beta\). The offset \(n\) controls where exponential amplification begins. The scales \(e_w\) and \(l_w\) independently weight each pathway’s contribution. When enabled, the residual \(\Delta_\phi\) adds a sign-independent correction; its learning rate is scaled by \(1/\beta\) to keep its effective step size commensurate with the other terms.

4.2.0.3 Comparison of combination modes.

We define two modes for combining the pathways: \[\begin{align} \text{Congruent:} \quad w_{\mathrm{eff}} &= S_\theta(w) + L_\theta(w) \tag{6} \\ \text{Mismatch:} \quad w_{\mathrm{eff}} &= \operatorname{sign}(w) \cdot \bigl(E_\theta(\abs{w}) + L_\theta(w)\bigr) \tag{7} \\ \end{align}\] In congruent mode 6 , the linear part is added outside the sign wrapping and thus independently preserves \(w\)’s sign. In mismatch mode 7 , both parts are inside the sign wrapping: for negative \(w\), the linear term (which is negative) reduces the magnitude. Figure 3 illustrates this difference: the two modes produce identical outputs for positive weights, but for negative weights, mismatch mode produces smaller effective magnitudes than congruent mode. This asymmetry is one mechanism we study for improving early optimization. In our width-1024 ablations, mismatch mode outperforms congruent, particularly when combined with congruent initialization (§4.3).

4.3 Mismatched Initialization↩︎

To ensure that effective weights at initialization match standard Xavier statistics [5], we initialize the raw weights by inverting the transform: given target weights \(W_{\mathrm{target}}\) from Xavier uniform initialization, we solve \(f(w) = W_{\mathrm{target}}\) via Newton’s method.

The key design choice is which combination rule to use for this inversion. In our width-1024 ablations, using the congruent formula 6 for inversion while using the mismatch formula 7 for the forward pass yields the strongest results. We call this mismatched initialization.

Under congruent inversion, Newton’s method finds raw weights \(w\) such that \(S_\theta(w) + L_\theta(w) = W_{\mathrm{target}}\). When these weights are then evaluated with the mismatch formula \(\operatorname{sign}(w)(E_\theta(\abs{w}) + L_\theta(w))\), the behavior diverges by sign:

• Positive weights (\(w > 0\)): Both formulas give the same result, since \(S_\theta + L_\theta = \operatorname{sign}(w)(E_\theta + L_\theta)\) when \(L_\theta > 0\). The initialization is correct.

• Negative weights (\(w < 0\)): The congruent formula gives \(-E_\theta + L_\theta\) (symmetric-exponential opposes, linear reinforces), while the mismatch formula gives \(-(E_\theta + L_\theta) = -(E_\theta - \abs{L_\theta})\) (both wrapped inside the negation, with the linear term’s magnitude reducing the total). The effective weight magnitude is reduced by \(2\abs{L_\theta}\) compared to the congruent target.

For our default parameters (\(\beta = 7.5\)), the linear pathway contributes roughly 10–15% of the total effective weight magnitude for typical Xavier-scale weights. As beta increases, this further drops. This means negative weights are under-initialized by approximately 20–25% in magnitude, while positive weights are initialized correctly. The result is an asymmetric initialization where positive weights start at full strength and negative weights start weaker.

We hypothesize the benefit comes primarily from symmetry breaking: the positive/negative asymmetry gives early gradients a preferred direction to act on, rather than requiring the network to break its own initial symmetry from a fully symmetric Xavier distribution. This may be reinforced by our scheduled scale learning rates (§4.5), which accelerate the learning rate of the scale parameters early on and may let the scales commit to a direction prior to the weights themselves beginning to converge, greater coupling their updates.

4.3.0.1 Xavier uniform over normal.

We find that Xavier uniform initialization consistently outperforms Xavier normal for the target weights before inversion. We attribute this to the bounded support of the uniform distribution: extreme values from the tails of the normal distribution can produce very large raw parameters after inversion through the symmetric-exponential pathway, destabilizing early training.

4.4 Structured Scale Parameters↩︎

Each learnable scale (\(e_w\), \(l_w\), \(m\), \(s\), \(n\)) is parameterized with a chosen factorization pattern. The optional residual \(\Delta_\phi\) is an additive correction, and is usually parameterized as a LoRA-style update. For a weight matrix \(W \in \R^{d_{\mathrm{out}} \times d_{\mathrm{in}}}\), available patterns include:

• Global: a single shared scalar.

• Row / Column: a vector along one axis (\(d_{\mathrm{out}}\) or \(d_{\mathrm{in}}\)).

• Row\(\times\)Col multiplicative: two rank-1 vectors \(r \in \R^{d_{\mathrm{out}}}\), \(c \in \R^{d_{\mathrm{in}}}\) whose outer product \(r \cdot c^\top\) gives the scale matrix. This is our default and recommended mode.

• LoRA: a low-rank additive perturbation \(\text{base} + AB\) where \(A \in \R^{d_{\mathrm{out}} \times r}\), \(B \in \R^{r \times d_{\mathrm{in}}}\), initialized with \(B = 0\) [11].

Each scale has a learning rate multiplier \(k\) that scales its learning rate relative to the base rate, allowing different components of the reparameterization to learn at different speeds. One of the main reasons for this is that the scale parameters are initialized around 1.0 and have to travel a significant distance in weight space for a meaningful effect, as per the prior explanation of 0.5 as a halving operation and 2.0 as a doubling operation.

In our main runs, the output pathway scales \(e_w\) and \(l_w\) use row_col_mult as an efficient trade-off: it is more expressive than a global scalar while remaining much cheaper than full elementwise scale parameters. The inner multiplier \(m\) also uses row_col_mult in our experiments, though simplifying it to a row-only or column-only pattern is a natural future ablation. For the offset \(n\), we typically use a column-wise scale, since offset effects are more sensitive to input-axis structure.

4.4.0.1 Weight decay on scale parameters.

We disable weight decay for scale parameters that primarily control the shape of the reparameterization, such as the curvature multiplier \(m\) and offset \(n\). However, we keep weight decay on the output pathway scales \(e_w\) and \(l_w\). These scales directly weight the symmetric-exponential and linear contributions, so they can increase the overall effective weight magnitude even if the raw weight \(w\) itself is decayed. Applying weight decay to \(e_w\) and \(l_w\) prevents the model from growing the effective weight unboundedly by growing pathway scales.

4.5 Exponential Pathway Learning Rate Annealing↩︎

The symmetric-exponential pathway has two controls that affect its strength: the outer scale \(e_w\) and the inner multiplier \(m\). While both increase the exponential contribution, they serve different roles: \(e_w\) can go negative (flipping the sign of the exponential part), while \(m\) controls the curvature (how sharply the exponential grows with \(\abs{w}\)).

These controls are partially redundant, making joint optimization difficult. We address this with a targeted learning rate annealing schedule: \(e_w\) starts with a high \(k\) value (fast learning) so it can quickly commit to a sign, then its learning rate decays over a cosine-in-log-space schedule, while \(m\) starts with a low \(k\) value and anneals upward to refine the curvature once \(e_w\) has stabilized.

Specifically, for a parameter with initial multiplier \(k_{\text{start}}\) and final multiplier \(k_{\text{end}}\), the effective multiplier at step \(t\) is: \[k(t) = \exp\!\Bigl(\ln k_{\text{start}} + \frac{\ln(k_{\text{end}}/k_{\text{start}})}{2}\bigl(1 - \cos(\pi \cdot \min(t / T, 1))\bigr)\Bigr)\] where \(T\) is the total annealing steps. This interpolates between \(k_{\text{start}}\) and \(k_{\text{end}}\) in log space along a cosine curve, providing smooth transitions. After step \(T\), the multiplier remains at \(k_{\text{end}}\).

5 Experiments↩︎

We evaluate SEL on autoregressive language modeling. All experiments use the same codebase and training infrastructure.

5.1 Experimental Setup↩︎

5.1.0.1 Architecture.

We use a modern transformer with GeGLU feedforward blocks [14] (hidden dimension \(\frac{8}{3}d\) rounded to the nearest multiple of 64), RoPE positional encodings [15], RMSNorm [16], and QK-normalization. SEL is applied to all linear projections: Q, K, V, output projection, and both FFN layers. Biases are enabled and also reparameterized.

5.1.0.2 Scale grid.

We evaluate the full Cartesian product of widths {1024, 2048, 3072} and depths {12, 24, 36}, for nine total configurations.

5.1.0.3 Training.

All models are trained on OpenWebText [17] with sequence length 1024. We use AdamW [18] with \(\beta_1 = 0.9\), \(\beta_2 = 0.999\), weight decay 0.01, gradient clipping at 1.0, and linear learning rate warmup for 250 steps followed by linear decay. Net batch size varies across runs; width-1024 configurations are trained for 250k steps, while width-2048 and width-3072 configurations are trained for 150k steps. Models are compiled with torch.compile (reduce-overhead mode, full graph) and trained with bf16 mixed precision and gradient checkpointing. Compiling is important for minimizing the overhead of our method. Given that our method is composed entirely of elementwise operations, compiling offers a significant benefit in reducing the memory-bound parts of the computation.

5.1.0.4 SEL configuration.

Unless otherwise noted, SEL experiments use: the \(\beta\) value selected by the initialization-scale heuristic described below, mismatch forward mode with congruent initialization, row_col_mult pattern for all scale parameters, \(k_{e_w} = 50\) (annealed to \(k_{e_w,\text{end}} = 8.0\)), \(k_m = 0.01\) (annealed to \(k_{m,\text{end}} = 0.5\)), \(k_{l_w} = 50\),(annealed to \(k_{l_w,\text{end}} = 8.0\)), optional LoRA-style residual \(\Delta_\phi\) with \(k_{\Delta} = 0.5\) when enabled, weight decay enabled for \(e_w\) and \(l_w\) but disabled for the shape-controlling scale parameters, Xavier uniform initialization, beta_as_mult enabled (fixed \(\beta\) appears in the exponent, while the old \(1/\beta\) normalizers are folded into the initial \(e_w\) and \(l_w\) scales).

5.2 Main Results↩︎

Table 2 expands the main comparison to show the two SEL variants run at each scale: mismatch without the optional residual/offset, and mismatch with the optional LoRA-style residual plus offset. At width 1024, the offset+LoRA variant is consistently strongest; at larger widths, the no-offset mismatch variant often matches or exceeds it. We therefore report the better of the two in Table 1.

Figure 4 shows evaluation loss curves across all nine configurations spanning three widths and three depths. SEL consistently outperforms the baseline throughout training at every scale (Table 1). At width 1024 (250k steps), improvements grow with depth: \(\Delta = -0.038\) at depth 12, \(-0.040\) at depth 24, and \(-0.044\) at depth 36. At width 2048 (150k steps), the gains are larger: \(\Delta = -0.047\), \(-0.042\), and \(-0.048\) for depths 12, 24, and 36. At width 3072 (150k steps), gains remain strong: \(\Delta = -0.045\), \(-0.052\), and \(-0.043\). The largest single improvement is \(\Delta = -0.052\) at 3072\(\times\)​24. Across all nine configurations, the baseline never reaches SEL’s final loss within the same training budget.

5.2.0.1 Choosing \(\beta\).

We set \(\beta\) using a simple initialization-scale heuristic rather than treating it as a broad sweep. Because Xavier uniform initialization has bounded support that shrinks with matrix width, larger matrices require larger \(\beta\) values for the symmetric-exponential curvature to become active near typical initial weight magnitudes. In practice, we choose \(\beta\) so that the exponential curve begins to noticeably depart from the linear regime around 1.5–2 initialization standard deviations, while remaining within the support of the Xavier uniform draw. This gives \(\beta = 7.5\) at width 1024, \(\beta \in \{12.0, 12.5, 15.0\}\) at width 2048, and \(\beta \in \{17.5, 20.0\}\) at width 3072.

5.3 Ablation Studies↩︎

We explore a few ablations over specific design choices: initialization asymmetry, the necessity of the linear pathway, and the learning-rate schedule for scale parameters.

5.3.1 Initialization Asymmetry↩︎

At width 1024, we run a comparison varying the combination rule (congruent vs mismatch) along with the optional offset/LoRA components (Table 3). The mismatch forward mode improves over congruent mode at every depth, and adding offset/LoRA further improves the mismatch runs at this scale. The asymmetric sign-wrapping and initialization appears to be useful at all scales, though the main-grid results in Table 2 benefit from offset/LoRA changes at larger widths.

The same pattern holds in the newly run 3072\(\times\)​12 comparison at \(\beta=20\) (Table 4). Mismatch improves over congruent mode both with and without the optional offset/LoRA branch, though in this wider setting the no-offset mismatch run is strongest.

5.3.2 Linear Pathway Necessity↩︎

A natural question is whether the linear pathway \(L_\theta(w)\) is truly necessary, or whether the symmetric-exponential pathway alone is sufficient. To test this, we compare the full method against an exponential-only variant that removes the linear pathway entirely: \[w_{\mathrm{eff}} = S_\theta(w) + \Delta_\phi\] omitting the \(L_\theta(w)\) term from Eq. 5 .

Figure 5 compares the full method against the exponential-only variant over their shared training range. The exponential-only run was stopped early after failing to approach the full model’s learning curve: at 31k steps it reached eval loss 3.870, while the full model was already at 3.142 at the same step and continued to 2.925 by 99k steps. This supports the view that the linear pathway provides a necessary effect.

We hupothesize that the linear term \(L_\theta(w) = w \cdot l_w\) may provide one of two (or both) plausible stabilizing mechanisms: (1) local additive route: even when the symmetric-exponential pathway’s contribution is small (e.g., for near-zero weights or when the offset \(n\) suppresses activation), the linear pathway preserves a direct dependence of effective weights on raw weights; (2) pathway balance: because \(e_w\) and \(l_w\) are learned, the model can adjust how much it relies on the curved symmetric-exponential branch versus the direct linear branch.

5.3.3 LR Annealing↩︎

The scale parameters may need to move quickly enough early in training to select a useful regime, then slow down once the transform has settled. We therefore test whether the schedule in §4.5 matters by comparing the default schedule against fixed learning-rate multipliers. The primary comparison is default annealing vs fixed multipliers.

Table 5 and Figure 6 compare the default annealed schedule against fixed multipliers. At the width-1024 curvature setting, the annealed run is slightly better at the end of training (2.894 vs.) with essentially identical runtime. The scale trajectories show that most of the pathway rebalance occurs early: \(e_w\) rises above its initialization, \(l_w\) falls quickly, and \(m\) remains close to one.

5.3.4 Curvature Parameterization↩︎

The inner multiplier \(m\) controls the effective curvature \(\beta m\) in the exponential pathway. This raises two parameterization questions. First, should the base curvature be treated as a fixed scalar, with \(m\) initialized near one, or should the learned inner multiplier absorb the base \(\beta\) value directly? The latter is more naturally a multiplicative quantity: moving from \(\beta=7\) to \(\beta=14\) and from \(\beta=7\) to \(\beta=3.5\) are symmetric changes in log space, but not in linear parameter space. We therefore parameterize the absorbed-\(\beta\) variant with a positive exponential scale, so additive raw updates map to multiplicative changes in the effective curvature of the exponential pathway.

Second, when \(m\) changes the exponent \(\beta m \abs{w}\), should it also change the near-zero gain of the transform? By default, increasing \(m\) increases both curvature and the local derivative at the origin. The normalize_by_inner_mult variant instead divides the exponential and linear pathways by \(m\), so \(m\) more directly controls curvature/growth rate while leaving the small-signal normalization closer to fixed. We evaluate the resulting \(2\times2\) design in a 100k-step ablation (Table 6).

The combined variant performs best in this sweep, suggesting that log-space curvature updates and inner-multiplier normalization are complementary rather than redundant. For future work, it may be worth repeating the main experiments with the combined variant.

5.4 Analysis↩︎

5.4.1 Weight Distribution Under the Transform↩︎

Figure 7 compares standard baseline weights against mapped SEL weights at initialization and after training for matched 1024\(\times\)​12 runs. At initialization, the baseline raw weights and SEL effective weights are close by construction. After training, the SEL transform produces a visibly different, more heavy-tailed effective-weight distribution than the baseline parameterization.

5.4.2 Scale Parameter Evolution↩︎

The scale trajectories in Figure 6 show the change in \(e_w\), \(l_w\), and \(m\) throughout training. In the annealed run, averaged across tracked effective-scale tensors, \(e_w\) rises from 1.0 to about 1.38, while \(l_w\) falls from 1.0 to about 0.54 by the final evaluation. The inner multiplier \(m\) changes much less, ending near 0.98. Most of the visible pathway rebalance occurs early in training, especially the initial rise in \(e_w\) and drop in \(l_w\).

5.4.3 Transform Shape Visualization↩︎

Figure 8 plots the learned transform by module role for the \(\beta=7.5\) analysis run. We observe in many cases the median example has a tilt, induced by modification of scale parameters, in slope at origin, pointing slightly upwards. The visualization supports our hypothesis that the symmetric-exponential shape is utilized, or even made more extreme, evidenced by the downweighting of the linear path while upweighting of the exponential path.

6 Discussion↩︎

6.0.0.1 Why symmetric-exponential + linear?

The symmetric-exponential pathway provides the magnitude-dependent sensitivity and multiplicative updates: small weights remain in a near-linear regime, while larger raw weights enter a curved regime where linear raw-parameter update steps can produce a larger effective-weight change. The linear pathway gives the model a direct additive route from raw to effective weights, and the learned pathway scales allow training to choose how much to rely on each branch. This is loosely analogous to a residual path in the limited sense that gradients need not pass only through the curved branch, though both pathways are functions of the same underlying weight. The exponential-only ablation proposed in §5.3.2 is designed to test whether both components are essential.

Viewed through the lens of the optimizer-initialization mismatch described in Section 3, SEL can be understood as making output space parameter updates proportional to weight magnitude. Under standard parameterization with Adam, a weight at magnitude \(0.01\) and one at magnitude \(1.0\) receive updates of similar absolute size, but the small weight is perturbed by \(100\times\) more in relative terms. SEL’s symmetric-exponential pathway counteracts this by increasing \(\partial w_{\mathrm{eff}} / \partial w_{\mathrm{raw}}\) with \(\abs{w}\): a comparable raw-space step corresponds to a larger movement in effective weight space once the parameter has entered the curved regime. Thus the parameterization offers the relative scaling that a standard parameterization with Adam would not have, without requiring per-parameter learning rate tuning. \(\mu\)P [8] addresses a related but separate axis of the same mismatch: it rescales per-tensor learning rates from static, width-dependent init statistics/typical dimension magnitudes, whereas SEL adapts effective step size per-parameter on the fly as weights grow. The two are complementary and freely composable.

6.0.0.2 The mismatch insight.

A surprising finding in our ablations is that mismatched initialization outperforms the congruent versions. The specific asymmetry introduced by the mismatch (weaker negative weights) may provide a useful inductive bias or early symmetry-breaking effect.

6.0.0.3 Structured scales as lightweight adaptation.

The structured scale parameters add a small number of auxiliary parameters (two vectors of size \(d_{\mathrm{out}}\) and \(d_{\mathrm{in}}\) per scale, per layer) that modulate the reparameterization. This is significantly cheaper than full element-wise scales and provides enough expressivity to adapt the transform per-row and per-column. The row_col_mult pattern is balanced between expressivity and cost for the output pathway scales. Each output neuron and each input feature has its own multiplicative adjustment to the exponential/linear balance. We also experimented with additive row+column composition, which is appealing for quantities that behave like log-space shifts, such as the inner multiplier or offset. However, additive row+column scales introduce a gauge symmetry: adding a constant to the row vector and subtracting it from the column vector leaves the combined scale unchanged. In practice this underperformed the multiplicative row\(\times\)column form, so we use row_col_mult by default.

6.0.0.4 Future work: pretrained conversion and finetuning.

Because SEL changes only the parameterization of weights, conversion is possible in both directions. After training, effective weights can be materialized and deployed as ordinary linear layers. Conversely, a standard pretrained weight matrix can be inverted into the SEL raw parameter space by solving \(f(W_{\mathrm{raw}}) = W_{\mathrm{pre}}\), yielding an equivalent model at the moment of conversion. This makes continued pretraining and finetuning in SEL space a natural direction for future work.

6.0.0.5 Training and inference cost.

A key practical advantage of SEL over architectural modifications (e.g., mixture-of-experts, nonlinear branches) is that it changes the parameterization without changing the deployed architecture. During training, the transform consists of element-wise operations on weight tensors, so compiled implementations using torch.compile, Triton kernels, and fused AdamW keep the overhead modest. Table 7 shows preliminary H200 profiling for representative small and large configurations. Forward/backward overhead is small, while the fused AdamW step shows the most visible increase. Combining per-step overhead with the step reductions in Table 1, SEL still yields estimated wall-clock speedups to matched validation loss.

These measurements indicate that the transform’s training cost is small relative to the reduction in steps. After training, one computes \(W_{\mathrm{eff}} = f(W_{\mathrm{raw}})\) once and converts to standard linear layers. All auxiliary parameters (structured scales, the curvature \(\beta\), etc.)are folded into the effective weights and discarded, giving zero additional inference cost.

6.0.0.6 Alternative formulations: sinh.

The hyperbolic sine function \(\sinh(x) = (e^x - e^{-x})/2\) shares many properties with symexp: it is an odd function that behaves approximately linearly for small \(\abs{x}\) (since \(\sinh(x) \approx x\) for \(\abs{x} \ll 1\)) and grows exponentially for large \(\abs{x}\). A key advantage of sinh over symexp is that it is fully smooth everywhere, including at the origin, whereas symexp (and by extension SEL) has a non-differentiable point at \(w = 0\) due to the \(\abs{w}\) term. A sinh-based reparameterization \(w_{\mathrm{eff}} = \sinh(\beta w) / \beta\) would provide similar magnitude-dependent curvature while guaranteeing continuous gradients. We leave systematic comparison of sinh-based transforms to future work, noting that preliminary experiments suggest comparable performance with potentially improved optimization smoothness.

6.0.0.7 Interaction with weight decay.

Weight decay is itself magnitude-aware: applying \(w \leftarrow w - \lambda w = w(1 - \lambda)\) shrinks parameters in proportion to their current value, so larger parameters receive larger absolute shrinkage. SEL introduces a different kind of magnitude dependence through the raw-to-effective map: as \(\abs{w_{\mathrm{raw}}}\) grows, the local sensitivity \(\partial w_{\mathrm{eff}} / \partial w_{\mathrm{raw}}\) increases, so comparable raw-space steps can produce larger movements in effective weight space. Thus both mechanisms act in a magnitude-aware manner.

6.0.0.8 Weight decay on auxiliary scales.

Decaying the raw weights alone does not constrain every route by which effective weights can grow: the output pathway scales \(e_w\) and \(l_w\) directly multiply the symmetric-exponential and linear contributions. Thus, they can increase effective magnitudes even when \(w_{\mathrm{raw}}\) is decayed. We therefore apply weight decay to \(e_w\) and \(l_w\), while disabling it for geometry-shaping parameters such as the inner multiplier \(m\) and offset \(n\).

6.0.0.9 Outliers at scale and their functional importance.

Recent work has shown that larger language models develop increasingly pronounced weight and activation outliers. These outliers are not demonstrated to be critical to model performance and degrade performance if removed. [13] documented that outlier features emerge at scale. Models under \(\sim\)​6B parameters have few outliers, but larger models develop specific dimensions with extreme values that dominate attention. [19] found “massive activations” in LLMs that can be 100,000\(\times\) larger than typical values; these activations remain constant regardless of input and function as indispensable bias terms. Most strikingly, [20] showed that pruning a single “super weight” can destroy an LLM’s ability to generate coherent text, increasing perplexity by three orders of magnitude. These super weights, typically found in mlp.down_proj of early layers, create persistent super activations that suppress stopword likelihoods in final logits.

This body of work suggests that during training, some parameters must grow to extreme values and that these are the parameters most disadvantaged by linear additive updates. SEL’s exponential amplification directly addresses this: as weights grow, their effective gradients are amplified, accelerating progress toward large-magnitude targets. At scale, where initialization values shrink further (due to larger matrix dimensions) while the need for extreme outliers persists or intensifies, larger \(\beta\) values may be needed to provide greater curvature and accelerated update speed to reach final values.

6.0.0.10 Multiplicative computation, multiplicative parameterization.

A useful intuition for SEL comes from considering how parameters affect computation. LayerNorm scale parameters provide a clean example: initialized at \(\gamma = 1.0\), moving to \(\gamma = 0.5\) halves the output scale while moving to \(\gamma = 2.0\) doubles it. These are symmetric operations in terms of their computational effect (\(\div 2\) vs \(\times 2\)), yet asymmetric in optimization distance (\(0.5\) vs \(1.0\)). Under Adam’s near-constant update magnitudes, “doubling” takes twice as many steps as “halving.” The natural parameterization for such multiplicative quantities is logarithmic: with \(\gamma_{\mathrm{eff}} = \exp(\gamma_{\mathrm{raw}})\), both operations require \(\abs{\Delta \gamma_{\mathrm{raw}}} = \log 2\).

We suggest this intuition extends, at least partially, to matrix weights. In \(y = Wx\), each weight \(w_{ij}\) multiplicatively scales \(x_j\)’s contribution to \(y_i\). A weight growing from \(0.01 \to 0.1\) and one growing from \(0.1 \to 1.0\) both represent \(10\times\) multiplicative changes, but the latter requires covering \(10\times\) more optimization distance. Not all aspects of the function implemented by the composed neural network may act in a relative manner: biases are additive, activation functions have thresholds at absolute values. However, SEL attempts to respect both regimes. The offset \(n\) provides explicit control over this transition. This “soft log-space” parameterization may explain why the symmetric-exponential pathway helps weights reach large-magnitude targets that the network’s computation requires.

7 Limitations↩︎

• Implementation dependence: The training transform is element-wise and is applied on every forward pass, so its overhead depends on how well these operations are fused by torch.compile/Triton or similar compilers. The additional scale parameters also increase the number of optimizer tensors; foreach or fused optimizer implementations reduce the overhead compared with Python loops over many small tensors.

• Hyperparameter sensitivity: The method introduces several hyperparameters (\(\beta\), \(k\) values for each scale, annealing schedule). We use row_col_mult as a robust structured-scale default and choose \(\beta\) with the width-dependent initialization heuristic above, but further work is needed to understand how these choices should scale beyond our tested model sizes.

• Scale: Our largest configuration (3072\(\times\)​36) is relatively small by modern standards. Validation at larger scales would strengthen the results.

• Task diversity: We evaluate only autoregressive language modeling. The method may behave differently on other tasks (classification, generation with different loss functions, multimodal models).

• Theoretical understanding: We provide intuitive explanations for why mismatched initialization helps but lack formal analysis. A rigorous understanding of the optimization landscape under this reparameterization remains open.

8 Conclusion↩︎

We introduced SymExpLin(SEL), a nonlinear weight reparameterization that decomposes transformer linear layer weights into symmetric-exponential and linear components. One useful insight is that a deliberate asymmetry between initialization and forward-pass combination rules can improve optimization in small-scale ablations, possibly by acting as early symmetry breaking. Learnable scale parameters allow modifying the the weight overall transform, and a targeted annealing schedule manages the transition between partially redundant exponential controls. SEL has modest training overhead in compiled implementations because its transform is element-wise, and adds no inference cost because it requires only a one-time weight materialization after training.

Overall, the remaining ablations support the same picture as the main grid: mismatch remains helpful at 3072\(\times\)​12, the annealed schedule gives a small but consistent improvement over fixed multipliers, and removing the linear pathway produces a clear failed-to-learn case.

9 Implementation Details↩︎

9.0.0.1 SymExpLin Transform (Full).

The complete forward pass for a weight matrix \(W \in \R^{d_{\mathrm{out}} \times d_{\mathrm{in}}}\) is:

1. Compute scaled tensor: \(z = \beta \cdot w \cdot m()\) where \(m()\) is the inner multiplier structured scale.

2. Compute shifted absolute value: \(a = \abs{z} + s()\) where \(s()\) is the shift structured scale.

3. Compute exponential part: \(E = (\exp(a - n()) - \exp(-n())) \cdot e_w()\) where \(n()\) is the offset and \(e_w()\) is the exponential scale.

4. Compute linear part: \(L = w \cdot l_w()\) where \(l_w()\) is the linear scale.

5. Combine (mismatch mode): \(w_{\mathrm{eff}} = \operatorname{sign}(w)(E + L) + \Delta_\phi\).

9.0.0.2 Newton’s Method Initialization.

Given target weights \(W_{\mathrm{target}}\) from Xavier uniform initialization, we solve for raw weights via \(n_{\mathrm{iter}} = 10\) iterations of Newton’s method under the congruent formula: \[\begin{align} f(x) &= \operatorname{sign}(x) \cdot e_{w,0}\bigl(\exp(\beta\abs{x}) - 1\bigr) + x \cdot l_{w,0} \\ f'(x) &= e_{w,0}\beta \cdot \exp(\beta\abs{x}) + l_{w,0} \\ x_{k+1} &= x_k - \frac{f(x_k) - W_{\mathrm{target}}}{f'(x_k)} \end{align}\] Here \(e_{w,0}\) and \(l_{w,0}\) are the initial exponential and linear pathway scales, set to \(1/\beta\) in the default normalized parameterization. The iteration converges rapidly (typically within 5–6 iterations) since \(f\) is monotonically increasing and smooth except at \(x = 0\).

9.0.0.3 Structured Scale Modes.

The row_col_mult mode stores two vectors \(r \in \R^{d_{\mathrm{out}}}\) and \(c \in \R^{d_{\mathrm{in}}}\), initialized so that \(r_i \cdot c_j = v\) for all \(i, j\) where \(v\) is the target value. The effective scale is \(r \cdot c^\top\), broadcast to match the weight matrix shape. For multiplicative composition, each component is initialized to \(\sqrt{v}\) (or \(v^{1/2}\) from the inverse transform if a nonlinear transform is applied).

10 Alternative Formulations↩︎

During development, we explored several alternative formulations before arriving at the version presented in the main paper. Here we document these variants for completeness.

10.1 Combination Modes↩︎

The main paper focuses on the congruent and mismatch modes for combining the symmetric-exponential and linear pathways. Two additional modes were explored:

10.1.0.1 Oppose mode.

In this variant, the linear pathway subtracts from the exponential: \[w_{\mathrm{eff}} = S_\theta(w) - L_\theta(w)\] where \(S_\theta(w)\) is the symmetric-exponential pathway and \(L_\theta(w) = w \cdot l_w\) is the linear pathway. This creates an asymmetry where the symmetric-exponential pathway must “overcome” the linear opposition. We observed reduced performance in this configuration, possibly because the opposing forces create optimization challenges.

10.1.0.2 Exp-only mode.

This variant removes the linear pathway entirely: \(w_{\mathrm{eff}} = S_\theta(w)\). Without the linear pathway, the transform behaves purely exponentially for all weight magnitudes. This showed faster initial progress but sometimes exhibited instability; the linear pathway appears to serve as a stabilizing influence.

10.2 Damped Exponential Formulation↩︎

An alternative formulation introduces explicit damping to prevent exponential blowup for large weight magnitudes: \[w_{\mathrm{eff}} = \operatorname{sign}(w) \cdot \frac{\exp(\beta\abs{w} - n) - \exp(-n)}{\beta \cdot \exp(\abs{w})} + w \cdot \left(1 - \exp(-n \cdot \exp(\abs{w}))\right)\]

This formulation has two key modifications:

10.2.0.1 Damping denominator.

The symmetric-exponential pathway is divided by \(\beta \cdot \exp(\abs{w})\) rather than just \(\beta\). For large \(\abs{w}\), this grows as \(\exp(\abs{w})\), which partially cancels the \(\exp(\beta\abs{w})\) in the numerator. The net effect is that the symmetric-exponential pathway grows as \(\exp((\beta - 1)\abs{w})\) for large weights, taming the otherwise unbounded exponential growth when \(\beta > 1\).

10.2.0.2 Input-dependent linear coefficient.

Instead of a constant linear pathway \(w / \beta\), the coefficient \((1 - \exp(-n \cdot \exp(\abs{w})))\) depends on the weight magnitude. For small \(\abs{w}\), this coefficient is small (suppressing the linear term), while for large \(\abs{w}\) it rapidly saturates to 1. This gives the linear pathway a naturally high contribution for weights that have already grown large, providing stability in the large-weight regime.

The damped formulation is more conservative than the version in the main paper: it provides magnitude-dependent curvature while explicitly preventing runaway growth. However, in practice we found that the simpler undamped formulation (with appropriate \(\beta\) tuning) achieved comparable or better results with fewer hyperparameters to tune.

11 Supplementary Figures↩︎

11.1 Transform Visualizations↩︎

11.2 Mismatched Initialization Analysis↩︎

11.3 Weight Distribution Analysis↩︎

References↩︎