1 Introduction↩︎

Training large language models (LLMs) in full precision (FP32) incurs substantial memory, compute, and energy costs. Modern accelerators are optimized for reduced-precision formats such as FP8 and FP4, via specialized hardware like tensor cores [1]. As a result, most large-scale training pipelines adopt mixed-precision training, executing forward and backward passes in reduced precision [2]. However, they still retain high-precision copies (known as master weights) for optimization, since training directly in low precision results in significant degradation of accuracy and is often unstable [3].

4pt

Recently, the precision of master weights has been reduced, e.g. BF16 instead of FP32, using techniques such as stochastic rounding (SR) [4] to keep them unbiased. While this reduces the memory overhead compared to FP32 master weights, they still exhibit degraded convergence or instability relative to full-precision FP32 training. This is because standard optimizers, such as Adam, fail to make progress at large weight magnitudes under low precision (larger quantization bins in Fig.3) , since updates smaller than the local quantization step are rounded to zero.

To overcome this failure mode, alternative optimizers have been proposed based on multiplicative updates, such as Madam [5]. These are effective at taking larger steps at higher magnitudes (larger quantization bins in Fig.3), where standard optimizers with additive updates fail to make progress. Madam has been shown to be effective for training under extremely low precision, without high-precision master weights. For instance, Madam can train under the logarithmic number system (LNS) [6], an extreme case of floating-point system where all the bits are allocated to the exponent, and mantissa is not present. While LNS has optimal hardware utilization in theory, it is not available in standard AI hardware.

Multiplicative methods such as Madam overcome the failure mode of Adam at large magnitudes (larger quantization bins in Fig.1), but they have their own failure modes. They struggle near zero, where relative updates produce small absolute changes and cannot move weights through zero when there is a sign change (Fig. 1).

Our approach: The additive and multiplicative updates thus suffer from complementary modes of failure. Additive updates get rounded to zero at large magnitudes when mantissa resolution is coarse. Multiplicative updates are able to overcome this and make progress at large magnitudes, but they cannot flip signs or escape exact zero initialization.(Fig. 1 and Table 1).

This motivates us to propose M+Adam. It is a new optimizer that combines Adam-style additive updates with Madam-style multiplicative updates for low-precision training, as outlined in Algorithm 2 and Fig. 3. We prove that M+Adamis a monotone descent method under standard smoothness assumptions, making it a valid optimizer.

We demonstrate stable training with BF16, FP8, and NVFP4 master weights across \(60\)M–\(1\)B LLaMA-style models and \(1\)–\(8\times\) Chinchilla training budgets. M+Adamtrains stably without needing any stochastic rounding (SR), and improves over AdamW and AdamW+SR, as seen in Table ¿tbl:tab:final-all-1x?.

Together, these results support our central claim: combined additive–multiplicative updates are especially useful when low-precision master weights becomes a dominant source of optimization error.

2 Related Work↩︎

We review prior work on multiplicative optimization and low-precision training most closely related to our approach.

2.1 Multiplicative Updates↩︎

Multiplicative update rules apply relative, scale-invariant parameter changes and have a long history in optimization [7]. In deep learning, Madam [5] adapts this principle as a multiplicative analogue of Adam, showing improved robustness and reduced sensitivity to learning-rate scaling. Recent work revisits multiplicative or hybrid updates in modern settings: [8] study additive–multiplicative hybrids; [6] combine multiplicative updates with a logarithmic number system to enable low-precision training without FP32 master weights; and [9] analyze log-normal multiplicative dynamics under low-precision forward passes. In contrast to log-based Madam approaches, M+Adamremains within the standard floating-point arithmetic.

2.2 Low-precision Training↩︎

Mixed-precision frameworks such as AMP [10] and Transformer Engine [11] accelerate BF16/FP8 arithmetic, but are largely orthogonal to optimizer design and typically rely on FP32 master weights for stable accumulation [2]. Standard additive optimizers such as AdamW can degrade when master weights are stored in reduced precision, since nearest rounding can cancel small model-weight updates [3]. Stochastic rounding preserves small updates in expectation and can mitigate deterministic rounding bias [12]. Kahan-style compensated updates provide another remedy by accumulating lost low-order update residuals in an auxiliary buffer, and have been shown to recover much of the accuracy lost in pure BF16 training [3]. While both stochastic rounding and Kahan compensation can improve accumulation accuracy, they do not change the underlying additive update geometry or address dynamic-range limitations directly.

Several complementary approaches reduce low-precision storage costs without changing the optimizer geometry. LoQT [13] trains most weights in 4-bit precision while maintaining a low-rank 16-bit subspace. A separate line of work compresses optimizer states through 8-bit or 4-bit moment quantization [14], [15]. [16] study stable FP8 LLM pretraining at trillion-token scale with FP32 weights and FP8 forward computation, including quantized optimizer moments, focusing on system-level scaling and mixed-precision pipelines. In contrast, M+Adammodifies the update rule itself through a joint additive–multiplicative formulation, targeting the stability of low-precision weight storage rather than optimizer-state compression.

Recent work [17] explores training with 4-bit weights and activations using micro-scaling formats such as MXFP4, where each block shares a common scale factor. These approaches mainly stabilize low-bit matrix multiplication by managing outliers and representational fidelity during computation. Because low-bit quantization is highly discrete, such systems often keep higher-precision weights and round to the target format before GEMMs. Our work addresses a complementary challenge: stable gradient-based parameter updates when the master weights themselves are stored in BF16, FP8, or NVFP4. Thus, outlier-aware quantization and M+Adamattack different bottlenecks: the former improves low-bit computation, while the latter improves numerical stability across optimization steps.

3 Background↩︎

We briefly review Adam and Madam, the two optimizers that M+Adambuilds upon.

Adam. Adam [18] maintains exponential moving averages of the first and second moments of the gradient and applies adaptive, element-wise additive updates. AdamW [19] decouples weight decay from this adaptive gradient step: \[\text{Adam}(w,g) \;=\; (1-\eta\lambda)w - \eta \frac{\mu}{\sqrt{\nu}+\epsilon}.\]

While effective in full precision, these additive updates can be rounded away in low-precision formats when their magnitudes fall below the local quantization step, which grows with \(|w|\) in floating-point formats such as BF16 or FP8.

Madam. Madam [5] retains Adam’s moment estimates but applies updates multiplicatively: \[\text{Madam}(w,g) \;=\; w \exp\!\Big(-\eta\,\frac{\mu}{\sqrt{\nu} + \epsilon}\Big).\] This yields scale-invariant behavior, with step sizes proportional to \(|w|\), but purely multiplicative updates preserve sign and cannot escape \(w=0\). We illustrate these complementary failure modes of AdamW and Madam in a controlled toy example (Sec. 5.1), which motivates combining local additive updates with multiplicative scale control.

4 M+Adam: Additive–Multiplicative Optimization↩︎

M+Adamis designed for low-precision weight storage, where the optimizer must update weights on a nonuniform floating-point grid. The method combines two complementary update mechanisms: an Adam-style additive branch that acts directly on the weights, and a Madam-style multiplicative branch that applies relative scale changes. This section derives the two branches, explains how they are combined, and presents the corresponding descent result.

4.1 Additive–Multiplicative Updates↩︎

Low-precision floating-point weights have nonuniform resolution: the spacing between representable values grows with the exponent. As a result, purely additive updates can become rounded back to zero at large magnitudes. When \(|w_t|\) is large, an additive update can be smaller than the local quantization bin and disappear after rounding; when \(|w_t|\) is small, however, additive updates remain important because they can update values near zero and change signs. M+Adamis designed around this observation: it combines an Adam-style additive correction with a Madam-style multiplicative scale update.

Let \(g_t=\nabla_w \mathcal{L}(w_t)\) be the gradient of weight \(w_t\). The additive branch is the standard Adam-style update \(u_t^{(a)}\) in the weight space, \[u_t^{(a)} = -\eta_a\,\dfrac{\hat{u}}{\sqrt{\hat{v}}+\epsilon}.\]

A multiplicative update is as follows element-wise: \[w_{t+1}=w_t \exp(a_t)= w_t 2^{a_t/\ln 2},\] where \(a_t\) is an adaptive preconditioned descent direction. Naturally, the update \(a_t\) is based on the gradient \(g_t^{(e)}\) with respect to the exponent. To obtain this, we use the mantissa \(m_t\) and exponent \(e_t\) in floating-point decomposition in the base-2 system: \[w_t = m_t 2^{e_t},\] The corresponding gradient simplifies as \[g_t^{(e)} = \frac{\partial \mathcal{L}}{\partial e} = (\ln 2)\, w_t g_t .\] Thus the gradient \(g_t^{(e)}\) can be expressed without the explicit exponent value, and we use this in our updates.

We then apply adaptive normalization to \(g_t^{(e)}\). In our implementation, the multiplicative branch uses a second-moment accumulator, \[\begin{align} v_t^{(e)} &= \beta_2 v_{t-1}^{(e)} + (1-\beta_2)\bigl(g_t^{(e)}\bigr)^2,\\ \widetilde{u}_t^{(m)} &= -\eta_m \frac{g_t^{(e)}}{\sqrt{\hat{v}_t^{(e)}}+\epsilon}. \end{align}\]

Thus, \(\widetilde{u}_t^{(m)}\) defines the branch’s proposed scale movement: \[\rho_t =\max(|w_t|,\tau_t), \qquad u_t^{(m)} = \operatorname{clip}\!\left(\widetilde{u}_t^{(m)}/ \rho_t\right)\] The threshold \(\tau_t>0\) prevents division by very small weights, and clipping keeps the relative factor well-defined. \[w_t \mapsto w_t + w_t u_t^{(m)}.\]

Finally, M+Adamcombines the multiplicative rescaling with the additive displacement: \[w_{t+1} = w_t+w_t u_t^{(m)}+u_t^{(a)}.\]

This structure directly targets the two failure modes shown in Fig. 1. Additive updates can change signs and update small-magnitude weights, but they may be lost to rounding at large magnitudes. Multiplicative updates preserve relative progress at large magnitudes, but by themselves are sign-preserving and cannot recover exact zeros. By combining both branches, M+Adammaintains scale-aware progress where additive updates are rounded away while retaining the local flexibility needed near zero. Algorithm 2 summarizes the core method. Optionally, clipping is added for more stability, which is not listed in Algorithm 2, as outlined in App. 8.

4.2 Theoretical Analysis↩︎

We analyze the deterministic exact-arithmetic core of Algorithm 2. Let \(\mathcal{L}(w)\) be the weight-space objective, let \(g=\nabla_w\mathcal{L}(w_t)\), and define the base-2 gradient with respect to exponent signal \[g^{(e)}=(\ln 2)w_t g,\] obtained by differentiating \(z\mapsto \mathcal{L}(w_t 2^z)\) at \(z=0\). Algorithm 2 forms a relative multiplicative update \(u^{(m)}\), sets \(s=1+u^{(m)}\), and updates \(w_{t+1}=w_t s+u^{(a)}\). For \(s>0\), writing \(z=\log_2s\) gives the equivalent scale-then-add form \(w_{t+1}=w_t 2^z+u^{(a)}\). We assume the following local smoothness bound in these coordinates: for all \((d,z)\) in a neighborhood of the update, \[\label{eq:scale-add-smooth} \begin{align} \mathcal{L}(w_t 2^z+d) \leq\;& \mathcal{L}(w_t) + \langle g,d\rangle + \langle g^{(e)},z\rangle \\ &+ \frac{L_a}{2}\|d\|^2 + \frac{L_e}{2}\|z\|^2 + L_{ae}\|d\|\,\|z\| . \end{align}\tag{1}\] Here \(L_a\) and \(L_e\) control curvature within the additive and exponent directions, while \(L_{ae}\) controls their interaction.

The theorem formalizes the local two-branch geometry of Algorithm 2: an additive weight-space displacement and a multiplicative scale change can be composed while preserving descent under the stated alignment and smoothness conditions. The proof is given in Appendix 11.

5 Empirical Findings↩︎

We evaluate M+Adamin two stages. First, a controlled BF16-stored matrix-fitting diagnostic isolates the update-level failure modes predicted by our analysis. Second, LLaMA-style pretraining on C4/EN tests whether the same additive–multiplicative geometry improves low-precision training across model scale, token budget, and precision regime. Our main LLM study covers 60M, 130M, and 350M models over \(1\), \(2\), \(4\), and \(8\times\) Chinchilla budgets, and we additionally evaluate 1B models at the \(1\times\) budget across all precision regimes. We consider BF16, FP8, and NVFP4 master-weight storage with BF16 or FP8 compute.

3.5pt

6pt

6pt

5.1 Toy Diagnostic: Isolating Low-Precision Failure Modes↩︎

We begin with a minimal convex diagnostic designed to highlight the complementary strengths of additive and multiplicative optimizers. The task is matrix fitting, \[\mathcal{L}(W)=c \|W-W^\star\|_F^4,\] where each step samples a minibatch of entries uniformly with replacement and \(c\) is a constant. Each optimizer step is computed in FP32, but the updated parameter is immediately rounded through BF16 before the next iteration. Since the objective is otherwise trivial, any failure reflects the interaction between the update geometry and BF16 storage. This directly tests the complementary regimes isolated by the theory. For the evaluation, we perform a gradient step using either Additive or Multiplicative, corresponding to the additive and multiplicative branches of M+Adam, respectively. We disable momentum and adaptive moments and tune the learning rate for simplicity. For M+Adam we have \(\eta_a = \eta_m\), and in the figures, the Additive and Multiplicative baselines have their optimal learning rate for each setting, whereas M+Adam reuses the same learning rate across all settings.

Fig. 4 summarizes four diagnostic regimes. In the sign-flip regime, additive steps are needed to change signs: Additive and M+Adam correct the sign, while Multiplicative remains sign-preserving. In the zero-revival regime, Multiplicative cannot escape from zero because its update is zero-absorbing, whereas Additive and M+Adam recover zero weights through the additive branch. In the small-weight regime, M+Adam retains the additive updates needed to reach the BF16-limited floor. In the large-weight regime, Additive stalls because its updates fall below the local BF16 spacing, while Multiplicative and M+Adam continue to make relative, scale-aware progress. Overall, the toy example isolates the intended complementarity: the additive branch enables sign changes, recovery from zero, and small-magnitude updates, while the multiplicative branch preserves progress when the quantization grid becomes too coarse. Full learning-rate sweeps and exact losses are provided in App. 9.

5.2 Language-Model Pretraining Setup↩︎

We evaluate M+Adamon LLaMA-style autoregressive language-model pretraining on C4/EN. For a model with \(P\) million trainable parameters, the \(1\times\) Chinchilla budget is \(T_{1\times}\approx 20P\times 10^6\) tokens, and the main scaling study evaluates \(n\times\) budgets for \(n\in\{1,2,4,8\}\). This design is important because optimizer rankings can depend on the data-to-model ratio, training horizon, and tuning protocol [20], [21]. The 60M–350M models are evaluated across the full \(1\)–\(8\times\) grid. The 1B model is evaluated at \(1\times\) Chinchilla across all precision regimes to test whether the observed gains persist beyond the largest model used in our full scaling sweep.

We study four increasingly aggressive low-precision regimes: , , , , where the notation denotes weight storage / compute precision. BF16/BF16 corresponds to end-to-end BF16 training without FP32 master weights. BF16/FP8 keeps BF16 weights but uses Transformer Engine FP8 GEMMs. FP8/FP8 further stores selected trainable weight matrices persistently in FP8 and uses FP8 compute. NVFP4/FP8 uses true NVFP4 master-weight storage with FP8 GEMMs. For the FP8 master-weight setting, we apply FP8 storage to attention and MLP linear layers, while embeddings, layer norms, and the language-model head remain in BF16. The selected matrices are quantized in blocks with FP16 scales. During training, the optimizer operates under the constraint that these weights are repeatedly quantized to FP8 after updates. In our FP8 experiments, this storage is simulated through explicit quantize/dequantize operations for compatibility with current Transformer Engine kernels.

We compare against AdamW in each regime. In BF16/BF16, we also include two stronger additive baselines: AdamW with stochastic rounding and AdamW with Kahan-style compensated accumulation. Stochastic rounding probabilistically rounds low-precision updates to nearby representable values so that small updates are preserved in expectation [4]. Kahan-style compensated accumulation keeps an auxiliary correction buffer that accumulates low-order update residuals lost to rounding, thereby improving additive update accumulation under BF16 storage [3]. We additionally report an FP32-weight/TF32-compute AdamW reference as a higher-precision point of comparison.

5.2.0.1 Hyperparameter tuning.

We use a coordinate-descent protocol  [20], [21] for each optimizer and precision regime. At \(1\times\) Chinchilla, we sweep learning rate, warmup length, \(\beta_1\), \(\beta_2\), \(\epsilon\), gradient clipping, and decoupled weight decay where applicable. For M+Adam, we additionally sweep the additive and multiplicative learning rates. To control cost at larger budgets and model sizes, we identify the hyperparameters that are scaling-sensitive and re-sweep only those in the remaining regimes, while holding the others fixed. We accept a hyperparameter change only when it improves final validation loss by at least \(\Delta_{\mathrm{loss}}\ge 3\times 10^{-3}\). Full grids, sweep traces, and best configurations are reported in App. 16.

5.3 Scaling Across Budgets and Precision Regimes↩︎

Fig. 5 reports C4/EN validation perplexity for 60M, 130M, and 350M models across \(1\)–\(8\times\) Chinchilla budgets. Each panel compares tuned AdamW and M+Adamacross the evaluated low-precision regimes.

Across model sizes, token budgets, and precision regimes, M+Adamconsistently improves over the corresponding AdamW baseline. The gap persists as the Chinchilla ratio increases, showing that the improvement is not merely an early-training effect. The same trend also holds as the precision regime becomes more aggressive. FP8 compute degrades absolute perplexity for both optimizers, and FP8 weights make the problem harder still, but M+Adamretains a clear advantage.

5.4 Main \(1\times\) Chinchilla Results↩︎

Table ¿tbl:tab:final-all-1x? summarizes the main \(1\times\) Chinchilla results across 60M, 130M, 350M, and 1B models. M+Adamimproves over AdamW in every low-precision regime and at every model size. In BF16/BF16, M+Adamreduces perplexity relative to AdamW by \(2.8\%\), \(6.6\%\), \(13.3\%\), and \(9.6\%\) for 60M, 130M, 350M, and 1B models, respectively. It also improves over AdamW+Kahan and AdamW+SR. Kahan compensation and stochastic rounding both help additive updates survive low-precision accumulation, but neither changes the underlying additive update geometry. This suggests that explicitly separating additive and multiplicative update mechanisms may provide benefits beyond improvements to the additive update alone, as in SR or Kahan compensation.

When compute is reduced to FP8 while weights remain BF16, M+Adamcontinues to improve over AdamW, with relative perplexity reductions of \(3.2\%\), \(6.2\%\), \(6.9\%\), and \(4.0\%\). The FP8-weight/FP8-compute regime is more challenging because the optimizer must repeatedly update weights that are quantized to FP8. Even in this setting, M+Adamimproves over AdamW by \(3.4\%\), \(2.7\%\), \(10.5\%\), and \(3.0\%\). The 350M result is especially notable: M+Adamwith FP8 weights and FP8 compute reaches 19.395 perplexity, narrowing the gap to BF16-weight AdamW despite using a coarser weight format.

The NVFP4/FP8 regime amplifies the failure mode of purely additive updates because the representable bins are even coarser. M+Adamgives its largest gains in this setting, reducing perplexity over AdamW by \(8.7\%\), \(7.3\%\), \(16.6\%\), and \(7.2\%\) for 60M, 130M, 350M, and 1B models. These results directly support our central claim: as weight precision becomes more restrictive, additive–multiplicative updates become increasingly valuable.

Finally, the 1B results suggest that the gains extend beyond the 60M–350M sweep: even at \(1\times\) Chinchilla, M+Adamimproves over AdamW across all evaluated precision regimes.

5.5 Hyperparameter Transfer and Scaling Sensitivity↩︎

M+Adamintroduces additional hyperparameters relative to AdamW, especially separate learning rates for the additive and multiplicative branches. A natural concern is whether these require exhaustive re-tuning across model sizes and token budgets. Our coordinate-descent results suggest that they do not: after tuning a base configuration, only a small subset of hyperparameters is sufficiently scaling-sensitive to warrant re-sweeping, while the remaining settings transfer across regimes. For the NVFP4 experiments, we also investigated whether the FP4 block scale should influence the relative contribution of the additive and multiplicative updates, since the block scale changes the effective numerical resolution during training. In practice, we found that using fixed global coefficients for the two update branches is sufficient, and therefore keep this simpler formulation in the final algorithm. Additional analysis of the block-scale effect is provided in  16.

This transferability is consistent with the optimizer design. The multiplicative branch controls relative scale changes, while the additive branch provides local weight-space updates, so a single learning rate does not need to simultaneously handle both scale adaptation and local corrections. Under a matched restricted re-tuning protocol, in which both the baselines and M+Adamre-sweep only the hyperparameters identified as scaling-sensitive (Table 2), M+Adamcontinues to achieve lower validation perplexity across the regimes reported in App. 16. This suggests that the improvements are not simply an artifact of giving M+Adama larger hyperparameter search space, but persist under a comparable tuning budget.

5.6 Optimizer-State Overhead and Apollo-E Compression↩︎

M+Adamremoves FP32 master weights but adds one optimizer state: a second-moment accumulator for the multiplicative branch. Since prior work shows that optimizer states can often be compressed with low-rank approximations [22]–[24], we test whether this added state can be compressed independently.

Apollo [22] compresses optimizer states using low-rank gradient projections, storing and updating the optimizer statistics in a reduced-dimensional subspace. We apply Apollo only to the exponent/multiplicative second-moment accumulator; we refer to this variant as Apollo-E, where “E” denotes the exponent branch. We use rank \(r=4\) for both 130M and 350M models. The additive AdamW-style branch and the additive–multiplicative update rule are otherwise unchanged. Table 3 shows that Apollo-E keeps the optimizer-state footprint close to BF16-state AdamW while retaining most of M+Adam’s perplexity gains. Detailed Apollo-E variants and rank sweeps are given in App. 15.

6 Limitations and Future Work↩︎

M+Adamfocuses on optimization geometry for low-precision weights rather than optimizer-state compression or systems-level acceleration. The current prototype introduces one additional optimizer state for the multiplicative branch and incurs runtime overhead relative to AdamW, e.g. roughly \(6\%\!-\!9\%\) at 1B scale (App. 14). At lower precision, overhead also comes from simulating FP8/FP4 weight storage through explicit quantize/dequantize steps, since fully native low-bit parameter-update support is still limited in current training stacks. Our Apollo-E results suggest that the additional state can be compressed independently, while fused kernels and native low-precision update paths could further reduce the cost of the multiplicative branch.

Our theoretical analysis studies an idealized additive–multiplicative update and does not fully model practical ingredients such as stochastic gradients, Adam-style moments, clipping, or distributed training. Finally, while we evaluate up to 1B-parameter LLaMA-style models across BF16, FP8, and FP4 regimes, larger-scale studies, longer-horizon FP8/FP8 training, and broader downstream evaluations remain important future work. Another natural direction is to study whether M+Adamcan be combined with stochastic rounding or Kahan-style compensated accumulation.

7 Conclusion↩︎

We present M+Adam, an optimizer that combines additive Adam-style updates with multiplicative Madam-style updates. This is effective under low-precision arithmetic: the additive branch provides local updates, while the multiplicative branch makes scale-aware relative updates in cases when the additive updates are rounded off to zero. Our analysis provides an idealized descent guarantee, and our experiments show that this geometry targets a concrete failure mode of low-precision training. Additive updates can get lost to rounding under coarse quantization, while multiplicative updates struggle with zeros and sign changes; combining them mitigates both failure modes. Across 60M-1B parameter LLaMA-style models, multiple Chinchilla budgets, and BF16/FP8/FP4 regimes, M+Adamconsistently improves over AdamW, with the largest gains in the most aggressive low-precision settings. Overall, M+Adamshows that low-precision training benefits from optimizers aligned with floating-point structure. By directly addressing quantization-induced update failures, it provides a principled step toward end-to-end low-precision training without relying on FP32 master weights.

Impact Statement↩︎

This work aims to advance the field of machine learning by enabling more efficient and stable low-precision training of large neural networks. While improved training efficiency may have downstream environmental and economic implications, we do not foresee any direct negative societal impacts arising specifically from this work.

Acknowledgements↩︎

Sebastian Loeschcke is supported by the Danish Data Science Academy, which is funded by the Novo Nordisk Foundation (NNF21SA0069429) and VILLUM FONDEN (40516). Anima Anandkumar is supported in part by the Bren endowed chair, and the AI2050 Senior Fellow program at Schmidt Sciences. We acknowledge TACC and Caltech HPC for computing resources.

8 Detailed Algorithm↩︎

We provide pseudocode for our method in Algorithm 6. Our presentation treats M+Adamas a weight-space optimizer with two coupled update components: an additive component and a multiplicative component. All operations are performed elementwise.

At each step, we first compute the stochastic gradient \[g = \nabla_w \mathcal{L}(w).\] The additive component follows an Adam-style update in weight space, using first and second moments to form a preconditioned direction and an additive update \(u^{(a)}\). The multiplicative component is designed to capture relative changes in the weights: it starts from the scale-sensitive quantity \[g^{(e)} = (\ln 2)\,w g,\] applies variance normalization, and converts the resulting proposal into a relative multiplicative update \(u^{(m)}\). For optional exponent-step clipping, this relative change can be mapped into a log-base-2 multiplicative increment \[\Delta e = \log_2(1+u^{(m)}).\] The two components are then composed directly in weight space as \[w^{+} = w + w u^{(m)} + u^{(a)}.\]

The additive component keeps Adam-style first and second moments, whereas the multiplicative component uses second-moment normalization only. This design preserves the intended additive–multiplicative behavior while improving numerical stability in finite precision. We also allow optional multiplicative-step clipping, multiplicative weight decay, additive weight decay, and optional final weight clipping. \[\mathrm{clamp}_a(x)=\max(-a,\min(x,a))\] where needed. This differs from standard global gradient-norm clipping: our cap acts on the multiplicative update component itself, rather than on the raw gradient. A small \(\epsilon\) stabilizes divisions in both components, and a small threshold \(\tau>0\) is used when converting multiplicative weight proposals into relative changes near zero.

9 BF16-Stored Matrix Fitting to Isolate Low-Precision Failure Modes↩︎

We describe the minimal convex diagnostic used to isolate how update geometry interacts with low-precision weight storage. The goal is explanatory rather than competitive: we remove data and architecture and study only whether an optimizer update survives repeated BF16 storage.

9.0.0.1 Setup.

We minimize \[\min_{W\in\mathbb{R}^{n\times n}} \mathcal{L}(W) = c \|W-W^\star\|_F^4, \label{eq:toy95obj95app}\tag{3}\] We adopt this loss rather than the typical squared error, to prevent gradient descent from converging in a single step. For this reason we also introduce stochastic samples of the entries and calculate the loss as the average of the sampled entries. We pick \(B\) samples uniformly at random with replacement. Each step is computed in FP32, but the parameter is immediately rounded through BF16 before the next iteration: \[\widetilde{W}_{t+1}=\mathrm{Update}(W_t,G_t), \qquad W_{t+1}=\mathrm{RoundBF16}(\widetilde{W}_{t+1}). \label{eq:bf1695roundtrip95app}\tag{4}\] Thus, failures in this experiment come only from the interaction between the update rule and BF16 storage.

9.0.0.2 Update rules.

Additive is a simple additive gradient step: \[u_t^{(a)} = -\eta_a g_t . \label{eq:adam95add95app}\tag{5}\]

For Multiplicative, we first form the coordinate-wise gradient with respect to exponent signal \[g_t^{(e)} = (\ln 2)\,w_t g_t .\] We then apply the multiplicative update \[\widetilde{u}_t^{(m)} = -\eta_m g_t^{(e)},\] \[\rho_t = \max(|w_t|,\tau),\] \[u_t^{(m)} = \operatorname{clip} \left( \frac{\widetilde{u}_t^{(m)}}{\rho_t}, -r_{\max},r_{\max} \right),\] \[w_{t+1} = \operatorname{bf16} \left( w_t+w_t u_t^{(m)} \right).\] We set \(r_{\max}=0.75\) and \(\tau=10^{-8}\).

For M+Adam, we follow the same steps as above but apply both branches: \[w_{t+1} = \operatorname{bf16} \left( w_t+w_t u_t^{(m)}+u_t^{(a)} \right).\]

9.0.0.3 Diagnostic regimes.

We use five regimes, each chosen to expose a specific low-precision failure mode:

1. Sign flip: success requires changing signs.

2. Zero revival: success requires escaping exact zero initialization.

3. Small weights: success requires fine local updates near zero.

4. Large weights: success requires updates that remain effective under coarse BF16 spacing.

5. Mixed scales: success requires handling small and large coordinates simultaneously.

• Sign flip. \[n=4096,\qquad c=1.\] Targets are random signs: \[w_i^\star \in \{-1,+1\},\] and the initialization has the opposite sign: \[w_{0,i}=-w_i^\star.\]

• Zero revival. \[n=4096,\qquad c=1.\] Targets are random signs: \[w_i^\star \in \{-1,+1\},\] and all weights start exactly at zero: \[w_{0,i}=0.\]

• Small weights. \[n=4096,\qquad c=10^8.\] Targets are small Gaussian values: \[w_i^\star = 10^{-4} z_i,\qquad z_i\sim\mathcal{N}(0,1),\] and all weights start at zero: \[w_{0,i}=0.\]

• Large weights. \[n=4096,\qquad c=10^{-7}.\] Targets are positive large-scale values: \[w_i^\star = 900\max(|z_i|,1/900), \qquad z_i\sim\mathcal{N}(0,1),\] so \(w_i^\star\ge 1\). All weights start at \[w_{0,i}=100.\]

• Mixed scales. \[n=4096.\] The first half uses the small-weights regime: \[i\le 2048:\qquad w_i^\star = 10^{-4}z_i,\quad w_{0,i}=0,\quad c_i=10^8.\] The second half uses the large-weights regime: \[i>2048:\qquad w_i^\star = 900\max(|z_i|,1/900),\quad w_{0,i}=100,\quad c_i=10^{-7}.\]

For fairness, we tune learning rates separately for each optimizer and regime. For the main diagnostic figure, M+Adam sets \(\eta_a=\eta_m\) and uses one fixed shared learning rate across all regimes. The additive component handles sign changes, recovery from zero, and small-magnitude updates, while the multiplicative component controls scale-aware progress at large magnitudes. Fig. 7 shows the best sweep curves.

9.0.0.4 Main observations.

The results support four qualitative conclusions. First, multiplicative-only updates fail when the solution requires sign changes or escaping exact zero initialization; these are structural limitations, not learning-rate artifacts. Second, additive updates are effective for sign changes, zero revival, and local updates, but can be rounded away at large magnitudes because BF16 spacing grows with \(|W_t|\). Third, mixed-scale problems expose a conflict for purely additive updates: a step size that affects large entries can be too coarse for small entries, while a step size that updates small entries may not survive rounding for large entries. Finally, M+Adam avoids the main weakness of either branch alone: the additive component handles local updates, sign changes, and zero revival, while the multiplicative component gives scale-aware progress at large magnitudes.

9.0.0.5 Summary.

This toy problem is intentionally simple, but it provides a plausible mechanism behind the main empirical LLM results. Purely additive updates fail when quantization bins become too wide; purely multiplicative updates fail near zero and across sign changes. M+Adam combines the two geometries, giving relative progress at large magnitudes while retaining the flexibility of additive optimization near zero.

10 Hyperparameter Sensitivity↩︎

We claim reduced hyperparameter sensitivity in BF16. As direct evidence, we report learning-rate sensitivity for optimizers AdamW and M+Adam under the same BF16 training setting and token budget (1\(\times\)Chinchilla), across three model sizes (60M/130M/350M). For AdamW we sweep the global learning rate. For M+Adam we separately sweep the mantissa step size lr\(_m\) and exponent step size lr\(_e\) around the tuned configuration (holding all other hyperparameters fixed as in the sweep tables). We plot the resulting validation losses; for readability under occasional divergence/outliers, the visualization uses a log-scaled view of loss deviations so that stable regions remain visually comparable. These 1D sweeps provide a minimal, direct sensitivity check supporting the claim of improved robustness.

11 Proofs↩︎

11.1 Proof of the Scale-Then-Add Descent Theorem↩︎

Relation to Algorithm 2. Algorithm 2 computes an additive update \(u^{(a)}\) and a relative multiplicative update \(u^{(m)}\). With \[s=1+u^{(m)},\] its core composition is \[\begin{align} w_{t+1} &= w_t+w_t u^{(m)}+u^{(a)} \\ &= w_t s+u^{(a)} . \end{align}\] When \(s>0\) elementwise, this is equivalently \[w_{t+1}=w_t 2^z+u^{(a)}, \qquad z=\log_2 s .\] The theorem analyzes this same scale-then-add map in the local coordinates \((d,z)\), with \(d=u^{(a)}\). The adaptive moments, normalization, thresholding, and clipping in Algorithm 2 determine the concrete branch outputs \(u^{(a)}\) and \(u^{(m)}\); the descent statement only requires these outputs to satisfy the stated alignment and size conditions. Thus the theorem captures the local additive–multiplicative geometry of the algorithm without committing to a particular implementation of the branch preconditioners.

12 Model Architectures↩︎

10pt

13 Memory Comparison↩︎

Table 5 summarizes the per-parameter memory accounting. In its uncompressed version, M+Adamreplaces FP32 master weights with lower-precision weights, but introduces an additional optimizer state for the multiplicative branch. As a result, the BF16-master version has the same memory footprint as AdamW with FP32 master weights in this accounting, while the FP8-master version is slightly smaller. Thus, M+Adamshould not be interpreted as an optimizer-state memory-reduction method by itself. Its contribution is instead to improve the numerical stability of low-precision weight updates and reduce reliance on FP32 master weights. Our Apollo-E results in App. 15 further suggest that the added multiplicative state can be compressed independently, making optimizer-state compression complementary to the proposed additive–multiplicative update geometry.

14 Compute Resources and Runtime Overhead↩︎

We report representative compute resources and wall-clock runtimes for the main \(1\times\) Chinchilla experiments in Tables 6 and 7. All runs were executed on NVIDIA H200 GPUs using distributed training. The reported wall-clock time measures end-to-end training time for the corresponding final configuration, including forward and backward passes, optimizer updates, evaluation overhead, and logging overhead as used in our training pipeline. Per-update time is computed as wall-clock time divided by the number of optimizer steps.

The overhead numbers compare our current M+Adamimplementation against AdamW within the same training stack and kernel configuration. They should therefore be interpreted as implementation-level overheads rather than intrinsic lower bounds for the additive–multiplicative update rule. In particular, our current implementation simulates some low-precision storage paths, such as FP8/FP4 master weights, through explicit quantize/dequantize operations for compatibility with existing Transformer Engine kernels. These software-simulation steps can introduce additional overhead that would not necessarily remain in a fused or fully native low-precision implementation.

These measurements are intended to quantify the practical overhead of M+Adamand AdamW+SR relative to AdamW in our implementation. They do not include the full research compute used for preliminary experiments, failed runs, debugging, or hyperparameter sweeps. Thus, the total project compute was larger than the compute required to reproduce the final reported configurations.

4pt

Table 6 shows that the overhead of M+Adamdepends on the precision regime and model size. The overhead is smallest in the FP8/FP8 and 1B settings, where the cost of the additional multiplicative branch is relatively small compared to the total training step. In contrast, the BF16/BF16 and BF16/FP8 settings at smaller scales show larger relative overheads, reflecting that the optimizer computation and extra state operations occupy a larger fraction of total runtime when the model is smaller or compute kernels are less dominant.

4pt

Table 7 provides the corresponding runtime comparison for AdamW with stochastic rounding in the BF16/BF16 regime. In our implementation, AdamW+SR also introduces nontrivial overhead relative to standard AdamW. This comparison is useful because AdamW+SR is a strong additive baseline for mitigating BF16 round-to-nearest effects, but its runtime cost is not negligible. Together, Tables 6 and 7 show that M+Adamimproves low-precision optimization stability at the cost of additional runtime, while also making clear that stochastic-rounding baselines carry their own systems overhead.

15 Apollo-E Compression for the Multiplicative State↩︎

M+Adamintroduces one additional optimizer state beyond AdamW: the second-moment accumulator used by the multiplicative branch. This state improves the stability of exponent-style updates, but also increases optimizer-state memory. To test whether this overhead is intrinsic to the method, we evaluate Apollo-E, a compressed variant that replaces only this extra multiplicative second-moment state with an Apollo-style approximation. The additive AdamW-style branch is unchanged.

Apollo-E compresses the multiplicative-state normalizer rather than changing the additive–multiplicative update rule. We evaluate two modes: mini, which uses the most aggressive tensor-wise compression, and channel, which uses a channel-wise compressed representation with rank controlling the auxiliary dimension. Larger ranks store slightly more state but can better approximate the dense multiplicative normalizer.

Table 8 reports optimizer-state memory in bytes per trainable parameter. These numbers count tensors stored in the optimizer state only; they do not include model weights, gradients, activations, temporary buffers, CUDA allocator effects, or communication buffers. Thus, the table should be interpreted as optimizer-state accounting rather than measured peak GPU memory.

4pt

Apollo-E reduces the optimizer-state overhead of M+Adamto a small increase over BF16-state AdamW while preserving most of the optimization gain. For 130M models, Apollo-E uses roughly \(4.74\)–\(4.82\) bytes per parameter, substantially below FP32-state AdamW, while improving perplexity over both BF16-state and FP32-state AdamW. For 350M models, Apollo-E uses only \(4.36\)–\(4.44\) bytes per parameter and remains much closer to full M+Adamperformance than to BF16 AdamW. These results show that the additional multiplicative state is not fundamental memory overhead: it can be compressed independently of the additive–multiplicative update geometry.

16 Hyperparameter Ablation↩︎

We report hyperparameter sweep results for the optimizers studied in the main text. Each table is organized as follows: the first row lists the approximately best configuration found in the sweep, and the subsequent rows report one-dimensional ablations that vary a single hyperparameter while holding all others fixed at this reference setting. Reported losses are the final C4/EN validation losses. We consider four precision regimes throughout: (i) BF16 weights + BF16 compute, where model parameters, activations, and optimizer state are stored in BF16; (ii) BF16 weights + FP8 compute, where parameters are maintained in BF16 master weights while Transformer Engine executes GEMMs in FP8; (iii) FP8 weights + FP8 compute, where parameters are stored in FP8 master weights and Transformer Engine executes GEMMs in FP8; and (iv) FP32 weights + TE32 compute, where parameters are stored in FP32 and Transformer Engine executes GEMMs in TE32.

16.0.0.1 FP4 block-scale branch weighting.

For block-scaled FP4/NVFP4 master weights, a dequantized weight in block \(b\) can be written approximately as \(w_i \approx q_i s_b\), where \(q_i\) is the FP4 payload value and \(s_b\) is the block scale. Since \(s_b\) determines the absolute spacing of the FP4 grid in weight space, it provides a simple time-varying signal for format-aware branch weighting. However, this scale has different meanings for the two M+Adam branches. For the additive branch, moving from payload value \(q_i\) to a neighboring value \(q_i^+\) requires an absolute displacement proportional to \(|q_i^+ s_b - q_i s_b| = s_b |q_i^+ - q_i|\), so a block-scale-aware correction can be interpreted as compensating for the absolute resolution of the additive grid. In contrast, the multiplicative branch operates in relative/log-weight space. The relevant adjacent-grid ratio is \(\frac{|q_i^+ s_b|}{|q_i s_b|} = \frac{|q_i^+|}{|q_i|}\), so the block scale cancels. Therefore, in the final implementation we keep the multiplicative branch controlled by a global branch scale rather than directly modulating its learning rate by the local FP4 block scale. Empirically, in a 60M NVFP4 diagnostic sweep, the simple block-scale learning-rate correction did not improve over the corresponding FP4 true-master baseline, so we treat this as an implementation diagnostic rather than an additional component of M+Adam.

16.1 Sweeping Results for AdamW (BF16 weights + BF16 compute)↩︎

4pt

4pt

4pt

4pt

4pt

4pt

16.2 Sweeping Results for M+Adam(BF16 weights + BF16 compute)↩︎

4pt

4pt

4pt

4pt

4pt

4pt

16.3 Sweeping Results for AdamW (BF16 weights + FP8 compute)↩︎

4pt

4pt

4pt

4pt

4pt

4pt

16.4 Sweeping Results for M+Adam(BF16 weights + FP8 compute)↩︎

4pt

4pt

4pt

4pt

4pt

4pt

16.5 Hyperparameter Ablation in Phase II↩︎

4pt

4pt

4pt

4pt

4pt

4pt

4pt

4pt

4pt

4pt

4pt

4pt

4pt

4pt

4pt

4pt

4pt

4pt

4pt

4pt

4pt

4pt

4pt

4pt

16.6 Hyperparameter Ablation of FP8 master weights with FP8 compute↩︎

4pt

4pt

4pt

4pt

4pt

4pt

4pt

4pt

4pt

4pt

4pt

4pt

4pt

References↩︎