Geometric–Nongeometric Optimizer Calculus:
A Modular Language for Reachable Gradient Methods
July 08, 2026
Adaptive optimizers mix several mechanisms: a metric or preconditioner maps gradients to descent directions, while estimation, memory, step-size control, constraints, stochasticity, target modification, and discretization determine which directions are available and how they are used. We introduce geometric–nongeometric optimizer calculus, a modular language for auditing reachable gradient methods under explicit oracle, budget, state, and rule constraints. The geometric module is a positive cometric family that maps covectors to parameter-space directions; the nongeometric modules are information, memory, control, operator, noise, target, and discretization mechanisms. The main formal result is a direction-expressivity theorem: away from critical points, full positive-definite geometry expresses exactly the strict descent directions. We then define restricted direction residuals for admissible metric families, prove exact expressivity conditions for diagonal and block geometries, and separate this direction-level diagnostic from condition-number geometric complexity. The resulting design problem is a Pareto optimization over module budgets, not a single universal optimizer ordering. We also lift pointwise residuals to a trajectory-level residual complexity that couples direction mismatch with the variation of the explaining geometry. We include diagnostic prototypes only as evidence for the language: a high-information full-metric probe solves deterministic quadratic benchmarks to numerical precision, while a practical Muon-style PyTorch candidate gives small-scale evidence that matrix-operator updates can be audited by the calculus. The paper is a theory and benchmark-language manuscript; it does not claim large-scale optimizer state-of-the-art performance.
Gradient-based optimization is usually described through update formulas: stochastic gradient descent adds noise through mini-batches, momentum stores a velocity, Adam rescales coordinates using moment estimates, Shampoo and K-FAC use structured curvature approximations, and quasi-Newton methods build secant memories. These descriptions are operationally useful, yet they mix several roles. Some parts define a geometry: a positive map from the gradient covector to a descent direction. Other parts decide what information is visible, which past quantities are remembered, how step sizes are controlled, whether constraints or projections are applied, how much randomness is injected, whether the objective is changed, and how a continuous idea is discretized.
This paper proposes a language for separating these roles. We call it geometric–nongeometric optimizer calculus. The geometric module is a positive cometric or preconditioner family, possibly restricted to diagonal, block, Kronecker, low-rank, or proxy forms. The nongeometric side is split into seven modules: information, memory, control, operator, noise, target, and discretization. The split is designed as an audit language for reachable gradient methods. It lets us ask whether a given update direction is explained by a metric family, which residual mechanisms remain, and how the answer changes under oracle, memory, communication, or compute budgets.
This paper is the optimizer-level companion to two preceding dynamic geometric optimization papers. Optimization geometrodynamics studies optimization as a coupled evolution of parameter trajectory, transported distribution, and time-varying positive-definite metric, including invariant obstructions and affine-invariant metric costs [1]. Restricted dynamic geometric complexity then studies condition-number reachability inside diagonal, block, Kronecker, low-rank, and proxy metric families [2]. The present paper does not require those papers as prerequisites: its definitions, direction-expressivity results, restricted residuals, and diagnostic prototypes are stated self-containedly. We use the distinction between full geometry, restricted geometry, and residual mechanisms as motivation, then lift it from metric evolution to optimizer design. In this setting, full positive-definite geometry is an expression envelope, restricted geometry is an implementable family, and nongeometric modules are budgeted mechanisms that cannot be collapsed into a positive parameter-space metric without losing information.
The question is not whether every possible algorithmic behavior can be called geometry. If an arbitrary residual vector is allowed, any update can be encoded and the language becomes vacuous. The useful question is:
Given an oracle class, a budget, a state class, and admissible rules, can reachable gradient methods be decomposed into a positive geometric module and structured nongeometric modules in a way that exposes expression gaps, mechanism budgets, and optimizer-design tradeoffs?
We address this by giving a formal module language, direction-level expressivity theorems, restricted residual diagnostics, and a Pareto formulation of optimal optimizer design. We do not prove a universal representation theorem for arbitrary optimizer code. The formal claims are local or class-relative: they hold after the oracle, budget, state variables, rules, target, and geometric family have been declared.
The main contributions are:
We define a geometric–nongeometric optimizer calculus for continuous and discrete gradient methods. The geometric module is a positive cometric family; the nongeometric modules are information, memory, control, operator, noise, target, and discretization.
We prove a pointwise full-geometry expressivity theorem: when \(g=\nabla f(\theta)\neq 0\), a direction \(u\) is exactly \(u=-Pg\) for some \(P\succ 0\) if and only if \(g^\top u<0\). Thus full positive-definite geometry expresses exactly strict descent directions away from critical points.
We define restricted direction residuals \(\rho_{\mathcal{F},h}(u;\alpha)\) for a metric family \(\mathcal{F}\) under a reference metric \(h\), and prove exact diagonal and block expressivity conditions. The residual gives a direction-level analogue of restricted dynamic geometric complexity for the optimizer-level setting.
We separate two diagnostics: condition-number geometric complexity, which measures curvature reachability, and direction residual, which measures how much of a proposed update is explained by a metric family. Their four quadrants distinguish geometric preconditioning from memory, projection, noise, target modification, and discretization effects.
We formulate optimal optimizer design as a constrained Pareto problem over module budgets. We also define trajectory-level residual complexity, which asks whether an entire trace admits a coherent geometric shadow with bounded metric variation. Expression-class inclusions imply oracle inequalities only when the performance functional and feasible budgets are shared.
We report diagnostic prototypes that illustrate the language: a full-metric probe for deterministic quadratics, a Krylov-subspace geometric variant, a residual-selected quasi-Newton hybrid, and a Muon-style engineering candidate. These prototypes support the framework as a research language; they leave practical optimizer competitiveness to a separate empirical study.
This is a theory and benchmark-language paper. The proved results concern direction expressivity, residual diagnostics, and class/budget distinctions. The experiments are diagnostic and small scale: they show how the calculus can attribute optimizer behavior in controlled settings. Establishing a new optimizer as broadly competitive on modern vision or language benchmarks would require a separate empirical paper with stronger baselines, tuned schedules, wall-clock accounting, and large compute budgets.
Riemannian optimization supplies the coordinate-free language of gradients, metrics, retractions, and Hessians [3], [4]. Natural gradient methods use Fisher geometry [5], mirror descent uses Bregman geometry [6], [7], and classical numerical optimization studies conditioning, preconditioning, line search, trust regions, and quasi-Newton updates [8], [9]. Our paper uses this geometric viewpoint at the optimizer-language level: the geometric module is the positive map from covectors to descent vectors.
AdaGrad, Adam, and AdamW use coordinate-wise adaptive scaling and moment estimation [10]–[12]. K-FAC and Shampoo use structured matrix or tensor preconditioners [13], [14], while Adafactor and SM3 reduce memory through factored or compressed statistics [15], [16]. SOAP combines Shampoo-style preconditioning with Adam-like second-moment tracking in the preconditioner’s eigenbasis [17], making it especially close to the geometry-plus-statistics split used here. These methods naturally fit our calculus as combinations of a restricted geometric family with information, memory, and control modules. The restricted direction residual introduced below is meant to diagnose how much of an update is explainable by the chosen geometric family.
Momentum and acceleration can be studied through discrete stability polynomials, ODE limits, or variational formulations [18]–[21]. Stochastic optimization separates oracle noise from deterministic descent dynamics [9]. Modern optimizers such as Lion and Sophia combine learned or designed update rules with signs, momentum, curvature proxies, clipping, and schedules [22], [23]. In our language, these mechanisms should be recorded explicitly as memory, noise, operator, target, or control modules when they cannot be represented as positive parameter-space metric descent.
Muon-style optimizers apply Newton–Schulz orthogonalization to momentum matrices, producing updates that are closer to matrix geometry than coordinate-wise diagonal scaling [24]. Recent work studies Muon as a practical pretraining optimizer and as a non-Euclidean gradient method with several variants [25], [26]. We use this family only as an engineering reference point for a diagnostic prototype. The formal results in this paper do not depend on Muon, and the small-scale experiments below should not be read as comparisons against Muon, SOAP, or other strong matrix optimizers.
Optimization geometrodynamics studies optimization as a coupled evolution of parameter trajectory, transported distribution, and time-varying metric. Its full positive-definite benchmark identifies invariant obstructions and affine-invariant metric costs [1]. Restricted dynamic geometric complexity then asks what condition-number targets are reachable inside diagonal, block, Kronecker, low-rank, or proxy metric families [2]. The present paper is the optimizer-level companion to these two papers: it asks which part of an algorithmic update is geometric, which part is a structured nongeometric mechanism, and how these parts should be budgeted when designing optimizers. The preceding papers provide background terminology and motivation; the formal calculus and residual results below are stated independently for the optimizer-level setting.
Let \(\mathcal{M}\) be a finite-dimensional smooth parameter manifold and \(f:\mathcal{M}\to\mathbb{R}\) an objective. In Euclidean sections we write \(\mathcal{M}=\mathbb{R}^d\) and \(g=\nabla f(\theta)\). A gradient method is never reachable in isolation: it depends on the information oracle, the computation and memory budget, the state variables it is allowed to carry, and the rules it must obey.
Definition 1 (Reachable optimizer class). Let \(\mathcal{O}\) be an oracle class, \(\mathcal{B}\) a budget class, \(\mathcal{S}\) a state class, and \(\mathcal{R}\) an admissible rule class. The reachable optimizer class \[\mathfrak A(\mathcal{O},\mathcal{B},\mathcal{S},\mathcal{R})\] is the set of update laws whose queried information, stored state, internal randomness, target transformations, and parameter updates are generated under these constraints.
The four arguments in 1 are part of the problem statement. For example, exact gradients, stochastic gradients, Hessian-vector products, and function values define different oracle classes. A method that is reachable with \(d\) extra gradient probes per step may be unreachable under a one-gradient-per-step budget.
The definition is semantic rather than syntactic. Two implementations belong to the same reachable class when their allowed queries, state transitions, randomness, target transformations, and parameter updates satisfy the same declared constraints, even if their code is organized differently. Claims about inclusion, optimality, or residual attribution are therefore meaningful only after these constraints and the visible covector have been fixed.
At a point \(\theta\in\mathcal{M}\), a Riemannian metric maps tangent vectors to covectors. Its inverse, or cometric, maps covectors to tangent vectors. The geometric module in this paper is a positive cometric family \[\mathcal{P}_{\mathcal{F}}(\theta)\subset \{P_\theta:T_\theta^*\mathcal{M}\to T_\theta\mathcal{M}\;:\; P_\theta \succ 0\}.\] For a covector \(\alpha_\theta=\mathrm{d}f_\theta\), the geometric descent direction is \[u_{\rm geo}(\theta)=-P_\theta \alpha_\theta .\] In Euclidean coordinates, \(P_\theta\) is a symmetric positive-definite preconditioner and \(u_{\rm geo}=-P_\theta \nabla f(\theta)\).
Typical families include full SPD matrices, diagonal matrices, fixed block-diagonal matrices, Kronecker-factored matrices, low-rank corrections, Fisher or Gauss–Newton proxy families, and learned or sketched curvature families. A geometric module therefore consists of \[\mathcal{G}=(\mathcal{F},P_0,\Gamma),\] where \(\mathcal{F}\) is the family, \(P_0\) the initial cometric, and \(\Gamma\) an update rule or flow for the cometric.
The remaining mechanisms are split into seven modules: \[\text{Nongeometry} = \text{Information}+\text{Memory}+\text{Control}+\text{Operator} +\text{Noise}+\text{Target}+\text{Discretization}.\]
Gradient estimates, mini-batches, curvature proxies, Hessian-vector probes, matrix sketches, bias corrections, and any other rule that determines which covector or curvature data are visible.
Momentum, Nesterov lookahead, secant pairs, historical gradient subspaces, restart variables, and other state that carries past information into the current update.
Learning rates, damping, trust regions, line search, accept–reject rules, schedules, and metric-update speeds.
Projection, proximal maps, clipping, normalization, decoupled weight decay, parameter repair, and nonsmooth transformations of candidate updates.
Mini-batch sampling noise when modeled explicitly, Langevin perturbations, dropout-induced optimization noise, and escape perturbations.
Regularized, smoothed, penalized, barrier, homotopy, curriculum, or surrogate objectives that differ from the original \(f\).
Euler or implicit steps, splitting schemes, finite precision, approximate matrix roots, and implementation-level approximations to a continuous model.
This finer split prevents the residual from becoming a shapeless container. For instance, momentum is a memory mechanism under the parameter-space metric convention used here, while a target homotopy belongs to the target module and must be reported as a problem transformation.
A continuous-time model can be written as \[\dot{\theta}_t =-P_t\mathrm{d}f_t(\theta_t)+R_t, \qquad \dot{z}_t=\mathcal{Z}(z_t,\theta_t,f,\mathcal{O}_t,\xi_t), \label{eq:continuous-composition}\tag{1}\] where \(f_t\) is the target-module objective, \(P_t\in\mathcal{P}_{\mathcal{F}}\) is the geometric module, \(R_t\) collects direct nongeometric velocity contributions, and \(z_t\) stores nongeometric state. The metric update can itself be driven by nongeometric state: \[\dot{P}_t=\Gamma(P_t,\theta_t,z_t,f_t,\mathcal{O}_t).\]
In discrete time, a broad one-step template is \[\theta_{k+1} = \mathcal{O}_k\!\left( \theta_k-\eta_k P_k\widehat g_k+r_k \right), \label{eq:discrete-composition}\tag{2}\] where \(\widehat g_k\) comes from the information module, \(P_k\) from geometry, \(\eta_k\) from control, \(r_k\) from memory/noise/discretization residuals, and \(\mathcal{O}_k\) from the operator module. The target module determines whether \(\widehat g_k\) estimates \(\nabla f(\theta_k)\) or a transformed objective \(\nabla f_k(\theta_k)\).
The decomposition of a real optimizer is not unique. We use the following lexicographic audit order:
Fix the target \(f_k\) that the algorithm is actually using.
Fix the visible covector \(\widehat g_k\) delivered by the information module.
Within the declared family \(\mathcal{F}\), choose \(P_k\) to best explain the observed direction by positive geometry.
Attribute global scale, damping, and accept–reject choices to control.
Attribute the remaining residual to memory, operator, noise, target distortion, or discretization according to its source.
This audit order is a convention, yet it is mathematically useful because it forces the geometric claim to be tested before residual mechanisms are used to explain the update.
This section studies a single point in Euclidean space. The goal is to identify which directions can be explained by a positive parameter-space geometry and which directions require nongeometric modules.
Proposition 1 (Pointwise full-SPD geometricization). Let \(g\in\mathbb{R}^d\) be nonzero and let \(u\in\mathbb{R}^d\). There exists \(P\in\mathbb{S}_{++}^d\) such that \[u=-Pg\] if and only if \[g^\top u<0.\]
1 says that full positive-definite geometry expresses exactly strict descent directions away from critical points. It also identifies the boundary. If \(g^\top u\ge 0\), the direction is neutral or ascending to first order and cannot be a pure positive-metric gradient descent direction. If \(g=0\) and \(u\neq0\), the motion cannot be produced by any positive parameter-space cometric applied to \(\mathrm{d}f_\theta\). This is an existential expressivity statement. The constructed matrix may be ill-conditioned and expensive to estimate or store, so the proposition does not by itself imply a practical full-matrix method.
Corollary 1 (Critical-point boundary). If \(\mathrm{d}f_\theta=0\), then \(-P_\theta\mathrm{d}f_\theta=0\) for every positive cometric \(P_\theta\). Any nonzero motion at an exact critical point must come from a nongeometric module, a target transformation, an extended state-space geometry, or a nonpositive/non-gradient mechanism.
Full SPD geometry is an expression envelope. Implementable optimizers use restricted families. Let \(h\) be a reference metric on \(T_\theta\mathcal{M}\) and let \(\left\lVert \cdot\right\rVert_h\) be its norm. For a tangent vector \(u\) and covector \(\alpha\), define the restricted direction residual \[\rho_{\mathcal{F},h}(u;\alpha) = \inf_{P\in\mathcal{P}_{\mathcal{F}}(\theta)} \left\lVert u+P\alpha\right\rVert_h . \label{eq:direction-residual}\tag{3}\] When a scalar step size is separated into the control module, a scaled version is \[\rho_{\mathcal{F},h}^{\rm scale}(u;\alpha) = \inf_{\eta>0,\;P\in\mathcal{P}_{\mathcal{F}}(\theta)} \left\lVert u+\eta P\alpha\right\rVert_h . \label{eq:scaled-direction-residual}\tag{4}\] The residual is zero when the family can approximate the direction arbitrarily well. Exact expression requires the infimum to be attained. This distinction matters for open cones such as \(p_i>0\) in diagonal families.
For diagnostics that should penalize nearly singular explanations, one should replace the open family by a bounded family such as \[\mathcal{P}_{\mathcal{F}}^{[\lambda,L]}(\theta) = \{P\in\mathcal{P}_{\mathcal{F}}(\theta):\lambda I\preceq P\preceq L I\}, \qquad 0<\lambda\le L<\infty .\] The unbounded residual in 3 is useful for expression analysis, while a bounded residual is more appropriate for engineering comparisons with explicit conditioning, storage, or stability budgets. For bounded diagonal geometry, this engineering residual has the closed-form coordinate projection in 7.
If \(\mathcal{P}_{\mathcal{F}}\) is a cone closed under positive scaling, the extra minimization over \(\eta\) in 4 is redundant. The scaled residual is included for normalized families, bounded families, or families whose parameterization fixes a metric scale.
For \(u\neq0\), an empirical geometric explanation rate is \[\operatorname{GER}_{\mathcal{F},h}(u;\alpha) = 1- \frac{\rho_{\mathcal{F},h}(u;\alpha)}{\left\lVert u\right\rVert_h}. \label{eq:ger}\tag{5}\] The ratio can be negative if the best geometric approximation is worse than the zero explanation. A clipped version can be used as an engineering score, while the unclipped version is cleaner for analysis. When the infimum is not attained, GER should be reported as an approximation rate, not as an exact geometric representation certificate.
Consider Euclidean coordinates, \(h=I\), and the diagonal family \[\mathcal{P}_{\rm diag}=\{\operatorname{diag}(p_1,\ldots,p_d):p_i>0\}.\]
Proposition 2 (Diagonal exact expressivity and residual). Let \(g,u\in\mathbb{R}^d\). There exists \(P\in\mathcal{P}_{\rm diag}\) such that \(u=-Pg\) if and only if, for every coordinate \(i\), \[g_i=0 \Rightarrow u_i=0, \qquad g_i\neq0 \Rightarrow u_i g_i<0.\] Moreover, \[\rho_{\rm diag,I}(u;g)^2 = \sum_{i:g_i=0} u_i^2 + \sum_{i:g_i\neq0,\;u_i g_i>0} u_i^2 .\] If \(g_i\neq0\) and \(u_i=0\) for some coordinate, the infimum in that coordinate is zero but is not attained by any strictly positive \(p_i\).
This proposition shows that diagonal geometry is coordinate-dependent. A rotation of the parameterization can change which directions satisfy the coordinate-wise sign condition.
Let \(\{B_1,\ldots,B_m\}\) be a partition of coordinates and let \[\mathcal{P}_{\rm block} = \{ \operatorname{blockdiag}(P_{B_1},\ldots,P_{B_m}) : P_{B_j}\succ0\}.\] Write \(g_{B_j}\) and \(u_{B_j}\) for block restrictions.
Proposition 3 (Block exact expressivity). There exists \(P\in\mathcal{P}_{\rm block}\) such that \(u=-Pg\) if and only if, for every block \(B_j\), \[g_{B_j}=0 \Rightarrow u_{B_j}=0, \qquad g_{B_j}\neq0 \Rightarrow g_{B_j}^\top u_{B_j}<0.\]
The block condition interpolates between the coordinate-wise diagonal condition and the global full-SPD condition. It is also a useful audit rule: if a transformer layer, convolutional block, or parameter group violates the block descent condition, the update contains a nongeometric contribution at that block scale.
[prop:full-spd-geometricization,cor:critical-boundary,prop:diagonal-expressivity,prop:block-expressivity] imply that several common optimizer behaviors should remain outside the parameter-space positive geometry module:
Momentum can create short-time neutral or ascending steps. These steps are memory contributions relative to the current objective and metric.
Noise can move an iterate away from an exact critical point. Such motion is not generated by \(-P\mathrm{d}f\) at that point.
Projection and proximal maps can replace an infeasible descent step by a boundary motion or a fixed point satisfying a normal-cone condition.
Target transformations can make \(-P\mathrm{d}f_k\) a descent direction for \(f_k\) while it has a large residual relative to the original \(f\).
Keeping these mechanisms explicit is necessary for a useful optimizer language, because their benefits and risks follow different budget and stability laws.
Restricted dynamic geometric complexity and restricted direction residuals measure different things. Both are needed to audit optimizer mechanisms.
For a strongly convex quadratic \[f(\theta)=\frac{1}{2}\theta^\top H\theta-b^\top \theta, \qquad H\succ0,\] a metric or cometric changes the spectrum that governs local linear convergence. Prior restricted-complexity analysis studies quantities schematically of the form \[D_{K,\mathcal{F}}(S_0;H) = \inf \operatorname{Len}_{\operatorname{AI}}(S_t) \quad \text{subject to } S_t\in \mathcal{F},\; \kappa(S_T^{1/2}HS_T^{1/2})\le K,\] with the precise expression depending on whether the family is represented by metrics, cometrics, quotient coordinates, or proxy variables. This quantity asks how much geometric deformation is required to reach a target condition-number class.
The residual \(\rho_{\mathcal{F},h}(u;\alpha)\) in 3 asks a different question: given an observed algorithmic direction \(u\) and a visible covector \(\alpha\), how much of \(u\) is explained by positive geometry inside \(\mathcal{F}\)? It can be computed even when the Hessian is unavailable. It is therefore suitable for auditing real training traces, where one usually has gradients and parameter updates but not full curvature.
The pair \[\mathcal{D}_{\mathcal{F}} = \left(D_{K,\mathcal{F}},\rho_{\mathcal{F},h}\right)\] has four qualitatively different regimes:
| Curvature | Direction residual | Interpretation |
|---|---|---|
| low | low | Geometry improves conditioning and explains the update. |
| low | high | The family can handle curvature, while the update uses memory, noise, operator, or target mechanisms. |
| high | low | The current direction is explainable, while the family may remain weak for long-term curvature conditioning. |
| high | high | The family struggles both as a curvature model and as a direction explanation. |
This table is a diagnostic device. A low direction residual is not a convergence certificate; it only says that the direction is geometrically explainable. A high residual is not a defect by itself; it may reveal a useful memory, exploration, projection, or target mechanism.
The target module must be controlled. If an optimizer may replace \(f\) by an arbitrary easier objective, optimizer comparison loses meaning. We therefore separate four admissible target transformations:
A path \(f_\lambda\) with a declared endpoint \(f_1=f\) and a schedule for returning to the original objective.
A transformation whose critical points are related to those of \(f\) by a stated inclusion, correspondence, or error bound.
A regularizer or penalty with a reported magnitude and transfer effect on the original objective.
A surrogate objective accompanied by a descent, transfer, or approximation certificate for \(f\).
Residuals should specify whether they are computed relative to the active target \(f_k\) or the original objective \(f\).
Let \(\mathfrak A_{\rm all}\) be a broad reachable class, let \(\mathfrak A_{\rm fullgeo}\) be the class whose directions are pure full-SPD geometry relative to the declared target and information module, and let \(\mathfrak A_{\mathcal{F}{\rm geo}}\) be the analogous class restricted to a family \(\mathcal{F}\). If \[\mathfrak A_{\mathcal{F}{\rm geo}} \subseteq \mathfrak A_{\rm fullgeo} \subseteq \mathfrak A_{\rm all}\] and all classes are evaluated by the same performance functional \(J\), then \[J_{\rm all}^\star \le J_{\rm fullgeo}^\star \le J_{\mathcal{F}{\rm geo}}^\star, \qquad J_{\mathcal{C}}^\star=\inf_{\mathcal{A}\in\mathcal{C}}J(\mathcal{A}).\] This is an expression-class statement. It does not imply that a full-SPD method is preferable under engineering constraints. If full SPD matrices are too expensive to estimate, store, communicate, or invert, the budget-feasible class can exclude them. Optimizer design therefore needs both expression inclusions and budget-feasible inclusions.
The calculus turns optimizer design into a constrained control problem over modules. The important object is not a total ordering of all optimizers, but a Pareto frontier under explicit budgets and tasks.
For an optimizer \(\mathcal{A}\) run for \(N\) steps, define a module-budget vector \[\mathbf{B}(\mathcal{A}) = (B_{\rm geo},B_{\rm info},B_{\rm mem},B_{\rm ctrl}, B_{\rm op},B_{\rm noise},B_{\rm target},B_{\rm disc}).\] The components can count different resources: affine-invariant metric length, number of gradient calls, Hessian-vector products, sketch dimension, optimizer state size, synchronization cost, line-search evaluations, projection cost, random samples, target distortion, or numerical approximation error. They need not be commensurate.
For example, a geometric deformation cost can be \[B_{\rm geo} = \sum_{k=0}^{N-1} d_{\operatorname{AI}}(P_k,P_{k+1}),\] when the family carries an affine-invariant SPD geometry. An information budget can count oracle calls: \[B_{\rm info} = \#\{\text{gradient calls}\} \;+\; c_{\rm hvp}\#\{\text{Hessian-vector products}\} \;+\; c_{\rm eval}\#\{\text{function evaluations}\}.\] For large-scale training, memory and communication budgets may dominate the choice of admissible geometry.
Let \(\Pi\) be a task distribution over objectives, data, initializations, and stopping criteria. Let \(J(\mathcal{A};\Pi)\) be a performance functional such as expected final loss, time-to-accuracy, validation loss, wall-clock time, or a multi-objective score. Given a budget set \(\mathbb{B}\), the constrained design problem is \[\min_{\mathcal{A}\in\mathfrak A(\mathcal{O},\mathcal{B},\mathcal{S},\mathcal{R})} J(\mathcal{A};\Pi) \quad \text{subject to} \quad \mathbf{B}(\mathcal{A})\in\mathbb{B} . \label{eq:optimal-optimizer}\tag{6}\] When the budget components are not scalarized, the output is a Pareto set: an optimizer is dominated only if another optimizer improves performance without exceeding any declared budget component.
Pointwise residuals explain one update direction. To audit an optimizer trace, the geometric explanations should also be coherent over time. Let \[\mathcal{T}_N=\{(\theta_k,\alpha_k,u_k,h_k)\}_{k=0}^{N-1}, \qquad \theta_{k+1}=\theta_k+u_k,\] where \(\alpha_k\) is the declared visible covector, such as a gradient or mini-batch gradient. An explanatory path for a family \(\mathcal{F}\) is a sequence \[E=\{(\eta_k,P_k)\}_{k=0}^{N-1}, \qquad \eta_k>0,\quad P_k\in\mathcal{P}_{\mathcal{F}}(\theta_k).\] It induces a geometric shadow increment and residual \[v_k(E)=-\eta_k P_k\alpha_k, \qquad r_k(E)=u_k-v_k(E)=u_k+\eta_kP_k\alpha_k .\] For \(1\le p,q<\infty\), define \[R_p(E;\mathcal{T}_N) = \left(\sum_{k=0}^{N-1}\left\lVert r_k(E)\right\rVert_{h_k}^p\right)^{1/p}, \qquad V_q(E) = \left(\sum_{k=0}^{N-2}d_{\mathcal{F}}(P_{k+1},P_k)^q\right)^{1/q},\] with the usual maximum-norm variants for \(p=\infty\) or \(q=\infty\). Here \(d_{\mathcal{F}}\) is the declared geometry distance, for example affine-invariant SPD distance, log-diagonal distance, or a distance in proxy variables. The trajectory-level residual complexity is \[\operatorname{TRC}_{p,q}^{B_{\rm geo}}(\mathcal{T}_N;\mathcal{F}) = \inf_{E:\,V_q(E)\le B_{\rm geo}} R_p(E;\mathcal{T}_N). \label{eq:trajectory-residual-complexity}\tag{7}\] Thus the trace must be explained not only by admissible pointwise preconditioners, but by an admissible path of preconditioners with bounded geometric variation. This prevents a rich family from explaining every step through unrelated or nearly singular matrices. In this paper we use 7 as a finite-trace diagnostic after the family, distance, bounds, and variation budget have been declared. General compactness, attainment, and statistical consistency questions for broad metric families are left as part of the longer theory of restricted geometric traces; the diagnostic cases below use finite-dimensional bounded families with explicit optimization problems.
Let \[U_p(\mathcal{T}_N) = \left(\sum_{k=0}^{N-1}\left\lVert u_k\right\rVert_{h_k}^p\right)^{1/p}.\] When \(U_p(\mathcal{T}_N)>0\), the trajectory geometric explanation rate is \[\operatorname{TGER}_{p,q}^{B_{\rm geo}}(\mathcal{T}_N;\mathcal{F}) = 1- \frac{\operatorname{TRC}_{p,q}^{B_{\rm geo}}(\mathcal{T}_N;\mathcal{F})}{U_p(\mathcal{T}_N)}.\] The raw version \(\operatorname{TRC}_{p}^{\rm raw}\) is obtained by removing the variation constraint. The coherence gap \[\operatorname{CG}_{p,q}^{B_{\rm geo}}(\mathcal{T}_N;\mathcal{F}) = \operatorname{TRC}_{p,q}^{B_{\rm geo}}(\mathcal{T}_N;\mathcal{F}) - \operatorname{TRC}_{p}^{\rm raw}(\mathcal{T}_N;\mathcal{F})\] measures the extra residual paid to make the explanation temporally coherent. A low raw residual but high coherence gap indicates pointwise geometry that requires rapidly changing metrics; a high raw residual with low coherence gap indicates a persistent nongeometric mechanism relative to the chosen family.
Proposition 4 (Trajectory shadowing). In Euclidean space with a fixed norm, let \(E\) be any explanatory path and let \(\bar\theta_0=\theta_0\), \(\bar\theta_{k+1}=\bar\theta_k+v_k(E)\). Then, for \(1\le p<\infty\) and every \(m\le N\), \[\left\lVert \theta_m-\bar\theta_m\right\rVert \le \sum_{k=0}^{m-1}\left\lVert r_k(E)\right\rVert \le m^{1-1/p}R_p(E;\mathcal{T}_N).\] For \(p=\infty\), the last term is replaced by \(m R_\infty(E;\mathcal{T}_N)\).
Proposition 5 (Smooth descent with residual budget). Let \(f\) be \(L\)-smooth in Euclidean norm. Suppose \(g_k=\nabla f(\theta_k)\) and \[u_k=-Q_k g_k+r_k,\qquad Q_k=\eta_kP_k\succeq0,\] with \[g_k^\top Q_k g_k\ge m_k\left\lVert g_k\right\rVert^2, \qquad \left\lVert Q_k g_k\right\rVert\le M_k\left\lVert g_k\right\rVert.\] Then \[f(\theta_{k+1}) \le f(\theta_k) -m_k\left\lVert g_k\right\rVert^2 +\left\lVert g_k\right\rVert\left\lVert r_k\right\rVert +\frac{L}{2}\left(M_k\left\lVert g_k\right\rVert+\left\lVert r_k\right\rVert\right)^2 .\]
For bounded diagonal geometry, 7 has a computable convex specialization. With \(P_k=\operatorname{diag}(p_{k,1},\ldots,p_{k,d})\), \(\lambda\le p_{k,i}\le \Lambda\), fixed positive \(\eta_k\), and quadratic variation in the diagonal weights, one solves \[\min_{\lambda\le p_{k,i}\le \Lambda} \sum_{k,i}(u_{k,i}+\eta_k p_{k,i}g_{k,i})^2 +\beta\sum_{k,i}(p_{k+1,i}-p_{k,i})^2 .\] This turns trajectory residual complexity into a practical diagnostic: report raw residual, coherent residual, coherence gap, and geometry variation before attributing the unexplained part to memory, operator, noise, target, or discretization modules.
The calculus suggests a structured search process:
Choose the target family allowed for the task.
Select the cheapest geometry family whose curvature complexity and direction residual are acceptable.
Allocate information budget to estimate the chosen geometry.
Add memory and control only where residuals or stability diagnostics show a need.
Add operator, noise, and target modules when constraints, exploration, or problem deformation are part of the task specification.
Track discretization error separately from the mathematical update.
This search process is a research program. It does not assume that a richer geometry always wins, because information, memory, communication, and implementation budgets can make a restricted family preferable.
The following statement records the only optimizer-ordering principle used in this paper.
Proposition 6 (Budgeted class monotonicity). Let \(\mathcal{C}_1\subseteq\mathcal{C}_2\) be two optimizer classes and let \(J\) be the same performance functional on both. If the same budget-feasible set is imposed on both classes, then \[\inf_{\mathcal{A}\in\mathcal{C}_2}J(\mathcal{A}) \le \inf_{\mathcal{A}\in\mathcal{C}_1}J(\mathcal{A}).\] If the two classes have different feasible budget sets, the inequality need not compare the engineering optima.
This elementary proposition is included to prevent a common overextension: full-SPD geometry is an expression superset of restricted geometry, while the budget-feasible optimizer frontier may favor a cheaper family.
The prototypes in this section are included to show how the calculus guides optimizer construction and auditing. They are not presented as final large-scale optimizers.
Consider deterministic strongly convex quadratics \[f(x)=\frac{1}{2}x^\top Hx-b^\top x,\qquad H\succ0.\] The prototype GNG-FullMetricProbe uses one gradient at the current point and \(d\) coordinate gradient probes: \[\nabla f(x+e_i)-\nabla f(x)=He_i.\] It reconstructs \(H\) and takes the full metric step \[x_+ = x-H^{-1}\nabla f(x).\] The module decomposition is:
| Module | Choice |
|---|---|
| Geometry | full SPD inverse metric \(H^{-1}\) |
| Information | one initial gradient plus \(d\) coordinate probes |
| Memory | none |
| Control | one full metric step |
| Operator/noise/target | none, original objective |
| Discretization | explicit finite-dimensional solve |
This method is appropriate only for the diagnostic setting where full coordinate gradient probes are affordable and the objective is exactly quadratic. It illustrates the expression envelope: enough information buys the full geometry and eliminates the quadratic in one step.
| Optimizer | Calls | Mean final gap | Converged |
|---|---|---|---|
| Default check: \(d=32\), two instances per problem family | |||
| GNG-FullMetricProbe v2 | 33 | \(1.26\times10^{-12}\) | 6 / 6 |
| GNG-KrylovMetric | 33 | \(1.70\) | 0 / 6 |
| FIFO-LBFGS | 151 | \(1.88\) | 0 / 6 |
| RSDLR-BFGS-Hybrid | 156.67 | \(7.29\) | 0 / 6 |
| Adam-tuned | 161 | \(75.83\) | 0 / 6 |
| GD-oracle | 161 | \(572.99\) | 0 / 6 |
| HeavyBall-oracle | 161 | \(4.14\times10^{7}\) | 0 / 6 |
| Dimension check: \(d=64\), one instance per problem family | |||
| GNG-FullMetricProbe v2 | 65 | \(0.0\) | 3 / 3 |
| GNG-KrylovMetric | 65 | \(2.68\) | 0 / 3 |
| FIFO-LBFGS | 186.67 | \(1.71\) | 0 / 3 |
| RSDLR-BFGS-Hybrid | 221 | \(15.01\) | 0 / 3 |
| Adam-tuned | 221 | \(209.77\) | 0 / 3 |
| GD-oracle | 221 | \(1.26\times10^{3}\) | 0 / 3 |
| HeavyBall-oracle | 208.67 | \(2.54\times10^{9}\) | 1 / 3 |
GNG-KrylovMetric replaces the full coordinate probe by sequential directional probes. Each step obtains \[Hp=\nabla f(x+p)-\nabla f(x)\] and takes an exact quadratic line-search step along a conjugate direction. In exact arithmetic this is closely related to conjugate gradient; in the calculus it is a growing subspace geometry with one directional probe per iteration. The finite-precision benchmark in 1 shows that this version did not match the full probe on the current high-condition-number instances, despite using the same order of gradient calls.
RSDLR-BFGS-Hybrid uses a bounded diagonal inverse metric and low-rank secant memory selected partly by a diagonal residual score. It did not outperform FIFO-LBFGS in the diagnostic benchmark. This failure is informative: directions with large diagonal residual are not automatically the most useful secant memories for future quasi-Newton progress. A stronger low-budget method should combine residual, recency, curvature diversity, and positive-definite stability.
The Krylov row should also be read narrowly. It is a prototype subspace metric with finite-precision directional probes and implementation choices, not a replacement claim for standard conjugate gradient on exact SPD quadratics.
The prototype GNGMuon combines matrix orthogonalized momentum for two-dimensional parameter tensors with an AdamW path for vector and scalar parameters. Its module decomposition is:
| Module | Choice |
|---|---|
| Geometry | matrix-shaped candidate update after orthogonalization |
| Information | mini-batch gradient |
| Memory | matrix momentum; AdamW moments for scalar/vector parameters |
| Control | learning rate, matrix scale, weight decay, Newton–Schulz steps |
| Operator | Newton–Schulz polar-factor approximation |
| Noise | mini-batch stochasticity |
| Target | supervised cross-entropy |
| Discretization | PyTorch finite-step implementation |
Under the strict convention of 3, the Newton–Schulz polar factor of a momentum buffer is not automatically an SPD cometric applied to the current visible gradient. The prototype is therefore best viewed as a matrix-operator and memory-heavy engineering candidate whose update can be audited by the calculus, rather than as a proof that Muon-style steps are pure positive geometry.
On a small MNIST subset with \(1024\) training examples, \(512\) test examples, one epoch, batch size \(128\), and three seeds, the best small-grid AdamW setting observed in the repository obtained mean accuracy about \(72.6\%\), while GNGMuon obtained about \(77.0\%\). This is early engineering evidence only. It does not compare against Muon, Lion, Sophia, Shampoo/SOAP, or tuned task-specific schedules, and it does not include large-scale wall-clock or memory accounting.
To check that the calculus can audit ordinary training traces, we also record the first eight MNIST-subset training updates for AdamW, SGD/Nesterov, and GNGMuon. For each step we compute the actual parameter update \(u_k=\theta_{k+1}-\theta_k\) and the visible raw gradient \(g_k\), then report the open diagonal GER and the bounded diagonal GER with \(p_i\in[10^{-6},1]\) from 7. This is an audit diagnostic, not a performance comparison.
| Optimizer | Mean open diagonal GER | Mean bounded diagonal GER |
|---|---|---|
| AdamW | \(0.387\) | \(0.153\) |
| SGD/Nesterov | \(0.732\) | \(0.720\) |
| GNGMuon | \(0.460\) | \(0.209\) |
The gap between open and bounded GER shows why the bounded diagnostic matters: an update may look partly diagonal-geometric only through scales outside the declared engineering range. The trace also illustrates that optimizers can be compared by mechanism attribution even when their accuracy numbers are too small-scale to support optimizer-performance claims.
We also compute a trajectory-level bounded-diagonal diagnostic on the same eight-step traces. The raw bounded TGER uses the \(\ell_2\) trace residual with an independently chosen bounded diagonal scale at each step. The coherent bounded TGER uses one shared bounded diagonal scale over all eight steps, which is the \(B_{\rm geo}=0\) specialization of 7 . The normalized coherence gap is \((\operatorname{TRC}_{\rm coherent}-\operatorname{TRC}_{\rm raw})/U_2\).
| Optimizer | Raw bounded TGER | Coherent bounded TGER | Norm. coherence gap |
|---|---|---|---|
| AdamW | \(0.150\) | \(0.120\) | \(0.030\) |
| SGD/Nesterov | \(0.686\) | \(0.423\) | \(0.263\) |
| GNGMuon | \(0.203\) | \(0.120\) | \(0.084\) |
The coherent diagnostic asks whether the same restricted geometry can shadow the whole short trace. On this run, bounded pointwise diagonal explanations for SGD/Nesterov are strong, but their no-variation coherent shadow pays a larger gap; AdamW and GNGMuon have lower raw bounded TGER and smaller absolute coherence gaps. These numbers are mechanism audits for a short trace, not claims about optimizer quality.
The prototypes support three modest conclusions:
When full curvature information is available and cheap, the geometric module can dominate performance on quadratic problems.
When information is limited, the choice of subspace geometry and memory selection rule matters as much as the nominal geometry family.
Engineering candidates need nongeometric controls, operator choices, and implementation budgets before they can be evaluated as practical optimizers.
These conclusions are aligned with the calculus: optimizer quality is a budgeted combination of geometry and nongeometry.
The paper uses a narrow and auditable convention: geometry means a positive parameter-space cometric applied to the visible covector. Momentum may be geometric in an enlarged phase space, and proximal methods may be geometric under a different variational structure. Those are valuable extensions. The present convention keeps the parameter update audit simple and makes the boundary statements in 4 precise.
A large residual does not mean an optimizer is poor. It means the chosen metric family does not explain the direction. The residual may be useful memory, exploration, projection, target deformation, or discretization. Residuals become actionable only when paired with module budgets and stability or transfer certificates.
Diagonal and block residuals depend on the parameterization. This is a feature for auditing coordinate-based optimizers such as Adam-like methods, and a limitation for coordinate-free claims. A coordinate-invariant diagnostic requires a declared reference metric \(h\) and a family defined intrinsically on the manifold.
Continuation, smoothing, regularization, and surrogates are powerful, yet they change the optimization problem unless transfer conditions are stated. The calculus therefore reports target transformations separately from metric geometry. This separation is important when comparing algorithms whose training losses, regularizers, augmentations, or curricula differ.
The diagnostic prototypes are intentionally small. The deterministic quadratic benchmark checks the full-geometry expression story under a high information budget. The MNIST subset check shows that a geometry-aware engineering candidate can be promising. Neither experiment resolves broad performance against current strong optimizers. That would require tuned baselines such as AdamW, SGD/Nesterov, Lion, Sophia, Muon, and Shampoo/SOAP across modern vision and language workloads, with matched training budgets, seeds, throughput, memory, and hyperparameter search.
The most immediate theoretical directions are:
closed-form residuals for bounded diagonal, block, Kronecker, and low-rank families;
separation examples proving all four quadrants of curvature complexity and direction residual;
Lyapunov or regret bounds with explicit nongeometric residual budgets;
empirical GER traces for AdamW, Muon, Shampoo, and related optimizers;
module-level Pareto frontiers on controlled quadratics, convex finite sums, and small neural networks.
The formal statements in [sec:direction-expressivity] [sec:restricted-diagnostics] [sec:optimal-optimizer] are proved in 11.
The diagnostic quadratic benchmark is implemented in the repository under experiments/toy/gng_optimizer_benchmark.py. The Muon-style prototype is implemented under experiments/optimizers/gng_muon.py, with a small training entry
under experiments/mnist/train_gng_muon.py. The trace residual audit is implemented under experiments/mnist/trace_residual_diagnostic.py. The reported prototype numbers should be treated as diagnostic observations from these
scripts, not as a definitive benchmark suite. The MNIST script records the training configuration and fixes Python, NumPy, PyTorch, and data-loader seeds, but the current repository results do not record hardware, CUDA/cuDNN versions, or a complete
hyperparameter-search log.
This paper studies optimization language, theory, and small diagnostic experiments. It does not introduce a deployed system or a dataset involving human subjects. The main ethical risk is overclaiming optimizer performance from insufficient evidence. We therefore state the scope of the empirical results explicitly and reserve broad performance claims for future work with large-scale, fairly tuned comparisons.
If \(u=-Pg\) with \(P\succ0\) and \(g\neq0\), then \[g^\top u=-g^\top Pg<0.\] Conversely, suppose \(g^\top u<0\) and define \(b=-u\). Then \(g^\top b>0\). Choose an orthonormal basis with \(e_1=g/\left\lVert g\right\rVert\). In this basis write \[b=\beta e_1+c,\qquad c\perp e_1, \qquad \beta=e_1^\top b=\frac{g^\top b}{\left\lVert g\right\rVert}>0.\] For \(d=1\), set \(P=[\beta/\left\lVert g\right\rVert]\). For \(d>1\), define \[\widetilde{P} = \begin{pmatrix} \beta/\left\lVert g\right\rVert & c^\top/\left\lVert g\right\rVert\\ c/\left\lVert g\right\rVert & \gamma I \end{pmatrix}\] in the chosen basis. Then \[\widetilde{P}(\left\lVert g\right\rVert e_1)=b.\] The Schur complement of the upper-left block is \[\gamma I-\frac{cc^\top}{\left\lVert g\right\rVert\,\beta}.\] Choosing \(\gamma>\left\lVert c\right\rVert^2/(\left\lVert g\right\rVert\,\beta)\) makes this complement positive, so \(\widetilde{P}\succ0\). Transforming back to the original coordinates gives an SPD matrix \(P\) with \(Pg=b=-u\), hence \(u=-Pg\).
If \(\mathrm{d}f_\theta=0\), linearity gives \(P_\theta\mathrm{d}f_\theta=0\) for every positive cometric \(P_\theta\). Thus a pure positive-geometry update is zero. Any nonzero tangent vector at that point must enter through another module or through a different modeling convention.
The equation \(u=-Pg\) with \(P=\operatorname{diag}(p_1,\ldots,p_d)\) and \(p_i>0\) is equivalent coordinate-wise to \[u_i=-p_i g_i.\] If \(g_i=0\), exact expression requires \(u_i=0\). If \(g_i\neq0\), exact expression requires \(p_i=-u_i/g_i>0\), equivalently \(u_i g_i<0\).
For the residual, the squared Euclidean objective separates: \[\inf_{p_i>0}\sum_i (u_i+p_i g_i)^2 = \sum_i \inf_{p_i>0}(u_i+p_i g_i)^2 .\] If \(g_i=0\), the term is \(u_i^2\). If \(g_i\neq0\) and \(u_i g_i<0\), the positive choice \(p_i=-u_i/g_i\) gives zero. If \(g_i\neq0\) and \(u_i g_i>0\), the unconstrained minimizer is negative, so the infimum over \(p_i>0\) is the boundary value \(u_i^2\), approached as \(p_i\downarrow0\). If \(g_i\neq0\) and \(u_i=0\), the infimum is zero, again approached as \(p_i\downarrow0\), but it is not attained by a strictly positive \(p_i\). This proves the formula and the attainment statement.
Proposition 7 (Bounded diagonal residual). Fix \(0<\lambda\le L<\infty\) and let \[\mathcal{P}_{\rm diag}^{[\lambda,L]} = \{\operatorname{diag}(p_1,\ldots,p_d):\lambda\le p_i\le L\}.\] For \(g,u\in\mathbb{R}^d\) and \(h=I\), \[\rho_{{\rm diag}^{[\lambda,L]},I}(u;g)^2 = \sum_{i=1}^d (u_i+p_i^\star g_i)^2,\] where, for \(g_i\neq0\), \[p_i^\star = \min\!\left\{L,\max\!\left\{\lambda,-\frac{u_i}{g_i}\right\}\right\},\] and for \(g_i=0\) any \(p_i^\star\in[\lambda,L]\) gives the same term \(u_i^2\).
Proof. The squared residual separates by coordinate: \[\inf_{\lambda\le p_i\le L}\sum_i (u_i+p_i g_i)^2 = \sum_i \inf_{\lambda\le p_i\le L}(u_i+p_i g_i)^2 .\] If \(g_i=0\), the coordinate objective is constant and equal to \(u_i^2\). If \(g_i\neq0\), the unconstrained minimizer of the one-dimensional convex quadratic is \(-u_i/g_i\). Projecting this minimizer onto the interval \([\lambda,L]\) gives \(p_i^\star\), and substituting the coordinate minimizers gives the stated formula. ◻
For a block-diagonal \(P\), the equation \(u=-Pg\) separates over blocks: \[u_{B_j}=-P_{B_j}g_{B_j}.\] If \(g_{B_j}=0\), exact expression requires \(u_{B_j}=0\). If \(g_{B_j}\neq0\), 1 applied inside the block gives existence of \(P_{B_j}\succ0\) exactly when \(g_{B_j}^\top u_{B_j}<0\). Combining the block solutions gives the desired block-diagonal matrix.
By construction, \[\theta_m-\bar\theta_m = \sum_{k=0}^{m-1}\left(u_k-v_k(E)\right) = \sum_{k=0}^{m-1} r_k(E).\] The first inequality follows from the triangle inequality: \[\left\lVert \theta_m-\bar\theta_m\right\rVert \le \sum_{k=0}^{m-1}\left\lVert r_k(E)\right\rVert.\] For \(1\le p<\infty\), Holder’s inequality gives \[\sum_{k=0}^{m-1}\left\lVert r_k(E)\right\rVert \le m^{1-1/p} \left(\sum_{k=0}^{m-1}\left\lVert r_k(E)\right\rVert^p\right)^{1/p} \le m^{1-1/p}R_p(E;\mathcal{T}_N).\] For \(p=\infty\), the same argument gives \[\sum_{k=0}^{m-1}\left\lVert r_k(E)\right\rVert \le m\max_{0\le k<N}\left\lVert r_k(E)\right\rVert = mR_\infty(E;\mathcal{T}_N).\]
By \(L\)-smoothness, \[f(\theta_{k+1}) \le f(\theta_k)+g_k^\top u_k+\frac{L}{2}\left\lVert u_k\right\rVert^2.\] Using \(u_k=-Q_kg_k+r_k\), \[g_k^\top u_k = -g_k^\top Q_k g_k+g_k^\top r_k \le -m_k\left\lVert g_k\right\rVert^2+\left\lVert g_k\right\rVert\left\lVert r_k\right\rVert.\] Also, \[\left\lVert u_k\right\rVert \le \left\lVert Q_kg_k\right\rVert+\left\lVert r_k\right\rVert \le M_k\left\lVert g_k\right\rVert+\left\lVert r_k\right\rVert.\] Substituting these two bounds into the smoothness inequality proves the claim.
Let \(\mathcal{C}_1\subseteq\mathcal{C}_2\) and impose the same feasible budget set on both. The feasible set for \(\mathcal{C}_1\) is a subset of the feasible set for \(\mathcal{C}_2\). Taking the infimum of the same functional \(J\) over a larger set cannot increase the value. If the feasible budget sets differ, this set-inclusion argument no longer applies.
The deterministic quadratic benchmark uses three problem families:
rotated strongly convex quadratics;
diagonal-plus-low-rank quadratics;
block-coupled quadratics.
The script records final objective gap, gradient-call count, and convergence under a gradient-norm tolerance. The default repository command is
python experiments/toy/gng_optimizer_benchmark.py --output experiments/toy/gng-resultsand the larger diagnostic run used during development was
python experiments/toy/gng_optimizer_benchmark.py --output experiments/toy/gng-results `
--dim 64 --instances 4 --memory 8 --max-iter 220 --grad-tol 1e-8The paper reports the later optimized diagnostic checks because they use GNG-FullMetricProbe v2, which removes an unnecessary final gradient validation call and therefore uses \(d+1\) gradient calls. The benchmark script also runs HeavyBall-oracle; the row is included in the updated table because it is part of the saved summary. Its poor performance on these high-condition-number instances should not be read as a general statement about heavy-ball methods. The Adam-tuned row reports the selected run from a six-point learning rate sweep in the script; the table’s gradient-call column does not charge the separate tuning sweep as an optimizer budget.
The method evaluates \(g_0=\nabla f(x_0)\) and then evaluates gradients at \(x_0+e_i\) for \(i=1,\ldots,d\). For a deterministic quadratic, these probes recover the columns of \(H\). The update \(x_+=x_0-H^{-1}g_0\) is the exact Newton step. The final gradient can be computed internally as \(g_0+H(x_+-x_0)\), so no final oracle call is needed.
The method performs sequential directional probes to obtain \(Hp\) along search directions. It uses exact quadratic line search and conjugacy-style updates. Its interpretation in this paper is a growing Krylov subspace metric, not a new claim about conjugate-gradient theory. The reported implementation is not presented as a standard CG baseline.
The method combines a bounded diagonal inverse metric with a limited secant memory. It keeps a mixture of recent secant pairs and pairs with high residual under the current diagonal model. In the present benchmark it did not outperform FIFO-LBFGS.
The engineering prototype GNGMuon was checked with the repository training entry on a small MNIST subset: \[\text{train subset}=1024,\quad \text{test subset}=512,\quad \text{epochs}=1,\quad \text{batch size}=128.\] The comparison used three seeds after a small learning-rate grid. The best AdamW grid point found was \(lr=0.003\); the GNGMuon setting used \(lr=0.03\) and matrix scale \(1.5\). The observed test accuracies were:
| Seed | AdamW | GNGMuon |
|---|---|---|
| 7 | \(70.90\%\) | \(79.10\%\) |
| 8 | \(76.76\%\) | \(78.52\%\) |
| 9 | \(70.12\%\) | \(73.24\%\) |
| Mean | \(72.59\%\) | \(76.95\%\) |
The sample standard deviations across these three seeds are about \(3.63\) percentage points for AdamW and \(3.23\) percentage points for GNGMuon. The
script fixes Python, NumPy, and PyTorch seeds and uses a seeded DataLoader generator with num_workers=0; the repository run did not record GPU model, CUDA/cuDNN versions, PyTorch/torchvision versions, or deterministic cuDNN flags.
This experiment is intentionally reported as a small diagnostic. It lacks the baseline set, tuning budget, duration, hardware accounting, and workload diversity needed for a broad optimizer-performance claim.
The trace-level residual audit in 7 is produced by:
python experiments/mnist/trace_residual_diagnostic.py --dataset mnist `
--max-steps 8 --train-subset 1024 --test-subset 512 --batch-size 128 `
--seed 7 --output tmp/trace_residual_mnist_8stepThe script records per-step loss, gradient norm, update norm, open diagonal residual, bounded diagonal residual, the corresponding GER values, and trace-level bounded-diagonal residual summaries. The run used \(p_i\in[10^{-6},1]\) for the bounded diagonal family and wrote trace_residual_summary.json, trace_residuals.csv, and trace_residual_ger.png. The first reported table uses the mean GER over
the eight recorded updates for each optimizer. The trajectory-level table uses the \(\ell_2\) residual over the same eight updates: the raw bounded TGER allows an independently selected bounded diagonal scale at each step,
whereas the coherent bounded TGER uses one shared bounded diagonal scale over the whole trace, corresponding to the \(B_{\rm geo}=0\) case of 7 .