1 Introduction↩︎

Stochastic Gradient Langevin Dynamics (SGLD) turns an empirical risk into a noisy sampling dynamics [1], [2]. Given data \[z=(z_1,\ldots,z_N), \qquad F_z(w)=\frac{1}{N}\sum_{i=1}^N f(w,z_i),\] the reference overdamped Langevin diffusion is \[\label{eq:langevin} \,\mathrm{d}X_t=-\nabla F_z(X_t)\,\mathrm{d}t+\sqrt{2\beta^{-1}}\,\mathrm{d}B_t,\tag{1}\] with Gibbs invariant law \[\label{eq:gibbs} \pi_z(\,\mathrm{d}w)=\Lambda_z^{-1}e^{-\beta F_z(w)}\,\mathrm{d}w.\tag{2}\] For polynomial-growth objectives, the continuous Langevin drift may be stabilizing, while the fixed-stepsize explicit Euler update can still be unstable. A simple example is \[F(x)=\frac{1}{4}x^4-\frac{1}{2}x^2, \qquad F'(x)=x^3-x,\] where the far-field Euler increment may overshoot. Taming is therefore needed to make fixed-stepsize explicit simulation meaningful in the superlinear-drift regime.

The design issue is not only how to stabilize the drift. In stochastic-gradient SGLD, the taming factor may also depend on the current stochastic-gradient realization. This coupling can change the conditional mean of the update. We therefore treat the denominator as a design object: its dependence structure determines whether the stochastic-gradient oracle is preserved, and its activation pattern determines where the drift is modified. Deterministic envelopes separate these roles by stabilizing the far-field drift while keeping the oracle structure transparent.

A useful way to locate this question is to separate finite-time sampling error into two parts. If \(\pi_\eta\) denotes an invariant law of the numerical chain, then for any suitable distance \(d\), \[\label{eq:intro-finite-time-split} d(\mathcal{L}(X_k),\pi_z) \le d(\mathcal{L}(X_k),\pi_\eta) + d(\pi_\eta,\pi_z).\tag{3}\] The first term is the mixing error of the numerical chain toward its own invariant law. The second term is the stationary bias created by the discretization, the taming denominator, and the stochastic-gradient oracle; fixed-stepsize SGLD bias and finite-time SGLD analyses have been studied in, for example, [3], [4]. This paper targets the second term: the stationary effect of denominator design in a fixed-step sampling kernel. The numerical experiments show that this component can change substantially across denominator choices.

The construction is analysis-driven. We first isolate the bias mechanism caused by a random denominator, and then design the deterministic envelope to remove that first-order oracle-bias channel. Localized and hybrid envelopes apply the same principle to deterministic taming: instead of damping the drift globally, they choose where taming should turn on.

1.0.0.1 Deterministic-envelope principle.

We use denominators of the form \[1+\eta^\alpha A_\eta(x),\] where the envelope depends only on the current state and the fixed dataset. It does not depend on the current stochastic-gradient realization. Then \[\mathbb{E}\left[\frac{g(x,\xi)}{1+\eta^\alpha A_\eta(x)}\mid x\right] =\frac{\nabla F_z(x)}{1+\eta^\alpha A_\eta(x)}.\] Deterministic taming removes the bias caused by a gradient-dependent denominator, but it still introduces a deterministic drift modification. The design goal is to keep this modification mild where the chain typically moves, while retaining enough stabilization in the far tail. Thus the improvement is structural: stabilization is separated from nonlinear transformation of the stochastic-gradient oracle.

1.0.0.2 Local and hybrid envelopes.

A global hard envelope can be stable but overly conservative. To reduce unnecessary common-region taming, we use a localized deterministic-envelope template rather than a single prescribed rule: \[\label{eq:intro-hybrid-envelope} A_\eta^{c_s,c_h}(x) =c_s\big(\bar A(x)-R\big)_+^\gamma +c_h\big(\bar A(x)-S_\eta\big)_+, \qquad c_s,c_h\in\{0,1\},\quad 0<\gamma\le1,\tag{4}\] where \(\bar A\) is a computable deterministic growth indicator. This template includes local soft \((c_s,c_h)=(1,0)\), local hard \((0,1)\), and hybrid \((1,1)\) envelopes. A nonlocalized global envelope instead uses the growth indicator everywhere, for example \(A_\eta(x)=\bar A(x)\) up to a fixed scaling. Thus the local soft, local hard, and hybrid rules are not different oracle principles; they are different activation patterns for the same deterministic-envelope idea across typical, intermediate, and tail regions.

1.0.0.3 Contributions.

The contributions are organized around this denominator-design viewpoint:

• We isolate the taming denominator as an oracle-level transformation. If the denominator is computed from the same stochastic-gradient realization as the numerator, it can couple the taming factor to the oracle noise and create a systematic mean shift. This elementary mechanism, including its variance and covariance forms, motivates deterministic-envelope designs.

• We formulate a structure-preserving framework for tamed denominators. Global, local soft, local hard, and hybrid envelopes are activation patterns of the same deterministic-envelope template, with the hybrid rule combining mild typical-region behavior and hard-tail protection.

• We establish the stationary theory for this framework. The analysis gives Lyapunov stability, invariant-measure existence, small-stepsize stationary consistency, and a Poisson-residual decomposition separating oracle-dependent taming bias from the remaining deterministic-envelope residual.

• We validate the denominator-design principle in controlled numerical examples. The experiments show the stationary distortions caused by gradient-dependent denominators, the bias reduction obtained by localized deterministic envelopes, and the stabilizing role of the hybrid construction in high-growth and empirical-risk settings.

1.0.0.4 Relation to existing tamed schemes.

Taming and truncation are standard stabilization tools for SDEs with non-globally Lipschitz coefficients [5]–[9] and for Langevin-type algorithms with superlinear potentials [10]–[12]. TULA-style methods provide deterministic stabilization for Langevin drifts, while TUSLA-type analyses give finite-time guarantees for stabilized stochastic-gradient Langevin dynamics, including sampling, optimization, and nonconvex learning guarantees [13], [14].

The present paper takes a different viewpoint. We study the denominator as part of the stochastic-gradient oracle in a fixed-step sampling kernel. This makes the distinction between a denominator fixed before the oracle noise is sampled and a denominator coupled to the sampled stochastic gradient explicit. A concrete gradient-dependent example is the tamed stochastic-gradient direction \[\frac{\nabla f(\xi_n,w_n)}{1+\alpha_n\|\nabla f(\xi_n,w_n)\|},\] where the normalization is applied to the current stochastic-gradient realization [15]. Table 1 summarizes the comparison.

1.0.0.5 Organization of the analysis.

Section 2 develops the oracle-transformation viewpoint. Section 3 defines the polynomial-growth Langevin model, deterministic-envelope recursions, and global/local/hybrid envelope family. Sections 4–5 establish moment stability, invariant-measure existence, and small-stepsize stationary consistency. Section 6 decomposes exact-gradient finite-stepsize stationary bias into Euler and deterministic-envelope residuals. Section 7 extends the framework to stochastic-gradient kernels, Section 8 gives numerical evidence, and Section 9 discusses the stationary-bias viewpoint and extensions.

2 Oracle Transformations and Denominator-Induced Bias↩︎

In this section we view taming denominators as transformations of the stochastic-gradient oracle. For an oracle output \(v\), a deterministic-envelope transformation is \[\mathcal{T}_\eta^{\rm det}(x,v) =\frac{v}{1+\eta^\alpha A_\eta(x)}.\] A gradient-dependent transformation is instead \[\mathcal{T}_\lambda^{\rm gd}(v) =\frac{v}{1+\lambda\|v\|}.\] The distinction is structural: conditional on \(x\), a deterministic-envelope denominator is fixed, whereas a gradient-dependent denominator remains coupled to the current oracle realization.

Let \(g(x,\xi)\) be a stochastic-gradient oracle satisfying \[\label{eq:oracle-unbiased-intro} \mathbb{E}[g(x,\xi)\mid x]=\nabla F_z(x).\tag{5}\] For a deterministic envelope \(A_\eta(x)\), define the tamed oracle \[G_\eta^{\rm det}(x,\xi)=\frac{g(x,\xi)}{1+\eta^\alpha A_\eta(x)}.\]

Because the denominator is state-measurable, a deterministic envelope preserves the conditional mean of the stochastic-gradient oracle relative to the modified drift. This matters for fixed-stepsize sampling because a denominator-induced drift perturbation changes the Markov transition whose invariant law is being approximated. The next result quantifies the corresponding drift bias for gradient-dependent denominators; later sections use this diagnostic to motivate the envelope family and the stationary residual decomposition.

3 Model, Assumptions, and Deterministic-Envelope Recursions↩︎

We now turn the oracle-level mechanism into the concrete tamed Langevin kernels used in the rest of the paper. This section fixes the deterministic setting: the drift assumptions, the envelope conditions, and the exact-gradient fixed-stepsize recursion. The stochastic-gradient kernel is introduced later, after the exact-gradient stationary structure has been established.

3.1 Langevin model and structural assumptions↩︎

Throughout, the dataset \(z\) is fixed and \[b(x)=b_z(x)=-\nabla F_z(x).\] The Langevin generator is \[\label{eq:generator} \mathcal{L}_zf(x)=\langle b(x),\nabla f(x)\rangle+\beta^{-1}\Delta f(x).\tag{6}\] Constants may depend on structural parameters such as \(d,\beta,p\), polynomial-growth orders, and Lyapunov constants, but not on the iteration index or on sufficiently small \(\eta\), unless explicitly stated.

The assumptions have separate roles. Assumptions 1–2 describe a polynomial-growth dissipative drift. Assumption 3 describes an envelope that controls the far-field size of the explicit drift increment. The compatibility bound ?? is the key point; it will be used below to make the tamed increment effectively linear.

Here “deterministic” means that the denominator depends only on the current state and the fixed dataset. It is not computed from the fresh stochastic-gradient sample used in the current update. This convention becomes the oracle-preservation condition in Section 7.

Localized envelopes below are built from a deterministic growth indicator \(\bar A\). It is used only to decide where the envelope becomes active; the actual envelope in the update is \(A_\eta\). For example, \(\bar A\) may satisfy \[\|\nabla F_z(x)\|\le \bar A(x)\le C(1+\|x\|^{r+1}).\]

3.2 Localized deterministic-envelope family↩︎

We next define a family of localized deterministic envelopes. The oracle principle is unchanged: the denominator is still fixed by the current state and dataset. The different members only specify where the deterministic correction becomes active.

The threshold \(R\) defines the safe region \[\mathcal{S}_R \mathrel{\vcenter{:}}= \{x:\bar A(x)\le R\}.\] If \(S_\eta>R\), the intermediate region is \[\mathcal{I}_{R,S_\eta} \mathrel{\vcenter{:}}= \{x:R<\bar A(x)\le S_\eta\}.\] The far-tail region is \[\mathcal{T}_{S_\eta} \mathrel{\vcenter{:}}= \{x:\bar A(x)>S_\eta\}.\] We call these envelopes localized because they are inactive on the safe region and become active only after the prescribed growth thresholds are crossed. The exponent \(\gamma\) defines the soft component: it lets taming turn on gradually after the safe region, so that the intermediate region can be controlled without unnecessary drift modification. This soft component is not meant to be the extreme-tail stabilizer; when \(\gamma<1\), it may grow too slowly to control rare large excursions by itself. The second term is therefore the hard-tail safeguard, active only in \(\mathcal{T}_{S_\eta}\). The hybrid envelope combines soft intermediate control with hard-tail protection. The local soft, local hard, and hybrid envelopes are \[A_\eta^{\rm soft} \mathrel{\vcenter{:}}= A_\eta^{1,0},\qquad A_\eta^{\rm hard} \mathrel{\vcenter{:}}= A_\eta^{0,1},\qquad A_\eta^{\rm hyb} \mathrel{\vcenter{:}}= A_\eta^{1,1}.\]

In Lemma 1 below, we choose \(S_\eta=c_S\eta^{-\alpha}\) precisely to obtain the effective-linearity bound ?? . Its fixed-stepsize interpretation is discussed in the envelope-design remarks below.

Later results for a deterministic envelope \(A_\eta\) apply to any localized envelope satisfying the stated conditions.

3.3 Envelope compatibility of localized rules↩︎

The later stability theory uses the effective-linearity consequence of Assumption 3. Global envelopes satisfy this condition by assumption. Localized rules require a separate check. The check below assumes that the growth indicator itself has global-envelope-type far-field control. It then shows that rules with a hard-tail safeguard, such as local hard and hybrid envelopes, retain the needed control. Local soft rules do not satisfy this bound automatically; Remark 6 explains the resulting far-tail limitation and the motivation for adding a hard-tail safeguard.

The preceding proof also gives a useful diagnostic for local soft rules.

3.4 Exact-gradient deterministic-envelope recursion↩︎

We now define the exact-gradient Markov chain that serves as the base stationary object. The stochastic-gradient kernel is introduced later in Section 7, after the exact-gradient stationary theory and residual decomposition have been established.

Fix \(\alpha\in(0,1/2)\). For a constant stepsize \(\eta\in(0,\eta_0]\), define \[\label{eq:tamed-drift} b^T_\eta(x)=\frac{b(x)}{1+\eta^\alpha A_\eta(x)},\tag{7}\] where \(A_\eta\) may be the global envelope \(A\) from Assumption 3 or any localized deterministic envelope satisfying the same effective-linearity condition. The exact-gradient tamed Langevin recursion is \[\label{eq:tamed-chain} Y_{k+1}=Y_k+\eta b^T_\eta(Y_k)+\sqrt{2\beta^{-1}\eta}\,\xi_{k+1},\tag{8}\] where \((\xi_k)\) are independent standard Gaussian vectors in \(\mathbb{R}^d\). We denote its Markov kernel by \(Q_\eta\). Equivalently, if \(X_0=x\) and \[X_1=x+\Delta_\eta(x),\qquad \Delta_\eta(x):=\eta b^T_\eta(x)+\sqrt{2\beta^{-1}\eta}\,\xi,\] then, for every bounded measurable test function \(\varphi\), \[\label{eq:kernel-conditional-expectation} Q_\eta\varphi(x) =\mathbb{E}\big[\varphi(X_1)\mid X_0=x\big] =\mathbb{E}\big[\varphi(x+\Delta_\eta(x))\big].\tag{9}\] This conditional-expectation form will be used in the Lyapunov drift argument below. In terms of \(F_z\), the Hessian-envelope form is \[\label{eq:hessian-tamed} Y_{k+1}=Y_k- \eta\frac{\nabla F_z(Y_k)}{1+\eta^\alpha\|\nabla^2F_z(Y_k)\|_{\mathrm{op}}} +\sqrt{2\beta^{-1}\eta}\,\xi_{k+1}.\tag{10}\]

4 Moment Stability and Invariant Measures↩︎

We now prove moment stability and invariant-measure existence for the exact-gradient deterministic-envelope kernel \(Q_\eta\). The key input is the effective-linearity bound in Assumption 3. For localized envelopes with a hard-tail safeguard, the corresponding bound is verified in Lemma 1; Remark 6 explains why sublinear local soft envelopes do not generally provide the same far-tail control.

We first collect two elementary estimates used in the Lyapunov drift argument. The first one controls the tamed drift under the envelope condition, while the second one converts one-step updates into increments of the polynomial Lyapunov function.

Theorem 9 motivates a convention for selecting invariant measures uniformly in the stepsize. We write the selected family as \((\pi_\eta)_{0<\eta\le\eta_0}\), where each \(\pi_\eta\) is an invariant probability measure of the corresponding kernel \(K_\eta\).

For the exact-gradient deterministic-envelope kernel \(Q_\eta\), Theorem 9 supplies this uniform bound for any such selection \((\pi_\eta)_{0<\eta\le\eta_0}\) of invariant probability measures, at each polynomial moment order covered there. The uniform \(V_q\)-moment bound is the tightness input used below: since \(V_q(x)\to\infty\) as \(\|x\|\to\infty\), every sequence \(\eta_j\downarrow0\) admits a subsequence for which \(\pi_{\eta_j}\Rightarrow\pi\). The phrase admissible family of invariant measures will be used in the same way for stochastic-gradient kernels once the corresponding Lyapunov estimate supplies the required uniform \(V_q\)-moment bound.

The preceding results were stated for a deterministic envelope satisfying Assumption 3. We now record the corresponding drift estimate for a localized exact-gradient kernel whose envelope has a hard-tail safeguard.

5 Weak Stationary Identification and Polynomial Observables↩︎

Section 4 gives invariant measures for each fixed-stepsize exact-gradient deterministic-envelope kernel \(Q_\eta\). Since \(Q_\eta\) is an Euler-type numerical kernel with a modified drift, its invariant measures need not equal the Gibbs law \(\pi_z\) at fixed \(\eta\). This section proves that every admissible family \((\pi_\eta)_{0<\eta\le\eta_0}\) for \(Q_\eta\) satisfies \[\pi_\eta\Rightarrow\pi_z,\qquad \eta\downarrow0,\] and then extends this convergence to polynomial-growth observables.

Admissibility makes \((\pi_\eta)_{0<\eta\le\eta_0}\) tight, so subsequential weak limits exist as \(\eta\downarrow0\). To identify these limits with the Gibbs law, we impose the following stationary identification condition for the limiting Langevin generator.

For compactly supported test functions, the generator limit only sees compact subsets of the state space. The condition \(\eta^\alpha A_\eta\to0\) locally uniformly means that the denominator perturbation disappears on each such compact set: \[\frac{1}{1+\eta^\alpha A_\eta(x)}\longrightarrow1 .\] Consequently, the tamed drift \(b(x)/(1+\eta^\alpha A_\eta(x))\) converges locally to the original drift \(b(x)\). Without this local vanishing, the limiting generator could contain a modified drift rather than \(\mathcal{L}_z\).

Weak convergence only guarantees convergence of integrals against bounded continuous test functions \(f\in C_b(\mathbb{R}^d)\). The examples and empirical risks used below may be unbounded, so we call such quantities observables when their expectations under invariant measures are compared. The next lemma records the standard compact–tail argument for polynomial-growth observables.

This corollary gives the stationary interpretation used in this paper: fixed-stepsize taming may bias the invariant law, but the bias disappears for polynomial-growth observables as \(\eta\to0\), provided the admissible family has a uniform moment bound of order strictly higher than the growth order of the observable.

6 Quantitative Stationary Residual Decomposition↩︎

Section 5 proves the qualitative limit \(\pi_{\eta}\Rightarrow\pi_z\) as \(\eta\downarrow0\). This section studies what remains when the stepsize is fixed. For an observable \(H\), we compare the two stationary expectations \[\pi_{\eta}(H)-\pi_z(H).\] The Poisson equation rewrites this difference as the average one-step generator residual of the numerical kernel. We then decompose that residual into an Euler discretization residual and a deterministic-envelope residual. The stochastic-gradient extension in Section 7 adds the oracle covariance channel.

The bound separates two residuals. The term \(\eta^{\gamma/2}\) is the Euler residual; it comes from the Taylor remainder in the one-step expansion of \(Q_\eta u_H\). The second term is the deterministic-envelope residual; it measures how much the denominator changes the drift on the region sampled by \(\pi_\eta\).

In this bound, the deterministic-envelope residual is the only part that depends explicitly on the denominator design. To isolate this channel, fix a reference probability law \(\mu\) and define, with the exponent \(m\) from Proposition 17, \[\label{eq:det-env-residual-functional} \mathcal{D}_\eta(A;\mu) :=\eta^\alpha\int_{\mathbb{R}^d}(1+\|x\|^m)A_\eta(x)\|b(x)\|\,\mu(\,\mathrm{d}x).\tag{16}\] This fixed-law functional measures the denominator-dependent drift modification under \(\mu\). The next proposition compares this residual channel for a global envelope and a localized hybrid envelope under the same reference law.

Proposition 18 compares the deterministic-envelope residual functional \(\mathcal{D}_\eta(A;\mu)\) under a fixed reference law \(\mu\). This functional measures how much the envelope changes the drift on the region sampled by \(\mu\). Localization can reduce this drift modification on the typical region, while the hard-tail component remains available for Lyapunov stability. The comparison is therefore made under the same reference law, not between the invariant measures of different kernels. It should not be used by itself to rank all members of the envelope family: an envelope that is almost inactive under \(\mu\) can have a small fixed-law residual while still turning on too late to give good finite-stepsize stability or stationary behavior.

Overall, this section decomposes fixed-stepsize bias into an Euler residual and a deterministic-envelope residual, in the same broad weak-error tradition as numerical SDE bias expansions [18], [21]. Section 7 adds the stochastic-gradient layer, where centered oracle noise appears through covariance and higher-order residual terms.

7 Deterministic-Envelope Stochastic-Gradient Extension↩︎

Section 6 decomposed the fixed-stepsize residual for the exact-gradient deterministic-envelope kernel into an Euler residual and a deterministic-envelope residual. This section extends the deterministic-envelope kernel to stochastic-gradient oracles. The key point is that the denominator is fixed conditional on the current state, so a centered oracle error remains centered after envelope scaling.

Let \[\label{eq:sg-oracle} g(x,U_z)=\nabla F_z(x)+\zeta(x,U_z), \qquad \mathbb{E}[\zeta(x,U_z)\mid x]=0.\tag{17}\] The deterministic-envelope stochastic-gradient update is \[\label{eq:sg-tamed-kernel} Y_{k+1}^{\mathrm{sg}} =Y_k^{\mathrm{sg}} -\eta\frac{g(Y_k^{\mathrm{sg}},U_{z,k+1})}{1+\eta^\alpha A_\eta(Y_k^{\mathrm{sg}})} +\sqrt{2\beta^{-1}\eta}\,\xi_{k+1}.\tag{18}\] Since \(b=-\nabla F_z\), the tamed stochastic drift can be written as \[\label{eq:sg-tamed-drift-decomposition} \widehat b^T_\eta(x,U_z) :=\frac{b(x)-\zeta(x,U_z)}{1+\eta^\alpha A_\eta(x)} =b^T_\eta(x)+\varepsilon_\eta(x,U_z), \qquad \varepsilon_\eta(x,U_z):=-\frac{\zeta(x,U_z)}{1+\eta^\alpha A_\eta(x)}.\tag{19}\] Because the denominator is deterministic conditional on \(x\), \[\label{eq:sg-centered-tamed-drift} \mathbb{E}[\widehat b^T_\eta(x,U_z)\mid x]=b^T_\eta(x), \qquad \mathbb{E}[\varepsilon_\eta(x,U_z)\mid x]=0.\tag{20}\] This identity will be used in both the Lyapunov estimate and the stationary residual decomposition below.

A mini-batch-dependent denominator generally does not have this property. If the same oracle realization appears in the denominator, then typically \[\label{eq:random-denominator-centering-failure} \mathbb{E}\left[ \frac{b(x)-\zeta(x,U_z)}{1+\eta^\alpha \widetilde{A}_\eta(x,U_z)} \mid x \right] \ne b^T_\eta(x).\tag{21}\] Thus random denominators may create a first-order oracle drift shift. A deterministic envelope eliminates this mean-shift channel.

The stochastic-gradient extension requires three basic controls: a centered oracle error, conditional moments, and envelope-compatible one-step increments. The next assumption records these requirements in this order.

With this oracle condition in place, the Lyapunov stability estimate extends to the stochastic-gradient kernel. The only new term is the envelope-scaled oracle increment, whose conditional mean and moments are controlled by Assumption 6.

Before stating the weak consistency result, define the conditional second moment of the envelope-scaled oracle perturbation by \[\label{eq:sg-oracle-second-moment} \begin{align} \mathsf{Var}_x(\varepsilon_\eta) &:= \mathbb{E}[\|\varepsilon_\eta(x,U_z)\|^2\mid x] \\ &= \mathbb{E}\left[ \left\| \frac{\zeta(x,U_z)}{1+\eta^\alpha A_\eta(x)} \right\|^2 \mid x\right]. \end{align}\tag{22}\] This scalar conditional second moment is the covariance scale used in the stochastic-gradient residual conditions below.

Proposition 19 gives the stability and tightness needed to identify stationary limits. We now identify the small-step limits of the stochastic-gradient invariant measures.

The consistency result above identifies the small-stepsize limit. We next give the corresponding finite-stepsize stationary error decomposition for Poisson observables.

Theorem 22 separates three stationary-error channels. The term \(\eta^{\gamma/2}\) is the Euler residual from the local Taylor remainder. The term \(\eta^\alpha A_\eta(x)\|b(x)\|\) is the deterministic-envelope residual from replacing \(b\) by \(b^T_\eta\). For the deterministic-envelope stochastic-gradient kernel considered here, the taming denominator does not create an additional first-order oracle-bias term; the oracle contribution appears through the covariance residual \(\eta\,\mathsf{Var}_x(\varepsilon_\eta)\). This is the structural advantage of a deterministic envelope: conditional on \(x\), the denominator is fixed, so the current stochastic-gradient realization enters only through the numerator. This separation also identifies which design choice acts on which residual channel: deterministic denominators remove the oracle-coupled mean-shift channel, localized activation reduces deterministic-envelope residuals in typical regions, and batch size or variance reduction controls the covariance channel.

8 Numerical Experiments↩︎

The experiments answer three concrete questions suggested by the theory. First, does using a denominator computed from the same stochastic gradient create an extra average drift bias? Second, how much stationary error is introduced by different deterministic-envelope choices? Third, do the same patterns persist in regression examples, implementable envelope rules, and a higher-dimensional stress test?

8.1 Experimental setup↩︎

The experiments isolate how denominator design affects stationary error. We compare gradient-dependent denominators, global deterministic taming, and localized deterministic-envelope designs on controlled high-growth and empirical-risk examples. Reported errors are measured against either an analytic reference or a long-run numerical reference for the corresponding invariant-law quantity.

When a numerical reference is used, we account for its Monte Carlo uncertainty. For an error of the form \[\Delta H=\widehat H_{\rm method}-\widehat H_{\rm ref},\] we use the propagated standard error \[\mathrm{SE}(\Delta H) = \left( \mathrm{SE}_{\rm method}(H)^2+ \mathrm{SE}_{\rm ref}(H)^2 \right)^{1/2}.\] Errors comparable to this scale are interpreted as reaching the numerical-reference uncertainty level, rather than as resolved nonzero stationary bias.

Table 2 records how envelope parameters are selected. The radial rows use fixed stress-test rules, and the analytic reference values are used only for reporting. In the quartic-regression examples, warm-up quantiles of a deterministic growth indicator estimate typical and tail regimes. The high-dimensional row is included as a stress-test setting rather than as an optimized sampler benchmark.

8.2 Bias from a random denominator↩︎

Experiment 1: One-step random-denominator bias.

This short diagnostic illustrates the local mechanism from Propositions 2 and 4. We use the scalar unbiased oracle \[g_b=\mu+\sigma b^{-1/2}Z, \qquad Z\sim N(0,1),\] with \(\mu=1\) and \(\sigma=3\), and compare a random-denominator transform with its deterministic-denominator counterpart, \[T_\lambda^{\rm rand}(g)=\frac{g}{1+\lambda |g|}, \qquad T_\lambda^{\rm det}(\mu)=\frac{\mu}{1+\lambda |\mu|}.\] The diagnostic reports \[\left|\mathbb{E}\,T_\lambda^{\rm rand}(g_b)-T_\lambda^{\rm det}(\mu)\right|,\] which is zero for a deterministic denominator but generally nonzero when the same noisy oracle realization enters the denominator.

Experiment 1 is only a local sanity check. The more important question is whether this denominator-induced mean drift shift is visible after the Markov chain reaches stationarity; Experiment 2 addresses that question.

Experiment 2: Stationary observable errors for the quartic target.

We now test the same denominator effect at the stationary-observable level. Before specifying the target, we first define the diagnostics used in the table. Let \(\widehat b_\eta(x,U)\) denote the tamed stochastic drift used by the method being tested, and let \(b_\eta^T(x)\) be the deterministic-envelope drift. The reported mean drift shift is \[\mathbb{E}_x\left\|\mathbb{E}_U[\widehat b_\eta(x,U)\mid x]-b_\eta^T(x)\right\|.\] It measures the extra first-order oracle-bias channel that appears for a random denominator and is removed by a deterministic denominator.

The reported \(\eta\)-noise variance is \[\eta\,\mathbb{E}_x\mathbb{E}_U\left[ \left\|\widehat b_\eta(x,U)-\mathbb{E}_U[\widehat b_\eta(x,U)\mid x]\right\|^2 \right].\] It measures the scale of the covariance-level stochastic-gradient contribution. Finally, the observable columns report the stationary errors \[|\Delta \mathbb{E}H| =\left|\mathbb{E}_{\pi_\eta^{\rm method}}H-\mathbb{E}_{\rm ref}H\right|,\] for \(H=x^2,x^4,F\), probing scale, tail sensitivity, and energy-level error. The deterministic-envelope residual associated with the choice of \(A_\eta\) is examined later in Experiment 4, where different deterministic envelopes are compared directly.

We instantiate these diagnostics on the one-dimensional quartic target \[F(x)=\frac{x^4}{4},\qquad b(x)=-x^3,\] for which the Gibbs moments are available analytically. The stochastic-gradient oracle is \[g(x,U)=b(x)+\sigma U, \qquad U\sim N(0,1),\] with \(\sigma=2.5\), and the stepsize is \(\eta=0.04\). We compare exact deterministic-envelope taming, stochastic-gradient taming with the same deterministic denominator, and stochastic-gradient taming with a denominator recomputed from the current oracle realization. Each row averages ten independent runs; the errors are absolute deviations from the analytic Gibbs values.

Table 3 shows the denominator effect at the level of stationary averages. With a deterministic denominator, the measured mean drift shift is zero. With a random denominator, the mean shift is nonzero, and the errors in \(x^4\) and \(F\) are larger in this experiment. Thus the one-step bias mechanism from Experiment 1 remains visible after the chain is run to stationarity.

Experiment 3: Batch-size scaling of the stochastic-gradient error.

The previous experiment uses a fixed oracle noise level. Experiment 3 checks a more specific prediction behind the covariance condition in Theorem 20 and Remark 21: increasing the batch size should reduce the \(\eta\)-scaled noise-variance contribution, while the deterministic denominator should still have no mean drift shift. We vary the batch-size proxy while keeping the same quartic target and stepsize \(\eta=0.04\). The stochastic-gradient oracle is \[g_{\ell_b}(x,U)=b(x)+\sigma \ell_b^{-1/2}U, \qquad U\sim N(0,1),\] so increasing \(\ell_b\) reduces only the oracle noise. Thus the \(\eta\)-scaled noise variance for the deterministic-denominator scheme should decrease approximately like \(1/\ell_b\), while the mean drift shift should remain absent.

Table 4 shows two effects. For the deterministic denominator, the mean drift shift is structurally zero and the \(\eta\)-scaled noise variance is nearly halved whenever \(\ell_b\) is doubled. For the random denominator, increasing \(\ell_b\) also reduces the extra mean drift shift, but this first-order bias is not absent at finite batch size. The observable error \(|\Delta \mathbb{E}x^4|\) moves in the same direction, so the batch-size trend is visible both at the drift level and at the stationary-observable level.

8.3 Choosing the deterministic envelope↩︎

Experiment 4: Deterministic-envelope choices for radial targets.

The next experiment returns to the selective-taming principle in Remark 5. Once the denominator is deterministic, the remaining design question is where the envelope should start acting. We use the controlled radial potential \[U_p(x)=\frac{\|x\|^p}{p},\qquad x\in\mathbb{R}^d,\] with \(d=5\). Under the Gibbs reference law, \(\pi(U_p)=d/p\), so we measure stationary risk error by \[\left|\widehat\mathbb{E}_{\pi_\eta}U_p-\frac{d}{p}\right|.\] This observable is used only as a finite-stepsize stationary check; no convergence-rate fitting is used.

We compare gradient-dependent denominator taming, global hard deterministic taming, global soft deterministic taming, local hard deterministic taming, and the hybrid deterministic envelope \[A^{\rm hyb}(x) =\big(\|\nabla U_p(x)\|-R\big)_+^{1/2} +\big(\|\nabla U_p(x)\|-S\big)_+ .\] The first term keeps the envelope inactive in the safe region and turns it on softly afterward; the second term is a hard-tail safeguard. Thus the experiment compares different choices of the taming range, rather than different stochastic-gradient oracles. The envelope parameters follow the fixed stress-test rules in Table 2.

Table 5 reports absolute error relative to \(d/p\), averaged over three independent repetitions. The main empirical comparison is the invariant-law error and stability behavior. Here “unstable” means that the run leaves the prescribed computational domain, produces non-finite values, or has moment estimates dominated by rare explosive excursions before the sampling horizon. The \(\mathcal{D}_\eta\)-scale column is included as a grouped diagnostic linking the experiment to the fixed-law residual mechanism in Proposition 18.

The results support the selective-taming principle. Global hard taming is stable but can be overly conservative, while the global soft row is mild in easier cases but becomes unstable in the hardest case, consistent with the far-tail limitation in Remark 6. The hybrid envelopes combine mild typical-region behavior with a hard-tail safeguard and give the smallest errors among the reported stable deterministic-envelope runs.

To make this interpretation concrete, Table 6 reports envelope-range diagnostics for the hardest radial case \(p,\eta)=(30,0.005)\). The effective drift factor is \[R_\eta(x)=\frac{1}{1+\eta^\alpha A_\eta(x)},\] so smaller values mean stronger taming. The first row measures how much drift is retained at the 99.9% reference quantile of \(\|\nabla U_p(X)\|\). The second row measures the far-tail deterministic step size \(\eta\|\nabla U_p\|R_\eta\) at \(\|\nabla U_p\|=10^5\).

Table 6 gives the local scale information behind these outcomes. Global hard taming already removes most of the drift at the 99.9% reference quantile, while the hybrid envelope retains more than half of it. Local soft envelopes are also mild near the reference mass, but their far-tail step remains large; the hard-tail part of the hybrid rule brings this step back to the hard-taming scale.

The local hard row shows the opposite failure mode: it is inactive on the sampled reference mass but can turn on too late for good finite-stepsize stationary behavior. Overall, the diagnostics match the design compromise in Remark 5: mild behavior near the typical region, with a hard safeguard only in the far tail.

8.4 Regression and implementable-envelope examples↩︎

Experiment 5: Non-radial quartic-regression target.

The radial tests above isolate the denominator and envelope choices in a setting with an analytic reference value. We next compare ordinary global taming with selective deterministic envelopes in a non-radial empirical-risk problem. The purpose is to check whether the advantage of the envelope family persists beyond radial examples.

We use the quartic-regression objective in ?? , repeated here for convenience: \[\label{eq:quartic-regression-experiment} F_z(w)=\frac{1}{n}\sum_{i=1}^n \frac{1}{4}(a_i^\top w-y_i)^4+\frac{\lambda}{2}\|w\|^2 .\tag{23}\] Its gradient depends cubically on the residuals \(a_i^\top w-y_i\), so the drift scale is nonuniform across directions and regions of the state space. This makes it a natural test case for selective taming: global taming can be stable but conservative, while local or hybrid envelopes can leave moderate-gradient regions less changed and still act when the drift scale is large.

The stochastic-gradient oracle is the mini-batch estimator \[g(w,B)=\frac{1}{|B|}\sum_{i\in B}\nabla_w \left[\frac{1}{4}(a_i^\top w-y_i)^4+\frac{\lambda}{2}\|w\|^2\right],\] with batches sampled uniformly from the dataset. In the first run we use \(d=4\), \(n=50\), ridge parameter \(\lambda=0.2\), Gaussian covariates scaled by \(d^{-1/2}\), and mini-batch size \(|B|=6\).

Since no closed-form Gibbs reference is available, we use small-stepsize exact-gradient Langevin runs as numerical references: \(\eta_{\rm ref}=10^{-3}\), 80 parallel chains, 5000 iterations, and 1500 burn-in iterations. Eight independent reference runs give reference empirical-risk average approximately \(1.46\), with standard error about \(2.2\times10^{-2}\). The sampled chains use 40 parallel chains, 2500 iterations, and 800 burn-in iterations. A run is declared unstable if \(\|w_k\|>25\) or if the empirical risk becomes non-finite.

To isolate the denominator-design effect, Table 7 first uses the full-gradient envelope \(\bar A_{\rm diag}(w)=\|\nabla F_z(w)\|\) for deterministic-envelope methods. Values close to the propagated uncertainty scale \(\mathrm{SE}(\Delta H)\) should be interpreted as reaching the numerical-reference noise floor rather than as meaningful nonzero stationary bias.

The local hard and hybrid rows are close to the numerical-reference noise floor, while random-denominator and global hard taming remain visibly farther from the exact-gradient reference. Thus, in this non-radial example, selective deterministic envelopes improve on ordinary global taming, with local hard and hybrid both effective. The slightly smaller error of local hard in the smaller-stepsize run is consistent with the envelope interpretation: when tail excursions are rare, a local hard rule can be nearly inactive on the sampled typical region and therefore introduce less deterministic shrinkage, while the hybrid rule trades this for additional tail robustness.

These risk means are stationary averages, not optimized training losses. Several stabilized schemes may still move toward low-risk regions when viewed as optimization methods, but their fixed-step invariant laws can differ. A random denominator changes the transition kernel by coupling the current mini-batch gradient to the taming factor, and this mean distortion is visible in stationary averages such as \(\pi_\eta(F_z)\).

The same qualitative separation appears in a moderately larger synthetic instance with \(n=500\), \(d=10\), and mini-batch size \(|B|=20\). The reference is again a small-stepsize exact-gradient Langevin numerical average for the same empirical objective, with estimated reference standard error about \(9.6\times10^{-2}\). Under the propagated-error calculation, the hybrid and local hard rows remain several uncertainty scales closer to the reference than the random-denominator and global-hard rows. All listed runs had zero blow-up.

Experiment 6: Implementable deterministic envelopes.

Experiment 6 replaces the full-gradient diagnostic envelope by a cheaper deterministic growth bound. The full-gradient envelope \(\bar A_{\rm diag}(w)=\|\nabla F_z(w)\|\) is useful for mechanism diagnosis, while an implementable rule can use a deterministic polynomial scale. For the quartic regression objective, we use \[\bar A_{\rm poly}(w)=C_z(1+\|w\|^3),\] which is independent of the current mini-batch and has negligible cost compared with a mini-batch gradient evaluation.

Table 9 compares this polynomial envelope with the full-gradient envelope. Since the polynomial envelope operates on a different numerical scale, the local and hybrid thresholds are rescaled. The reference standard error is about \(2.2\times10^{-2}\); for the ideal hybrid row, the reported error is below the propagated numerical-reference uncertainty scale, while the polynomial hybrid and global-hard errors remain much larger.

The ideal and polynomial hybrid rows use the same hybrid activation pattern; they differ only in the deterministic growth indicator. The ideal row uses the full-gradient scale \(\|\nabla F_z(w)\|\), whereas the polynomial row uses the cheaper proxy \(C_z(1+\|w\|^3)\). Thus Experiment 6 tests whether the selective-envelope idea remains useful under a coarser, implementable scale. The polynomial hybrid row retains a clear advantage over the global-hard and local-hard polynomial rows, although its gap from the ideal hybrid row shows that envelope scale matching still matters.

To see why, Table 10 reports the effective drift factor \[R_\eta(w)=\frac{1}{1+\eta^\alpha A_\eta(w)}\] at two reference quantiles of the implementable polynomial scale. Larger values of \(R_\eta\) mean that more of the original drift is retained after taming.

The table helps explain the pattern in Table 9. The global hard polynomial envelope is already very aggressive over much of the reference mass: at the 95% quantile it keeps only about 10% of the drift. The local and hybrid polynomial envelopes leave that region essentially unchanged. At the 99.9% quantile the hybrid envelope has started to act, but it remains much milder than the global hard polynomial denominator.

This is the selective-taming design of Remark 5 in an implementable form. The envelope family gives different ways of choosing where taming turns on; for this quartic-regression problem, the hybrid polynomial rule is a better match to the observed growth-scale separation than the ordinary global polynomial denominator.

8.5 A high-dimensional stress test↩︎

Experiment 7: High-dimensional quartic-regression stress test.

Finally, we include a calibrated high-dimensional stress test with the same quartic-regression structure, using \(n=1000\), \(d=50\), mini-batch size \(|B|=50\), and \(\eta=8\times10^{-4}\). The numerical reference is an averaged small-stepsize exact-gradient run for the same empirical objective, computed from twelve independent reference chains; the estimated reference empirical risk is \(8.69\), with standard error about \(8.3\times10^{-2}\).

The hybrid thresholds are selected from warm-up quantiles of the deterministic growth indicator, with \(R\approx Q_{0.6}\), \(S\approx Q_{0.99}\), and \(\gamma=0.5\), so that the safe, soft, and hard-tail regimes are all represented. The selected thresholds are then held fixed for the reported production runs. In Table 11, entries are relative to the averaged exact-gradient numerical reference, and the safe/soft/tail columns report sampled regime fractions.

Table 11 is a stress test of the envelope family. The high-dimensional reference itself has a non-negligible Monte Carlo uncertainty. The selected deterministic-envelope choice remains stable and gives the smallest reported stationary error, but its error, about \(3.13\times10^{-1}\), is comparable to the propagated uncertainty scale \(3.57\times10^{-1}\). Thus the selected hybrid row is already at the numerical-reference resolution scale for this experiment.

By contrast, the random-denominator, global-hard, and local hard rows remain several propagated standard errors away from the numerical reference, so their larger discrepancies are still informative method-level differences. Thus, even in this higher-dimensional stress setting, the hybrid envelope still shows a clear advantage among the reported denominator designs. This suggests that the selective-envelope idea remains useful in moderately high dimension, while the choice of envelope parameters becomes more sensitive to dimension.

Overall, the numerical results support the interpretation developed above. Random-denominator taming and global hard deterministic taming can stabilize the update, but they may introduce large finite-stepsize stationary error for different reasons. Local and hybrid envelopes instead choose where taming turns on.

The examples also show that the best member of the envelope family can be problem-dependent. In the moderate quartic-regression example, local hard and hybrid can be statistically comparable; in the high-dimensional stress test, the hybrid rule is more robust among the reported choices. The polynomial-envelope rows show an additional scale effect: matching polynomial growth can be enough for stability control but still produce poor finite-stepsize error if the envelope is too conservative on the typical region.

9 Discussion↩︎

9.0.0.1 Oracle-structure viewpoint.

This paper studied tamed denominators in fixed-stepsize stochastic-gradient Langevin sampling from an oracle-structure viewpoint. The main observation is that a denominator depending on the same stochastic-gradient realization as the numerator is not a neutral stabilizer: it changes the stochastic oracle and can create a stationary mean distortion even when the original stochastic gradient is unbiased. Deterministic envelopes separate stabilization from current oracle randomness by fixing the denominator conditional on the state.

9.0.0.2 Deterministic and selective envelopes.

The deterministic-envelope framework turns this observation into a denominator-design principle. The denominator is a problem-dependent component of the Markov transition: it controls both the coupling to stochastic-gradient noise and the region in which stabilization is activated. Global deterministic envelopes remove the oracle-coupled mean-shift channel, but they may still damp the drift too strongly in typical regions. Localized and hybrid envelopes address this second issue by choosing where taming turns on. In particular, the hybrid construction combines mild behavior near the sampled mass with a hard-tail safeguard for rare excursions.

The theory reflects this separation. Weak stationary consistency identifies the limiting Gibbs law, while the stationary error decomposition separates the Hölder discretization remainder, the deterministic-envelope residual, and the stochastic-gradient covariance residual. This decomposition also suggests a residual-budget view of denominator design: avoid the oracle-coupled mean-shift channel, keep deterministic-envelope residuals small in typical regions, and retain far-tail control for stability. Turning this view into systematic rules for envelope selection is a natural next step. The numerical experiments support the same interpretation: random denominators create visible drift shifts, batch-size scaling controls the covariance contribution, selective envelopes reduce stationary error relative to ordinary global taming, and the hybrid design remains effective in the higher-dimensional stress test.

9.0.0.3 Outlook.

The present framework suggests a broader denominator-design program for tamed stochastic-gradient Langevin kernels. For a tamed stochastic-gradient oracle, the denominator should be chosen as a deterministic envelope rather than as a fixed stabilization formula or a function of the current oracle realization. The stationary decomposition gives a practical guide for this design: avoid the oracle-coupled mean-shift channel, keep deterministic-envelope residuals small in typical regions, and retain enough far-tail control for stability.

Several questions remain open. The most immediate one is how to turn this residual-budget view into systematic rules for choosing the envelope scale, activation thresholds, soft-growth exponent, and implementable growth proxy. The experiments indicate that these choices depend on the growth profile of the objective and become more delicate as dimension increases. A further direction is to extend the same deterministic-envelope principle to more general stochastic-gradient oracles, as long as the denominator remains fixed before the current oracle noise is sampled. The central point is that the denominator should be a deterministic, problem-dependent component of the Markov transition, not an additional stochastic transformation of the oracle.

References↩︎