1 Introduction↩︎

Stochastic gradient descent (SGD) and its variants are widely used as optimization algorithms for complex, non-convex functions, which often contain many under-performing local minima. One well-studied aspect of SGD, which is theorized to contribute to the method’s success, is the presence of noise [1]–[6], typically through the mini-batch process, whereby a noisy estimate of the true gradient signal is taken to update the weights of the model. This introduction of stochasticity helps to escape local minima and saddle points that otherwise may trap an optimizer.

Consequently, it has been recognized that deliberate noise injection, typically the addition of Gaussian noise to the data, further improves robustness and generalization [5], [7]–[12]. Adding Gaussian noise to the weights is employed for sharpness-aware-minimization [13] and, after training, to create a diverse ensemble of models [14]. However, for optimization methods with noise injection one has to decide on a noise schedule, and each choice comes with downsides. A monotonic decaying schedule restricts the optimizers capacity to escape suboptimal minima late in training, whereas “curriculum” ([15], which starts with no noise and then switches to high noise) or “anti-curriculum” (which does the opposite) schedules either lose their regularization power or underfit the clean data, respectively: these schedules risk forgetting what they fit to during the early training phase. But this need not be the case. Rather than decaying the noise, or turning it on or off completely, we propose an interleaved schedule with several sharp transitions between clean epochs and corrupted epochs. By evaluating both additive (Gaussian) and non-additive (impulse) noise perturbations within this schedule, we demonstrate that interleaved noise injections improves clean, corrupt, and out-of-distribution (OOD) performance. While the latter two performance gains are not surprising, the improvement on clean data (compared to training only on clean data) deserves further investigation because it suggests using noise as a tool to improve not only the robustness but the outright training objective. The remainder of this paper presents a theoretical derivation of the effects of injected noise, empirical evidence for our claim of superior performance, and an investigation into the interplay between neural network architecture and the most effective type of noise corruption.

Contributions↩︎

• Interleaved noise curriculum: We introduce an interleaved noise curriculum for stochastic optimization, which outperforms standard curriculum and anti-curriculum methods on clean, corrupted, and out-of-distribution classification tasks.

• Theoretical formulation of impulse noise regularization: We present a second-order Taylor expansion to demonstrate how intermittently injecting noise acts as a regularizer. In particular, we derive that impulse noise approximates directional Jacobian regularization.

• Empirical improvements and orthogonality to existing regularizers: We test the interleaved curriculum against other corruption and robustness regularization techniques on CIFAR-100 and ImageNet-1k datasets on WideResNet, ResNet50 and ViT architectures. Interleaved noise injection improves clean accuracy, corrupted accuracy, and robustness across all tests. We show that layering the interleaved noise schedule on top of existing regularization methods further improves performance across all three domains.

• Gradient-norm stabilization: A primary challenge in optimization with noise injection is the volatility of the gradient signal and its magnitude during noisy epochs. We present a dynamic scaling method to constrain the magnitude of the optimizer updates to prevent vanishing and/or exploding gradients for noise-injection optimization.

• Architecture specific noise preference: We provide empirical evidence showing convolutional-based networks benefit more from impulse noise, whereas attention-based models perform better under Gaussian noise. The specific noise distribution reduces overfitting for the specific inductive biases of each architecture. We show that injection with the appropriate noise distribution for a specific architecture always results in model improvements.

2 Related work↩︎

2.0.0.1 Simulated (and parallel) tempering

Simulated tempering involves simultaneously performing stochastic optimization on some loss surface over a range of fixed temperatures [16], [17]. These methods aim to reduce the occurrence of optimizers becoming stuck in a sub-optimal local minima. As the action of adding noise to samples is equivalent to increasing the temperature, we see our work in this tradition, with the main contribution coming from temporal modulation of the noise.

2.0.0.2 Noise-injection for regularization:

[18] explored injecting noise into datasets for MCMC algorithms and found this method is able to speed up convergence when using monotonically decaying noise curricula, akin to simulated annealing. [5] explored the effect of simulating stochastic gradient noise, finding that using their “positive-negative momentum” method improved generalization performance over a variety of image classification tasks. The role of noise as a regularization method is also explored in [4] who empirically find that SGD noise due to mini-batch sampling is Hessian-dependent and pushes the optimizer towards flatter minima. This result hints at noise structure, as well as magnitude, being important factors affecting regularization effectiveness. [19] introduce Noise Injection Node Regularization (NINR), which adds additional noise-injecting nodes into the architecture of NNs. This acts to adaptively penalize the loss sharpness, which leads to flatter solutions but introduces additional training times and architecture alterations.

2.0.0.3 Sharpness-aware-minimization

Stochastic noise can be injected directly into the model weights during optimization. Classical weight noise approaches add Gaussian perturbations to the parameters to encourage the optimizer to settle in flat minima, thereby improving generalization. More recently, this concept has been formalized by Sharpness-Aware Minimization (SAM) [13], which explicitly seeks parameters whose neighborhood has, ideally, uniformly low loss. While weight noise and SAM operate on the parameter space to implicitly or explicitly minimize the trace of the Hessian with respect to the weights, we will show that impulse noise acts equivalently but operates entirely on the input space. This allows our method to approximate directional Jacobian regularization and minimize input-loss sharpness, specifically guarding against corruptions and out-of-distribution input shifts rather than parameter perturbations.

2.0.0.4 Curriculum learning:

[15] first introduced the concept of curriculum learning for the field of machine learning, typically proposing a curriculum for data samples then moves monotonically from easy (clean) to hard (noisy or complex). Conversely, anti-curriculum and golden-sample methods work to expose the optimizer to the most difficult samples first. Interleaving is the process of alternating between distinct tasks or concepts during learning, and has been consistently shown to improve long-term retention and generalization compared to blocked, continuous practice in student learning [20]–[22]. We draw from these lines of work and propose an interleaved curriculum, which aims to improve the overall performance of trained models.

3 Methods↩︎

In this section, we formalize our interleaved noise curriculum and show its regularization effects on the optimization landscape through a second-order Taylor expansion. We then introduce our gradient-norm stabilization routine which allows for stable training even at very high noise levels. We end by describing the range of experiments we perform in this study.

3.1 Interleaved noise regularization↩︎

The noise injection schedule \(N(t)\) determines whether (or how much) noise to add to the training data at epoch \(t\). Unlike standard simulated annealing, where \(N(t)\) is always active, we choose an interleaved schedule, which injects noise for \(L\) epochs after \(P\) epochs without noise: \[\label{eq:injection} N(t) = I(t) \sigma_t \;\text{with}\;I(t)= \begin{cases} 1\;\text{if } t \;\text{mod}\;(P+L) \geq P & (\text{noise injection}) \\ 0\;\text{otherwise} & (\text{clean data}) \end{cases}\tag{1}\] where \(\sigma_t\) represents the amount of noise, and \(I(t)\) is a square-wave indicator function for the injection.

3.1.1 Impulse noise↩︎

Assuming normalized input data \(\mathbf{x} \in [0, 1]^D\), the corrupted sample \(\tilde{\mathbf{x}}_t\) is generated via a stochastic mask: \[\tilde{\mathbf{x}}_t = (1 - \mathbf{m_t}) \odot \mathbf{x} + \mathbf{m_t} \odot \mathbf{n},\] where \(\mathbf{n} \sim \text{Bernoulli}(\tfrac{1}{2})\) represents the impulse noise \(\{0,1\}\), and \(\mathbf{m_t} \in \{0, 1\}^D\) is the binary corruption mask sampled element-wise such that \(\mathbf{m}_t \sim \text{Bernoulli}(\sigma_t)\). That means, \(\sigma_t\) indicates the per-pixel probability of receiving a noise corruption that removes all information of the initial pixel value.

3.1.1.1 Expected perturbation

The perturbation of each sample image \(\mathbf{x}\) is given by; \[\begin{align} \boldsymbol{\delta} &= \tilde{\mathbf{x}} - \mathbf{x} = \mathbf{m_t} \;\odot (\mathbf{n} - \mathbf{x}) \end{align}\] Because the noise instance and the mask are uncorrelated, the expected perturbation over a full batch is the expected probability of pixel corruption, multiplied by the expected difference between the clean and noisy image: \[\nonumber \mathbb{E}[\boldsymbol{\delta}] = \mathbb{E}[\mathbf{m_t}] \cdot \left(\mathbb{E}[\mathbf{n} - \mathbf{x}] \right) = I(t)\sigma_t\mathbb{E}[(\tfrac{1}{2} - \mathbf{x})]\]

To see the approximate effect the noise-injection has on the optimization trajectory, we take a second-order Taylor expansion of the loss function \(\mathcal{L}(\theta, \tilde{\mathbf{x}})\) around the clean training sample \(\mathbf{x}\): \[\label{eq:taylor} \mathcal{L}(\theta, \tilde{\mathbf{x}}) \approx \mathcal{L}(\theta, \mathbf{x}) + \boldsymbol{\delta}^\top \nabla_\mathbf{x} \mathcal{L}(\theta, \mathbf{x}) + \frac{1}{2}\boldsymbol{\delta}^\top \nabla_\mathbf{x}^2\mathcal{L}(\theta, \mathbf{x})\boldsymbol{\delta}\tag{2}\] We can take the expectation over a mini-batch, \[\label{zhiyucrt} \mathbb{E}_{x,\boldsymbol{\delta}}[\mathcal{L}(\theta, \tilde{\mathbf{x}})] \approx \underbrace{\mathbb{E}_{\mathbf{x}}\left[\mathcal{L}(\theta, \mathbf{x})\right]}_{\text{clean loss}} + \underbrace{I(t)\sigma_t\mathbb{E}_{\mathbf{x}}\left[(\tfrac{1}{2}-\mathbf{x})^\top \nabla_x \mathcal{L}(\theta, \mathbf{x})\right]}_{\text{Jacobian regularization}} + \underbrace{\frac{I(t)\sigma_t}{2}\mathbb{E}_{\mathbf{x}}\left[\sum_i [\mathbf{H_x}]_{ii}(\tfrac{1}{2} - \mathbf{x}_i + \mathbf{x}_i^2)\right],}_{\text{weighted curvature regularization}}\tag{3}\] which yields three separate components: the loss function of the clean data \(\mathbf{x}\), a regularization term proportional to both the magnitude of the noise level \(\sigma_t\) and the input sensitivity of the model to the data \(\nabla_\mathbf{x} \mathcal{L}(\theta, \mathbf{x})\), and a loss term proportional to the weighted summation of the Hessian diagonal (curvature) which again is modulated by the noise level. Because the mask entries are independent, the off-diagonal second-order terms scale as \(\sigma_t^2\) compared to linear scaling with \(\sigma_t\) of the diagonal terms, hence we drop them in 2 .¹

3.1.1.2 First order regularization

With Cauchy-Schwarz we can provide an upper-bound on our first-order regularization term: \[\label{eq:regulariser} \mathbb{E}_{\mathbf{x}}\left[(\tfrac{1}{2}-\mathbf{x}])^\top \nabla_x\mathcal{L}(\theta, \mathbf{x})\right] \leq \mathbb{E}_{\mathbf{x}}\big[\|\tfrac{1}{2} - \mathbf{x} \|_2\;\cdot \| \nabla_x \mathcal{L}(\theta, \mathbf{x})\|_2 \big]\tag{4}\] Training with impulse noise invokes a magnitude-modulated Jacobian regularization, which has been shown to improve robustness against common data corruptions and improve the identification of large scale features in deep neural networks [23]–[26]. The magnitude of the first-order regularizer is maximized for high contrast \(x_i \in\{0,1\}\) pixels and zero for gray \(x_i=0.5\) pixels. This means our regularizer will more aggressively reduce the sensitivity to high contrast (extreme pixel values) features, while retaining high sensitivity for low contrast features.

3.1.1.3 Second order regularization

We can also upper-bound the second-order regularization term, connecting the minimization of the trace to minimizing the spectral norm (sharpness) of the Hessian matrix: \[\boldsymbol{\delta}^\top \mathbf{H_x}\boldsymbol{\delta} \leq \lambda_{\rm{max}}(\mathbf{H_x})\|\boldsymbol{\delta} \|^2_2\]

\[\mathbb{E}_{\boldsymbol{\delta}|\mathbf{x}}\left[\boldsymbol{\delta}^\top \mathbf{H_x}\boldsymbol{\delta} \right] \leq \lambda_{\rm{max}} \cdot\sigma_t\sum_i(\tfrac{1}{2} - \mathbf{x}_i + \mathbf{x}_i^2)\] By minimizing the expected local sharpness in 2 , impulse noise injection implicitly minimizes the sharpness (spectral norm) weighted by pixel-level contrast, ideally leading to flatter, more generalizable solutions [2]. This is similar to regularization approaches suggested in Sharpness-Aware-Minimization [13], but it is important to point out that we minimize the sharpness w.r.t. the input data \(\mathbf{x}\) instead of the weights \(\boldsymbol{\theta}\). Because impulse noise stochastically removes local information, the training is thus regularized against overfitting to high-frequency spatial features. Instead, the training encourages learning large-scale features that remain robust even to high levels of impulse noise.

3.1.2 Gaussian noise schedule↩︎

For interleaved Gaussian noise, we sample from a zero-mean Gaussian distribution, \(\tilde{\mathbf{x}}_t = \mathbf{x} + \mathbf{n}_t\), where \(\mathbf{n}_t\sim\mathcal{N}(\mathbf{0}_D, \sigma_t\mathbf{1}_D)\). The expected perturbation is thus \(\mathbb{E}[\boldsymbol{\delta}] = 0\), and the expectation of the loss function over a mini-batch reduces to \[\label{eq:taylor-gauss} \mathbb{E}_{x,\boldsymbol{\delta}}[\mathcal{L}(\theta, \tilde{\mathbf{x}})] \approx \underbrace{\mathbb{E}_{\mathbf{x}}\left[\mathcal{L}(\theta, \mathbf{x})\right]}_{\text{clean loss}} + \underbrace{\frac{I(t)\sigma_t^2}{2} \mathbb{E}_{\mathbf{x}}\left[\rm{\mathbf{Tr}}(\mathbf{H_x})\right]}_{\text{curvature regularization}}.\tag{5}\] We no longer get a Jacobian regularization term but retain the familiar second-order curvature regularization, whose strength is again modulated by the noise curriculum. This term results in a uniform curvature penalty, independent of local pixel contrast.

3.2 Gradient-norm stabilization↩︎

The general update rule (ignoring preconditioning terms) for SGD-type algorithms is: \[\begin{align} \mathbf{g}_t = \eta_t \nabla_\theta \mathcal{L}(\theta_{t}, \mathbf{x}) \end{align}\] where \(\eta_t\) is the time-dependent learning rate. A potential issue with our interleaved noise schedule is that for a constant learning rate, the magnitude of the noisy gradient update \(\tilde{\mathbf{g}}_t\) may deviate significantly from \(\mathbf{g}_t\).

For example, if we were to introduce an image of pure noise, the magnitude of the gradient term \(\nabla_\theta \mathcal{L}(\theta_t, \mathbf{\tilde{x}})\) would represent an average of \(n\) random vectors where \(n\) is the batch size. As \(n \to \infty\) this would cause \(\nabla_\theta \mathcal{L} \to 0\). Alternatively, at late training times when \(\nabla_\theta \mathcal{L}(\theta_t, \mathbf{x})\) is small due to descending into a minima, even a small perturbation to \(\mathbf{x}\) due to the impulse noise could lead to significant increases in the gradient norm. Hence, we modulate the noisy update magnitude \(\tilde{\mathbf{g}}_t\) with a scalar prefactor term \(p_t\), defined as \[\begin{align} \label{eq:stabalisation} p_t = f \cdot \frac{\|\nabla_\theta \mathcal{L}(\theta_{t-1}, \mathbf{x}) \|_2 }{\|\nabla_\theta \mathcal{L}(\theta_t, \tilde{\mathbf{x}}) \|_2}, \end{align}\tag{6}\] where the magnitude of the gradient norm of the last clean update (averaged over a full epoch) determines the magnitude of the current noisy update. We also introduce a tuning parameter \(f\), which scales the overall strength of the noisy gradient updates (5.3.1 investigates the effect of this parameter). The update step during the noisy epochs is then \[\begin{align} \tilde{\mathbf{g}}_t = \eta_t p_t\nabla_\theta \mathcal{L}(\theta_{t}, \tilde{\mathbf{x}}). \end{align}\]

4 Experiments↩︎

We evaluate the proposed interleaved curriculum using both impulse and Gaussian noise 1) against the standard training on clean data, 2) in comparison to alternative noise injection curricula, and 3) against more complex noise injection and augmentation methods, specifically:

1. Cutout [27] to determine whether generalization benefits from noise injection exceed those from masking large patches;

2. Sharpness-Aware Minimization [13] to test implicit input-space curvature minimization against explicit weight-space curvature minimization;

3. AugMix [28], which performs a series of structural augmentations to the input images, to determine benefits for corrupted and OOD data errors;

4. Vital Phase Augmentation [29], a novel approach that introduces phase shifts during training, to compare frequency-space augmentation to spatial-domain regularization.

In each case, we measure the performance on clean, corrupted, and OOD datasets.

4.0.0.1 Architectures and training details

For our CIFAR-100 experiments, we utilize a WideResNet architecture [30]. For ImageNet-1k evaluations, we test both ResNet50 [31] and Vision Transformer [32] architectures to ensure our findings generalize across convolutional and attention-based models. We train the WideResNet from scratch for 100 epochs; for the ResNet50 and ViT we begin with pre-trained models and perform fine-tuning for 50 and 30 epochs for the ResNet50 and ViT, respectively. The ResNet50 architecture follows from [31] and we use the ViT-B/16 variant with a patch size of \(16 \times 16\) and an embedding dimension of 768. We perform hyper-parameter tuning over the learning rate for all results reported. For the CIFAR-100 trials our baseline method is AdamW, while for ImageNet-1k we use SGD with momentum for the ResNet50 model and AdamW for the ViT. For AugMix we use identical hyperparameters for the severity and Jensen-Shannon divergence loss as reported in [28].

For our noise schedules, we inject impulse or Gaussian noise to all training samples for one full epoch, once every 5 epochs, in all trials \((P=5)\)². We note that this results in a square wave curriculum with \(156\) noisy gradient updates every 5 epochs for the CIFAR-100 trials using a batch size of 256, and \(8008\) noisy updates for both ImageNet trials which use a batch size of 128. For all reported results we use the gradient norm scaling factor \(f=0.4\), which was approximately the best performing trial in each case. We explicitly avoid a large hyperparameter sweep over \(P\) and \(f\) for our presented results to show the practical usefulness of this method without the need for extra costly tuning. Preliminary ablation tests (see Sections 5.3) showed performance was robust for a wide range of \(P\) and \(f\).

We train all models on NVIDIA-A100 GPUs. The training times remain unchanged because our method (unlike SAM, AugMix or VIPAug) does not require any additional forward or backwards passes over the data nor any additional clean epochs. The only extra computational cost stems from tracking the last clean gradient norm and calculating the gradient-norm scaling in 6 .

4.0.0.2 Corruption and robustness testing

To assess model robustness against common corruptions, we use the CIFAR-100-C and ImageNet-C benchmark datasets introduced by [33]. These datasets apply 15 distinct types of corruptions across five severity levels. To test our noise injection method for improvements in OOD performance, we compare the inference performance of the ResNet50 and ViT models, trained on the ImageNet-1k dataset, on the ImageNet-R dataset [34]. This dataset consists of 200 subclasses of ImageNet-1k which have been altered to test for out-of-distribution robustness. We report standard clean top-1 error, mean corruption error (mCE), and structural mCE (which removes the noise-based corruption classes). All results are averaged over 5 random seeds with bootstrapped errors shown.

5 Results↩︎

We report error rates (\(\downarrow\) is better) in all tables and accuracy (\(\uparrow\) is better) in all figures.

5.1 Comparison to clean training and other noise injection curricula↩︎

For this section, we train the WideResNet model on CIFAR-100 data, using either only clean, or only noisy, or a mix of clean and noisy data, time-ordered as an interleaved curriculum (\(P=10, L=5\); see 1 ). The dataset was split into a 70/10/20 train/validation/test split, and each dataset was split further into a clean and noisy set, where the latter had impulse noise injected at a level of \(\sigma=0.65\).

1 shows the accuracy on clean and noisy test data. It is evident that dataset switches are quickly picked up by the model: the transition from clean to noisy initially sharply degrades clean test accuracy, but only for a short period, while producing significant benefits in the long run. In particular: Training with interleaved noise improves long-term clean test accuracy beyond training on clean data only, even though the model probes the clean data for fewer epochs. We interpret this outcome as the result of increased robustness during training with noisy data, leading to minimizers with wider, shallower minima [11], [34], [35]. On the flip side, the interleaved scheme also shows drastic improvement on noisy data: When the training is stopped during a noise-injection period, the long-term accuracy on noisy data outperforms training on noisy data only. We interpret this behavior as the benefit of having access to high-quality samples even if the target data distribution is noisy.

The last result has implications for curriculum and transfer learning, which often employ only one or a small number of sharp transitions in dataset quality or optimization objective. We therefore expand the above test to include a suite of proposed curricula for noise injection. 2 shows that the interleaved schedule consistently achieves the highest final accuracy on both the clean and noisy test datasets. Common alternative curricula either lose their regularization capacity late in training, or fail to converge on the clean data in the finite amount of epochs. We interpret these results as the benefits of repeated “kicks” out from narrow minima into wider ones as well as the repeated “reminders” of what accurate data looks like.

5.2 Comparison to other augmentation methods↩︎

The previous section demonstrated the effectiveness of interleaved noise for a specific model and interleaved period (\(P=10, L=5\)), which may appear fine-tuned. In this section we demonstrate that a fixed, simple choice of a single noisy epoch for every five clean epochs, i.e. \(P=5, L=1\), reaps the same benefits, even when compared to more advanced and computationally costly augmentation schemes. An ablation study on the best choice for \(P\) is presented in 5.3.3.

5.2.1 CIFAR-100↩︎

We compare the performance of our method on the WideResNet model and the CIFAR-100 and CIFAR-100-C datasets. The baseline is a standard AdamW training run, which is enhanced with either Gaussian or impulse noise injection. The alternative methods use training with Cutout, SAM, VIPAug, and AugMix augmentations. ¿tbl:tab:cifar95results? shows that the inclusion of interleaved Gaussian or impulse noise improves clean accuracy, mCE, and structural mCE for all methods, when the right noise distribution is selected. The best performance comes from using AugMix with interleaved impulse noise, which reduced the clean error by \(1.4\%\), mCE by \(16.1\%\), and structural mCE by \(13.2\%\) compared to the baseline AdamW run. We note that much of this improvement for the corruption robustness comes from AugMix alone, but the noise injection further improves the results by \(0.9\%\), \(0.4\%\) and \(0.7\%\) across clean, mCE and structural mCE, respectively. But even just our simple noise injection method improves performance to levels comparable or better than these more expensive schemes.

5.2.2 ImageNet-1k↩︎

We now investigate if these results hold up for larger models by fine-tuning pretrained ResNet50 and ViT architectures on the ImageNet-1k dataset. We report the clean error, corruption, and structural corruption error as well as the OOD accuracy via testing on the ImageNet-R dataset [34]. ¿tbl:tab:imagenet95results? shows that for the overwhelming majority of baseline regularization methods, the addition of either Gaussian or impulse noise with our interleaved curriculum reduces the error across all four performance metrics. Interestingly, for the ResNet50 models the impulse noise appears to work best to reduce the clean, corrupt, structural, and OOD error rates, while for the ViT Gaussian noise performs significantly better across all baselines. We will explore the reasons for this architecture dependence in 6.2.

In 3 we show a breakdown of the test performance by the specific type of corruption from [33]. For a clearer comparison, we only show the best performing baseline regularizer with and without noise injection (full results of per-type corruption accuracy are listed in 9). Injection of impulse noise outperforms both baseline AugMix and the Gaussian noise injection at every severity level across all metrics.

We visualize the test accuracy of the same runs used for ¿tbl:tab:imagenet95results? in 4, showing the performance of the ResNet50 and ViT models on the clean (ImageNet-1k), corrupt (ImageNet-C), and OOD (ImageNet-R) test sets. It confirms the clear trend of interleaved noise injection consistently improving across performance metrics and optimization baselines.

5.3 Ablations↩︎

We now present ablation tests over a set of hyperparameters we have introduced in this paper.

5.3.1 Gradient-norm scaling↩︎

We perform ablations over the gradient-norm scaling factor \(f\) with the WideResNet model on the CIFAR-100 and CIFAR-100-C datasets. 1 shows results with varying levels of \(f\) as well as a trial with no gradient-norm stabilization. We find that reducing the value of \(f\) to around 0.2 tends to increase the error on the clean data, however increasing \(f\) increases the performance on the corrupt data. The general trend shows that increasing \(f\) past \(0.4\) reduces the corruption error rates, but at the cost of higher clean error rates. For impulse noise, the clean error rate never rises above \(25.4 \pm.3\). The performance of both the Gaussian and impulse noise injection with no gradient-norm stabilization degrades noticeably on clean data, however the corruption performance improves.

5.3.2 Noise level↩︎

The amount of noise \(\sigma_t\) added during each noisy epoch is another hyperparameter. 5 shows an ablation over both the gradient-norm scaling factor and the level of noise injected during the noisy epochs. We also show a trial (in green) for linearly decaying the amount of noise injected. In general, we find a gradient-norm scaling factor of roughly \(f\approx0.4\) performs best across all noise levels in terms of clean accuracy. We show a visual example of the noise levels when utilizing salt-and-pepper noise for CIFAR-100 samples in 11.

5.3.3 Interleaved period↩︎

We also evaluate the sensitivity of the results to the interleaved period \(P\), which determines the frequency of the noisy epochs (while leaving \(L=1\) for simplicity). We perform trials again over the WideResNet on the CIFAR-100 dataset varying the period from 1 to 100 in 6. We find the best results occur for \(P\in [5,10]\). Importantly, and as expected, lower values (more frequent noise) can harm clean accuracy but improve corruption performance, and conversely, larger values limit the regularization strength, thus reducing the corruption performance until noise injection has little to no effect on performance, regardless of noise distribution.

6 Discussion↩︎

6.1 Interleaved curricula↩︎

Our results demonstrate that noise injection, which has traditionally been invoked to improve optimization robustness, is more effective than previously known. It provides a tool to improve the performance under the optimization objective itself, but it has to be treated with nuance. In particular, the specific curriculum used to inject noise is as important as the noise distribution. Monotonic decay schedules lose their regularizing effect late in training, trapping the optimizer in local minima. Conversely, permanently applying heavy noise prevents the network from exploiting fine-grained features. We find the anti-curriculum loses the robustness benefit in the late training stages while standard curriculum fails to converge to accurate solutions on clean data (see 2). Our proposed interleaved noise schedule resolves this conflict by mixing exploration during noisy epochs with exploitation during the clean epochs. Noise acts as a regularizer, forcing the optimizer out of sharp, overfitting minima, while the clean epochs allow the model to refine its decision boundaries without the interference from data distortion. Interleaving these periods prevents the model from catastrophically forgetting either of these regimes.

6.2 Architecture-specific noise preferences↩︎

¿tbl:tab:cifar95results? and ¿tbl:tab:imagenet95results? suggest that the choice of Gaussian vs impulse noise appears to be dependent on the underlying model architecture. The ResNet architectures work best with the impulse noise, while the vision transformer performs best the with Gaussian noise. We find the interaction between noise type and architecture intuitive in the context of the inductive biases of each architecture. The convolutional layers of the ResNet models create a strong locality bias. Impulse noise randomly corrupts individual pixel intensities, preventing the network from over-fitting to localized pixel-level features. Vision transformers, on the other hand, treat images as sequences of patches and thus lack this form of locality bias, instead learning primarily global features. Because Gaussian noise applies perturbations uniformly across all of these patches, it tends to smooth out low-contrast global features, which forces the attention mechanism to emphasize higher frequency, local features. Recent work by [36] has demonstrated that explicitly enforcing locality structure in ViTs via architectural changes can significantly improve high frequency spatial representation, and we believe that applying interleaved Gaussian noise achieves precisely this training goal.

7 shows a gradient saliency map for the ResNet50 and an attention map for the ViT, before and after fine-tuning with interleaved noise injection. We observe that the ResNet50 models initially concentrate heavily on a small number of important pixels, in clean and noisy samples. After the fine-tuning with our impulse noise curriculum, the focus is more diffuse and appears to capture a much larger area of the main feature, especially for the noisy sample (2nd row). Conversely, the pretrained ViT shows very little focus on the main feature, especially for the noisy sample, where the attention is drawn to weak, spurious features of the snow corruption. After the interleaved noise injection with Gaussian noise, the focus becomes significantly sharper with the majority of the attention being directed at the telephone pole. Interestingly, the same noise injection yields better generalization to OOD samples (3rd row) as well, as evidenced by a shifted focus to the dog in the middle (the actual chow-chow) instead of the other dog on the left edge of the image.

6.3 Combination with alternative augmentations↩︎

Despite being powerful on its own, another advantage of noise injection (interleaved or otherwise) lies in its ability to be combined, at virtually no extra cost, with existing training and augmentation pipelines. Our results demonstrate that the interleaved noise curriculum acts to further improve the performance of established robustness-enhancing training techniques, such as Cutout, AugMix, SAM, and VIPAug.

As SAM operates entirely in the weight space, it is not surprising that our method (which operates in data space) provides complementary benefits. But we have clearly seen that our method also improves the performance of spatial and frequency-space data augmentations like Cutout, AugMix, and VIPAug. These augmentations directly alter the structure of the input samples to encourage invariance under the respective equivalence transformations. So, why does a simple data-space noise injection provide additional benefits beyond these more complex data-space augmentations?

We believe the improvements from interleaved noise injection primarily come from altering the optimization dynamics through the regularizer. Specifically, from improving the ability to escape sub-optimal local minima of the loss surface. By injecting noise into the data, the gradient signal is randomized, causing a random walk with drift. The drift component stems from the original gradient, which is not fully destroyed by the noise. If the optimizer is stuck in a sharp minimum, the random component will allow it to escape, so that it ends in larger, flatter minima that are harder to leave through a random walk. Finding such flat minima has been found to lead to better overall solutions as well as increased generalization and robustness [35]. The gradient-norm scaling is therefore a critical component of our method because it provides a balance between the random and the drift component.

7 Conclusions↩︎

In this study we introduce an interleaved noise injection scheme and demonstrate that periodically shifting between clean and noisy training epochs improves the test performance on clean data compared to the customary optimization that trains only with clean data. This benefit comes at essentially zero computational cost, does not require more training data or additional optimization epochs, and requires only minimal hyperparameter tuning.

We provide a theoretical analysis to demonstrate that noise injection is equivalent to regularization. Specifically, we show that impulse noise introduces penalties that are proportional to the Jacobian and the Hessian of the loss with respect to the data; for Gaussian noise only a previously known curvature term remains.

The benefits of these penalties can best be realized through an interleaved schedule compared to alternative curricula, which either transition only once from noisy to clean data or linearly change the noise level. We believe that repeatedly switching between noisy and clean data encourages exploration during the noisy epochs without entirely forgetting the features of the clean data. This combination allows optimizers to escape sub-optimal local minima, reducing overfitting and improving performance on in-domain, out-of-domain, and corrupted samples.

We also find that the type of noise is best chosen to combat the failure modes of different model architectures. While interleaved noise injection generally improves training performance, convolutional architectures tend to benefit more from impulse noise, whereas attention-based models benefit more from Gaussian noise.

Lastly, we show that our method can also be combined with other common forms of corruption and augmentation to further improve their performance with minimal added computational overhead.

Overall, interleaved noise injection provides a simple, yet effective tool to improve model performance and robustness against noise and real-world distribution shifts.

Software and Data↩︎

We use jax [37], pytorch [38], and tensorflow [39] for model architecture and data-loading. We use the publicly available CIFAR-100, CIFAR-100-C, ImageNet-1k, ImageNet-C, and ImageNet-R datasets [33], [34].

Impact Statement↩︎

This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.

8 Effect of gradient-norm stabilization↩︎

We find \(f= 0.4\) to provide the best results in our tests, i.e., we take steps a bit less than half the magnitude taken during the clean epochs, but we find any value between 0.2 and 0.8 to provide similar results. We present an ablation over the scale-factor \(f\) in 5.3. Setting \(f > 1\) means that during a noisy epoch, the optimizer will potentially move far away from desirable regions of the loss surface. Setting \(f\) too small simply means that very small updates are performed during the noisy epochs, hence the benefits of our method vanish.

In 8 we show the effect on the magnitude of the gradient norm, and hence the change to the effective learning rate via 6 . We show this for a ResNet50 fine-tuned with interleaved noise injection on ImageNet-1k with impulse noise in the left column and Gaussian noise in the right. Note that further details on all trials and architecture setups are in the following section. We color the noisy epochs in gray shading, and see that this corresponds to strong spikes in \(\| \nabla_{\theta} \mathcal{L}\|\) which can have strong destabilizing effects if not scaled down particularly in late training. We apply 6 to the base \(\eta\) to control the effective total update magnitude via using the new scaling \(\eta\) as shown in the colored dashed lines in the top rows of 8. We observe similar spiking when using a ViT for ImageNet-1k, and a WideResNet on CIFAR-100 which we show in 9 and 10 in the appendix.

9 Performance with corruption severity↩︎

In ¿tbl:tab:table4? we show the performance of the ResNet50 and ViT models with the suite of robustness regularizers with the range of 5 corruption severities. We see that, as expected, the error rate increases with severity.

In 2 we show the same results but by corruption type which is explained in detail in [28].

10 Noise period ablation results↩︎

3 shows an ablation over the noise injection period P run on the CIFAR-100 dataset with the WideResNet model. For the results, we chose P based on the strongest performance on the clean data, resulting in a period of 5 used for all trials (we note P=10 also performs strong for the Gaussian noise, but is worse for impulse noise).

11 Noise corruption visualization↩︎

We show a visualisation of different impulse noise levels on the CIFAR-100 dataset in 11.

References↩︎

---

1. Note the maximum \(\sigma_t\) we use is 0.65, so that, for large enough batch sizes, the off-diagonal terms vanish in expectation.↩︎

2. Barring the results shown in Figures 1, and 2 which use \(P=10\) and \(L=5\), but are for visualization purposes and not included in any tabulated results.↩︎