1 Introduction↩︎

The field of training neural network is dominated by gradient decent optimizers for a long time, which use first order method. Typical ones include SGD [1] and SGD with momentum (SGDM) [2], which are simple yet efficient algorithms and enjoy even better resulting convergence than many recently proposed optimizers. However, it suffers the disadvantage of low speed in initial stage and poor performance in sparse training datasets. This shortcoming can not be ignored since with the development of deep learning, the amount of data becomes much larger, and the model becomes much more complex. The time to train a network is also considered an important metric when evaluating an optimizer. To address this issue, optimizers with adaptive learning rate have been proposed which use nonuniform stepsizes to scale the gradient while training, and the usual implementation is scaling the gradient by square roots of some kind of combination of the squared values of historical gradients. By far the most used are Adam [3] and AdamW [4] due to their simplicity and high training speed in early stage. Despite their popularity, Adam and many variants like of it (such as RAdam [5]) is likely to achieve worse generalization ability than non-adaptive optimizers, observing that their performance quickly plateaus on validation sets.

To achieve a better tradeoff, researchers have made many improvements based on SGD and Adam family optimizers. One attempt is switching from adaptive learning rate methods to SGD, based on the idea of complementing each other’s advantages. However, a sudden change from one optimizer to another in a set epoch or step is not applicable because different algorithms make characteristic choices at saddle points and tend to converge to final points whose loss functions nearby have different geometry [6]. Therefore, many optimizers based on this idea seek for a smooth switch. The representative ones are AdaBound [7] and SWATS [8]. The second attempt is proposing new method to further accelerate SGDM, including introducing power exponent (pbSGD [9]), aggregated momentum (AggMo [10]) and warm restarts (SGDR [11]). The third attempt is modifying the process of optimizers with adaptive learning rate to achieve better local optimum, which is the most popular field in recent researches [12], [13]. Due to space constraints, please see more related work in Appendix 7.

We focus in this paper on the use of historical and future information about the optimization process of the model, both of which we argue are important for models to reach their optimal points. To this end, we introduce a bidirectional view, backward-looking and forward-looking. In the backward-looking view, EMA is an exponentially decreasing weighted moving average, which is used as a trend-type indicator in terms of the optimization process. And since the training uses a mini-batch strategy, each batch is likely to have deviations from the whole, so it may mislead the model to the local optimal point. Inspired by stock market indicators, DEMA [14] is an exponential average calculated on the traditional EMA and current input, which can effectively maintain the trend while reducing the impact caused by short-term bias. We thus replace the traditional exponential moving average (EMA) with double exponential moving average (DEMA). It is worth noting that our usage is not equivalent to the original DEMA, but rather a variant of it. In the forward-looking part, since we observe that a constant weight adopted by the original Lookahead optimizer [15] to control the scale of fast weights and slow weights in each synchronization period makes the early stage training slow and lossy, we propose a new dynamic strategy which adopts an asymptotic weight for improvement. By applying these two ideas, we propose Admeta optimizer with AdmetaR and AdmetaS implementations based on RAdam and SGDM respectively.

Extensive experiments have been conducted on computer vision (CV), natural language processing (NLP) and audio processing tasks, which demonstrate that our method achieves better convergence results compared to other recently proposed optimizers. Further analysis shows that AdmetaS achieves higher performance than SGDM and AdmetaR achieves better convergence results and maintains high speed in the initial stage compared to other adaptive learning rate methods. We further find that the DEMA and dynamic looking strategy can improve performance compared to EMA and constant strategy, respectively. In addition, we provide convergence proof of our proposed Admeta in convex and non-convex optimizations. The code is available at https://github.com/Chernyn/Admeta-Optimizer.

2 Admeta↩︎

2.1 Background↩︎

The role of the optimizer in model training is to minimize the loss on the training set and thus drive the learning of model parameters. Formally, consider a loss function \(f: \mathbb{R}^d \rightarrow \mathbb{R}\) that is bounded below greater than zero, where \(\mathbb{R}\) represents the field of real numbers, \(d\) denotes the dimension of the parameter and thus \(\mathbb{R}^d\) denotes d-dimensional Euclidean space. The optimization problem can be formulated as: \(\min_{\theta \in \mathcal{F}} f(\theta)\), where \(\theta\) indicates a parameter whose domain is \(\mathcal{F}\) and \(\mathcal{F} \subset \mathbb{R}^d\). If we define the optimum parameter of the above loss function as \(\theta^*\), then the optimization objective can be written as: \[{\theta^*=\mathop{\arg\min}\limits_{\theta \in \mathcal{F}}f(\theta).}\] Optimizers iteratively update parameters to make them close to the optimum as training step \(t\) increases, that is to make: \(\lim_{t \rightarrow \infty}\|\theta_t-\theta^*\|=0\).

The stochastic gradient algorithm SGD [1] optimizes \(f\) by iteratively updating parameter \(\theta_t\) at step \(t\) in the opposite direction of the stochastic gradient \(g(\theta_{t-1};\xi_t)\) where \(\xi_t\) is the input variables of the \(t\)-th mini-batch in training datasets. For the sake of clarity, we abbreviate \(g(\theta_{t-1};\xi_t)\) as \(g_t\) for the rest of the paper unless specified. SGD optimization aims to calculate the updated model parameters based on the previous model parameters, the current gradient and the learning rate. Define the learning rate as \(\alpha_t\), the update process is summarized as follows: \[\begin{align} \theta_t = \theta_{t-1} - \alpha_t g_t. \end{align}\] Original SGD tends to vibrate along the process due to the mini-batch strategy and not using of past gradients. What’s more, this disadvantage also results in its long-time plateaus in valleys and saddle points, thus slowing the speed. To smooth the oscillation and speed up convergence rate, momentum, also known as Polyak’s Heavy Ball [16], is introduced to modify SGD. Momentum at step \(t\) is often denoted as \(m_t\) and obtained by iterative calculation with a dampening coefficient \(\beta\). Thus, the update process of SGD with momentum (SGDM) [2] becomes as follows: \[\begin{align} & m_t=\beta m_{t-1} + (1 - \beta) g_t, \label{eq:sgdm}\\ & \theta_t = \theta_{t-1} - \alpha_t m_t, \end{align}\tag{1}\]

Although momentum works well, the uniform stepsize on every parameter is also another factor to limit the speed, especially in large datasets and sparse datasets. To further accelerate the update, adaptive learning rate optimizer is introduced which adopts an individual stepsize for each parameter based on their unique update process. Since a smoothing mechanism is employed in the calculation of stepsize, two dampening coefficients, \(\beta_1\) and \(\beta_2\), are introduced for balancing the current and historical information. Adam [3], a typical adaptive learning rate optimizer, is implemented as follows: \[\begin{align} & m_t= \beta_1 m_{t-1} + (1 - \beta_1) g_t, \tag{2}\\ & v_t = \beta_2 v_{t-1} + (1 - \beta_2) g_t^2,\\ & \theta_t = \theta_{t-1} - \alpha_t {m_t}/\sqrt{v_t}, \tag{3} \end{align}\] where \(m_t\) indicates the first momentum, corresponding to the momentum in SGDM; \(v_t\) indicates the second momentum.

To emphasize the functionality of \(v_t\), we call it adaptive item for the rest of the paper. Adam may sometimes converge to bad local optimum, partly due to its large variance in the early stage. To fix this issue, RAdam [5] introduces a further rectified item \(r_t\) and splits the update process into two sub-processes sequentially connected: \[\begin{align} &\rho_\infty = 2/(1-\beta_2)-1,\\ &\rho_t=\rho_\infty-2t\beta_2^t/(1-\beta_2^t),\\ &r_t \leftarrow \sqrt{\frac{(\rho_t-4)(\rho_t-2)\rho_\infty}{(\rho_\infty-4)(\rho_\infty-2)\rho_t}},\\ & \theta_t = \left\{ \begin{array}{ll} \theta_{t-1} - \alpha_t m_t, & \rho_t\leq 4 \\ \theta_{t-1} - \alpha_t r_t m_t/\sqrt{v_t}, & \rho_t > 4 \\ \end{array} \right.. \label{eq:radam95bias} \end{align}\tag{4}\]

2.2 Backward-looking↩︎

In fact, the calculation of momentum \(m_t\) in Eq. (1 ) and Eq. (2 ) is an exponential moving average (EMA) on gradient \(g_t\). EMA, also known as exponential weighted moving average, can be used to estimate the local mean value of variables, so that the update of variables is related to historical values over a period of time. Formally, EMA is expressed as: \[S_t = \beta S_{t-1} + (1 - \beta) p_t,\] where the variable \(S\) is denoted as \(S_t\) at time \(t\) and \(p_t\) are the newly assigned values. Particularly, \(S_t = p_t\) without using EMA. In Eq. (1 ), SGDM employs EMA to take a moving average of the past gradients. While in Eq. (2 ), Adam and RAdam further apply EMA on the square of past gradients to construct the adaptive item. In the EMA, the moving average of the variable \(S\) at time \(t\) is roughly equal to the average of the values \(p\) over the past \(1 / (1 - \beta)\) steps. This makes the moving average vary more at the beginning, so a bias correction is proposed and used in Adam (Eq. (3 )) and in RAdam (Eq. (4 )) when \(\rho > 4\).

EMA can be regarded as obtaining the average values of the variables over time. Compared with the direct assignment of values to variables, the change curve of the values obtained by moving average is smoother and less jittery, and the moving average does not fluctuate greatly when inputting outliers, which is very important for the optimization using sampled mini-batch. Although efficient, EMA is not necessarily the best strategy for using historical information when it comes to the backward-looking part. Although it can effectively suppress the vibration caused by mini-batch training by performing the moving average on \(g_t\), it also brings a lag time that affects the convergence speed and increases with the length of the moving average. What’s more, it can result in the overshoot problem [17], one possible reason is that EMA might make the wrong use of historical gradients in the final stage and thus have a “burden" to converge to optimum.

Double Exponential Moving Average (DEMA), first proposed by [14], is a faster moving average strategy and was invented to reduce the lag time of EMA. Thus, motivated by the advantage of DEMA, we developed a DEMA variant for the model optimization. It is worth noting DEMA is not simply taking a moving average of historical gradients twice, instead, it takes the moving average of the linear combination of the current gradient the moving average of past gradients. The form of our DEMA variant can be written as: \[\begin{align} \label{DEMA32form} \mathbf{DEMA}=\mathbf{EMA}^{out}(\mu \mathbf{EMA}^{in} + \kappa g_t), \end{align}\tag{5}\] where \(\mu\) and \(\kappa\) are coefficients that control the scale of current gradient and only depend on \(\beta\).

From the formula \(\mathbf{EMA}=\Sigma_{i=1}^n \beta^{n-i}g_i\), past gradients follow a fixed proportionality, that is, the ratio of gradient weight at one time to gradient weight at the previous time is \(\beta\).

Due to the use of minibatch training strategy, the input is randomly sampled. The effect of each minibatch towards optimization is varied. Therefore, applying a fixed proportionality to past gradients is not a reasonable approach since it does not take into account the changeable situation. The disadvantage of overshoot that EMA usually has may also be caused by the above reasons [17]. Thus, we deal with the relationship between the historical gradients and the current gradient more flexibly by further controlling the proportion of past gradients. Our design of coefficients in DEMA is also for this purpose. Based on Eq. (5 ), our actual implementation on algorithm is: \[\begin{align} &I_t=\lambda I_{t-1} + g_t, \\ &h_t=\kappa g_t + \mu I_t + \nu, \\ &m_t=\beta m_{t-1} + (1 - \beta) h_t, \end{align}\]

where \(I_t\) is the output of \(\mathbf{EMA}^{in}\) with a 0 initial value and \(m_t\) is the output of \(\mathbf{EMA}^{out}\) also initiated with 0. \(\lambda\) and \(\beta\) are dampening coefficients of inner EMA and outer EMA respectively, \(\nu\) is a bias item, which is set to a small amount that decreases exponentially to 0 and chosen as \(\lambda^t g_1\). The bias item does not affect the convergence proof, so for the sake of brevity, it is omitted for the rest of this paper and the details can be seen in the code. Please refer to Appendix 8 for more comparison and discussion between EMA and DEMA.

2.3 Forward-looking↩︎

Focusing on gradient history, that is, backward-looking, the optimizer is conducive to alleviating the vibration problem in the optimization process and preventing it from being misled by local noise information. However, since the optimization problem of the deep neural network is very complex, the optimizer can make the optimization process more robust by pre-exploration, so as to obtain better optimization results, which is called forward-looking.

Based on Reptile algorithm and advances in understanding the loss surface, [15] proposed Lookahead optimizer, which introduces two update processes and averages fast and slow weights periodically. The algorithm can be expressed as the cycle of the following process: \[\begin{align} &\boldsymbol{Pre-exploration}: \theta_t = \mathrm{\small Optim}(\theta_{t-1})\\ &\boldsymbol{Synchronization}: (\textit{every}\;k\;\textit{steps}) \\ &\qquad \phi_t = \phi_{t-k} + \eta (\theta_{t-1} - \phi_{t-k})\\ &\qquad \theta_t = \phi_t \end{align}\]

where \(\mathrm{\small Optim}(\cdot)\) denotes a chosen optimizer, \(k\) denotes the synchronization period, or in other words, the period of forward-looking, \(\phi_t\) denotes the slow weights , \(\theta_t\) denotes the fast weight updated with a chosen optimizer, and \(\eta\) is a constant coefficient controlling the proportion of slow weights and fast weights in each synchronization. Generally, the chosen optimizer can be arbitrary.

We can get an intuitive explanation of Lookahead optimizer from the pseudo code above: Guided by fast weight \(\theta_t\), the slow weight \(\phi_t\) updates by taking linear interpolation between itself and the fast weight. Every time the fast weight updates \(k\) steps, the slow weight updates 1 step. The update direction of slow weight can be regarded as \(\theta_t-\phi_t\) from the equation. Therefore, \(\eta\) can also be interpreted as the stepsize of slow weight in each synchronization. In order not to be confused with the stepsize of fast weight, we rename the stepsize of slow weight as \(stepsize_s\). The recommended value of \(\eta\) in [15] is 0.5 and 0.8.

In the original Lookahead optimizer implementation, the fast and slow optimization processes were synchronized according to a given period, and parameters are fused at a fixed ratio during synchronization. However, optimization is a continuous process. In different optimization stages, fast optimization steps have different guiding effects on parameters. We argue that using fixed \(stepsize_s\) in each synchronization is not an optimal strategy, and may even lead to negative effects. For this consideration, we turn the constant \(\eta\) into a \(\eta_t\) that changes over step monotonously and asymptotically. Generally, \(\eta_t\) is a function that starts from 1 and converges to a set value and depends only on the step \(t\). In this setting, the proportion of slow weights increases and this part gradually turns into the original Lookahead method. In other words, the slow weights in our method adopt a faster \(stepsize_s\) at the beginning, and it asymptotically slows down as processing. Specifically, we define two asymptotic functions for \(\eta_t\): \[\begin{align} \eta_t = 0.5*\left(1+\frac{1}{0.01\sqrt{t}+1}\right), \\ \eta_t = 0.8*\left(1+\frac{1}{0.1\sqrt{t}+3.8}\right), \end{align}\] thus we call this as dynamic asymptotic lookahead. The two functions are designed to turn \(\eta_t\) from 1 to 0.5 and 0.8 respectively. Notably, these asymptotic functions may not be the best. We just find that it works well and maybe future work can be done to investigate a more suitable one. For the sake of clarity, we will use the latter one in the rest of the paper and the results of experiments trained from scratch are based on this function unless specified.

To illustrate the advantages of our dynamic lookahead strategy over no lookahead and the original constant lookahead, we give an optimization example in Figure 1. In region , which is around the early stage, the direction of the update is relatively stable and a large \(stepsize_s\) is needed. \(\theta_1 \rightarrow \theta_4\) denotes the update of fast weights. A constant lookahead method will slow the update process in each synchronization period, as can be seen in \(\theta_1 \rightarrow \theta_2\). In our method, fast weights share more proportion in each synchronization period in early stage, thus updating faster, as can be seen in \(\theta_1 \rightarrow \theta_3\).

In region , which is around the final stage, the direction of the update is relatively oscillated, and a small \(stepsize_s\) is needed. Fast weights tend to overshoot the optimum, as can be seen in \(\theta_5\rightarrow \theta_8\) . Lookahead optimizer can achieve better convergence result than general algorithm as it averages the weights to make them more close to the optimum point, as can be seen in \(\theta_5\rightarrow \theta_6\). In our method, the proportion of fast weights has already been reduced asymptotically to a set value, thus can achieve similar efficacy as Lookahead optimizer as can be seen in \(\theta_5\rightarrow \theta_7\).

From these analyses, we demonstrate that our dynamic lookahead strategy method improves the robustness of training.

2.4 Implementations of AdmetaR and AdmetaS↩︎

Since optimizers of the Adam family and SGD family have their own advantages and disadvantages, and the bidirectional looking optimizer framework and improvement we propose do not have too many restrictions on the basic optimizer, we have implemented improved versions AdmetaR and AdmetaS based on RAdam and SGDM optimizer. The final algorithm forms are shown in Algorithm 2 and 3. Detailed proof of convergence and convergence rate for our AdmetaR and AdmetaS is put in Appendix 9 and 10.

Notations:

• \(\alpha_t\): learning rate at step \(t\)

• \(\lambda, \beta, \beta_1, \beta_2\): the momentum coefficients

• \(\epsilon\): a small value used to avoid a zero denominator

• \(k\): synchronization period

• \(\prod_{\mathcal{F},M}(y) = \mathrm{argmin}_{x \in \mathcal{F}} \vert \vert M^{1/2} (x-y) \vert \vert\)

• \(\mu=25-10\left(\lambda+\frac{1}{\lambda}\right), \kappa=\frac{10}{\lambda}-9\)

3 Experiments↩︎

In this section, we demonstrate the effectiveness of our optimizer by turning to an empirical exploration of different datasets and different models to compare some popular optimizers. Specifically, we conduct experiments on typical CV, NLP, and audio processing tasks. Influenced by the Transformer structure, models are becoming deeper and larger, and therefore training is becoming more difficult. The current paradigm of pre-training-fine-tuning is mainly used for large models. Therefore, we compare optimizers not only in the training-from-scratch setup, but also in the fine-tuning setup.

In this section, we compare our proposed optimizer with several typical optimizers, including classic SGD [1] and Adam [3], our base, SGDM [2]¹ and RAdam [5], the current state-of-the-art AdaBelief [12], and the optimizer combined of many modules, Ranger [19]. Since we should compare these optimizers under the same condition, the model used may be different from the original paper of them, which may lead to different convergence results compared to the results reported in the original paper. Please refer to Appendix 11 for more experimental details.

3.1 Image Classification↩︎

Consistent with general optimizer researches [12], we conduct experiments on two image classification tasks, CIFAR-10 and CIFAR-100 [20] in CV field, and the results are presented in Table 1. For model baselines, we choose the popular and leading performance ResNet-110 [21] and PyramidNet [22], respectively. From the experimental results, whether in CIFAR-10 or CIFAR-100 dataset, and based on the ResNet-110 or PyramidNet model, SGDM achieves better results than SGD, indicating that backward-looking improves the optimization effect. EMA with rectified item in RAdam performs better than EMA in Adam, suggesting that a better backward-looking process can lead to performance gains. Comparing SGDM and RAdam, we find that SGDM has a performance advantage, showing that though Adam uses an adaptive learning rate to improve the speed of convergence, it is lossy for performance.

Among optimizers with adaptive learning rate, AdaBelief achieves better results than Adam and RAdam in CIFAR-10 with PyramidNet and CIFAR-100 with ResNet-110 and PyramidNet. Ranger, which combines forward and backward looking, achieves better performance than the backward-looking-only RAdam in CIFAR-10 and CIFAR-100 with PyramidNet. Our AdmetaR achieves consistent improvement over the optimizer baseline RAdam, which also confirms the gain of bidirectional looking for optimization. And AdmetaR has better results than Ranger, indicating that our bidirectional looking is better than Ranger’s simple combination of multiple optimization features. Our AdmetaS also performs better than SGDM, further demonstrating the adaptability of our approach, which not only performs well in Adam family, but also works in SGD family.

Following the previous practice [5], we visualize the optimization process of the ResNet-110 model with Adam, RAdam, SGDM, and our AdmetaS, AdmetaR optimizers on the CIFAR-10 and CIFAR-100 datasets in Figure 2. As can be seen from the training loss figure, the above optimizers can successfully train the model to converge to a stable state, but AdmetaS obtains the lowest training loss on CIFAR-10, while AdaBelief obtains the training loss on CIFAR-100. In terms of performance on the test set, AdmetaS has obtained the best convergence result, which shows that the lower the loss of the training set may not necessarily lead to the better performance on the model. In addition, from the accuracy of the test set, the convergence speed of the SGD family including SGDM and AdmetaS is generally slower than that of the Adam family (Adam, RAdam, Ranger, AdaBelief and AdmetaR), but the final convergence result of the SGD family is better than the Adam family. However, our AdmetaR achieves more comparable performance to the SGD family, while maintaining the advantage of the fast convergence of the Adam family. AdmetaR has the highest results on the test set in the early stage of optimization (\(< 80\) epoch), which demonstrates that bidirectional looking improves both accuracy and speed, making AdmetaR an efficient and effective optimizer implementation.

Compared to ResNet-110, PyramidNet has a more complicated structure and can achieve better results in these tasks. In cases where the model is strong enough, the selection of optimizer will not be the main factor for the final performance. As shown in Table 1, compared to Adam, RAdam and AdaBelief achieve just a bit of improvement on CIFAR-10 task and even achieve worse results on CIFAR-100 task, which also verifies our above claims. It also shows that some recently proposed methods are not always suitable when the structure is complex enough.

3.2 Natural Language Understanding↩︎

As a general AI component, the general capability for various tasks and various models is a basic requirement for optimizers. We evaluate the adaptability of our Admeta optimizer on the finetune training scenario with current popular pre-trained language models. Specifically, we conduct experiments based on the pre-trained language model BERT [23] on three natural language understanding tasks, GLUE benchmark [24], machine reading comprehension (SQuAD v1.1 and v2.0 [25]) and named entity recognition (NER-CoNLL03 [26]). We report results for two model sizes, BERT\(_\text{base}\) and BERT\(_\text{large}\) to explore whether model size has an effect on the optimizer.

In Table 2, we report the results on the development set of 8 datasets of the GLUE benchmark, where Acc, MCC, SCC are abbreviations of accuracy, Matthews Correlation and Spearman Correlation Coefficient, respectively. First, under the BERT-base model, compared with the basic optimizer RAdam, AdmetaR achieves consistent improvement. The most significant improvement is obtained on RTE and CoLA, which indicates that our AdmetaR optimizer exhibits greater stability for low-resource optimization. On the other seven datasets, some of them are slightly improved. This is because most of the parameters of the model in the pre-training-fine-tuning paradigm have converged to a certain extent in the pre-training stage, so the further advantage of the optimizer in finetune is not apparent. And when the model is switched to a larger BERT-large, most tasks receive performance gains, except for CoLA and RTE using AdamW optimizer. Due to the further increase in model parameters, the low-resource dataset is not enough to fine-tune the large model, it will even reduce the model performance. But RAdam with rectified item, Ranger with bidirectional looking, and our AdmetaR handle the low-resource challenge well, continue to improve performance, and take advantage of large models. Our AdmetaR achieves the best results on these two low-resource datasets, demonstrating the effectiveness of our bidirectional looking approach.

In Table 3, we further report the results of machine reading comprehension and named entity recognition. AdmetaR achieved improvements at both model sizes in SQuAD v1.1 dataset, while similar improvements were achieved in SQuAD v2.0 with more complex models, illustrating that our optimizer is model-independent. Named entity recognition has reached a very accurate level with the help of pre-trained language models, and our AdmetaR optimizer also brings performance improvements over such a strong baseline, showing that optimization is also a bottleneck that restricts further performance improvement in addition to model structure and data.

3.3 Audio Classification↩︎

Like images and natural language, speech is one of the mainstream fields of deep learning research. In speech processing, there are also a large number of pre-trained large models, such as Wav2vec [27]. To highlight the input-independent nature of the optimizer, we also conduct experiments on two typical tasks of audio classification, keyword spotting (SUPERB) [28] and language identification (Common Language) [29]. We employ Wav2vec 2.0\(_\text{base}\) as the baseline model and report the results of each optimizer in Table 4. In addition, we also list the training time of each optimizer to evaluate the impact of the bidirectional looking mechanism on the optimizer time overhead².

AdmetaR shows better classification accuracy than AdamW, RAdam, Ranger and AdaBelief, which is consistent with the experimental conclusions in the image and natural language tasks. Consistent results across image, natural language, and speech modalities verify the task-independence of our optimizer. Comparing the training time of AdmetaR with AdamW, RAdam, Ranger, and AdaBelief, our AdmetaR has different degrees of increase due to the additional computation and storage in the optimization process. Ranger and our AdmetaR increased the time most, but it can still be regarded as slight compared to the overall training time. Therefore, it can be concluded that the bidirectional looking mechanism adopted by Admeta optimizer will bring additional computational overhead and increase the training time, but compared with the overall training cost, it is very small. Admeta achieves better performance without increasing model parameters and training data, and does not have any impact on the inference time of the model, which achieves a better tradeoff.

4 Ablation Study↩︎

We perform an ablation study on various designs of Admeta in bidirectional looking in this section. -DEMA means removing the DEMA mechanism in backward-looking and using the original EMA. -LB means complete removal of backward-looking, -LF means complete removal of forward-looking. -LB-LF means to remove bidirectional looking at the same time. w/ constant LF means use the original Lookahead mechanism in the forward-looking. The results are evaluated using the ResNet-110 model on the test set of CIFAR-10. According to the results shown in Table 5, it can be found that the improvement of SGDM compared with SGD initially shows the advantage of backward-looking. And compared with Adam, RAdam reveals that the EMA with the rectified item in backward-looking is more suitable for the training of the model than the original EMA. Our Admeta (including AdmetaR and AdmetaS) achieved the best results. After removing DEMA and replacing dynamic lookahead with constant lookahead, respectively, the performance drops, indicating that both DEMA and dynamic asymptotic lookahead play an important role in stable optimization. After further removing the backward-looking, the forward-looking, and the bidirectional looking, the results drop further, validating our argument that bidirectional looking is beneficial for optimization. Another observation is that backward-looking and DEMA make a more significant contribution to the performance of SGDM compared to RAdam. This may show that our methods have a better complementarity for SGD-family optimizers.

5 Conclusion↩︎

In this paper, we introduce a bidirectional looking optimizer framework, exploring the use of historical and future information for optimization. For backward-looking, we introduce a DEMA scheme to replace the traditional EMA strategy, while for forward-looking, we propose a dynamic asymptotic lookahead strategy to replace the constant lookahead scheme. In this way, we propose the Admeta optimizer, and provide two implement versions, AdmetaR and AdmetaS, which are based on adaptive and non-adaptive momentum optimizers, RAdam and SGDM respectively. We verify the benefits of Admeta with intuitive examinations and various experiments, showing the effectiveness of our proposed optimizer. Please refer to Appendix 12 for future work discussion.

6 Limitation↩︎

Although improving the performance on different tasks, our method introduces additional computational complexity and requires more hyperparameters than some existing approaches. However, the selection range of hyperparameters can be preliminarily determined through the visual tool we proposed (Figure 5), which can slightly reduce the workload of tuning parameters.

langley00?

Appendix↩︎

7 Related Work↩︎

As an important part of machine learning and deep learning, optimizers have received much attention in recent years. The optimizer plays a prominent role in the convergence speed and the convergence effect of the model. To seek good properties like fast convergence, good generalization and robustness, many algorithms have been put forward recently, and they can be divided into four families according to their characteristics and motivation.

7.0.0.1 SGD Family

In this family, the optimizers adopt the method of update like \[\theta_t = \theta_{t-1} - \alpha_t m_t,\] where \(\theta_t\) denotes the parameter to be optimized at iteration step t and \(m_t\) refers to some combination of past gradients (such as EMA), which can be represented as \(f_1(g_1,g_2,...,g_t)\). Original SGD [1] directly minus the product of global learning rate and the gradient at each step. Despite of its simplicity, it is still widely used in many datasets. However, SGD is blamed for its low convergence rate and high fluctuation, thus many methods have been proposed to accelerate the speed and smooth the update process. One efficient optimizer to tackle this issue is SGDM [2], which uses a exponential moving average (EMA, also known as momentum) to replace the gradient with an exponential weight decay of past gradients. SGDM-Nesterov [18] is a variant of SGDM which modifies the momentum by computing gradient based on the approximation of the next position and thus changing the descent direction. Experiments have shown that Nesterov momentum tends to achieve a higher speed and performance.

7.0.0.2 Adam Family

The Adam family optimizers usually update parameters by \[\theta_t = \theta_{t-1} - \alpha_t m_t / \sqrt{v_t},\] where \(v_t\) is the adaptive item and can be represented as \(f_2(g_1^2,g_2^2,...,g_t^2)\). Compared to SGD family, instead of using a uniform learning rate, this kind of optimizer computes an individual learning rate for each parameter due to the effect of the denominator \(\sqrt{v_t}\) in the equation. \(v_t\) is usually a dimension-reduction approximation to the matrix which contains the information of second order curvature, such as Fisher matrix [30].

Adadelta [31], Adagrad [32] and RMSprop [33] are early optimizers in this family. A stand out generation is Adam [3] which combines the RMSprop with Adagrad. It has been widely used in a wide range of datasets and works well even with sparse gradients. However, there are problems with Adam with respect to convergence and generalization, thus many methods have been proposed to make improvements

Based on the large variance in the early stage that may lead to a bad optimum, heuristic warmup [34], [35] and RAdam [5] are proposed, of which the former starts with a small initial learning rate and the latter introduces a rectified item. To fix the convergence error, [36] proposed AMSGrad which requires the non-decreasing property of the second momentum. In fact, this method can be interpolated into other Adam family algorithms to guarantee the convergence in convex situations. Considering the curvature of the loss function, AdaBelief [12] and AdaMomentum [37] are proposed. More recently, there are still numerous studies devoted to improving Adam, such as AdaX [13] and AdaFamily [38]. However, we notice that most researchers put a solid emphasis on modifying the second momentum term, i.e., the adaptive item and ignore the possibility to make a relative overall change to the algorithms.

7.0.0.3 Stochastic Second-Order Family

In the stochastic second-order optimizers, parameters are updated using second-order information related to Hessian matrix. The update process is typically written as \[\theta_t=\theta_{t-1}-\alpha_t H^{-1}m_t,\] where \(H\) is the Hessian matrix or approximation matrix to it. Ideally, they can achieve better results than the first order optimizers (like Adam family and SGD family), but their practicality is limited due to the large computational cost of the second order information, like the Fisher / Hessian matrix. Some methods have been proposed using low-rank decomposition and approximating to hessian diagonal to reduce the cost, like Apollo [39], AdHessian [40] and Shampoo [41].

7.0.0.4 Other Optimizers

There are some algorithms that are not convenient to be categorized into the above families and we list some examples here. Motivated by PID controller, SGD-PID [17] takes an analogy between the gradient and the input error in an automatic control system. Analysis shows that it can reduce the overshoot problem in SGD and SGD variants. Furthermore, [42] applied PID to Adam and proposed the AdaPID optimizer.

Lookahead [15] optimizer updates two sets of weight wherein "fast weights" function as a guide to search for the direction and "slow weights" follow the guide to achieve better optimization. Ranger [19] optimizer further combines RAdam and Lookahead to get a compound algorithm and shows a better convergence performance.

7.0.0.5 Discussion

To show the advantage of bidirectional looking, we propose Admeta optimizer. Specifically, it is based on the idea of considering backward-looking and forward-looking, wherein DEMA plays a important role in the former aspect and dynamic asymptotic forward-looking strategy serves for the latter aspect.

In practical use, we provide two versions, AdmetaS and AdmetaR, using the framework of Admeta and based on SGDM and RAdam respectively. Specifically, AdmetaS replaces the traditionally used EMA in backward-looking part of SGDM with DEMA and adds the forward-looking part which is derived from Lookahead optimizer. AdmetaR is based on RAdam in the same way. The second order family is also introduced above because the framework of Admeta can also be applied in this family, and it is remained as the future work.

8 EMA vs. DEMA↩︎

To corroborate our analysis of EMA and DEMA, we compared the optimization process of EMA and DEMA on the SGD optimizer according to the practice of [43]. Using the same learning rate \(\alpha\) and starting from the same starting point, the convergence process is shown in Figure 5. The decent surface in the figure is the convex quadratic, which is a useful model despite its simplicity, for it comprises an important structure, the “valleys", which is often studied as an example in momentum-based optimizers. As demonstrated in Figure 5, on the one hand, DEMA achieves faster speed than EMA, which can be easily seen by comparing the distance to the optimal point at the same time; on the other hand, DEMA achieves better convergence results than EMA as can be seen in the distance between the point of convergence and optimum.

9 Proof of Convergence↩︎

In this section, following [44], [45] and [36], we provide detailed proofs of convergence for AdmetaR and AdmetaS optimizers in convex and non-convex situations.

9.1 Convergence Analysis in Convex and Non-convex Optimization↩︎

Optimization problem For deterministic problems, the problem to be optimized is \(\mathrm{min}_{\theta\in \mathcal{F}} f(\theta)\), where \(f\) denotes the loss function. For online optimization, the problem is \(\mathrm{min}_{\theta\in \mathcal{F}} \sum_{t=1}^T f_t(\theta)\), where \(f_t\) is the loss function of the model with the given parameters at the \(t\)-th step.

The criteria for judging convergence in convex and non-convex cases are different. For convex optimization, following [36], the goal is to ensure \(R(T)=o(T)\), i.e., \(\lim_{T \rightarrow \infty}R(T)/T=0\). For non-convex optimization, following [44], the goal is to ensure \(\operatorname*{min}_{t \in [T]} \mathbb{E} \Big \vert \Big \vert \nabla f(\theta_t ) \Big \vert \Big \vert^2=o(T)\).

Before formally proving the theorems, here list some remarks and preparations.
Remark 1. For brevity, we omit the rectified item of AdmetaR in the proof. However, it does not influence the proof since it can be integrated into the learning rate.

Remark 2. Following [7], the bias correction \(1/(1-\beta_1^t)\) of the first momentum \(m_t\) is omitted in the convergence of AdmetaR. Since \(1/(1-\beta_1^t)\) is bounded above 1 and below 10, the order of the terms used is not affected, thus hardly affecting the proof.

Remark 3. The forward-looking part is not considered in the proof. On the one hand, explanations and proofs of constant Lookahead have been given in [15] and [46], which can be imitated by our dynamic method. On the other hand, forward-looking part is exactly the interpolation of fast weights and slow weights at each synchronization period, and the fast weights are updated by the given optimizer. Therefore, the convergence proof is equivalent to only proving the convergence of fast weights.

Remark 4. The condition in the theorem that \(v_t \leq v_{t+1}, \forall t \in [T]\) does not necessarily hold in the practice of our method. Dropping this condition may lead to a non-convergence result, which can be seen in [36]. However, the counterexample given by this article is a very artificial design, which may not represent the case in practice. Many optimizers that do not meet this condition can eventually converge in the training process and further exploration may show that this condition is not necessary.

Remark 5. If we fix the number of steps, the training is a finite process over finite data, thus the iteration is bounded.

In this way, we can redefine \(G_{\infty}\) by enlarging it and the bounded stochastic gradient assumption in the theorem is equivalent to assuming \(\|g_t\|_{\infty}, \|I_t\|_{\infty}, \|h_t\|_{\infty}, \| m_t\|_{\infty} \leq G_{\infty}\).

Remark 6. As for non-convex optimization, in the same way, the bounded noisy gradient assumption is equivalent to \(\|g_t\|, \|I_t\|, \|h_t\|, \| m_t\| \leq H\) where \(H\) is a constant independent of \(T\). This remark will be used in several places in the following proof.

This is a common mathematical inequality, so the proof is omitted here.

9.2 Convergence Analysis of AdmetaR for Convex Optimization↩︎

9.3 Convergence Analysis of AdmetaR for Non-convex Optimization↩︎

9.4 Convergence Analysis of AdmetaS for Convex Optimization↩︎

9.5 Convergence Analysis of AdmetaS for Non-convex Optimization↩︎

9.6 Convergence Analysis of Forward-looking↩︎

In this section, based on [46], we further analysis forward-looking part to complete the convergence proof of Admeta optimizer.

According to [15], Lookahead is an algorithm that can be combined with any standard optimization method. The same is true for dynamic lookahead method in forward-looking part. What’s more, optimizers with forward-looking is essentially processing with two loops as discussed in the main text. The fast weight is updated by optimizers, while the slow weight is updated by interpolating with fast weight every given period. In other words, the slow weight is updated passively. Therefore, though the slow weight is relevant to optimizers, it is almost irrelevant to the selection of optimizers. For this reason, we only prove the convergence of forward-looking of AdmetaS, which can be easily extended to the AdmetaR.

Remarks:(some preliminaries)
Based on the design of the asymptotic dynamic weight \(\eta_t\) of the forward-looking part, it can be concluded that when it runs for a long time, \(\eta_t\) is highly close to the set point, at which we can safely assume that \(\eta_t\) is a constant and thus we denote it as \(\eta\). In this way, the analysis of a dynamic lookahead is the same as the case of static lookahead.

According to algorithm of Admeta, the slow weight \(\phi_t\) updates every k steps. We can assume that the slow weight is trained in sync with fast weight. For this purpose, all we should do is to stipulate \(\phi_{\tau k+l}=\phi_{\tau k}\), where \(k\) denotes the synchronization period, \(\tau \in \mathbb{N}^*\) and \(0\leq l <k\).

Define \(y_t=\eta \theta_t +(1-\eta)\theta_t\), then according to the update of \(\theta_t\) and \(\phi_t\), we have \[\begin{align} y_{t+1}=y_t-\eta \alpha_t m_t \end{align}\] and on each period of synchronization, we have \[\begin{align} &y_{\tau k}-\theta_{\tau k}=(1-\eta)(\phi_{\tau k}-\theta_{\tau k})=0\\ &y_{\tau k}-\phi_{\tau k}=\eta(\theta_{\tau k}-\phi_{\tau k})=0 \end{align}\]

10 Analysis of Convergence Rate↩︎

For convex situation, we adopt the regret function to estimate the convergence rate. And for non-convex situation, we adopt the minimum of the expectation of the squared gradient to estimate the convergence, which are corresponding to the proof of convergence since the process of the convergence proof is actually the process of finding the convergence rate.

From Table 6, we notice that the convergence rates for all optimizers for convex case are of magnitude of \(\mathcal{O}(1/\sqrt{T})\) and for non-convex are of \(\mathcal{O}(\text{log} T/\sqrt{T})\) , which means in essence, algorithms based on gradient decent follow a similar rate constraint. However, the convergence speed of different optimizers may attribute to many other factors, such as on the implementation. Therefore additional statistical experiments are needed for analysis, as we did in Table 4.

11 Experimental Details↩︎

11.1 Hyperparameter Tuning↩︎

For Admeta optimizer, we first determined a rough value range for learning rate and lambda with the toy model according to the visualization as in Figure 8. While for other baseline optimizers, we refer to the recommended/default hyperparameter settings in the original paper. In this way, we get the rouge range of the hyperparameter in optimizers. Then, we search the hyperparameters in the adjacent interval, which is listed in the following three subsections.

11.2 Image Classification↩︎

We conduct image classification experiments on CIFAR-10 and CIFAR-100 datasets, which are trained on a single NVIDIA RTX-3090 GPU. Typical architectures like ResNet-110 and PyramidNet are employed as the baseline models. In the ResNet-110 architecture, there are 54 stacked identical \(3 \times 3\) convolutional layers with 54 two-layer Residual Units [21].

While in the PyramidNet architecture, there are 110 layers with a widening factor of 48 [22]. We set the training batch size to 128 and the validation batch size to 256. Both models are trained with 160 epochs. Milestone schedule is adopted as the learning rate decay strategy, with learning rate decaying at the end of \(80\)-th and \(120\)-th epochs by 0.1.

We report the hyperparameters tuning for our proposed Admeta and other optimizers for reproduction of our experiments. For all optimizers, the weight decay is fixed as \(1e-4\). The searching scheme of hyperparameter settings for each optimizer is concluded as follows:

• For SGD and SGDM, the momentum is fixed as \(0.9\), and the best-performing learning rate is searched from \(\{0.01, 0. 05, 0.1\}\) and recommended values in original paper. For our AdmetaS, the \(\lambda\) is set to fixed \(0.9\) and we search the best-performing \(\beta\) from \(\{0.1, 0.2, 0.3, 0.4\}\) and learning rate from \(\{0.01, 0.05, 0.1\}\).

• For all adaptive learning rate optimizers, hyperparameters \(\beta_1\), \(\beta_2\) and \(\epsilon\) are set to \(\beta_1=0.9\), \(\beta_2=0.999\) and \(\epsilon=\text{1e-9}\) respectively. For Adam, RAdam and AdaBelief optimizer, the learning rate is searched from \(\{0.1, 0.01, 0.001\}\). For Ranger, \(\eta\) and \(k\) are set to \(\eta=0.5\) and \(k=6\) according to [19]. The learning rate is searched from \(\{0.1, 0.01, 0.001\}\). And for our AdmetaR, the setting of \(k\) is the same as Ranger, and we search \(\lambda\) from \(\{0.05, 0.1, 0.2, 0.3, 0.4\}\) and learning rate from \(\{0.1, 0.05, 0.01\}\).

The resulting hyperparameters reported in the paper are shown in Table 7, where \(LR\) is the abbreviation of learning rate.

11.3 Natural Language Understanding↩︎

In the NLU experiments, we employ a pre-trained language model BERT [23], [50] as our backbone. There are two model sizes for BERT: BERT\(_\text{base}\) and BERT\(_\text{large}\), where the base model size has 12 Transformer layers with 768 hidden size, 12 self-attention heads and 110M model parameters and the large model size has 24 Transformer layers with 1024 hidden size, 16 self-attention heads and 340M parameters [51].

In natural language understanding, we perform experiments on three modeling types of tasks: text classification, machine reading comprehension and token classification. The text classification uses the GLUE benchmark as the evaluation data set, the machine reading comprehension uses SQuAD v1.1 and v2.0, and the token classification uses the NER-CoNLL03 named entity recognition data set [52].

We train the eight tasks in GLUE benchmark for 3 epochs on a single NVIDIA RTX-3090 GPU, except for MRPC, which is trained for 5 epochs due to its relatively small data size [53]. The maximum sequence length is set to 128 and the training batch size is set to 32. SQuAD v1.1 and SQuAD v2.0 are trained for 2 epochs with two GPUs. The maximum sequence length is set to 384 and the training batch size per device is set to 12. And NER-CoNLL03 is trained for 3 epochs on a single GPU. The training batch size per device is set to 8.

Because of the pre-training-fine-tuning paradigm, we only employ the adaptive learning rate optimizer. We set \(\beta_1\), \(\beta_2\), \(\epsilon\) and weight decay of these optimizers to 0.9, 0.999, and 0.0 respectively. \(\eta\) and \(k\) are set to 0.5 and 6 in the Ranger optimizer and Admeta uses the same value of \(k\) as Ranger. We perform hyperparameter tuning on the learning rate and \(\lambda\), and the resulting hyperparameters reported in the paper are shown in Table 8 and 9.

11.4 Audio Classification↩︎

Based on Wav2vec [27], the Wav2vec 2.0 [54] is a framework for self-supervised learning of speech representations which is composed of 3 modules: feature encoder, contextualized representations and quantization module. In the feature encoder, there are 7 blocks with temporal convolutions that have 512 channels for each block and the relative positional embeddings of the convolutional layer modeling has kernel size of 128 and 16 groups.

Among the configurations of Wav2vec 2.0, we choose Wav2vec 2.0\(_\text{base}\) model, which has 12 Transformer blocks, 95M parameters and 8 attention heads, with model dimension of 768 and inner dimension (FFN) of 3072. We finetune Wav2vec 2.0\(_\text{base}\) for keyword spotting and language identification on SUPERB dataset [28] and Common Language [29] dataset respectively. The dataset size of keyword spotting is smaller than that of language identification, so we use a single NVIDIA RTX-3090 GPU for training on the SUPERB dataset, and use four GPUs for parallel training on the Common Language dataset. The keyword spotting model is trained for 5 epochs with training batch size 32 and language identification model for 10 epochs with training batch size 8 per device.

Due to the same reason as in NLU experiments, i.e. the pre-training-fine-tuning paradigm, we only employ adaptive learning rate optimizers here. For all optimizers chosen, we fix \(\beta_1=0.9, \beta_2=0.999, \epsilon=1e-8\) and set weight decay to 0.0. The learning rate is searched from {5e-5, 8e-5, 1e-4, 3e-4, 5e-4, 8e-4}, and for AdmetaR, \(\lambda\) is searched from {0.05, 0.1 0.2}. The resulting hyperparameters reported in the paper are shown in Table 10.

12 Future Work↩︎

In the future work, for backward-looking part, though DEMA provides a more flexible way to deal with past gradients, it is still unable to intelligently judge the value of certain historical gradient information, such as discarding some obviously unreasonable gradients caused by noise. A better optimizer may have the ability to forget these wrong information and take advantage of what works, just working like human brains. For forward-looking part, our method takes the constant coefficient into a dynamic one. It is kind of like milestone scheme of learning rate decay strategies to some extent. However, several experiments [39], [55] have shown that cosine strategy [11] works better. Therefore, we will follow the cosine scheme and propose a new forward-looking strategy that may work even better.

13 Performance of SGDM and AdmetaS on Finetune Setting↩︎

In this section, we test the performance of SGDM and AdmetaS on fintune setting and the results are shown in Table  11. For keyword spotting (SUPERB) [28] task, we train the models for 5 epochs and use Wav2vec\(_\text{base}\) [27] as the baseline model. And for CIFAR-10 [20] task, we train the model for 40 epochs from the checkpoint already trained with Adam using learning rate of 0.001 for 160 epochs. The baseline model of CIFAR-10 is ResNet-110 [21] with deep CNN architecture. We report the results of best hyperparameter settings for SGD and AdmetaS via grid searching.

From Table 11, we notice that in SUPERB task, compared to adaptive learning rate methods, SGDM achieves worse results in SUPERB task, but not by much, which shows that SGDM can also be used in finetune setting. While AdmetaS can achieve better result than any other learning rate methods used in our experiment, demonstrating the advantage of our approach. This phenomenon contradicts the mainstream view that SGD family is not suitable for finetune task. While for CIFAR-10 task, SGDM and AdmetaS both improve the performance compared to the start point. However, they are both obviously worse than the performance of training the task from scratch using SGDM and AdmetaS respectively, which shows that pre-training is a very strong approach that makes the model achieve a good state.

The reason why AdmetaS performs better than SGDM in finetune setting may lie in two aspects. On the one hand, DEMA scheme in the backward-looking part reduces the overshoot problem that may do harm especially near convergence. On the other hand, the forward-looking part improves the stability of the training process.

14 Influence of Different Learning Rates in SGD Family Optimizers↩︎

Since the learning rate of 0.5 for SGDM is a recommended value in [22] but not in [21], to alleviate the influence of different learning rates, we also try the performance of SGDM with a learning rate of 0.5 in the ResNet-110 network and the results are listed in Table 12.

The results show that choosing a large learning rate for SGDM may increase the performance, as shown that when setting the learning rate to 0.5 instead of 0.1, the recommended value in ResNet-110. However, this is not always true since the performance on CIFAR-10 when using the learning rate of 0.5 does not get prompted.

References↩︎