Deep Learning Hyperparameter Tuning: From Hand-Crafted Alchemy to Automated Search

Tuning is both a black art and a crucial step in deep learning training—so much so that the training process is jokingly called “alchemy,” and quite a few algorithm engineers call themselves “tuning monkeys.” This post first covers some manual tuning techniques, then moves on to methods and tools for automated tuning.

Manual Tuning

Take ImageNet classification as an example. Most of the hyperparameters you tune during training are optimization-related: batch size (BS hereafter), the choice of optimizer, and the optimizer-related learning rate (LR hereafter), learning-rate decay schedule, momentum, weight-decay coefficient, and so on.

Batch Size and Learning Rate

There are two angles to choosing BS. From the hardware side: too large a BS easily causes out-of-memory (OOM), while too small a BS fails to fully utilize compute and slows down training. From the optimization side: a larger BS gives a better estimate of the gradient over the whole dataset, making optimization more stable and less prone to divergence, so a larger BS is usually paired with a larger LR to speed up training. A common rule of thumb is to scale BS and LR proportionally: for instance, if BS=256 corresponds to LR=0.1, then with 4 GPUs at BS=256 each (total BS=1024), the LR should be scaled up to 0.4. Additionally, BS is usually set to $2^n$ to improve parallel efficiency.

Optimizers

Optimizing a neural network is, at its core, estimating the weights $w$ to minimize an objective function $Q(w)$ . With $n$ samples:

Q(w)=\frac{1}{n}\sum_{i=1}^{n} Q_i(w)

Standard gradient descent updates over all samples ( $\eta$ is the learning rate):

w := w-\eta\nabla Q(w)

But in deep learning $n$ is very large, so it is impossible to traverse the full set at every step. Instead, we use a mini-batch (whose size is the BS, denoted $m$ ) to estimate the gradient—this is stochastic gradient descent (SGD) with mini-batches:

w := w-\frac{\eta}{m}\sum_{i=1}^{m}\nabla Q_i(w)

Below, $\nabla\hat{Q}(w)$ denotes this mini-batch average gradient. In practice, momentum is often introduced, carrying a fraction $\alpha$ of the previous step’s update into the current one:

\Delta w := \alpha\,\Delta w-\eta\,\nabla\hat{Q}(w),\qquad w := w+\Delta w

Nesterov momentum is similar, the difference being that it first takes a step along the momentum direction and then computes the gradient at that position (in PyTorch the default is $\alpha=0$ , i.e. no momentum; I myself often set $\alpha=0.9$ ):

\Delta w := \alpha\,\Delta w-\eta\,\nabla\hat{Q}(w+\alpha\,\Delta w),\qquad w := w+\Delta w

SGD’s learning rate stays constant throughout. The methods below, by contrast, can adapt the learning rate. Adagrad scales the learning rate by the accumulated sum of squared gradients, with no extra hyperparameter besides $\eta$ :

v := v+\nabla\hat{Q}(w)^2,\qquad w := w-\frac{\eta}{\sqrt{v}}\,\nabla\hat{Q}(w)

RMSprop uses a decay factor $\gamma$ to fade the historical accumulation (PyTorch default $\gamma=0.99$ ):

v := \gamma v+(1-\gamma)\nabla\hat{Q}(w)^2,\qquad w := w-\frac{\eta}{\sqrt{v}}\,\nabla\hat{Q}(w)

Adam is essentially RMSprop with the gradient’s momentum added on top. Besides $\eta$ , it has two more hyperparameters, $\beta_1,\beta_2$ (PyTorch defaults $\beta_1=0.9,\beta_2=0.999$ ):

\begin{aligned} m &:= \beta_1 m+(1-\beta_1)\nabla\hat{Q}(w), \quad v := \beta_2 v+(1-\beta_2)\nabla\hat{Q}(w)^2\\ \hat{m} &:= \frac{m}{1-\beta_1^{t}}, \quad \hat{v} := \frac{v}{1-\beta_2^{t}}, \quad w := w-\eta\,\frac{\hat{m}}{\sqrt{\hat{v}}+\epsilon} \end{aligned}

Learning-Rate Decay and Weight Decay

Every optimizer requires tuning at least one learning rate $\eta$ . The LR is probably the most important hyperparameter in training; it is tied to the BS, the optimizer, the data, and the model, with no one-size-fits-all answer. The usual guideline is to use the largest learning rate that does not cause divergence. The LR typically first warms up (gradually increasing) and then decays, with decay schedules such as step, exponential, cosine, and so on:

Different learning-rate decay schedules — Common learning-rate decay schedules: step, exponential, cosine.

Besides preset schedules, you can also use automatic learning-rate control: watch a training metric (e.g. training loss) and preset some rules—for instance, decay the LR proportionally when the loss stops dropping for a long time, decay it when the loss suddenly increases, or scale it up when the loss is dropping very slowly. Such rule-based methods are very useful in automated deep learning; the downside is that designing the rules themselves takes considerable time and they need to cover as many tasks as possible. I used this approach in the winning solution for the image track of the NeurIPS 2019 AutoDL competition.

Finally, there is weight decay: adding a penalty term to the objective function (usually L1/L2, PyTorch defaults to L2) to prevent overfitting. The coefficient is generally small; I often use $4\times10^{-5}$ . One small trick is to not apply weight decay to BN-layer parameters—BN parameters are more sensitive than those of convolutional/fully-connected layers, and applying weight decay to them actually degrades accuracy slightly.

Automated Hyperparameter Optimization

Automated hyperparameter optimization falls under the umbrella of AutoML (which also includes meta-learning, neural architecture search (NAS), and so on). Here we only discuss hyperparameter optimization (HPO) done via black-box optimization. Black-box methods fall roughly into two categories: model-free methods and Bayesian optimization.

Model-Free Methods

The simplest is grid search: define a set of candidate values for each hyperparameter, take the Cartesian product, and validate each combination one by one. The problem is that as the number of hyperparameters grows, the computation explodes exponentially—the “curse of dimensionality.”

An alternative is random search: sample randomly in the solution space until the given budget is exhausted. When some hyperparameters are important and others are not, random search is often more efficient than grid search; it is also easier to parallelize (no communication needed during the search) and serves as a decent baseline—given enough compute, the result approaches the optimum arbitrarily closely, and it can be used as an initial value for other algorithms.

Comparison of grid search and random search — With the same number of samples, random search explores more values along the important dimension.

There are also population-based algorithms (genetic/evolutionary/particle swarm): maintain a population of hyperparameters and iterate toward a better next generation through local operations such as selection, crossover, and mutation—they are simple, can handle various data types, and parallelize well. A particularly strong one is CMA-ES (Covariance Matrix Adaptation Evolution Strategy): it samples from a multivariate Gaussian whose mean and covariance are continuously optimized through evolution, and it performs very well on the Black-Box Optimization Benchmark (BBOB).

Bayesian Optimization

The biggest advantage of Bayesian optimization is that it finds a decent hyperparameter combination in very few steps, making it especially suitable for scenarios where the objective function itself is expensive to evaluate (such as deep learning). It has two parts: a probabilistic surrogate model that models the black-box objective, and an acquisition function that decides the next point to observe—the latter must trade off exploration against exploitation. The whole process is iterative: each round first fits the surrogate model to the existing observations, then uses the acquisition function to pick the next point.

The most common surrogate model is the Gaussian process (GP): expressive, smooth, and computationally closed-form. For a new point $\lambda$ , the GP’s predicted output follows a Gaussian distribution whose mean and variance are (where $\mathbf{k}_*$ is the covariance vector between the new point and the historical points, $\mathbf{K}$ is the covariance matrix among the historical points, and $\mathbf{y}$ is the vector of historical observations):

\mu(\lambda)=\mathbf{k}_*^{T}\mathbf{K}^{-1}\mathbf{y},\qquad \sigma^2(\lambda)=k(\lambda,\lambda)-\mathbf{k}_*^{T}\mathbf{K}^{-1}\mathbf{k}_*

The acquisition function commonly used is Expected Improvement (EI), whose goal is to find the point with the largest expected improvement as the next observation:

\mathbb{E}[I(\lambda)]=(f_{min}-\mu(\lambda))\,\Phi\!\Big(\tfrac{f_{min}-\mu(\lambda)}{\sigma(\lambda)}\Big)+\sigma(\lambda)\,\phi\!\Big(\tfrac{f_{min}-\mu(\lambda)}{\sigma(\lambda)}\Big)

where $f_{min}$ is the current best observation and $\Phi,\phi$ are the CDF and PDF of the standard normal. Since each round must find the point that maximizes the acquisition function, the acquisition function must be far cheaper to compute than the objective function—when the objective is a compute-heavy deep learning task, Bayesian optimization is a great fit.

The standard GP also has shortcomings: high computational complexity (fitting $O(n^3)$ , prediction $O(n^2)$ ), so it struggles once observations pile up; and standard kernel functions are not flexible enough in high dimensions. Hence various extended kernels (random embedding, cylindrical kernels, additive kernels), as well as replacing the GP with a neural network or random forest—with many observations, neural networks are usually faster and parallelize better, while random forests are better at handling discrete hyperparameters and have lower complexity (fitting $O(n\log n)$ , prediction $O(\log n)$ ).

Tools

Quite a few tools already integrate the algorithms above, with visualization interfaces and cross-platform support, such as RAY, NNI, and Auto-sklearn. Among them, NNI (Neural Network Intelligence) is a lightweight yet powerful toolkit that can automatically perform feature engineering, architecture search, hyperparameter tuning, and model compression. It supports a variety of training environments—local machine, remote servers, Kubeflow, AML, and more—and its interface lets you intuitively monitor the time cost and effectiveness of the entire optimization process:

The NNI user interface — NNI’s web interface, which lets you intuitively monitor the optimization process.

References

Deng, Jia, et al. ImageNet: A Large-Scale Hierarchical Image Database. CVPR, 2009.
He, Tong, et al. Bag of Tricks for Image Classification with Convolutional Neural Networks. CVPR, 2019.
Duchi, John, et al. Adaptive Subgradient Methods for Online Learning and Stochastic Optimization (Adagrad). JMLR, 2011.
Kingma, Diederik P., Ba, Jimmy. Adam: A Method for Stochastic Optimization. arXiv:1412.6980, 2014.
Sutskever, Ilya, et al. On the Importance of Initialization and Momentum in Deep Learning. ICML, 2013.
Bergstra, James, Bengio, Yoshua. Random Search for Hyper-Parameter Optimization. JMLR, 2012.
Hansen, Nikolaus. The CMA Evolution Strategy: A Comparing Review. 2006.
Rasmussen, Carl Edward. Gaussian Processes in Machine Learning. 2003.
Jones, Donald R., et al. Efficient Global Optimization of Expensive Black-Box Functions. Journal of Global Optimization, 1998.
Li, Lisha, et al. Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization. JMLR, 2017.
Falkner, Stefan, et al. BOHB: Robust and Efficient Hyperparameter Optimization at Scale. arXiv:1807.01774, 2018.
Zhang, Quanlu, et al. Retiarii: A Deep Learning Exploratory-Training Framework (NNI). OSDI, 2020.

Technology

2020 · 12 · 12