Image Data Augmentation: From Random Crop to AutoAugment

Data augmentation is a method frequently used during training to prevent overfitting in computer vision tasks. This post first introduces several common image augmentations, then covers some automated data augmentation methods.

Common Image Data Augmentation

Random resized crop randomly crops the input image and then scales it to a given size. In PyTorch this is torchvision.transforms.RandomResizedCrop(size, scale, ratio): size is the final size to scale to; scale is a range giving the proportion of the original image area that the random crop region can cover; ratio is also a range, giving the aspect ratio range of the crop region.

Cutout randomly masks out a square portion of the image (setting the pixel values to 0):

Effect of Cutout on CIFAR10
Effect of Cutout on CIFAR-10: random masking with a square.

Random erase is similar to Cutout, also randomly masking out a portion, the difference being that it uses a rectangle rather than a square:

Effect of Random erase
Random erase: random masking with a rectangle.

Mixup creates some “virtual” training samples by randomly weighted mixing of two inputs and their labels:

x~=λxi+(1−λ)xj,where xi,xj are raw input vectorsy~=λyi+(1−λ)yj,where yi,yj are one-hot label encodings\begin{aligned} \tilde{x} &= \lambda x_i + (1-\lambda) x_j, &&\text{where } x_i, x_j \text{ are raw input vectors} \\ \tilde{y} &= \lambda y_i + (1-\lambda) y_j, &&\text{where } y_i, y_j \text{ are one-hot label encodings} \end{aligned}

mixup: linearly mix two images and their labels with a random weight λ∈[0,1].

Beyond these, there are many other common techniques: random flip, random rotation, adjusting contrast, brightness, sharpness, and so on.

Automated Data Augmentation

AutoAugment organizes a variety of data augmentation operations and their magnitudes into a search space, then automatically searches for the best-performing combination (each combination contains two operations, and each operation has two parameters: “probability” and “magnitude”).

AutoAugment search space
AutoAugment’s search space: combinations of augmentation operations and their magnitudes.

Its search approach is much like NAS—using an RNN to sample policies and reinforcement learning to update. Each candidate policy requires training a network to validate its effect, making it very time-consuming.

AutoAugment search process
AutoAugment: RNN sampling + reinforcement learning updates, with each policy validated by training a network.

Applying this combinatorial augmentation to ImageNet gives the following results:

AutoAugment results on ImageNet
Results of applying the policies found by AutoAugment to ImageNet.

Fast AutoAugment is much faster and once won the AutoCV competition; I also used it to win the image track of the AutoDL competition. Its speedup mainly comes from splitting the dataset into multiple parts, searching for the best augmentation policy on each separately, and finally merging these policies together. In the competition I also replaced the original paper’s Bayesian optimization with random search, and used AutoAugment’s search results as the search space.

Fast AutoAugment workflow
Fast AutoAugment: search augmentation policies in parallel over blocks, then merge.

One final note: besides training-time augmentation, you can also do test-time augmentation (TTA)—augment the test sample, run inference on each augmented version separately, and then ensemble the multiple results as the final output.

References

  • DeVries, Terrance, Taylor, Graham W. Improved Regularization of Convolutional Neural Networks with Cutout. arXiv:1708.04552, 2017.
  • Zhong, Zhun, et al. Random Erasing Data Augmentation. AAAI, 2020.
  • Zhang, Hongyi, et al. mixup: Beyond Empirical Risk Minimization. arXiv:1710.09412, 2017.
  • Shorten, Connor, Khoshgoftaar, Taghi M. A Survey on Image Data Augmentation for Deep Learning. Journal of Big Data, 2019.
  • Cubuk, Ekin D., et al. AutoAugment: Learning Augmentation Policies from Data. arXiv:1805.09501, 2018.
  • Lim, Sungbin, et al. Fast AutoAugment. NeurIPS, 2019.
  • Zoph, Barret, Le, Quoc V. Neural Architecture Search with Reinforcement Learning. arXiv:1611.01578, 2016.
  • Technology