The Evolution of Convolutional Network Architectures: From LeNet to SENet

To design efficient networks, you first need to understand how the field has, over the years, made networks deeper and stronger step by step. This post follows the timeline of the ILSVRC (ImageNet Large Scale Visual Recognition Challenge) and walks through several milestones in convolutional neural network architecture. (All accuracy figures below come from torchvision.)

LeNet-5: The Starting Point of Convolutional Networks

Convolutional neural networks date back to LeCun’s 1998 paper, which proposed LeNet-5 for handwritten character recognition. The corresponding dataset is the familiar MNIST—50,000 handwritten characters of digits 0–9, each $28\times28$ with a single grayscale channel.

The MNIST dataset — The MNIST handwritten digit dataset.

Apart from its fully connected layers, LeNet-5 has only 5 layers (3 convolutions + 2 pooling), making it the earliest CNN used for image recognition.

The LeNet-5 architecture — The structure of LeNet-5: convolution + pooling + fully connected.

AlexNet: Igniting Deep Learning

The real rise of CNNs is inseparable from ILSVRC—the large-scale recognition challenge on ImageNet (the ILSVRC2012 dataset spans 1,000 classes, with 1.28 million training images and 50,000 test images, totaling over 100GB). In 2012, AlexNet won. Being able to train a deep CNN at the scale of ImageNet owed much to the development of GPUs: the highly parallelizable nature of CNNs, combined with the parallel compute power of GPUs, allowed deep CNNs to shine in large-scale recognition. AlexNet has 5 convolutional layers and 3 max-pooling layers—not many more layers than LeNet-5—but each layer computes at a higher resolution, requiring far more computation. It ultimately achieved 56.55% Top-1 and 79.09% Top-5.

The AlexNet architecture — The structure of AlexNet.

GoogLeNet / Inception: Multi-Branch Modules

In 2014, Google’s GoogLeNet (i.e. Inception V1, later followed by V2/V3/V4) won ILSVRC2014. It introduced the Inception module: processing the input with several different convolutional layers and then concatenating the results. On ILSVRC2012, GoogLeNet reached 69.78% Top-1 and 89.53% Top-5.

The Inception module (naive version): multiple convolutions in parallel, then concatenated.

The Inception module with dimensionality reduction: a 1×1 convolution reduces dimensions first to cut down computation.

VGGNet: Making the Network “Very Deep”

Emerging at almost the same time as GoogLeNet, VGGNet upgraded deep convolution to very deep convolution right in its paper title. The differences between versions lie only in the number of convolutional layers and whether Batch Normalization is used. VGG19 + BN reached a peak of 74.24% Top-1 and 91.85% Top-5.

The VGGNet architecture — The different versions of VGGNet.

Batch Normalization and the Normalization Family

Some versions of VGG make use of Batch Normalization (BN) layers. BN performs normalization and an affine transformation on the input’s four-dimensional tensor, effectively suppressing covariate shift. This allows the use of larger learning rates to speed up training, while also making training more stable and less prone to divergence.

Input: Values of $x$ over a mini-batch $\mathcal{B} = \{x_{1\ldots m}\}$ ; parameters to be learned $\gamma, \beta$ .

Output: $\{y_i = \mathrm{BN}_{\gamma,\beta}(x_i)\}$ .

\begin{aligned} \mu_{\mathcal{B}} &\leftarrow \frac{1}{m}\sum_{i=1}^{m} x_i && \text{// mini-batch mean} \\ \sigma_{\mathcal{B}}^{2} &\leftarrow \frac{1}{m}\sum_{i=1}^{m} (x_i - \mu_{\mathcal{B}})^2 && \text{// mini-batch variance} \\ \widehat{x}_i &\leftarrow \frac{x_i - \mu_{\mathcal{B}}}{\sqrt{\sigma_{\mathcal{B}}^{2} + \epsilon}} && \text{// normalize} \\ y_i &\leftarrow \gamma\,\widehat{x}_i + \beta \equiv \mathrm{BN}_{\gamma,\beta}(x_i) && \text{// scale and shift} \end{aligned}

How Batch Normalization is computed.

BN has become an indispensable part of modern deep learning. Similar techniques include Layer Normalization, Instance Normalization, and Group Normalization, which differ mainly in the dimensions and scope over which normalization is performed:

Different normalization methods — BN / LN / IN / GN: different dimensions and scopes of normalization.

ResNet: Residual Connections That Train Hundreds of Layers

The ResNet of 2015 is arguably one of the most exciting results of recent years, sweeping the classification, detection, and segmentation tracks of ILSVRC2015 and COCO2015. The residual learning it proposed makes even deeper networks easy to train—the paper reached hundreds of layers and even attempted over a thousand, with experiments showing that networks this deep do not encounter optimization difficulties when fitting the training data (though they will overfit). On ILSVRC2012, ResNet-152 reached 78.32% Top-1 and 94.06% Top-5.

The residual module of ResNet: adding an identity shortcut so gradients propagate back more easily.

SENet: Adding Attention to Channels

The SENet (Squeeze-and-Excitation Network) of 2016 won the final ILSVRC (2017). It introduced the Squeeze-and-Excitation module, which fuses information across the different channels of a feature map—itself a form of attention mechanism.

The SE module can be conveniently embedded into other architectures. For instance, after embedding it into ResNet’s residual module, a SE-ResNet of the same depth consistently beats ResNet by more than 1% in Top-1 on ILSVRC2012.

The SE module embedded into a ResNet residual module — Embedding the SE module into ResNet’s residual module.

This string of milestones—AlexNet bringing deep networks into view, VGG making them deep, Inception introducing multiple branches and BN, ResNet introducing residuals, and SENet introducing attention—forms the foundation of all subsequent architecture design. But most of them were born to “chase accuracy”; deploying to mobile devices in pursuit of lower latency and memory calls for dedicated compact architecture design.

References

LeCun, Yann, et al. Gradient-Based Learning Applied to Document Recognition. Proceedings of the IEEE, 1998.
Krizhevsky, Alex, Sutskever, Ilya, Hinton, Geoffrey E. ImageNet Classification with Deep Convolutional Neural Networks (AlexNet). NeurIPS / CACM, 2012/2017.
Szegedy, Christian, et al. Going Deeper with Convolutions (GoogLeNet). CVPR, 2015.
Simonyan, Karen, Zisserman, Andrew. Very Deep Convolutional Networks for Large-Scale Image Recognition (VGG). arXiv:1409.1556, 2014.
Ioffe, Sergey, Szegedy, Christian. Batch Normalization. arXiv:1502.03167, 2015.
Ba, Jimmy Lei, et al. Layer Normalization. arXiv:1607.06450, 2016.
Ulyanov, Dmitry, et al. Instance Normalization: The Missing Ingredient for Fast Stylization. arXiv:1607.08022, 2016.
Wu, Yuxin, He, Kaiming. Group Normalization. ECCV, 2018.
He, Kaiming, et al. Deep Residual Learning for Image Recognition (ResNet). CVPR, 2016.
Hu, Jie, Shen, Li, Sun, Gang. Squeeze-and-Excitation Networks (SENet). CVPR, 2018.
Deng, Jia, et al. ImageNet: A Large-Scale Hierarchical Image Database. CVPR, 2009.

Technology

2020 · 12 · 01