MobileNetV2 Close Reading: Inverted Residuals and Linear Bottlenecks

Today I did a close reading of the MobileNetV2 paper. Its core changes are the inverted residual structure and the linear bottleneck, striking a balance among accuracy, FLOPs, and latency. The paper runs experiments on three kinds of tasks: classification, detection, and segmentation. Below I organize my thoughts from reading the paper.

Why ReLU Is Dropped at the Low-Dimensional Bottleneck

This is the question you most need to think through before everything else in the paper. The paper’s starting point is this: for the manifold of interest (the feature distribution we care about) to be fully preserved within some subspace of the high-dimensional activation space, two conditions must hold:

If the feature distribution is still non-zero after ReLU, then it corresponds to a linear transformation;
ReLU can fully preserve the information of the input distribution only when that input distribution lies within some subspace of the input space.

What these two points mean is: adding ReLU at the low-dimensional bottleneck causes the truncation on the negative half-axis to irreversibly collapse feature information, and later layers cannot recover it. Xception had already shown experimentally that following a depthwise conv directly with ReLU degrades performance, which is consistent with the direction of the analysis above.

Inverted Residual Structure

Based on the analysis above, MobileNetV2’s bottleneck design is “narrow-wide-narrow,” exactly the opposite of ResNet’s “wide-narrow-wide,” which is why it’s called an inverted residual. The specific flow is:

The input is a low-dimensional bottleneck;
A 1×1 conv expands it to high dimension by the expansion rate;
A depthwise conv (3×3) operates in the high-dimensional space;
A 1×1 linear conv (with no activation) maps it back to low dimension;
The residual connection links the two low-dimensional bottlenecks at the ends.

This structure has a small memory footprint and is well suited to mobile deployment. Both ends of the residual connection are low-dimensional tensors, so peak memory is far smaller than that of an ordinary ResNet bottleneck.

That final 1×1 conv adds no ReLU; the paper calls it a linear conv, for exactly the reason analyzed earlier: the output is low-dimensional, and adding ReLU here would irreversibly destroy information. The paper gives a formal proof, though the process is fairly involved.

The paper also points out that this structure allows the input domain and output domain to be decoupled to some degree.

Choosing the Expansion Rate

The expansion rate uses a single constant across the whole network, typically between 5 and 10, with little difference among the choices; small networks can take a slightly smaller value and large networks a slightly larger one.

Overall Network Structure

The whole network first uses an initial conv to raise the channels to 32 (or 64), then stacks 19 residual bottleneck layers, each with BN added.

The figure below is the complete network configuration given in the paper, where t is the expansion rate of each stage:

Input	Operator	$t$	$c$	$n$	$s$
$224^2 \times 3$	conv2d	-	32	1	2
$112^2 \times 32$	bottleneck	1	16	1	1
$112^2 \times 16$	bottleneck	6	24	2	2
$56^2 \times 24$	bottleneck	6	32	3	2
$28^2 \times 32$	bottleneck	6	64	4	2
$14^2 \times 64$	bottleneck	6	96	3	1
$14^2 \times 96$	bottleneck	6	160	3	2
$7^2 \times 160$	bottleneck	6	320	1	1
$7^2 \times 320$	conv2d 1x1	-	1280	1	1
$7^2 \times 1280$	avgpool 7x7	-	-	1	-
$1 \times 1 \times 1280$	conv2d 1x1	-	k	-	-

Here $t$ is the expansion rate, $c$ the number of output channels, $n$ the number of repeats in the stage, and $s$ the stride of the first block.

The bottleneck module comes in two cases: the stride=1 version has a residual connection, while the stride=2 version (used for downsampling) does not:

The two MobileNetV2 bottleneck block structures — Figure 1: The two bottleneck blocks. Left is the stride=1 version (input goes through 1x1 expansion + Relu6 → 3x3 depthwise + Relu6 → 1x1 linear projection, then added back to the input as a residual); right is the stride=2 version (same three layers but with depthwise stride=2 and no residual connection).

ReLU6

Inside the module (after the depthwise conv in the high-dimensional space), ReLU6 is used rather than ordinary ReLU. ReLU6 clips activation values to a maximum of 6.

The reason is quantization friendliness on mobile: mobile devices commonly use float16 or even int8 inference, and if the dynamic range of the activations is very large (it can reach hundreds or even thousands), the limited precision of float16 introduces representation error. Restricting activations to the fixed interval [0, 6] lets float16’s numerical resolution be fully used, with smaller precision loss.

Experimental Results

On ImageNet, with a 224×224 input and a width multiplier of 1, the whole network has about 300M multi-adds and around 3.4M parameters. The multi-scale test results in the paper are as follows:

Performance curves of MobileNetV2 vs MobileNetV1, ShuffleNet, NAS — Figure 2: Accuracy-vs-compute curves of MobileNetV2 against MobileNetV1, ShuffleNet, and NASNet on ImageNet (x-axis: Multiply-Adds, y-axis: Top-1 accuracy). Each resolution (96–224) uses width multipliers 0.35, 0.5, 0.75, 1.0, with an extra 1.4 at 224.

Paper Notes
MobileNet
Network Architecture

2019 · 03 · 12