Notes on MobileNetV3 and the Lottery Ticket Hypothesis

After reading the MobileNetV3 paper, my overall impression is that it doesn’t pack the punch of those earlier classic works — but if performance is good and it deploys, that’s a good thing. Let’s start with the experimental results:

Line plots of Pixel 1 latency versus ImageNet top-1 accuracy, split into small models (20–40ms) and large models (40–150ms); MobileNetV3 small/large sit above MnasNet, ProxylessNAS, and MobileNetV2 to form the optimal frontier — Figure 1: The trade-off between Pixel 1 latency and top-1 ImageNet accuracy. All models use input resolution 224; V3-Large and V3-Small use width multipliers 0.75, 1, and 1.25 to trace the optimal frontier. Latencies were measured on a single large core of the same device using TFLite.

The overall idea is to obtain a seed architecture from MnasNet and then fine-tune it. The Large version directly takes MnasNet’s search result, while the Small version modifies the objective function — lowering the weight on latency and increasing the importance of accuracy — and re-runs the search.

The objective function used by MnasNet is:

ACC(m) \times \left[\frac{LAT(m)}{TAR}\right]^{w}

where the magnitude of w controls the importance of latency.

NetAdapt Fine-Tuning

The second step uses NetAdapt to fine-tune the seed architecture. NetAdapt’s main goal is to find the best accuracy subject to a latency constraint, taking a trained seed architecture as input:

Start with a seed network architecture found by platform-aware NAS.
For each step:
1. Generate a set of new proposals. Each proposal represents a modification of the architecture that yields at least a $\delta$ reduction in latency compared to the previous step.
2. For each proposal, reuse the pre-trained model from the previous step and populate the newly proposed architecture, truncating and randomly initializing missing weights as appropriate; fine-tune each proposal for $T$ steps to get a coarse estimate of the accuracy.
3. Select the best proposal according to some metric.
Iterate the previous step until the target latency is reached.

Starting from the seed architecture, NetAdapt generates a batch of candidate architectures whose latency drops by at least delta. The seed’s weights are then aligned to each candidate by truncation or random initialization, fine-tuned, and the one with the best accuracy is kept:

\frac{\Delta Acc}{|\Delta latency|}

This process is repeated multiple times, eventually yielding the model with the best accuracy at the target latency.

Changes to the Head, Tail, and Activation Function

After obtaining the above model, some manual adjustments were made, mainly to the head, the tail, and the activation function.

Head change: halve the number of output channels of the initial convolution.

Tail change, as shown below:

Diagram comparing the original last stage with the efficient last stage: the efficient version moves Avg-Pool earlier, right after the 1x1 expansion, dropping three expensive layers — Figure 2: Comparison of the original last stage and the efficient last stage. The more efficient last stage drops three expensive layers at the end of the network with no loss of accuracy.

The activation function is changed to:

\text{h-swish}[x] = x \, \frac{\text{ReLU6}(x+3)}{6}

The resulting overall architecture is as follows:

Input	Operator	exp size	#out	SE	NL	s
$224^2 \times 3$	conv2d	-	16	-	HS	2
$112^2 \times 16$	bneck, 3x3	16	16	-	RE	1
$112^2 \times 16$	bneck, 3x3	64	24	-	RE	2
$56^2 \times 24$	bneck, 3x3	72	24	-	RE	1
$56^2 \times 24$	bneck, 5x5	72	40	✓	RE	2
$28^2 \times 40$	bneck, 5x5	120	40	✓	RE	1
$28^2 \times 40$	bneck, 5x5	120	40	✓	RE	1
$28^2 \times 40$	bneck, 3x3	240	80	-	HS	2
$14^2 \times 80$	bneck, 3x3	200	80	-	HS	1
$14^2 \times 80$	bneck, 3x3	184	80	-	HS	1
$14^2 \times 80$	bneck, 3x3	184	80	-	HS	1
$14^2 \times 80$	bneck, 3x3	480	112	✓	HS	1
$14^2 \times 112$	bneck, 3x3	672	112	✓	HS	1
$14^2 \times 112$	bneck, 5x5	672	160	✓	HS	1
$14^2 \times 112$	bneck, 5x5	672	160	✓	HS	2
$7^2 \times 160$	bneck, 5x5	960	160	✓	HS	1
$7^2 \times 160$	conv2d, 1x1	-	960	-	HS	1
$7^2 \times 960$	Pool, 7x7	-	-	-	HS	-
$1^2 \times 960$	conv2d 1x1, NBN	-	1280	-	HS	1
$1^2 \times 1280$	conv2d 1x1, NBN	-	k	-	-	-

Table 1: Specification for MobileNetV3-Large. SE denotes whether there is a Squeeze-And-Excite in that block; NL denotes the type of non-linearity used, where HS is h-swish and RE is ReLU; NBN denotes no batch normalization; s denotes stride.

Input	Operator	exp size	#out	SE	NL	s
$224^2 \times 3$	conv2d, 3x3	-	16	-	HS	2
$112^2 \times 24$	bneck, 3x3	16	16	✓	RE	2
$56^2 \times 24$	bneck, 3x3	72	24	-	RE	2
$28^2 \times 24$	bneck, 3x3	88	24	-	RE	1
$28^2 \times 40$	bneck, 5x5	96	40	✓	HS	1
$14^2 \times 40$	bneck, 5x5	240	40	✓	HS	1
$14^2 \times 40$	bneck, 5x5	240	40	✓	HS	1
$14^2 \times 40$	bneck, 5x5	120	48	✓	HS	1
$14^2 \times 48$	bneck, 5x5	144	48	✓	HS	1
$14^2 \times 96$	bneck, 5x5	288	96	✓	HS	2
$7^2 \times 96$	bneck, 5x5	576	96	✓	HS	1
$7^2 \times 96$	bneck, 5x5	576	96	✓	HS	1
$7^2 \times 96$	conv2d, 1x1	-	576	✓	HS	1
$7^2 \times 576$	Pool, 7x7	-	-	-	HS	7
$1^2 \times 576$	conv2d 1x1	-	1280	-	HS	1
$1^2 \times 1280$	conv2d 1x1	-	k	-	HS	-

Table 2: Specification for MobileNetV3-Small; notation is the same as in Table 1.

The SE block is embedded into the block as follows:

MobileNetV3 block diagram: 1x1 expansion, 3x3 depthwise convolution, an SE branch of Pool plus two FC layers (ReLU, hard-σ), then a 1x1 projection with a residual connection — Figure 3: MobileNetV2 + Squeeze-and-Excite. Unlike the original SE, here the squeeze-and-excite is applied inside the residual layer, with different non-linearities chosen per layer.

The structure of the stride=2 block in MobileNetV2 is shown below. In V3, stride=2 and stride=1 are likewise distinguished by whether there is a residual connection, and the residual and SE modules do not conflict:

The two MobileNetV2 blocks: the stride=1 block has a 1x1 expansion, a 3x3 depthwise convolution, and a 1x1 linear projection with a residual Add; the stride=2 block has the same structure but no residual connection — Figure 4: The stride=1 and stride=2 blocks of MobileNetV2, distinguished by whether they carry a residual connection.

THE LOTTERY TICKET HYPOTHESIS: Finding a Trainable Sparse Sub-Network

The core idea of this paper is that an iterative, unstructured pruning procedure can find a small network, and this iterative pruning process is itself a way of training the large network.

The resulting small network is roughly 10%–20% of the size of the full large network. If this small network is given the same initialization as the large network, it can be trained faster to match the accuracy of the large network — this is the so-called “lottery.”

The overall training-and-iteration process is shown below. This process is typically iterated many times, and the key point is that it is reinitialized to theta_0 each time:

Identifying winning tickets. We identify a winning ticket by training a network and pruning its smallest-magnitude weights. The remaining, unpruned connections constitute the architecture of the winning ticket. Unique to our work, each unpruned connection’s value is then reset to its initialization from the original network before it was trained. This forms our central experiment:

Randomly initialize a neural network $f(x; \theta_0)$ (where $\theta_0 \sim \mathcal{D}_\theta$ ).
Train the network for $j$ iterations, arriving at parameters $\theta_j$ .
Prune $p\%$ of the parameters in $\theta_j$ , creating a mask $m$ .
Reset the remaining parameters to their values in $\theta_0$ , creating the winning ticket $f(x; m \odot \theta_0)$ .

Moreover, this effect becomes more pronounced as the degree of pruning increases: the more you prune, the harder the resulting small network is to train under random initialization, and the more important the initialization becomes — whereas keeping the same initialization as the pre-pruning network makes it easy to train.

Comparison with “Rethinking the Value of Network Pruning”

Another paper, “Rethinking the Value of Network Pruning,” argues that random initialization can in fact achieve very good performance, provided the learning rate is appropriate.

The two papers also differ in a few minor ways:

That paper mainly uses structured pruning, with unstructured pruning only on smaller datasets — different from the lottery paper’s all-unstructured pruning.
That paper mainly prunes large networks, whereas the lottery work is mainly done on small networks.
The optimization approaches also differ; the optimization strategy used in the lottery paper is not exactly the mainstream configuration for classification tasks.

That paper finds that under structured pruning there is none of the difference observed in the lottery work — the structures obtained in the lottery setting show little difference across different initializations; under unstructured pruning, the main difference comes from differences in learning rate.

Test-accuracy-versus-prune-ratio curves for VGG-16 and ResNet-50 on CIFAR-10, split into (a) iterative pruning and (b) one-shot pruning, comparing winning tickets against random init at learning rates 0.1 and 0.01 — Figure 5: Iterative/one-shot unstructured pruning of VGG-16 and ResNet-50 on CIFAR-10, compared against the Lottery Ticket Hypothesis, with each point averaged over 5 runs. Using the winning ticket as initialization only helps when the learning rate is small (0.01), yet that small learning rate itself yields lower accuracy than the widely used large learning rate (0.1).

Paper Notes
MobileNet
Pruning

2019 · 05 · 09

Notes on MobileNetV3 and the Lottery Ticket Hypothesis

MobileNetV3: Engineering Refinements on Top of NAS

NetAdapt Fine-Tuning

Changes to the Head, Tail, and Activation Function

THE LOTTERY TICKET HYPOTHESIS: Finding a Trainable Sparse Sub-Network

Comparison with “Rethinking the Value of Network Pruning”