Designing Compact Networks: Taking Convolution Apart

The classic networks (AlexNet, VGG, Inception, ResNet, SENet, and so on) were mostly built to “chase accuracy.” But to actually deploy them—with a smaller runtime memory footprint, lower latency, and higher throughput—you need purpose-built compact network architectures. This piece walks through the design ideas behind a few of the main lines of work.

MobileNet: Depthwise Separable Convolution

MobileNet is more or less the starting point of compact network design. As the name suggests, it targets mobile deployment, and its biggest contribution is splitting the conventional convolution into two steps: depthwise convolution and $1\times1$ convolution (pointwise convolution).

The filters of standard convolution, depthwise convolution, and 1×1 convolution — The filters of standard convolution vs. depthwise convolution vs. 1×1 convolution.

The MobileNet architecture — The overall MobileNet architecture.

We can analyze this quantitatively. Suppose a $D_F\times D_F\times M$ tensor is turned by convolution into a $D_F\times D_F\times N$ tensor, with kernel size $D_K$ . The FLOPs of a standard convolution are:

\text{FLOPs}_{\text{std}}=D_K \times D_K \times M \times N \times D_F \times D_F

After splitting into depthwise + $1\times1$ :

\text{FLOPs}_{\text{dw+pw}}=D_K \times D_K \times M \times D_F \times D_F + M \times N \times D_F \times D_F

The ratio of the two is:

\frac{\text{FLOPs}_{\text{dw+pw}}}{\text{FLOPs}_{\text{std}}}=\frac{1}{N}+\frac{1}{D_K\times D_K}

The most common choice is the $3\times3$ convolution ( $D_K=3$ ), and when the number of output channels $N$ is large, this split can cut FLOPs to as little as $\tfrac{1}{9}$ of a standard convolution. But note: a 9× drop in FLOPs does not mean a 9× speedup in practice—inference frameworks usually implement convolution with im2col+GEMM or Winograd, and depthwise separable convolution doesn’t effectively reduce memory access cost, so the measured speedup is actually quite limited. The parameter-count analysis is the same, with a ratio of $\tfrac{1}{N}+\tfrac{1}{D_K^2}$ as well; the reduction in parameters is “more real” than the reduction in FLOPs, because it genuinely saves space when stored on disk—unlike FLOPs, which are constrained by software implementation and hard to translate into a real speedup.

ShuffleNet: Grouped 1×1 Convolution + Channel Shuffle

ShuffleNet came a few months after MobileNet, and it goes a step further on the $1\times1$ convolution: besides keeping the depthwise $3\times3$ convolution, it also groups the standard $1\times1$ convolution, and after grouping it introduces channel shuffle so that information from each group of channels can be fused in the next layer. Note that channel shuffle isn’t a truly random permutation—in PyTorch, for example, it’s implemented with reshape + permute.

Channel shuffle in ShuffleNet — Channel shuffle: with grouped convolution alone, the groups don’t communicate with each other (a); after the shuffle, channel information is fused across groups (c).

The whole of ShuffleNet is built by stacking multiple blocks:

The ShuffleNet block: replace the 1×1 convolution with a grouped convolution and add channel shuffle.

The ShuffleNet architecture — The overall ShuffleNet architecture.

Quantitatively: channel shuffle itself neither adds nor removes FLOPs/parameters, but it changes the in-memory layout of the feature maps, imposing strided memory access cost on the next layer. And a $1\times1$ convolution with $G$ groups has FLOPs of $\tfrac{M\times N\times D_F\times D_F}{G}$ —compared with a standard $1\times1$ convolution, both FLOPs and parameter count drop to $\tfrac{1}{G}$ . For a fixed total network FLOPs, more groups means you can use more filters; achieving the best accuracy requires a careful trade-off between the two.

MobileNetV2: Inverted Residual

MobileNetV2 brings ResNet’s residual connection and bottleneck into MobileNet, but uses the opposite inverted bottleneck: first a $1\times1$ convolution to expand the channel count, then a depthwise separable convolution, and finally a second $1\times1$ convolution to bring the channels back down to the input size.

The MobileNetV2 architecture — MobileNetV2: t is the expansion ratio, c is the output channels, n is the number of repeats, and s is the stride.

It didn’t invent any new compact operation; instead, it folds residual connections, bottlenecks, and other techniques that boost fitting capacity into a compact network, pushing accuracy a bit further. It’s worth mentioning that MobileNetV2 later became the backbone for many architecture-search methods (both MobileNetV3 and EfficientNet below are built on it).

ShuffleNetV2: Four Guidelines Built Around Memory Access Cost

ShuffleNetV2, in its paper, places its emphasis on an often-overlooked metric—memory access cost (MAC)—and from it derives four guidelines for designing compact networks.

Guideline 1: Keep a convolution’s input and output channel counts as equal as possible. Take the $1\times1$ convolution as an example: with input/output channels $c_1,c_2$ and a feature map of $h\times w$ , we have $\text{FLOPs}=h\,w\,c_1\,c_2$ . For fixed FLOPs, the memory access cost is

\text{MAC}=h\,w\,(c_1+c_2)+c_1 c_2 \ \ge\ 2\sqrt{h\,w\cdot\text{FLOPs}}+\frac{\text{FLOPs}}{h\,w}

By the AM–GM inequality, MAC has a lower bound, attained when $c_1=c_2$ . Experiments confirm this: at the same FLOPs, the measured speed is fastest when the input/output channel ratio is $1:1$ (the conclusion holds on both GPU and ARM).

c1:c2	(c1,c2) GPU x1	GPU x1	GPU x2	GPU x4	(c1,c2) ARM x1	ARM x1	ARM x2	ARM x4
1:1	(128,128)	1480	723	232	(32,32)	76.2	21.7	5.3
1:2	(90,180)	1296	586	206	(22,44)	72.9	20.5	5.1
1:6	(52,312)	876	489	189	(13,78)	69.1	17.9	4.6
1:12	(36,432)	748	392	163	(9,108)	57.6	15.1	4.4

Guideline 1: at the same FLOPs, a 1:1 input/output channel ratio is fastest.

Guideline 2: Too many groups increases MAC. For a $1\times1$ convolution with $g$ groups:

\text{MAC}=h\,w\,(c_1+c_2)+\frac{c_1 c_2}{g}=h\,w\,c_1+\frac{\text{FLOPs}\cdot g}{c_1}+\frac{\text{FLOPs}}{h\,w}

For fixed FLOPs, MAC grows as $g$ grows.

g	c GPU x1	GPU x1	GPU x2	GPU x4	c CPU x1	CPU x1	CPU x2	CPU x4
1	128	2451	1289	437	64	40.0	10.2	2.3
2	180	1725	873	341	90	35.0	9.5	2.2
4	256	1026	644	338	128	32.9	8.7	2.1
8	360	634	445	230	180	27.8	7.5	1.8

Guideline 2: at the same FLOPs, more groups means slower.

Guideline 3: Too much fragmentation reduces parallelism. Too many small serial/parallel branches in a network weaken the degree of parallel computation. But deeper structures often have higher accuracy, so you have to trade off between accuracy and parallel speedup.

Serial and parallel structures — Fragmented structures with different counts and serial/parallel arrangements.

structure	GPU c=128	GPU c=256	GPU c=512	CPU c=64	CPU c=128	CPU c=256
1-fragment	2446	1274	434	40.2	10.1	2.3
2-fragment-series	1790	909	336	38.6	10.1	2.2
4-fragment-series	752	745	349	38.4	10.1	2.3
2-fragment-parallel	1537	803	320	33.4	9.1	2.2
4-fragment-parallel	691	572	292	35.0	8.4	2.1

Guideline 3: at the same FLOPs, fewer fragments is faster; for the same number of fragments, serial is faster than parallel.

Guideline 4: Don’t ignore the cost of element-wise operations. The time of an operation is made up of two parts, MAC and FLOPs. A convolution’s FLOPs far exceed its MAC, but in operations with very small FLOPs—like element-wise add and ReLU—MAC is the dominant cost and can’t be ignored.

ReLU	short-cut	GPU c=32	GPU c=64	GPU c=128	CPU c=32	CPU c=64	CPU c=128
yes	yes	2427	2066	1436	56.7	16.9	5.0
yes	no	2647	2256	1735	61.9	18.8	5.2
no	yes	2672	2121	1458	57.3	18.2	5.1
no	no	2842	2376	1782	66.3	20.2	5.4

Guideline 4: element-wise operations like ReLU and residual connections also carry a non-trivial cost.

Accordingly, the ShuffleNetV2 block is almost entirely different from V1: it no longer uses grouped $1\times1$ convolution, switching instead to channel split (which keeps the residual connection while avoiding the MAC increase from grouped convolution—Guideline 2); and the Channel Split, Concat, and Channel Shuffle within the block can even be fused into a single operation to reduce MAC (Guideline 4).

MobileNetV3: h-swish

MobileNetV3 is obtained by applying automated search (MnasNet + NetAdapt compression) on top of MobileNetV2; it mainly changes the number of convolution layers, the kernel sizes, and the channel counts, and uses SE modules in some layers. In addition, it swaps the activation function in the deeper layers from ReLU to hard-swish:

\text{h-swish}(x)=x\,\frac{\text{ReLU6}(x+3)}{6}

The original swish is $\text{swish}(x)=x\cdot\sigma(x)$ , but the sigmoid is expensive to compute; hard-swish uses an approximation to get nearly the same effect at far lower cost.

The hard-swish activation function — hard-swish: approximating swish with ReLU6 to avoid the heavy computation of the sigmoid.

The structure of MobileNetV3-small (there is also a large version).

EfficientNet: Compound Scaling

EfficientNet was born at almost the same time as MobileNetV3 and is also derived from an MnasNet search, but the search uses only a simple grid search. Its core is a network scaling method: since a compact network can reach decent accuracy at very small FLOPs, then to get higher accuracy you can just scale it up. The scaling spans three dimensions: width, depth, and input resolution.

Scaling a neural network — The three kinds of scaling—width / depth / resolution—and their compound scaling.

The scaling factor for each dimension is decoupled into a relative factor and an overall factor $\phi$ :

\text{depth}:d=\alpha^{\phi},\quad \text{width}:w=\beta^{\phi},\quad \text{resolution}:r=\gamma^{\phi}

\text{s.t.}\quad \alpha\cdot\beta^2\cdot\gamma^2\approx2,\quad \alpha,\beta,\gamma\ge1

The reason for the constraint $\alpha\beta^2\gamma^2\approx2$ : FLOPs grow quadratically with width and resolution and linearly with depth, so constraining $\alpha\beta^2\gamma^2$ to a constant lets any scaling keep FLOPs at a controllable level. During the search, $\phi$ is fixed at $1$ and a grid search finds the best $\alpha,\beta,\gamma$ ; afterward, tuning $\phi$ scales the network to any FLOPs.

Compound scaling vs. single-dimension scaling — Compound scaling achieves higher accuracy than scaling a single dimension, at the same FLOPs.

The EfficientNet-B0 architecture — EfficientNet-B0 (the base network for scaling, found by an MnasNet search), scaling up to B1–B7.

GhostNet: Generating “Ghost” Features with Cheap Operations

GhostNet also takes the decomposition route, but from a distinctive starting point: visualizing feature maps reveals that many channels’ feature maps are very similar to one another—so there’s no need to compute them all with an expensive standard convolution.

Many of the feature maps from a standard convolution are near-duplicates — Among the feature maps computed by a standard convolution, quite a few are highly similar to each other.

So GhostNet splits the standard convolution into two steps: first compute part of the output with a standard convolution using fewer filters, then “generate” the other part on top of it with a cheap operation (such as a linear transform or depthwise separable convolution), and finally concat them together.

Standard convolution vs. the convolution in a Ghost module — Standard convolution (a) vs. the Ghost module (b): a small standard convolution + cheap operations to generate the remaining features.

Quantitative analysis: with kernel $k$ , input channels $c$ , and output $n\times h\times w$ , the standard convolution’s FLOPs are $n\,h\,w\,c\,k^2$ . If the Ghost module computes $m$ channels in the first step, uses a depthwise separable convolution of size $d$ for the cheap operation, and lets $s=\tfrac{n}{m}$ , then the speedup ratio is

R_{\text{FLOPs}}=\frac{n\,h\,w\,c\,k^2}{m\,h\,w\,c\,k^2+(n-m)\,h\,w\,d^2}=\frac{c\,k^2}{\tfrac{1}{s}c\,k^2+\tfrac{s-1}{s}d^2}\approx s

(The approximation uses $d\approx k$ and $s\ll c$ .) The compression ratio for parameter count is likewise about $s$ .

A block in GhostNet — The GhostNet block: similar to MobileNetV2, expanding the feature map in the middle and then compressing it.

From MobileNet taking convolution apart, to ShuffleNetV2 fixating on memory access, to GhostNet reusing similar features—the through-line of compact network design has always been repeatedly trading off among FLOPs, memory access, and accuracy. And MobileNetV3 and EfficientNet have already replaced “designed by hand” with “found by automated search,” which is precisely the direction of neural architecture search (NAS).

References

Howard, Andrew G., et al. MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. arXiv:1704.04861, 2017.
Zhang, Xiangyu, et al. ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices. CVPR, 2018.
Ma, Ningning, et al. ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design. ECCV, 2018.
Sandler, Mark, et al. MobileNetV2: Inverted Residuals and Linear Bottlenecks. CVPR, 2018.
Howard, Andrew, et al. Searching for MobileNetV3. arXiv:1905.02244, 2019.
Tan, Mingxing, Le, Quoc V. EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. arXiv:1905.11946, 2019.
Han, Kai, et al. GhostNet: More Features from Cheap Operations. CVPR, 2020.
Tan, Mingxing, et al. MnasNet: Platform-Aware Neural Architecture Search for Mobile. CVPR, 2019.
Jia, Yangqing, et al. Caffe: Convolutional Architecture for Fast Feature Embedding. ACM MM, 2014.
Lavin, Andrew, Gray, Scott. Fast Algorithms for Convolutional Neural Networks (Winograd). CVPR, 2016.

Technology

2020 · 12 · 06