A Side-by-Side Comparison of Attention Mechanisms in CV, with Single-Path NAS Notes

I recently read several papers on attention mechanisms in CV, along with a newly released NAS work. Let me organize and compare their respective ideas, and record some of my own thoughts along the way.

SENet: Weighting Channels

The core idea of Squeeze-and-Excitation Networks is to consider the correlations between channels. It adds an SE module after the feature map—first applying global average pooling to each channel (squeeze), then passing through FC layers to learn a weight for each channel (excitation), and finally using these weights to recalibrate the original feature map. Essentially, it adds an attention over the different channels.

Squeeze-and-Excitation module: a feature map goes through squeeze (global pooling), excitation (FC learns weights), and scale (per-channel recalibration) — Figure 1: The Squeeze-and-Excitation module. Squeeze applies global pooling per channel to obtain a channel descriptor, Excitation uses FC layers to learn per-channel weights, and Scale recalibrates the original feature map channel-wise with those weights.

SENet scores quite high on ImageNet, and its effect is very noticeable.

Model	224×224 top-1 err.	224×224 top-5 err.	320×320 / 299×299 top-1 err.	320×320 / 299×299 top-5 err.
ResNet-152 [13]	23.0	6.7	21.3	5.5
ResNet-200 [14]	21.7	5.8	20.1	4.8
Inception-v3 [20]	-	-	21.2	5.6
Inception-v4 [21]	-	-	20.0	5.0
Inception-ResNet-v2 [21]	-	-	19.9	4.9
ResNeXt-101 (64×4d) [19]	20.4	5.3	19.1	4.4
DenseNet-264 [17]	22.15	6.12	-	-
Attention-92 [60]	-	-	19.5	4.8
PyramidNet-200 [77]	20.1	5.4	19.2	4.7
DPN-131 [16]	19.93	5.12	18.55	4.16
SENet-154	18.68	4.47	17.28	3.79

Non-local NN: Self-Attention over Spatial Pixels

Non-local Neural Networks takes a different approach: it looks for correlations between pixels in the feature map. This is a self-attention mechanism that lets the feature at every position interact with the features at all other positions, capturing long-range dependencies. This kind of attention mechanism really can achieve fairly good results. After reading this paper, I wondered whether this aspect could also be taken into account in NAS—incorporating this kind of attention operation into the search space as a candidate.

Non-local block: the input passes through θ, φ, g (each a 1×1×1 conv); θ and φ are matrix-multiplied and softmaxed into attention, multiplied with g, passed through a 1×1×1 conv, and added back via a residual connection — Figure 2: Structure of a Non-local block. The input X passes through three 1×1×1 convolutions θ, φ, g; the features from θ and φ are matrix-multiplied and softmaxed into pairwise attention weights, multiplied with g, passed through a 1×1×1 conv, and added residually to the input to yield Z.

CBAM: Chaining Channel and Spatial Attention

CBAM: Convolutional Block Attention Module can be seen as an extension of the SE module, split into two parts.

Channel attention part: Apply global average pooling and global max pooling separately, pass both paths through the same FC, add them up, then apply sigmoid to obtain a weight for each channel.

Spatial attention part: Apply average pooling and max pooling along the channel dimension, concatenate the two results, pass them through a convolutional layer plus sigmoid, and obtain an attention map over the spatial dimension.

The two parts are chained in sequence: channel attention first, then spatial attention.

CBAM's two sub-modules: top is channel attention (MaxPool/AvgPool each through a shared MLP then summed), bottom is spatial attention (channel-wise pooling concatenated then through a conv layer) — Figure 3: CBAM’s two sub-modules. Top: channel attention, applying both max and average pooling to the features, each through a shared MLP and then summed. Bottom: spatial attention, applying max and average pooling along the channel dimension and concatenating them, then through a conv layer to obtain a spatial attention map.

Dual Attention Network: Dual Attention for Segmentation

Dual Attention Network applies attention over both the channel and the spatial dimensions of the feature map simultaneously in semantic segmentation, with the two paths running in parallel and finally fused together. The design idea is similar to CBAM, except the application scenario is switched to segmentation, and the two paths are in parallel rather than chained.

Dual Attention Network: ResNet features split into two branches, the top a position attention module (spatial attention) and the bottom a channel attention module, merged by sum fusion — Figure 4: Structure of the Dual Attention Network. Features from ResNet are split into two parallel branches: the top position attention module models attention over the spatial dimension, the bottom channel attention module models attention over the channel dimension, and the two are combined by sum fusion for segmentation.

Single-Path NAS: Compressing Multi-Path Search into a Single Path

The main idea of Single-Path NAS: Designing Hardware-Efficient Mobile Networks is to place all the multi-path operation choices into a single convolutional kernel, which amounts to a fairly fine-grained form of weight sharing, so that only one path needs to be maintained during training.

Comparison: left is prior NAS's multi-path search space (each candidate conv is a separate path), right is the proposed single-path search space (all convs merged into one superkernel) — Figure 5: Multi-path vs. single-path search space. Left: prior NAS treats each candidate convolution as a separate path, which is expensive. Right: Single-Path NAS merges all candidate operations into one “superkernel” and only makes a kernel-level selection.

In terms of time it isn’t particularly fast either, taking 30 TPU hours to search directly on ImageNet, with mediocre performance. The search space is based on MobileNet-v2, mainly searching over the kernel size of the depthwise conv.

Single-Path NAS search space in three levels: ConvNet macro-architecture (a sequence of blocks), MBConv micro-architecture (with a searchable depthwise conv), and the single-path searchable superkernel — Figure 6: The Single-Path NAS search space. Built on a MobileNet-v2-like hierarchical search space, from the ConvNet macro-architecture down to the MBConv micro-architecture; the core idea is a searchable “superkernel” that encodes all MBConv types, without listing each candidate operation as a separate path.

The final search results are as follows. It feels like it’s following the ProxylessNAS work, only replacing multi-path with a single path.

Layer-by-layer diagram of the hardware-efficient ConvNet found by Single-Path NAS, each layer annotated with its MBConv kernel size and expansion ratio — Figure 7: The hardware-efficient ConvNet found by Single-Path NAS. Each layer is annotated with its MBConv kernel size and expansion ratio; it reaches 74.96% top-1 accuracy on ImageNet with 79.48ms inference time on a Pixel 1 phone.

Search method details: An indicator function is used to determine whether to use a sub-kernel.

\mathbf{w}_k = \mathbf{w}_{3\times3} + \mathbb{1}(\text{use } 5\times5)\cdot \mathbf{w}_{5\times5\setminus 3\times3} \tag{1}

where $\mathbb{1}(\cdot)$ is the indicator function that encodes the architectural (NAS) choice, i.e., if $\mathbb{1}(\cdot)=1$ then $\mathbf{w}_k = \mathbf{w}_{3\times3} + \mathbf{w}_{5\times5\setminus 3\times3} = \mathbf{w}_{5\times5}$ , else $\mathbb{1}(\cdot)=0$ then $\mathbf{w}_k = \mathbf{w}_{3\times3}$ .

The indicator function uses a threshold $t_k = 5$ , which is learnable. The original indicator function is:

g(x,t) = \mathbb{1}(x > t)

To allow gradients to propagate back, the indicator function is relaxed into a sigmoid:

\hat{g}(x,t) = \sigma(x > t)

After relaxation it becomes:

\mathbf{w}_k = \mathbf{w}_{3\times3} + \mathbb{1}\!\left(\left\lVert \mathbf{w}_{5\times5\setminus 3\times3}\right\rVert^2 > t_{k=5}\right)\cdot \mathbf{w}_{5\times5\setminus 3\times3}

From this design, we can see that whether a given kernel is used is closely related to the weights themselves, mainly depending on the importance of the weights outside the sub-kernel. Overall, this method generalizes the entire MBConv layer into a searchable block.

The final experimental results also look mediocre.

Method	Top-1 Acc (%)	Top-5 Acc (%)	Mobile Runtime (ms)	Search Cost (epochs)
MobileNetV1 [11]	70.60	89.50	113	–
MobileNetV2 1.0x [17]	72.00	91.00	75.00	–
MobileNetV2 1.0x (our impl.)	73.59	91.41	73.57	–
Random search	73.78 ± 0.85	91.42 ± 0.56	77.31 ± 0.9	–
MnasNet 1.0x [20]	74.00	91.80	76.00	40,000
MnasNet 1.0x (our impl.)	74.61	91.95	74.65	40,000
ChamNet-B [6]	73.80	–	–	240‡
ProxylessNAS-R [4]	74.60	92.20	78.00	200*
ProxylessNAS-R (our impl.)	74.65	92.18	77.48	200*
FBNet-B [21]	74.1	-	-	90
FBNet-B (our impl.)	73.70	91.51	78.33	90
Single-Path NAS (proposed)	74.96	92.21	79.48	8 (3.75 hours)

Table 1: Single-Path NAS achieves state-of-the-art accuracy (%) under a similar mobile latency setting (≤80ms on Pixel 1), with up to 5,000× reduced search cost in number of epochs. *The search cost in epochs is estimated based on the claim that ProxylessNAS is 200× faster than MnasNet. ‡ChamNet does not detail the model derived under runtime constraints, so we cannot retrain or measure the latency.

Papers
Attention
NAS

2019 · 04 · 11