How to Measure Whether a Neural Network Is "Good" and "Fast"

When doing model optimization, the first step is usually not to start hacking on the network, but to get one thing clear: what exactly am I optimizing, and how will I know it has gotten better. As the saying goes—if you can not measure it, you can not improve it. So before talking about any acceleration or compression, we first need to take apart this word “performance.”

A neural network’s performance can be divided into two mutually independent lines:

Task-oriented performance: how well the model does on the task it is meant to solve, e.g. classification accuracy, detection mAP.
Efficiency-oriented performance: how much it costs to run the model, including time (latency) and space (memory).

Our goal is to push the model forward on both dimensions at once. Let’s look at each in turn.

Task-Oriented Performance Metrics

Convolutional neural networks (CNNs) are applied mostly in computer vision, so here I pick a few of the most common tasks—image classification, object detection, semantic segmentation, super-resolution—and see what metric each uses.

Classification comes in single-label and multi-label flavors. Single-label classification generally looks at Top-1 and Top-5 accuracy: both are “number of correctly classified samples / total number of samples,” the difference being that Top-1 requires the class the model assigns the highest probability to match the label, whereas Top-5 only requires that one of the top five highest-probability classes hit. But accuracy itself is not differentiable and can’t be used directly as an optimization objective; single-label classification typically uses the cross-entropy loss:

\text{Loss}(y, c) = -\log\frac{\exp(y[c])}{\sum_j \exp(y[j])}\cdot w[c]

where $y$ is the output probability vector (dimension equal to the number of classes), $c$ is the label class, and $w$ is the per-class weight. Multi-label classification, on the other hand, often treats each label as a binary classification problem, then reduces the losses over all labels to a single number by summing or averaging. Taking averaging, with total number of classes $N$ , as an example:

\text{Loss}(y, t)=\frac{1}{N}\sum_{n=1}^{N} l_n,\qquad l_n = -\big[\,t_n\log y_n+(1-t_n)\log(1-y_n)\,\big]\cdot w_n

Object detection most commonly uses mAP (mean Average Precision). It is obtained by averaging the AP (Average Precision) over each class, and the AP for each class equals the area under that class’s PR curve (Precision-Recall Curve). The algorithm goes roughly like this: after the model has predicted a batch of detection boxes over the whole dataset, for a specific class, first take out all predicted boxes of that class across all images and sort them by confidence from high to low; then compare them one by one against the ground-truth boxes—if a predicted box’s IoU (Intersection over Union) with some ground-truth box exceeds a threshold (usually 0.5), it counts as a TP (True Positive), otherwise it counts as an FP (False Positive), and that ground-truth box is marked as detected and no longer participates in subsequent matching. By setting different confidence thresholds, you obtain a series of TP/FP, and thus a series of precision and recall values:

\text{Precision} = \frac{TP}{TP+FP},\qquad \text{Recall} = \frac{TP}{TP+FN}

Plotting points with recall on the x-axis and precision on the y-axis and interpolating gives the PR curve; the area under the curve is the AP.

Illustration of the object detection task — Object detection: box the targets in the image and give their classes.

Computing IoU — IoU = intersection area of the predicted box and ground-truth box / union area.

The area under the PR curve is that class’s AP.

Semantic segmentation commonly uses mIoU (mean Intersection over Union): similar to detection, compute IoU per class (same algorithm as in the figure above), then average.

Illustration of the semantic segmentation task — Semantic segmentation: assign a class to every pixel.

Super-resolution, image restoration, image compression and similar tasks commonly use PSNR (Peak Signal to Noise Ratio). The higher the PSNR, the smaller the error between the compressed/restored image and the original; typical values in image compression fall between 30 dB and 50 dB:

\text{PSNR}=10\log_{10}\frac{\text{MAX}_I^2}{\frac{1}{3mn}\sum_{c\in\{R,G,B\}}\sum_{i=0}^{m-1}\sum_{j=0}^{n-1}\big[I_c(i,j)-K_c(i,j)\big]^2}

Efficiency-Oriented Performance

The efficiency line splits into two aspects: time and space. Because network training is done only once while inference is what gets run repeatedly after deployment, here we mainly discuss efficiency during inference (accelerating the training process itself is another topic).

Time: Latency, FLOPs, and Its Limits

Time efficiency mainly refers to inference latency—how long one forward pass takes. Latency depends on three things: the model itself, the software implementation, and the hardware platform. All three significantly affect the result when measuring, so when optimizing with respect to the model, you must measure in a stable software-and-hardware environment to rule out other interference: the same docker image, the same piece of hardware, a cache warmup before measuring, averaging over multiple runs, and so on.

But different researchers can rarely compare on a unified software-and-hardware platform, so an indirect metric, FLOPs (floating point operations, which some papers also call Mult-Adds), is commonly used in place of latency. A CNN’s FLOPs are concentrated mainly in the convolutional and fully connected layers. Convolutional layer:

\text{FLOPs}=H\times W \times (K\times K \times C_{in}+1)\times C_{out}

where $H,W$ are the height and width of the output feature map, $C_{in},C_{out}$ are the input/output channel counts, and $K$ is the convolution kernel size. Fully connected layer:

\text{FLOPs}=(2\times N_{in}-1)\times N_{out}

$N_{in},N_{out}$ are the input/output dimensions. Off-the-shelf tools already exist for computing a network’s FLOPs, for example the PyTorch-based torchprofile.

Naturally, an indirect metric has its limits: it can’t account for memory-access overhead, nor for the speedup brought by parallelization and high-performance operator implementations. To see this gap clearly, I measured FLOPs and the GPU latency on a unified platform for a batch of classic networks, and tallied the share of FLOPs/latency taken up by convolution, matrix multiplication (gemm), and other operations:

FLOPs vs. latency comparison for classic networks — FLOPs vs. latency for classic network architectures (input 224×224×3). Overall positively correlated, but with clear exceptions.

Overall, FLOPs and latency are positively correlated, but there are a few glaring exceptions, the most typical being that VGG16 and DenseNet161 are reversed. VGG16 has 15470.3M FLOPs and 1384.11 ms of GPU latency (the total time for 200 consecutive inferences after a warmup, converted to TensorRT); DenseNet161 has only 7757.6M FLOPs—half the FLOPs of VGG16, yet more than twice the latency. The reason is that FLOPs only count compute, not memory access. DenseNet161 contains a large number of Copy operations (about 10× the convolution operations after converting to TensorRT), and this overhead is completely invisible to FLOPs. Even after subtracting Copy, DenseNet161 is still slightly slower than VGG16: it has 160 convolution operations, whereas VGG16 has only 13—VGG16 has more FLOPs but far fewer convolution operations, so its memory-access overhead is smaller and it ends up faster. ShuffleNetV2 is similar: its Split, Reshape, and Transpose operations take up about 20% of the time, yet none of them show up in FLOPs.

Another phenomenon worth noting: in AlexNet, the gemm operations account for only 8.2% of FLOPs but a full 67.7% of latency (unfortunately the experiment ran inside a docker container without permission to access the NVIDIA GPU Performance Counters, so I couldn’t dig further into the kernel).

Finally, let’s look at depthwise convolution. In the figure below, MobileNetV2_d uses depthwise convolution, while MobileNetV2_n replaces all depthwise convolutions with ordinary convolutions. Compared with MobileNetV2_n, MobileNetV2_d has 18× fewer FLOPs but less than 4× the actual speedup—a layer-by-layer comparison shows that depthwise convolution drastically cuts FLOPs but does not reduce memory-access overhead much:

Per-layer FLOPs and latency of MobileNetV2 — Per-layer FLOPs vs. latency comparison for MobileNetV2.

Since the time is mainly spent in the convolutional and fully connected layers, on the GPU platform you can use NVIDIA Nsight Systems or TensorRT’s built-in Profiler to do profiling, directly measure the time cost of each part, and then optimize in a targeted way—because optimizing different parts yields different gains for the whole, which is Amdahl’s law of acceleration:

\text{Speedup}_{\text{overall}}=\frac{1}{(1-F_{\text{enhanced}})+\dfrac{F_{\text{enhanced}}}{S_{\text{enhanced}}}}

where $F_{\text{enhanced}}$ is the fraction of the original execution time taken by the optimized part, and $S_{\text{enhanced}}$ is the speedup of that part itself.

Profiling with NVIDIA Nsight Systems — Profiling the inference process with NVIDIA Nsight Systems.

Space: Parameter Count and Peak Memory

Space efficiency mainly concerns the runtime memory footprint. It likewise depends on the network architecture and the software implementation (for example, some inference frameworks promptly free feature maps already consumed by the next layer, or use an inplace implementation for depthwise convolution, or do FP16 inference). Here we assume all parameters are stored and computed in 32-bit floating point, and discuss only the network architecture itself.

Memory footprint has two sources: first, the network’s own parameter count (occupying memory at runtime and disk at storage time); second, the runtime peak memory, mainly from the feature maps in flight, which depends not only on the network architecture but also on the input size (batch, height/width, channels). The academic literature discusses parameter count a lot and peak memory little, but for multi-branch architectures, computing at a node with many branches requires keeping a large number of feature maps in memory at once, so the peak memory can be large even if such a network’s own parameter count is not.

Like FLOPs, the memory footprint can also be computed fairly accurately without relying on the runtime platform. First the parameter count, which likewise comes mainly from the convolutional and fully connected layers. Convolutional layer (with bias):

\#\text{Params}=(K\times K \times C_{in}+1)\times C_{out}

Fully connected layer:

\#\text{Params}=(N_{in}+1)\times N_{out}

As for peak memory, assuming that during inference only the network weights and the currently needed feature maps are stored, the maximum runtime memory footprint is approximately the memory occupied by all weight parameters plus the memory occupied by the largest single group of feature maps in flight.

Once the metrics on both lines—task accuracy and runtime efficiency—are clearly defined, all the subsequent architecture design, training tricks, pruning, and quantization finally have a common yardstick to align against.

References

He, Kaiming, et al. Deep Residual Learning for Image Recognition. CVPR, 2016.
Howard, Andrew G., et al. MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. arXiv:1704.04861, 2017.
Sandler, Mark, et al. MobileNetV2: Inverted Residuals and Linear Bottlenecks. CVPR, 2018.
Howard, Andrew, et al. Searching for MobileNetV3. arXiv:1905.02244, 2019.
Paszke, Adam, et al. PyTorch: An Imperative Style, High-Performance Deep Learning Library. NeurIPS, 2019.
Abadi, Martín, et al. TensorFlow: A System for Large-Scale Machine Learning. OSDI, 2016.

Technology

2020 · 11 · 15