One Weight Bitwidth to Rule Them All

Abstract

Weight quantization for deep ConvNets has shown promising results for applications such as image classification and semantic segmentation and is especially important for applications where memory storage is limited. However, when aiming for quantization without accuracy degradation, different tasks may end up with different bitwidths. This creates complexity for software and hardware support and the complexity accumulates when one considers mixed-precision quantization, in which case each layer’s weights use a different bitwidth. Our key insight is that optimizing for the least bitwidth subject to no accuracy degradation is not necessarily an optimal strategy. This is because one cannot decide optimality between two bitwidths if one has smaller model size while the other has better accuracy. In this work, we take the first step to understand if some weight bitwidth is better than others by aligning all to the same model size using a width-multiplier. Under this setting, somewhat surprisingly, we show that using a single bitwidth for the whole network can achieve better accuracy compared to mixed-precision quantization targeting zero accuracy degradation when both have the same model size. In particular, our results suggest that when the number of channels becomes a target hyperparameter, a single weight bitwidth throughout the network shows superior results for model compression.

Appendices

A Clipping Point for Quantization-aware Training

As mentioned earlier, ${\varvec{a}}\in \mathbb {R}^{C_{out}}$ denotes the vector of clipping factors which is selected to minimize $\Vert Q({\varvec{W}}_{i,:}) - {\varvec{W}}_{i,:} \Vert ^2_2$ by assuming ${\varvec{W}}_{i,:}~\sim \mathcal {N}(0, \sigma ^2{\varvec{I}})$. More specifically, we run simulations for weights drawn from a zero-mean Gaussian distribution with several variances and identify the best ${a}_i^*=\arg \min _{{a}_i}\Vert Q_{{a}_i}({\varvec{W}}_{i,:}) - {\varvec{W}}_{i,:} \Vert ^2_2$ empirically. According to our simulation, we find that one can infer ${a}_i$ from the sample mean $\bar{|{\varvec{W}}_{i,:}|}$, which is shown in Fig. 7. As a result, for the different precision values considered, we find $c = \frac{\bar{|{\varvec{W}}_{i,:}|}}{{a}_i^*}$ via simulation and use the obtained c to calculate ${a}_i$ on-the-fly throughout training.

Fig. 7.

Finding best ${a}_i$ for different precision values empirically through simulation using Gaussian with various $\sigma ^2$.

B Network Architectures

For the experiments in Sect. 4.2, the ResNets used are detailed in Table 4. Specifically, for the points in Fig. 1a, we consider ResNet20 to ResNet56 with width-multipliers of $0.5\times , 1\times , 1.5\times$, and $2\times$ for the 4-bit case. Based on these values, we consider additional width-multipliers $2.4\times$ and $2.8\times$ for the 2-bit case and $2.5\times , 3\times , 3.5\times ,$ and $3.9\times$ for the 1-bit case. We note that the right-most points in Fig. 1a is a $10\times$ ResNet26 for the 4 bits case. On the other hand, VGG11 is detailed in Table 6 for which we consider width-multipliers from $0.25\times$ to $2\times$ with a step of 0.25 for the 4 bits case (blue dots in Fig. 1c). The architecture details for Inv-ResNet26 used in Fig. 2b is shown in Table 5. The architecture of MobileNetV2 used in the CIFAR-100 experiments follows the original MobileNetV2 (Table 2 in [22]) but we change the stride of all the bottleneck blocks to 1 except for the fifth bottleneck block, which has a stride of 2. As a result, we down-sample the image twice in total, which resembles the ResNet design for the CIFAR experiments [8]. Similar to VGG11, we consider width-multipliers from $0.25\times$ to $2\times$ with a step of 0.25 for MobileNetV2 for the 4 bits case (blue dots in Fig. 1b).

Table 4. ResNet20 to ResNet56

Table 5. Inv-ResNet26

Table 6. VGGs

C Proof for Proposition 5.1

Based on the definition of variance, we have:

$$\begin{aligned} \mathrm {Var}(\frac{1}{d}\sum _{i=1}^d |{\varvec{w}}_i|)&:= \mathbb {E}\left[ \left( \frac{1}{d}\sum _{i=1}^d |{\varvec{w}}_i| \right) ^2 - \left( \mathbb {E}\frac{1}{d}\sum _{i=1}^d |{\varvec{w}}_i| \right) ^2\right] \\&= \mathbb {E}\left[ \left( \frac{1}{d}\sum _{i=1}^d |{\varvec{w}}_i| \right) ^2 - \frac{2\sigma ^2}{\pi } \right] \\&= \frac{1}{d^2} \mathbb {E}\left( \sum _{i=1}^d |{\varvec{w}}_i| \right) ^2 - \frac{2\sigma ^2}{\pi }\\&= \frac{\sigma ^2}{d} + \frac{d-1}{d} \rho \sigma ^2 - \frac{2\sigma ^2}{\pi }. \end{aligned}$$