Skip to main content

PTQ4ViT: Post-training Quantization for Vision Transformers with Twin Uniform Quantization

  • Conference paper
  • First Online:
Computer Vision – ECCV 2022 (ECCV 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13672))

Included in the following conference series:

Abstract

Quantization is one of the most effective methods to compress neural networks, which has achieved great success on convolutional neural networks (CNNs). Recently, vision transformers have demonstrated great potential in computer vision. However, previous post-training quantization methods performed not well on vision transformer, resulting in more than 1% accuracy drop even in 8-bit quantization. Therefore, we analyze the problems of quantization on vision transformers. We observe the distributions of activation values after softmax and GELU functions are quite different from the Gaussian distribution. We also observe that common quantization metrics, such as MSE and cosine distance, are inaccurate to determine the optimal scaling factor. In this paper, we propose the twin uniform quantization method to reduce the quantization error on these activation values. And we propose to use a Hessian guided metric to evaluate different scaling factors, which improves the accuracy of calibration at a small cost. To enable the fast quantization of vision transformers, we develop an efficient framework, PTQ4ViT. Experiments show the quantized vision transformers achieve near-lossless prediction accuracy (less than 0.5% drop at 8-bit quantization) on the ImageNet classification task.

Z. Yuan and C. Xue—contribute equally to this paper.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 119.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    Code is in https://github.com/hahnyuan/PTQ4ViT.

  2. 2.

    The ground truth y is not available in PTQ, so we use the prediction of floating-point network \(y_{FP}\) to approximate it.

  3. 3.

    The derivation of it is in Appendix.

References

  1. Banner, R., Nahshan, Y., Soudry, D.: Post training 4-bit quantization of convolutional networks for rapid-deployment. In: Wallach, H.M., Larochelle, H., Beygelzimer, A., d’Alché-Buc, F., Fox, E.B., Garnett, R. (eds.) Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, 8–14 December 2019. Vancouver, BC, Canada, pp. 7948–7956 (2019). https://proceedings.neurips.cc/paper/2019/hash/c0a62e133894cdce435bcb4a5df1db2d-Abstract.html

  2. Brown, T.B., et al.: Language models are few-shot learners. In: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., Lin, H. (eds.) Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, 6–12 December 2020. virtual (2020). https://proceedings.neurips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html

  3. Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End-to-End object detection with transformers. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12346, pp. 213–229. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58452-8_13

    Chapter  Google Scholar 

  4. Choi, J., Wang, Z., Venkataramani, S., Chuang, P.I., Srinivasan, V., Gopalakrishnan, K.: PACT: parameterized clipping activation for quantized neural networks. CoRR abs/1805.06085 (2018). http://arxiv.org/abs/1805.06085

  5. Choukroun, Y., Kravchik, E., Yang, F., Kisilev, P.: Low-bit quantization of neural networks for efficient inference. In: 2019 IEEE/CVF International Conference on Computer Vision Workshops, ICCV Workshops 2019, Seoul, Korea (South), 27–28 October 2019, pp. 3009–3018. IEEE (2019). https://doi.org/10.1109/ICCVW.2019.00363

  6. Devlin, J., Chang, M., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: Burstein, J., Doran, C., Solorio, T. (eds.) Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2–7, 2019, Volume 1 (Long and Short Papers), pp. 4171–4186. Association for Computational Linguistics (2019). https://doi.org/10.18653/v1/n19-1423

  7. Dosovitskiy, A., et al.: An image is worth 16x16 words: transformers for image recognition at scale. In: 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, 3–7 May 2021. OpenReview.net (2021). https://openreview.net/forum?id=YicbFdNTTy

  8. Fang, J., Shafiee, A., Abdel-Aziz, H., Thorsley, D., Georgiadis, G., Hassoun, J.H.: Post-training piecewise linear quantization for deep neural networks. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12347, pp. 69–86. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58536-5_5

    Chapter  Google Scholar 

  9. Gholami, A., Kim, S., Dong, Z., Yao, Z., Mahoney, M.W., Keutzer, K.: A survey of quantization methods for efficient neural network inference. CoRR abs/2103.13630 (2021). https://arxiv.org/abs/2103.13630

  10. Guo, Y., Yao, A., Zhao, H., Chen, Y.: Network sketching: exploiting binary structure in deep cnns. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, 21–26 July 2017, pp. 4040–4048. IEEE Computer Society (2017). https://doi.org/10.1109/CVPR.2017.430, http://doi.ieeecomputersociety.org/10.1109/CVPR.2017.430

  11. Han, K., et al.: A survey on visual transformer. CoRR abs/2012.12556 (2020). https://arxiv.org/abs/2012.12556

  12. Huang, L., Tan, J., Liu, J., Yuan, J.: Hand-transformer: non-autoregressive structured modeling for 3D hand pose estimation. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12370, pp. 17–33. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58595-2_2

    Chapter  Google Scholar 

  13. Jia, D., et al.: Efficient vision transformers via fine-grained manifold distillation. CoRR abs/2107.01378 (2021). https://arxiv.org/abs/2107.01378

  14. Li, Y., et al.: BRECQ: pushing the limit of post-training quantization by block reconstruction. In: 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, 3–7 May 2021. OpenReview.net (2021). https://openreview.net/forum?id=POWv6hDd9XH

  15. Liu, Z., et al.: Swin transformer: hierarchical vision transformer using shifted windows. CoRR abs/2103.14030 (2021). https://arxiv.org/abs/2103.14030

  16. Liu, Z., Wang, Y., Han, K., Zhang, W., Ma, S., Gao, W.: Post-training quantization for vision transformer, pp. 28092–28103 (2021). https://proceedings.neurips.cc/paper/2021/hash/ec8956637a99787bd197eacd77acce5e-Abstract.html

  17. Nagel, M., van Baalen, M., Blankevoort, T., Welling, M.: Data-free quantization through weight equalization and bias correction. In: 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), 27 October - 2 November 2019, pp. 1325–1334. IEEE (2019). https://doi.org/10.1109/ICCV.2019.00141

  18. Prato, G., Charlaix, E., Rezagholizadeh, M.: Fully quantized transformer for machine translation. In: Cohn, T., He, Y., Liu, Y. (eds.) Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16–20 November 2020. Findings of ACL, vol. EMNLP 2020, pp. 1–14. Association for Computational Linguistics (2020). https://doi.org/10.18653/v1/2020.findings-emnlp.1

  19. Russakovsky, O., et al.: ImageNet large scale visual recognition challenge. Int. J. Comput. Vision 115(3), 211–252 (2015). https://doi.org/10.1007/s11263-015-0816-y

    Article  MathSciNet  Google Scholar 

  20. Shen, S., et al.: Q-BERT: hessian based ultra low precision quantization of BERT. In: The 34th AAAI Conference on Artificial Intelligence, AAAI 2020, The 32nd Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The 10th AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, 7–12 February 2020, pp. 8815–8821. AAAI Press (2020). https://aaai.org/ojs/index.php/AAAI/article/view/6409

  21. Tang, Y., et al.: Patch slimming for efficient vision transformers. CoRR abs/2106.02852 (2021). https://arxiv.org/abs/2106.02852

  22. Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., Jégou, H.: Training data-efficient image transformers & distillation through attention. In: Meila, M., Zhang, T. (eds.) Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18–24 July 2021, Virtual Event. Proceedings of Machine Learning Research, vol. 139, pp. 10347–10357. PMLR (2021). http://proceedings.mlr.press/v139/touvron21a.html

  23. Vaswani, A., et al.: Attention is all you need. In: Guyon, I., et al. (eds.) Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4–9 December 2017. Long Beach, CA, USA, pp. 5998–6008 (2017). https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html

  24. Wang, H., Zhu, Y., Adam, H., Yuille, A.L., Chen, L.: Max-deeplab: end-to-end panoptic segmentation with mask transformers. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, 19–25 June 2021, pp. 5463–5474. Computer Vision Foundation/IEEE (2021). https://openaccess.thecvf.com/content/CVPR2021/html/Wang_MaX-DeepLab_End-to-End_Panoptic_Segmentation_With_Mask_Transformers_CVPR_2021_paper.html

  25. Wang, W., et al.: Pyramid vision transformer: a versatile backbone for dense prediction without convolutions. CoRR abs/2102.12122 (2021). https://arxiv.org/abs/2102.12122

  26. Wightman, R.: Pytorch image models (2019). https://doi.org/10.5281/zenodo.4414861, https://github.com/rwightman/pytorch-image-models

  27. Wu, D., Tang, Q., Zhao, Y., Zhang, M., Fu, Y., Zhang, D.: Easyquant: post-training quantization via scale optimization. CoRR abs/2006.16669 (2020). https://arxiv.org/abs/2006.16669

  28. Zhang, D., Yang, J., Ye, D., Hua, G.: LQ-Nets: learned quantization for highly accurate and compact deep neural networks. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11212, pp. 373–390. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01237-3_23

    Chapter  Google Scholar 

Download references

Acknowledgements

This work is supported by National Key R &D Program of China (2020AAA0105200), NSF of China (61832020, 62032001, 92064006), Beijing Academy of Artificial Intelligence (BAAI), and 111 Project (B18001).

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Guangyu Sun .

Editor information

Editors and Affiliations

Appendix

Appendix

1.1 Derivation of Hessian guided metric

Hessian guided metric introduces as small an increment on task loss \(L = CE(\hat{y}, y)\) as possible, in which \(\hat{y}\) is the prediction of the quantized model and y is the ground truth. Here y is approximated by the prediction of the floating-point model \(y_{FP}\), since no labels of input data are available in PTQ.

Quantization introduces a small perturbation \(\epsilon \) on weight W, whose effect on task loss \(\mathbb {E}[L(W)]\) could be analyzed with Taylor series expansion,

$$\begin{aligned} \mathbb {E}[L(\hat{W})]-\mathbb {E}[L(W)]\approx \epsilon ^T \bar{g}^{(W)}+\frac{1}{2}\epsilon ^T \bar{H}^{(W)}\epsilon . \end{aligned}$$
(8)

Since the pretrained model has converged to a local optimum, The gradients \(\bar{g}^{(W)}\) is close to zero and could be ignored. The Hessian matrix \(\bar{H}^{(W)}\) on weight could be computed by

$$\begin{aligned} \dfrac{\partial ^2\,L}{\partial w_i \partial w_j} = \dfrac{\partial }{\partial w_j} (\sum _{k=1}^{m} \dfrac{\partial L}{\partial O_k} \dfrac{\partial O_k}{\partial w_i}) = \sum _{k=1}^{m} \dfrac{\partial L}{\partial O_k} \dfrac{\partial ^2 O_k}{\partial w_i \partial w_j} + \sum _{k,l=1}^{m} \dfrac{\partial O_k}{\partial w_i} \dfrac{\partial ^2\,L}{\partial O_k \partial O_l} \dfrac{\partial O_l}{\partial w_j}. \end{aligned}$$
(9)

\(O = W^TX \in R ^m\) is the output of the layer, and \(\dfrac{\partial ^2 O_k}{\partial w_i \partial w_j} = 0\). So the first term of Eq. (9) is zero, and \(\bar{H}^{(W)} = J_O(W)^T \bar{H}^{(O)} J_O(W)\). Therefore, Eq. 8 could be further written as,

$$\begin{aligned} \mathbb {E}[L(\hat{W})]-\mathbb {E}[L(W)]\approx \dfrac{1}{2}(J_{O}(W) \epsilon )^T \bar{H}^{(O)} J_{O}(W) \epsilon \approx \dfrac{1}{2}(\hat{O}-O)^T\bar{H}^{(O)} (\hat{O}-O) \end{aligned}$$
(10)

Following Liu et al. [14], we use the Diagonal Fisher Information Matrix to substitute \(\bar{H}^{(O)}\). The optimization is formulated as:

$$\begin{aligned} \mathop {min}_{\varDelta _W} \mathbb {E}[ (\hat{O}-O)^T \text {diag}((\dfrac{\partial L}{\partial O_1})^2, \cdots , (\dfrac{\partial L}{\partial O_m})^2) (\hat{O}-O)]. \end{aligned}$$
(11)

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Yuan, Z., Xue, C., Chen, Y., Wu, Q., Sun, G. (2022). PTQ4ViT: Post-training Quantization for Vision Transformers with Twin Uniform Quantization. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds) Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol 13672. Springer, Cham. https://doi.org/10.1007/978-3-031-19775-8_12

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-19775-8_12

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-19774-1

  • Online ISBN: 978-3-031-19775-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics