Advertisement

Generative Low-Bitwidth Data Free Quantization

Conference paper
  • 1k Downloads
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12357)

Abstract

Neural network quantization is an effective way to compress deep models and improve their execution latency and energy efficiency, so that they can be deployed on mobile or embedded devices. Existing quantization methods require original data for calibration or fine-tuning to get better performance. However, in many real-world scenarios, the data may not be available due to confidential or private issues, thereby making existing quantization methods not applicable. Moreover, due to the absence of original data, the recently developed generative adversarial networks (GANs) cannot be applied to generate data. Although the full-precision model may contain rich data information, such information alone is hard to exploit for recovering the original data or generating new meaningful data. In this paper, we investigate a simple-yet-effective method called Generative Low-bitwidth Data Free Quantization (GDFQ) to remove the data dependence burden. Specifically, we propose a knowledge matching generator to produce meaningful fake data by exploiting classification boundary knowledge and distribution information in the pre-trained model. With the help of generated data, we can quantize a model by learning knowledge from the pre-trained model. Extensive experiments on three data sets demonstrate the effectiveness of our method. More critically, our method achieves much higher accuracy on 4-bit quantization than the existing data free quantization method. Code is available at https://github.com/xushoukai/GDFQ.

Keywords

Data free compression Low-bitwidth quantization Knowledge matching generator 

Notes

Acknowledgements

This work was partially supported by the Key-Area Research and Development Program of Guangdong Province 2018B010107001, Program for Guangdong Introducing Innovative and Entrepreneurial Teams 2017ZT07X183, Fundamental Research Funds for the Central Universities D2191240.

Supplementary material

504453_1_En_1_MOESM1_ESM.pdf (137 kb)
Supplementary material 1 (pdf 137 KB)

References

  1. 1.
    Abadi, M., et al.: Tensorflow: a system for large-scale machine learning. In: 12th \(\{\)USENIX\(\}\) Symposium on Operating Systems Design and Implementation (\(\{\)OSDI\(\}\) 16), pp. 265–283 (2016)Google Scholar
  2. 2.
    Banner, R., Nahshan, Y., Hoffer, E., Soudry, D.: Aciq: analytical clipping for integer quantization of neural networks (2018)Google Scholar
  3. 3.
    Banner, R., Nahshan, Y., Soudry, D.: Post training 4-bit quantization of convolutional networks for rapid-deployment. In: Proceedings of Advance Neural Information Processing System (2019)Google Scholar
  4. 4.
    Cai, Y., Yao, Z., Dong, Z., Gholami, A., Mahoney, M.W., Keutzer, K.: Zeroq: a novel zero shot quantization framework. In: Proceedings of IEEE Conference on Computer Vision Pattern Recognition (2020)Google Scholar
  5. 5.
    Cai, Z., He, X., Sun, J., Vasconcelos, N.: Deep learning with low precision by half-wave gaussian quantization. In: Proceedings of IEEE Conference on Computer Vision Pattern Recognition (2017)Google Scholar
  6. 6.
    Cao, J., Guo, Y., Wu, Q., Shen, C., Tan, M.: Adversarial learning with local coordinate coding. In: Proceedings of International Conference Machine Learning (2018)Google Scholar
  7. 7.
    Cao, J., Mo, L., Zhang, Y., Jia, K., Shen, C., Tan, M.: Multi-marginal wasserstein gan. In: Proceedings of Advances in Neural Information Processing Systems (2019)Google Scholar
  8. 8.
    Chen, H., et al.: Data-free learning of student networks. In: Proceedings of the IEEE International Conference on Computer Vision (2019)Google Scholar
  9. 9.
    Chen, T., et al.: Mxnet: a flexible and efficient machine learning library for heterogeneous distributed systems. arXiv preprint arXiv:1512.01274 (2015)
  10. 10.
    Choi, J., Wang, Z., Venkataramani, S., Chuang, P.I.J., Srinivasan, V., Gopalakrishnan, K.: Pact: Parameterized clipping activation for quantized neural networks. arXiv preprint arXiv:1805.06085 (2018)
  11. 11.
    Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: a large-scale hierarchical image database. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (2009)Google Scholar
  12. 12.
    Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)
  13. 13.
    Esser, S.K., McKinstry, J.L., Bablani, D., Appuswamy, R., Modha, D.S.: Learned step size quantization. In: Proceedings of International Conference on Learning Representations (2020)Google Scholar
  14. 14.
    Goodfellow, I., et al.: Generative adversarial nets. In: Proceedings of Advances in Neural Information Processing Systems (2014)Google Scholar
  15. 15.
    Guo, Y., et al.: Nat: neural architecture transformer for accurate and compact architectures. In: Proceedings of Advances in Neural Information Processing Systems (2019)Google Scholar
  16. 16.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (2016)Google Scholar
  17. 17.
    Hinton, G.E., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. arXiv:1503.02531 (2015)
  18. 18.
    Hubara, I., Courbariaux, M., Soudry, D., El-Yaniv, R., Bengio, Y.: Binarized neural networks. In: Proceedings of Advances in Neural Information Processing Systems (2016)Google Scholar
  19. 19.
    Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167 (2015)
  20. 20.
    Jacob, B., et al.: Quantization and training of neural networks for efficient integer-arithmetic-only inference. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (2018)Google Scholar
  21. 21.
    Jung, S., et al.: Learning to quantize deep networks by optimizing quantization intervals with task loss. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (2019)Google Scholar
  22. 22.
    Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: Bengio, Y., LeCun, Y. (eds.) Proceedings of International Conference on Learning Representations (2015)Google Scholar
  23. 23.
    Krizhevsky, A., Hinton, G., et al.: Learning multiple layers of features from tiny images (2009)Google Scholar
  24. 24.
    Lin, J., Gan, C., Han, S.: Defensive quantization: when efficiency meets robustness. arXiv preprint arXiv:1904.08444 (2019)
  25. 25.
    Lopes, R.G., Fenu, S., Starner, T.: Data-free knowledge distillation for deep neural networks. arXiv preprint arXiv:1710.07535 (2017)
  26. 26.
    Louizos, C., Reisser, M., Blankevoort, T., Gavves, E., Welling, M.: Relaxed quantization for discretized neural networks. In: Proceedings of International Conference on Learning Representations (2019)Google Scholar
  27. 27.
    Micaelli, P., Storkey, A.: Zero-shot knowledge transfer via adversarial belief matching. arXiv:1905.09768 (2019)
  28. 28.
    Mikolov, T., Karafiát, M., Burget, L., Černockỳ, J., Khudanpur, S.: Recurrent neural network based language model. In: Conference of the International Speech Communication Association (ISCA) (2010)Google Scholar
  29. 29.
    Miyato, T., Kataoka, T., Koyama, M., Yoshida, Y.: Spectral normalization for generative adversarial networks. arXiv preprint arXiv:1802.05957 (2018)
  30. 30.
    Nagel, M., Baalen, M.v., Blankevoort, T., Welling, M.: Data-free quantization through weight equalization and bias correction. In: Proceedings of the IEEE International Conference on Computer Vision (2019)Google Scholar
  31. 31.
    Nayak, G.K., Mopuri, K.R., Shaj, V., Babu, R.V., Chakraborty, A.: Zero-shot knowledge distillation in deep networks. In: Proceedings of the International Conference on Machine Learning (2019)Google Scholar
  32. 32.
    Nesterov, Y.E.: A method for solving the convex programming problem with convergence rate o (1/k\(^{\wedge }\) 2). In: Proceedings of the USSR Academy of Sciences, vol. 269, pp. 543–547 (1983)Google Scholar
  33. 33.
    Odena, A., Olah, C., Shlens, J.: Conditional image synthesis with auxiliary classifier GANs. In: Proceedings of International Conference on Machine Learning (2017)Google Scholar
  34. 34.
    Paszke, A., Gross, S., Chintala, S., Chanan, G.: Pytorch: tensors and dynamic neural networks in python with strong GPU acceleration. PyTorch: Tensors Dynamic Neural Networks in Python with strong GPU Acceleration, 6 (2017)Google Scholar
  35. 35.
    Rastegari, M., Ordonez, V., Redmon, J., Farhadi, A.: XNOR-Net: imagenet classification using binary convolutional neural networks. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9908, pp. 525–542. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-46493-0_32CrossRefGoogle Scholar
  36. 36.
    Sak, H., Senior, A.W., Beaufays, F.: Long short-term memory recurrent neural network architectures for large scale acoustic modeling. In: Conference of the International Speech Communication Association (ISCA), pp. 338–342 (2014)Google Scholar
  37. 37.
    Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., Chen, L.C.: Mobilenetv 2: inverted residuals and linear bottlenecks. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (2018)Google Scholar
  38. 38.
    Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: Bengio, Y., LeCun, Y. (eds.) Proceedings of International Conference on Learning Representations (2015)Google Scholar
  39. 39.
    Sung, W., Shin, S., Hwang, K.: Resiliency of deep neural networks under quantization. arXiv preprint arXiv:1511.06488 (2015)
  40. 40.
    Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the inception architecture for computer vision. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (2016)Google Scholar
  41. 41.
    Yang, J., et al.: Quantization networks. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (2019)Google Scholar
  42. 42.
    Yoo, J., Cho, M., Kim, T., Kang, U.: Knowledge extraction with no observable data. In: Proceedings of Advances in Neural Information Processing Systems, pp. 2701–2710 (2019)Google Scholar
  43. 43.
    Zeng, R., et al.: Graph convolutional networks for temporal action localization. In: Proceedings of the IEEE International Conference on Computer Vision (2019)Google Scholar
  44. 44.
    Zhang, C., Bengio, S., Hardt, M., Recht, B., Vinyals, O.: Understanding deep learning requires rethinking generalization. In: Proceedings of International Conference on Learning Representations (2017)Google Scholar
  45. 45.
    Zhang, D., Yang, J., Ye, D., Hua, G.: LQ-Nets: learned quantization for highly accurate and compact deep neural networks. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11212, pp. 373–390. Springer, Cham (2018).  https://doi.org/10.1007/978-3-030-01237-3_23CrossRefGoogle Scholar
  46. 46.
    Zhang, Y., et al.: From whole slide imaging to microscopy: deep microscopy adaptation network for histopathology cancer image classification. In: Shen, D., et al. (eds.) MICCAI 2019. LNCS, vol. 11764, pp. 360–368. Springer, Cham (2019).  https://doi.org/10.1007/978-3-030-32239-7_40CrossRefGoogle Scholar
  47. 47.
    Zhang, Y., Zhao, P., Wu, Q., Li, B., Huang, J., Tan, M.: Cost-sensitive portfolio selection via deep reinforcement learning. IEEE Trans. Knowl. Data Eng. (2020)Google Scholar
  48. 48.
    Zhao, R., Hu, Y., Dotzel, J., De Sa, C., Zhang, Z.: Improving neural network quantization without retraining using outlier channel splitting. In: Proceedings of the International Conference on Machine Learning (2019)Google Scholar
  49. 49.
    Zhou, S., Wu, Y., Ni, Z., Zhou, X., Wen, H., Zou, Y.: Dorefa-net: training low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint arXiv:1606.06160 (2016)
  50. 50.
    Zhuang, B., Liu, L., Tan, M., Shen, C., Reid, I.: Training quantized neural networks with a full-precision auxiliary module. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (2020)Google Scholar
  51. 51.
    Zhuang, B., Shen, C., Tan, M., Liu, L., Reid, I.: Towards effective low-bitwidth convolutional neural networks. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (2018)Google Scholar
  52. 52.
    Zhuang, B., Shen, C., Tan, M., Liu, L., Reid, I.: Structured binary neural networks for accurate image classification and semantic segmentation. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (2019)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.South China University of TechnologyGuangzhouChina
  2. 2.PengCheng LaboratoryShenzhenChina
  3. 3.Monash UniversityMelbourneAustralia

Personalised recommendations