Abstract
Data-free quantization is an effective way to compress deep neural networks under the situation where training data is unavailable, due to data privacy and security issues. Although Off-the-shelf data-free quantization methods achieve the relatively same accuracy as the fully-precision (FP) models for high-bit (e.g., 8-bit) quantization, low-bit quantization performance drops significantly to restrict their extensive applications. In this paper, we propose a novel data-free low-bit quantization method via Dynamic Multi-teacher Knowledge Distillation (DMKD) to improve the performance of low-bit quantization models. In particular, we first introduce a generator to synthesize the training data based on the input of random noise. The low-bit quantization models are then trained on these synthetic images by the dynamic knowledge from the FP model and the high-bit quantization models, which are balanced by learnable loss weight factors. The factors are controlled by a tiny learnable FP network to adaptively allocate the balanced weights for the knowledge from the FP model and the high-bit quantization models during training. For inference, we only kept the low-bit quantization model by safely removing other additional networks, such as the generator and the tiny model. Extensive experiments demonstrate the effectiveness of DMKD for low-bit quantization of widely-used convolutional neural networks (CNNs) on different benchmark datasets. Our DMKD ooon methods.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Ba, J., Caruana, R.: Do deep nets really need to be deep? In: NeurIPS (2014)
Banner, R., Nahshan, Y., Soudry, D.: Post training 4-bit quantization of convolutional networks for rapid-deployment. In: NeurIPS, vol. 32 (2019)
Cai, Y., Yao, Z., Dong, Z., Gholami, A., Mahoney, M.W., Keutzer, K.: ZeroQ: a novel zero shot quantization framework. In: CVPR, pp. 13169–13178 (2020)
Chen, H., et al.: Data-free learning of student networks. In: ICCV (2019)
Choi, J., Wang, Z., Venkataramani, S., Chuang, P.I.J., Srinivasan, V., Gopalakrishnan, K.: Pact: parameterized clipping activation for quantized neural networks. arXiv preprint arXiv:1805.06085 (2018)
Choi, K., Hong, D., Park, N., Kim, Y., Lee, J.: Qimera: data-free quantization with synthetic boundary supporting samples. In: NeurIPS, vol. 34, pp. 14835–14847 (2021)
Choi, K., et al.: It’s all in the teacher: zero-shot quantization brought closer to the teacher. In: CVPR, pp. 8311–8321 (2022)
Choi, Y., Choi, J., El-Khamy, M., Lee, J.: Data-free network quantization with adversarial knowledge distillation. In: CVPR Workshops (2020)
Courbariaux, M., Hubara, I., Soudry, D., El-Yaniv, R., Bengio, Y.: Binarized neural networks: training deep neural networks with weights and activations constrained to +1 or -1. arXiv preprint arXiv:1602.02830 (2016)
Gong, L., et al.: Adaptive hierarchy-branch fusion for online knowledge distillation. In: AAAI (2023)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR (2016)
Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 (2015)
Jacob, B., et al.: Quantization and training of neural networks for efficient integer-arithmetic-only inference. In: CVPR, pp. 2704–2713 (2018)
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
Krizhevsky, A., Hinton, G., et al.: Learning multiple layers of features from tiny images (2009)
Li, Y., et al.: Micronet: improving image recognition with extremely low flops. In: ICCV, pp. 468–477 (2021)
Liu, Y., Zhang, W., Wang, J.: Zero-shot adversarial quantization. In: CVPR (2021)
Lopes, R.G., Fenu, S., Starner, T.: Data-free knowledge distillation for deep neural networks. arXiv preprint arXiv:1710.07535 (2017)
Nagel, M., Amjad, R.A., Van Baalen, M., Louizos, C., Blankevoort, T.: Up or down? Adaptive rounding for post-training quantization. In: ICML, pp. 7197–7206 (2020)
Nagel, M., Baalen, M.V., Blankevoort, T., Welling, M.: Data-free quantization through weight equalization and bias correction. In: ICCV, pp. 1325–1334 (2019)
Nayak, G.K., Mopuri, K.R., Shaj, V., Radhakrishnan, V.B., Chakraborty, A.: Zero-shot knowledge distillation in deep networks. In: ICML, pp. 4743–4751 (2019)
Paszke, A., Gross, S., Chintala, S., Chanan, G.: Pytorch: tensors and dynamic neural networks in python with strong GPU acceleration. PyTorch 6(3), 67 (2017)
Rastegari, M., Ordonez, V., Redmon, J., Farhadi, A.: XNOR-Net: ImageNet classification using binary convolutional neural networks. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9908, pp. 525–542. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46493-0_32
Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. In: NeurIPS, pp. 91–99 (2015)
Romero, A., Ballas, N., Kahou, S.E., Chassang, A., Gatta, C., Bengio, Y.: Fitnets: hints for thin deep nets. In: ICLR (2015)
Russakovsky, O., et al.: Imagenet large scale visual recognition challenge. IJCV 115, 211–252 (2015)
Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., Chen, L.C.: Mobilenetv2: inverted residuals and linear bottlenecks. In: CVPR, pp. 4510–4520 (2018)
Tailor, S.A., Fernandez-Marques, J., Lane, N.D.: Degree-quant: quantization-aware training for graph neural networks. arXiv preprint arXiv:2008.05000 (2020)
Wang, P., Chen, Q., He, X., Cheng, J.: Towards accurate post-training network quantization via bit-split and stitching. In: ICML, pp. 9847–9856 (2020)
Wei, X., Gong, R., Li, Y., Liu, X., Yu, F.: QDrop: randomly dropping quantization for extremely low-bit post-training quantization. arXiv preprint arXiv:2203.05740 (2022)
Xiang, L., Ding, G., Han, J.: Learning from multiple experts: self-paced knowledge distillation for long-tailed classification. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12350, pp. 247–263. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58558-7_15
Xu, S., et al.: Generative low-bitwidth data free quantization. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12357, pp. 1–17. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58610-2_1
Yin, H., et al.: Dreaming to distill: data-free knowledge transfer via deepinversion. In: CVPR, pp. 8715–8724 (2020)
You, S., Xu, C., Xu, C., Tao, D.: Learning from multiple teacher networks. In: KDD (2017)
Yu, S., Chen, J., Han, H., Jiang, S.: Data-free knowledge distillation via feature exchange and activation region constraint. In: CVPR, pp. 24266–24275 (2023)
Zagoruyko, S., Komodakis, N.: Paying more attention to attention: improving the performance of convolutional neural networks via attention transfer. In: ICLR (2017)
Zhang, X., et al.: Diversifying sample generation for accurate data-free quantization. In: CVPR, pp. 15658–15667 (2021)
Zhong, Y., et al.: IntraQ: learning synthetic images with intra-class heterogeneity for zero-shot network quantization. In: CVPR, pp. 12339–12348 (2022)
Zhou, P., Mai, L., Zhang, J., Xu, N., Wu, Z., Davis, L.S.: M2KD: multi-model and multi-level knowledge distillation for incremental learning. arXiv preprint arXiv:1904.01769 (2019)
Zhou, S., Wu, Y., Ni, Z., Zhou, X., Wen, H., Zou, Y.: DoReFa-Net: training low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint arXiv:1606.06160 (2016)
Zhu, B., Hofstee, P., Peltenburg, J., Lee, J., Alars, Z.: Autorecon: neural architecture search-based reconstruction for data-free compression. arXiv preprint arXiv:2105.12151 (2021)
Acknowledgements
This work is supported by the National Natural Science Foundation of China (No. 62102151), Shanghai Sailing Program (21YF1411200), CCF-Tencent Open Research Fund, the Open Research Fund of Key Laboratory of Advanced Theory and Application in Statistics and Data Science, Ministry of Education (KLATASDS2305), the Fundamental Research Funds for the Central Universities.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Huang, C., Lin, S., Zhang, Y., Li, K., Zhang, B. (2024). Data-Free Low-Bit Quantization via Dynamic Multi-teacher Knowledge Distillation. In: Liu, Q., et al. Pattern Recognition and Computer Vision. PRCV 2023. Lecture Notes in Computer Science, vol 14432. Springer, Singapore. https://doi.org/10.1007/978-981-99-8543-2_3
Download citation
DOI: https://doi.org/10.1007/978-981-99-8543-2_3
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-99-8542-5
Online ISBN: 978-981-99-8543-2
eBook Packages: Computer ScienceComputer Science (R0)