Skip to main content
SpringerLink
Log in

Comparative Study of Different Reduced Precision Techniques in Deep Neural Network

  • Conference paper
  • First Online:
Proceedings of International Conference on Big Data, Machine Learning and their Applications

Part of the book series: Lecture Notes in Networks and Systems ((LNNS,volume 150))

Abstract

There has been rising interest in reduced precision in the training of deep neural networks (DNNs) that is used from the single-precision (FP32) to different precision format (FP16, FP8, bfloat16) due to the rapid increase in model sizes, which require less representational space when stored in lower precision. However, training a DNN in reduced precision format (FP16, FP8, and bfloat16) is challenging, because the data format may be inadequate for representing the gradients during backpropagation. In this research paper, we compare the various novel approaches to train a DNN using the different reduced precision formats and explore the challenges that we get during the training of DNN in reduced precision. Besides, we also examine various layers in the neural network which make a significant reduction in the backpropagation and also observe that when should get sufficient precision.

Equal contribution

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 129.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 169.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Micikevicius, P., Narang, S., Alben, J., Diamos, G., Elsen, E., Garcia, D., et al. (2017). Mixed precision training. arXiv preprint arXiv:1710.03740.

  2. Wang, N., Choi, J., Brand, D., Chen, C.-Y., & Gopalakrishnan, K. (2018). Training deep neural networks with 8-bit floating point numbers. In Advances in neural information processing systems (pp. 7675–7684).

    Google Scholar 

  3. Mellempudi, N., Srinivasan, S., Das, D., & Kaul, B. (2019). Mixed precision training with 8-bit floating point. arXiv preprint arXiv:1905.12334.

  4. He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 770–778).

    Google Scholar 

  5. Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., et al. (2016). Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144.

  6. Graves, A., Mohamed, A.-R., & Hinton, G. (2013). Speech recognition with deep recurrent neural networks. In 2013 IEEE international conference on acoustics, speech and signal processing (pp. 6645–6649). IEEE.

    Google Scholar 

  7. Hannun, A., Case, C., Casper, J., Catanzaro, B., Diamos, G., Elsen, E., et al. (2014). Deep speech: Scaling up end-to-end speech recognition. arXiv preprint arXiv:1412.5567.

  8. Matt Post. (2018). A call for clarity in reporting bleu scores. arXiv preprint arXiv:1804.08771.

  9. Ioffe, S., & Szegedy, C. (2015). Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv e-prints.

    Google Scholar 

  10. Training With Mixed Precision. https://docs.nvidia.com/deeplearning/sdk/mixed-precision-training/index.html.

  11. Amodei, D., & Hernandez, D. AI and compute. https://openai.com/blog/ai-and-compute.

  12. Neural Network Terminology. https://www.cs.toronto.edu/~lczhang/360/lec/w02/terms.html.

  13. Krizhevsky, Sutskever, A., Hinton, I., & Geoffrey. (2012). ImageNet classification with deep convolutional neural networks. Neural Information Processing Systems,25. https://doi.org/10.1145/3065386.

  14. He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 770–778).

    Google Scholar 

  15. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016). Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2818–2826).

    Google Scholar 

  16. Wu, Y., Schuster, M., Chen, Z., Le, Q.V., Norouzi, M., Macherey, W., et al. (2016). Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144.

  17. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., et al. (2017). Attention is all you need. In Advances in neural information processing systems (pp. 5998–6008).

    Google Scholar 

  18. Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., et al. (2015). Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115(3), 211–252.

    Article  MathSciNet  Google Scholar 

  19. Sennrich, R., Haddow, B., & Birch, A. (2016). Edinburgh neural machine translation systems for WMT 16. arXiv preprint arXiv:1606.02891.

  20. Kalamkar, D., Mudigere, D., Mellempudi, N., Das, D., Banerjee, K., Avancha, S., et al. (2019). A study of bfloat16 for deep learning training. arXiv preprint arXiv:1905.12322.

  21. Gupta, S., Agrawal, A., Gopalakrishnan, K., & Narayanan, P. (2015). Deep learning with limited numerical precision. In International conference on machine learning (pp. 1737–1746).

    Google Scholar 

  22. Micikevicius, P., Narang, S., Alben, J., Diamos, G., Elsen, E., Garcia, D., et al. (2017). Mixed precision training. arXiv preprint arXiv:1710.03740.

  23. UrsKöster, Webb, T., Wang, X., Nassar, M., Bansal, A. K., Constable, W., et al. (2017). Flexpoint: An adaptive numerical format for efficient training of deep neural networks. In Advances in neural information processing systems (pp. 1742–1752).

    Google Scholar 

  24. Das, D., Mellempudi, N., Mudigere, D., Kalamkar, D., Avancha, S., Banerjee, K., et al. (2018). Mixed precision training of convolutional neural networks using integer operations. arXiv preprint arXiv:1802.00930.

  25. Banner, R., Hubara, I., Hoffer, E., & Soudry, D. (2018). Scalable methods for 8-bit training of neural networks. In Advances in neural information processing systems (pp. 5145–5153).

    Google Scholar 

  26. Das, D., Mellempudi, N., Mudigere, D., Kalamkar, D., Avancha, S., Banerjee, K., et al. (2018). Mixed precision training of convolutional neural networks using integer operations. arXiv preprint arXiv:1802.00930.

  27. Cambier, L., Bhiwandiwalla, A., Gong, T., Nekuii, M., Elibol, O. H., & Tang, H. (2020). Shifted and squeezed 8-bit floating point format for low-precision training of deep neural networks. arXiv preprint arXiv:2001.05674.

Download references

Author information

Authors and Affiliations

  1. Computer Engineering Department, NIT, Kurukshetra, Haryana, India

    Rishi Raj Gupta & Virender Ranga

Authors
  1. Rishi Raj Gupta
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Virender Ranga
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Virender Ranga .

Editor information

Editors and Affiliations

  1. Department of Computer Science and Engineering, ABES Engineering College, Ghaziabad, Uttar Pradesh, India

    Shailesh Tiwari

  2. Department of Information Systems, Institut Teknologi Sepuluh Nopember, Surabaya, Indonesia

    Erma Suryani

  3. Singapore Institute of Technology, Singapore, Singapore

    Andrew Keong Ng

  4. Department of Computer Science and Engineering, Motilal Nehru National Institute of Technology Allahabad, Prayagraj, Uttar Pradesh, India

    K. K. Mishra

  5. Department of Electrical Engineering, Motilal Nehru National Institute of Technology Allahabad, Prayagraj, Uttar Pradesh, India

    Nitin Singh

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Gupta, R.R., Ranga, V. (2021). Comparative Study of Different Reduced Precision Techniques in Deep Neural Network. In: Tiwari, S., Suryani, E., Ng, A.K., Mishra, K.K., Singh, N. (eds) Proceedings of International Conference on Big Data, Machine Learning and their Applications. Lecture Notes in Networks and Systems, vol 150. Springer, Singapore. https://doi.org/10.1007/978-981-15-8377-3_11

Download citation

Publish with us

Policies and ethics

Search

Navigation