Abstract
Recently, training CNNs (Convolutional Neural Networks) on-chip has attracted much attention. With the development of the CNNs, the proportion of the BN (Batch Normalization) layer’s execution time is increasing and even exceeds the convolutional layer. The BN layer can accelerate the convergence of training. However, little work focus on the efficient hardware implementation of BN layer computation in training. In this work, we propose an accelerator, PRBN, which supports the BN and convolution computation in training. In our design, a systolic array is used for accelerating the convolution and matrix multiplication in training, and RBN (Range Batch Normalization) array based on hardware-friendly RBN algorithm is implemented for computation of BN layers. We implement PRBN on FPGA PYNQ-Z1. The working frequency of it is 50 MHz and the power of it is 0.346 W. The experimental results show that when compared with CPU i5-7500, PRBN can achieve 3.3\(\times \) speedup in performance and 8.9\(\times \) improvement in energy efficiency.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
He, K., et al.: Deep residual learning for image recognition. In: Computer Vision and Pattern Recognition, pp. 770–778 (2016)
Bottou, L.: Stochastic gradient descent tricks. In: Montavon, G., Orr, G.B., Müller, K.-R. (eds.) Neural Networks: Tricks of the Trade. LNCS, vol. 7700, pp. 421–436. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-35289-8_25
Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by reducing internal covariate shift. In: International Conference on Machine Learning, pp. 448–456 (2015)
Huang, G., et al.: Densely connected convolutional networks. In: Computer Vision and Pattern Recognition, pp. 2261–2269 (2017)
Jung, W., et al.: Restructuring batch normalization to accelerate CNN training. arXiv preprint arXiv:1807.01702 (2018)
Howard, A.G., et al.: Mobilenets: efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 (2017)
Banner, R., et al.: Scalable methods for 8-bit training of neural networks. In: Neural Information Processing Systems, pp. 5145–5153 (2018)
Kung, H.T., Leiserson, C.E.: Systolic arrays (for VLSI). In: Sparse Matrix Proceedings, vol. 1, pp. 256–282. Society for Industrial and Applied Mathematics, Philadelphia (1978)
LeCun, Y., et al.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)
Yang, Z., et al.: Bactran: a hardware batch normalization implementation for CNN training engine. IEEE Embed. Syst. Lett. 1 (2020)
Yang, Z., et al.: Systolic array based accelerator and algorithm mapping for deep learning algorithms. In: Zhang, F., Zhai, J., Snir, M., Jin, H., Kasahara, H., Valero, M. (eds.) NPC 2018. LNCS, vol. 11276, pp. 153–158. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-05677-3_16
Whatmough, P.N., et al.: FixyNN: efficient hardware for mobile computer vision via transfer learning. arXiv preprint arXiv:1902.11128 (2019)
Xiong, F., et al.: Towards efficient compact network training on edge-devices. IEEE Computer Society Annual Symposium on VLSI, pp. 61–67 (2019)
Sledevic, T.: Adaptation of convolution and batch normalization layer for CNN implementation on FPGA. In: 2019 Open Conference of Electrical, Electronic and Information Sciences (eStream), pp. 1–4. IEEE (2019)
Xie, F., et al.: Edge intelligence based co-training of CNN. In: 2019 14th International Conference on Computer Science & Education (ICCSE), pp. 830–834. IEEE (2019)
Donahue, J., et al.: DeCAF: a deep convolutional activation feature for generic visual recognition, vol. 50, no. 1, pp. 1–647 (2013)
Kukreja, N., et al.: Training on the edge: the why and the how. In: International Parallel and Distributed Processing Symposium, pp. 899–903 (2019)
Paul, G., Irvine, J.: Privacy implications of wearable health devices. In: Proceedings of the 7th International Conference on Security of Information and Networks. ACM (2014)
Deng, S., et al.: Edge intelligence: the confluence of edge computing and artificial intelligence. Networking and Internet Architecture. arXiv (2019)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Yang, Z., Wang, L., Zhang, X., Ding, D., Xie, C., Luo, L. (2020). PRBN: A Pipelined Implementation of RBN for CNN Training. In: Dong, D., Gong, X., Li, C., Li, D., Wu, J. (eds) Advanced Computer Architecture. ACA 2020. Communications in Computer and Information Science, vol 1256. Springer, Singapore. https://doi.org/10.1007/978-981-15-8135-9_9
Download citation
DOI: https://doi.org/10.1007/978-981-15-8135-9_9
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-15-8134-2
Online ISBN: 978-981-15-8135-9
eBook Packages: Computer ScienceComputer Science (R0)
