Abstract
Emerging edge intelligence applications require the server to retrain and update deep neural networks deployed on remote edge nodes to leverage newly collected data samples. Unfortunately, it may be impossible in practice to continuously send fully updated weights to these edge nodes due to the highly constrained communication resource. In this paper, we propose the weight-wise deep partial updating paradigm, which smartly selects a small subset of weights to update in each server-to-edge communication round, while achieving a similar performance compared to full updating. Our method is established through analytically upper-bounding the loss difference between partial updating and full updating, and only updates the weights which make the largest contributions to the upper bound. Extensive experimental results demonstrate the efficacy of our partial updating methodology which achieves a high inference accuracy while updating a rather small number of weights.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Ash, J.T., Zhang, C., Krishnamurthy, A., Langford, J., Agarwal, A.: Deep batch active learning by diverse, uncertain gradient lower bounds. In: International Conference on Learning Representations (ICLR) (2020). https://openreview.net/forum?id=ryghZJBKPS
Augustin, A., Yi, J., Clausen, T., Townsley, W.M.: A study of lora: long range & low power networks for the internet of things. Sensors 16(9) (2016). https://doi.org/10.3390/s16091466. https://www.mdpi.com/1424-8220/16/9/1466
Bhandare, A., et al.: Efficient 8-bit quantization of transformer neural machine language translation model. In: International Conference on Machine Learning (ICML), Joint Workshop on On-Device Machine Learning & Compact Deep Neural Network Representations (2019)
Brown, S., Sreenan, C.: Updating software in wireless sensor networks: a survey. Department of Computer Science, National University of Ireland, Maynooth, Technical report, pp. 1–14 (2006)
Courbariaux, M., Bengio, Y., David, J.P.: Binaryconnect: training deep neural networks with binary weights during propagations. In: Annual Conference on Neural Information Processing Systems (NeurIPS) (2015)
Evci, U., Gale, T., Menick, J., Castro, P.S., Elsen, E.: Rigging the lottery: making all tickets winners. In: International Conference on Machine Learning (ICML) (2021)
Frankle, J., Carbin, M.: The lottery ticket hypothesis: finding sparse, trainable neural networks. In: International Conference on Learning Representations (ICLR) (2019). https://openreview.net/forum?id=rJl-b3RcF7
Guo, T.: Cloud-based or on-device: an empirical study of mobile deep inference. In: 2018 IEEE International Conference on Cloud Engineering, IC2E 2018, Orlando, FL, USA, 17–20 April 2018, pp. 184–190. IEEE Computer Society (2018). https://doi.org/10.1109/IC2E.2018.00042
Han, S., Mao, H., Dally, W.J.: Deep compression: compressing deep neural networks with pruning, trained quantization and Huffman coding. In: International Conference on Learning Representations (ICLR) (2016)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2016)
Horowitz, M.: 1.1 computing’s energy problem (and what we can do about it). In: 2014 IEEE International Solid-State Circuits Conference Digest of Technical Papers (ISSCC) (2014)
Howard, A.G., et al.: Mobilenets: efficient convolutional neural networks for mobile vision applications. CoRR (2017)
Jung, S., Ahn, H., Cha, S., Moon, T.: Adaptive group sparse regularization for continual learning. CoRR (2020)
Kairouz, P., et al.: Advances and open problems in federated learning. CoRR (2019)
Krizhevsky, A., Nair, V., Hinton, G.: Cifar10/100 (Canadian institute for advanced research) (2009). http://www.cs.toronto.edu/kriz/cifar.html
LeCun, Y., Cortes, C.: MNIST handwritten digit database (2010). http://yann.lecun.com/exdb/mnist/
Lee, J., et al.: On-device neural net inference with mobile GPUs. CoRR (2019)
Li, A., et al.: LotteryFL: personalized and communication-efficient federated learning with lottery ticket hypothesis on non-IID datasets. CoRR (2020)
Lin, Y., Han, S., Mao, H., Wang, Y., Dally, W.J.: Deep gradient compression: Reducing the communication bandwidth for distributed training. In: International Conference on Learning Representations (ICLR) (2018). https://openreview.net/forum?id=SkhQHMW0W
Meng, Z., et al.: A two-stage optimized next-view planning framework for 3-D unknown environment exploration, and structural reconstruction. IEEE Robot. Autom. Lett. 2(3), 1680–1687 (2017)
Meyer, M., et al.: Event-triggered natural hazard monitoring with convolutional neural networks on the edge. In: International Conference on Information Processing in Sensor Networks (IPSN), IPSN 2019. Association for Computing Machinery (2019)
Nesterov, Y.: Introductory lectures on convex programming volume I: basic course. Lecture Notes 1, 25 (1998)
Paszke, A., et al.: Automatic differentiation in pytorch. In: NIPS Autodiff Workshop: The Future of Gradient-based Machine Learning Software and Techniques (2017)
Peste, A., Iofinova, E., Vladu, A., Alistarh, D.: AC/DC: alternating compressed/decompressed training of deep neural networks. In: Annual Conference on Neural Information Processing Systems (NeurIPS) (2021)
Pytorch: Pytorch example on resnet (2019). https://github.com/pytorch/vision/blob/master/torchvision/models/resnet.py. Accessed 15 Oct 2019
Qu, Z., Zhou, Z., Cheng, Y., Thiele, L.: Adaptive loss-aware quantization for multi-bit networks. In: The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2020)
Raghu, A., Raghu, M., Bengio, S., Vinyals, O.: Rapid learning or feature reuse? towards understanding the effectiveness of MAML. In: International Conference on Learning Representations (ICLR) (2020). https://openreview.net/forum?id=rkgMkCEtPB
Rastegari, M., Ordonez, V., Redmon, J., Farhadi, A.: XNOR-Net: ImageNet classification using binary convolutional neural networks. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9908, pp. 525–542. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46493-0_32
Renda, A., Frankle, J., Carbin, M.: Comparing fine-tuning and rewinding in neural network pruning. In: International Conference on Learning Representations (ICLR) (2020). https://openreview.net/forum?id=S1gSj0NKvB
Russakovsky, O., et al.: Imagenet large scale visual recognition challenge. Int. J. Comput. Vision 115(3), 211–252 (2015)
Shen, Z., Liu, Z., Qin, J., Savvides, M., Cheng, K.: Partial is better than all: revisiting fine-tuning strategy for few-shot learning. In: The AAAI Conference on Artificial Intelligence (AAAI) (2021)
Shokri, R., Shmatikov, V.: Privacy-preserving deep learning. In: Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security, CCS 2015 (2015). https://doi.org/10.1145/2810103.2813687
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: International Conference on Learning Representations (ICLR) (2015)
Stowell, D., Wood, M., Pamuła, H., Stylianou, Y., Glotin, H.: Automatic acoustic detection of birds through deep learning: the first bird audio detection challenge. Methods Ecol. Evol. 10, 368–380 (2018). https://doi.org/10.1111/2041-210X.13103
Varshney, A.: Enabling sustainable networked embedded systems. Ph.D. thesis, Uppsala University, Division of Computer Systems, Computer Architecture and Computer Communication (2018)
Wang, Z., K, K., Mayhew, S., Roth, D.: Extending multilingual bert to low-resource languages. CoRR (2020)
Yoon, J., Yang, E., Lee, J., Hwang, S.J.: Lifelong learning with dynamically expandable networks. In: International Conference on Learning Representations (ICLR) (2018). https://openreview.net/forum?id=Sk7KsfW0-
Zhou, H., Lan, J., Liu, R., Yosinski, J.: Deconstructing lottery tickets: zeros, signs, and the supermask. In: Annual Conference on Neural Information Processing Systems (NeurIPS) (2019)
Zhuang, F., et al.: A comprehensive survey on transfer learning. CoRR (2019)
Acknowledgement
Part of Zhongnan Qu and Lothar Thiele’s work was supported by the Swiss National Science Foundation in the context of the NCCR Automation. Part of Cong Liu’s work was supported by NSF CNS 2135625, CPS 2038727, CNS Career 1750263, and a Darpa Shell grant.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Qu, Z., Liu, C., Thiele, L. (2022). Deep Partial Updating: Towards Communication Efficient Updating for On-Device Inference. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds) Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol 13671. Springer, Cham. https://doi.org/10.1007/978-3-031-20083-0_9
Download citation
DOI: https://doi.org/10.1007/978-3-031-20083-0_9
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-20082-3
Online ISBN: 978-3-031-20083-0
eBook Packages: Computer ScienceComputer Science (R0)