Skip to main content
SpringerLink
Log in

Hardware and Software Optimizations for Capsule Networks

  • Chapter
  • First Online:
Embedded Machine Learning for Cyber-Physical, IoT, and Edge Computing

  • 328 Accesses

Abstract

Among advanced Deep Neural Network models, Capsule Networks (CapsNets) have shown high learning and generalization capabilities for advanced tasks. Their capability to learn hierarchical information of features makes them appealing in many applications. However, their compute-intensive nature poses several challenges for their deployment on resource-constrained devices. This chapter provides an optimization flow at the software and at the hardware level for improving the energy efficiency of the CapsNets’ execution.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 149.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Hardcover Book
USD 199.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., Kudlur, M., Levenberg, J., Monga, R., Moore, S., Murray, D.G., Steiner, B., Tucker, P.A., Vasudevan, V., Warden, P., Wicke, M., Yu, Y., Zheng, X.: TensorFlow: A system for large-scale machine learning. In: Keeton, K., Roscoe, T. (eds.) 12th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2016, Savannah, GA, USA, November 2-4, 2016, pp. 265–283. USENIX Association (2016). https://www.usenix.org/conference/osdi16/technical-sessions/presentation/abadi

  2. Ahmed, K., Torresani, L.: Star-caps: Capsule networks with straight-through attentive routing. In: Wallach, H.M., Larochelle, H., Beygelzimer, A., d’Alché-Buc, F., Fox, E.B., Garnett, R. (eds.) Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, pp. 9098–9107 (2019). https://proceedings.neurips.cc/paper/2019/hash/cf040fc71060367913e81ac1eb050aea-Abstract.html

  3. Capra, M., Bussolino, B., Marchisio, A., Shafique, M., Masera, G., Martina, M.: An updated survey of efficient hardware architectures for accelerating deep convolutional neural networks. Future Int. 12(7), 113 (2020). https://doi.org/10.3390/fi12070113

    Article  Google Scholar 

  4. Choi, J., Seo, H., Im, S., Kang, M.: Attention routing between capsules. In: 2019 IEEE/CVF International Conference on Computer Vision Workshops, ICCV Workshops 2019, Seoul, Korea (South), October 27–28, 2019, pp. 1981–1989. IEEE (2019). https://doi.org/10.1109/ICCVW.2019.00247

  5. Deb, K., Agrawal, S., Pratap, A., Meyarivan, T.: A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE Trans. Evol. Comput. 6(2), 182–197 (2002). https://doi.org/10.1109/4235.996017

    Article  Google Scholar 

  6. Devarakonda, A., Naumov, M., Garland, M.: ADABATCH: Adaptive batch sizes for training deep neural networks. CoRR abs/1712.02029 (2017). http://arxiv.org/abs/1712.02029

  7. Gu, J., Tresp, V.: Improving the robustness of capsule networks to image affine transformations. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, June 13–19, 2020, pp. 7283–7291. Computer Vision Foundation/IEEE (2020). https://doi.org/10.1109/CVPR42600.2020.00731. https://openaccess.thecvf.com/content_CVPR_2020/html/Gu_Improving_the_Robustness_of_Capsule_Networks_to_Image_Affine_Transformations_CVPR_2020_paper.html

  8. Hahn, T., Pyeon, M., Kim, G.: Self-routing capsule networks. In: Wallach, H.M., Larochelle, H., Beygelzimer, A., d’Alché-Buc, F., Fox, E.B., Garnett, R. (eds.) Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, pp. 7656–7665 (2019). https://proceedings.neurips.cc/paper/2019/hash/e46bc064f8e92ac2c404b9871b2a4ef2-Abstract.html

  9. Hanif, M.A., Hafiz, R., Shafique, M.: Error resilience analysis for systematically employing approximate computing in convolutional neural networks. In: Madsen, J., Coskun, A.K. (eds.) 2018 Design, Automation & Test in Europe Conference & Exhibition, DATE 2018, Dresden, March 19–23, 2018, pp. 913–916. IEEE (2018). https://doi.org/10.23919/DATE.2018.8342139.

  10. Hanif, M.A., Marchisio, A., Arif, T., Hafiz, R., Rehman, S., Shafique, M.: X-DNNs: systematic cross-layer approximations for energy-efficient deep neural networks. J. Low Power Electron. 14(4), 520–534 (2018). https://doi.org/10.1166/jolpe.2018.1575

    Article  Google Scholar 

  11. Hinton, G.E., Krizhevsky, A., Wang, S.D.: Transforming auto-encoders. In: Honkela, T., Duch, W., Girolami, M.A., Kaski, S. (eds.) Artificial Neural Networks and Machine Learning - ICANN 2011 - 21st International Conference on Artificial Neural Networks, Espoo, June 14–17, 2011, Proceedings, Part I, Lecture Notes in Computer Science, vol. 6791, pp. 44–51. Springer (2011). https://doi.org/10.1007/978-3-642-21735-7_6

  12. Hinton, G.E., Sabour, S., Frosst, N.: Matrix capsules with EM routing. In: 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, April 30–May 3, 2018, Conference Track Proceedings. OpenReview.net (2018). https://openreview.net/forum?id=HJWLfGWRb

  13. Krizhevsky, A.: Learning Multiple Layers of Features from Tiny Images. University of Toronto, Toronto (2012)

    Google Scholar 

  14. Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: Bartlett, P.L., Pereira, F.C.N., Burges, C.J.C., Bottou, L., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems 25: 26th Annual Conference on Neural Information Processing Systems 2012. Proceedings of a meeting held December 3–6, 2012, Lake Tahoe, Nevada, pp. 1106–1114 (2012). https://proceedings.neurips.cc/paper/2012/hash/c399862d3b9d6b76c8436e924a68c45b-Abstract.html

  15. LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998). https://doi.org/10.1109/5.726791

    Article  Google Scholar 

  16. Li, S., Chen, K., Ahn, J.H., Brockman, J.B., Jouppi, N.P.: CACTI-P: architecture-level modeling for SRAM-based structures with advanced leakage reduction techniques. In: Phillips, J.R., Hu, A.J., Graeb, H. (eds.) 2011 IEEE/ACM International Conference on Computer-Aided Design, ICCAD 2011, San Jose, California, November 7–10, 2011, pp. 694–701. IEEE Computer Society (2011). https://doi.org/10.1109/ICCAD.2011.6105405

  17. Loshchilov, I., Hutter, F.: SGDR: stochastic gradient descent with warm restarts. In: 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net (2017). https://openreview.net/forum?id=Skq89Scxx

  18. Marchisio, A., Hanif, M.A., Khalid, F., Plastiras, G., Kyrkou, C., Theocharides, T., Shafique, M.: Deep learning for edge computing: Current trends, cross-layer optimizations, and open research challenges. In: 2019 IEEE Computer Society Annual Symposium on VLSI, ISVLSI 2019, Miami, FL, July 15–17, 2019, pp. 553–559. IEEE (2019). https://doi.org/10.1109/ISVLSI.2019.00105

  19. Marchisio, A., Hanif, M.A., Shafique, M.: CapsAcc: An efficient hardware accelerator for CapsuleNets with data reuse. In: Teich, J., Fummi, F. (eds.) Design, Automation & Test in Europe Conference & Exhibition, DATE 2019, Florence, March 25–29, 2019, pp. 964–967. IEEE (2019). https://doi.org/10.23919/DATE.2019.8714922

  20. Marchisio, A., Nanfa, G., Khalid, F., Hanif, M.A., Martina, M., Shafique, M.: CapsAttacks: Robust and imperceptible adversarial attacks on capsule networks. CoRR abs/1901.09878 (2019). http://arxiv.org/abs/1901.09878

  21. Marchisio, A., Bussolino, B., Colucci, A., Hanif, M.A., Martina, M., Masera, G., Shafique, M.: FasTrCaps: An integrated framework for fast yet accurate training of capsule networks. In: 2020 International Joint Conference on Neural Networks, IJCNN 2020, Glasgow, July 19–24, 2020, pp. 1–8. IEEE (2020). https://doi.org/10.1109/IJCNN48605.2020.9207533

  22. Marchisio, A., Bussolino, B., Colucci, A., Martina, M., Masera, G., Shafique, M.: Q-CapsNets: A specialized framework for quantizing capsule networks. In: 57th ACM/IEEE Design Automation Conference, DAC 2020, San Francisco, CA, July 20–24, 2020, pp. 1–6. IEEE (2020). https://doi.org/10.1109/DAC18072.2020.9218746

  23. Marchisio, A., Massa, A., Mrazek, V., Bussolino, B., Martina, M., Shafique, M.: NASCaps: A framework for neural architecture search to optimize the accuracy and hardware efficiency of convolutional capsule networks. In: IEEE/ACM International Conference On Computer Aided Design, ICCAD 2020, San Diego, CA, November 2–5, 2020, pp. 114:1–114:9. IEEE (2020). https://doi.org/10.1145/3400302.3415731

  24. Marchisio, A., Mrazek, V., Hanif, M.A., Shafique, M.: Red-cane: A systematic methodology for resilience analysis and design of capsule networks under approximations. In: 2020 Design, Automation & Test in Europe Conference & Exhibition, DATE 2020, Grenoble, March 9–13, 2020, pp. 1205–1210. IEEE (2020). https://doi.org/10.23919/DATE48585.2020.9116393

  25. Marchisio, A., Mrazek, V., Hanif, M.A., Shafique, M.: DESCNet: developing efficient scratchpad memories for capsule network hardware. IEEE Trans. Comput. Aided Des. Integr. Circuits Syst. 40(9), 1768–1781 (2021). https://doi.org/10.1109/TCAD.2020.3030610

    Article  Google Scholar 

  26. Marchisio, A., Mrazek, V., Hanif, M.A., Shafique, M.: FEECA: design space exploration for low-latency and energy-efficient capsule network accelerators. IEEE Trans. Very Large Scale Integr. Syst. 29(4), 716–729 (2021). https://doi.org/10.1109/TVLSI.2021.3059518

    Article  Google Scholar 

  27. Marchisio, A., Bussolino, B., Salvati, E., Martina, M., Masera, G., Shafique, M.: Enabling capsule networks at the edge through approximate softmax and squash operations. In: 2022 IEEE/ACM International Symposium on Low Power Electronics and Design, ISLPED 2022, Boston, MA, August 1–3, 2022, pp. 1–6. IEEE (2022)

    Google Scholar 

  28. Mazzia, V., Salvetti, F., Chiaberge, M.: Efficient-CapsNet: Capsule network with self-attention routing. CoRR abs/2101.12491 (2021). https://arxiv.org/abs/2101.12491

  29. Michels, F., Uelwer, T., Upschulte, E., Harmeling, S.: On the vulnerability of capsule networks to adversarial attacks. CoRR abs/1906.03612 (2019). http://arxiv.org/abs/1906.03612

  30. Monday, H.N., Li, J., Nneji, G.U., Nahar, S., Hossin, M.A., Jackson, J.: Covid-19 pneumonia classification based on neurowavelet capsule network. Healthcare 10(3) (2022). https://doi.org/10.3390/healthcare10030422. https://www.mdpi.com/2227-9032/10/3/422

  31. Mrazek, V., Hrbacek, R., Vasícek, Z., Sekanina, L.: EvoApprox8B: Library of approximate adders and multipliers for circuit design and benchmarking of approximation methods. In: Atienza, D., Natale, G.D. (eds.) Design, Automation & Test in Europe Conference & Exhibition, DATE 2017, Lausanne, March 27–31, 2017, pp. 258–261. IEEE (2017). https://doi.org/10.23919/DATE.2017.7926993

  32. Mrazek, V., Vasícek, Z., Sekanina, L., Hanif, M.A., Shafique, M.: ALWANN: automatic layer-wise approximation of deep neural network accelerators without retraining. In: Pan, D.Z. (ed.) Proceedings of the International Conference on Computer-Aided Design, ICCAD 2019, Westminster, CO, November 4–7, 2019, pp. 1–8. ACM (2019). https://doi.org/10.1109/ICCAD45719.2019.8942068

  33. Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Köpf, A., Yang, E.Z., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., Chintala, S.: PyTorch: An imperative style, high-performance deep learning library. In: Wallach, H.M., Larochelle, H., Beygelzimer, A., d’Alché-Buc, F., Fox, E.B., Garnett, R. (eds.) Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, pp. 8024–8035 (2019). https://proceedings.neurips.cc/paper/2019/hash/bdbca288fee7f92f2bfa9f7012727740-Abstract.html

  34. Pearson, K., for National Eugenics, G.L.: “Note on Regression and Inheritance in the Case of Two Parents”. Proceedings of the Royal Society. Royal Society (1895). https://books.google.it/books?id=xst6GwAACAAJ

  35. Rajasegaran, J., Jayasundara, V., Jayasekara, S., Jayasekara, H., Seneviratne, S., Rodrigo, R.: DeepCaps: Going deeper with capsule networks. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, June 16–20, 2019, pp. 10725–10733. Computer Vision Foundation / IEEE (2019). https://doi.org/10.1109/CVPR.2019.01098. http://openaccess.thecvf.com/content_CVPR_2019/html/Rajasegaran_DeepCaps_Going_Deeper_With_Capsule_Networks_CVPR_2019_paper.html

  36. Sabour, S., Frosst, N., Hinton, G.E.: Dynamic routing between capsules. In: Guyon, I., von Luxburg, U., Bengio, S., Wallach, H.M., Fergus, R., Vishwanathan, S.V.N., Garnett, R (eds.) Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4–9, 2017, Long Beach, CA, pp. 3856–3866 (2017). https://proceedings.neurips.cc/paper/2017/hash/2cad8fa47bbef282badbb8de5374b894-Abstract.html

    Google Scholar 

  37. Smith, L.N., Topin, N.: Super-convergence: very fast training of residual networks using large learning rates. CoRR abs/1708.07120 (2017). http://arxiv.org/abs/1708.07120

  38. Tsai, Y.H., Srivastava, N., Goh, H., Salakhutdinov, R.: Capsules with inverted dot-product attention routing. In: 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26–30, 2020. OpenReview.net (2020). https://openreview.net/forum?id=HJe6uANtwH

  39. Wu, X., Cao, Y., Lu, H., Liu, S., Wang, D., Wu, Z., Liu, X., Meng, H.: Speech emotion recognition using sequential capsule networks. IEEE ACM Trans. Audio Speech Lang. Process. 29, 3280–3291 (2021). https://doi.org/10.1109/TASLP.2021.3120586

    Article  Google Scholar 

  40. Xiao, H., Rasul, K., Vollgraf, R.: Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms. CoRR abs/1708.07747 (2017). http://arxiv.org/abs/1708.07747

  41. Zhao, W., Peng, H., Eger, S., Cambria, E., Yang, M.: Towards scalable and reliable capsule networks for challenging NLP applications. In: Korhonen, A., Traum, D.R., Màrquez, L. (eds.) Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, July 28–August 2, 2019, Volume 1: Long Papers, pp. 1549–1559. Association for Computational Linguistics (2019). https://doi.org/10.18653/v1/p19-1150

Download references

Acknowledgements

This work has been supported in part by the Doctoral College Resilient Embedded Systems, which is run jointly by the TU Wien’s Faculty of Informatics and the UAS Technikum Wien.

Author information

Authors and Affiliations

  1. Technische Universität Wien (TU Wien), Vienna, Austria

    Alberto Marchisio & Alessio Colucci

  2. Politecnico di Torino, Turin, Italy

    Beatrice Bussolino, Maurizio Martina & Guido Masera

  3. Brno University of Technology, Brno, Czechia

    Vojtech Mrazek

  4. eBrain Lab, Division of Engineering, New York University Abu Dhabi, Abu Dhabi, UAE

    Muhammad Abdullah Hanif & Muhammad Shafique

Authors
  1. Alberto Marchisio
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Beatrice Bussolino
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Alessio Colucci
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Vojtech Mrazek
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Muhammad Abdullah Hanif
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Maurizio Martina
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Guido Masera
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Muhammad Shafique
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Alberto Marchisio .

Editor information

Editors and Affiliations

  1. Colorado State University, Fort Collins, CO, USA

    Sudeep Pasricha

  2. New York University Abu Dhabi, Abu Dhabi, Abu Dhabi, United Arab Emirates

    Muhammad Shafique

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Marchisio, A. et al. (2024). Hardware and Software Optimizations for Capsule Networks. In: Pasricha, S., Shafique, M. (eds) Embedded Machine Learning for Cyber-Physical, IoT, and Edge Computing. Springer, Cham. https://doi.org/10.1007/978-3-031-39932-9_12

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-39932-9_12

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-39931-2

  • Online ISBN: 978-3-031-39932-9

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation