Abstract
Convolutional neural networks (CNNs) have been proven to be effective for image recognition, which plays an important role in cyber security. In this paper, we focus on a promising neural network, capsule network, which aims at correcting the deficiencies of CNNs. Routing procedure between capsules, which serves as a key component in capsule networks, computes coupling coefficients with complicated steps iteratively. However, the expensive computational cost poses a bottleneck for extending capsule networks deeper and wider to approach higher performance on complex data. To address this limitation, we propose a novel routing algorithm named capsule-wise attention routing based on attention mechanism. With a successful reduction of computational cost in the routing procedure, we construct a deep capsule network architecture named CARNet. Our CARNets are proven experimentally to outperform other state-of-the-art capsule networks on SVHN and CIFAR-10 benchmarks while reducing the amount of parameters by 62% at most.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
The CapsNet we trained is wider than CapsNet for SVHN [11], which consists of a convolutional layer with 64 channels, a primary capsule layer with 16 6D-capsules and a final capsule layer with 10 8D-capsules.
References
Abadi, M., et al.: Tensorflow: large-scale machine learning on heterogeneous distributed systems. CoRR abs/1603.04467 (2016)
Chollet, F.: Xception: deep learning with depthwise separable convolutions. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, 21–26 July 2017, pp. 1800–1807 (2017)
Deliège, A., Cioppa, A., Droogenbroeck, M.V.: HitNet: a neural network with capsules embedded in a hit-or-miss layer, extended with hybrid data augmentation and ghost capsules. CoRR abs/1806.06519 (2018)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, 27–30 June 2016, pp. 770–778 (2016)
Hinton, G.E., Krizhevsky, A., Wang, S.D.: Transforming auto-encoders. In: Honkela, T., Duch, W., Girolami, M., Kaski, S. (eds.) ICANN 2011. LNCS, vol. 6791, pp. 44–51. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-21735-7_6
Hinton, G.E., Sabour, S., Frosst, N.: Matrix capsules with EM routing. In: 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, 30 April–3 May 2018, Conference Track Proceedings (2018)
Huang, G., Liu, Z., van der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, 21–26 July 2017, pp. 2261–2269 (2017)
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, 7–9 May 2015, Conference Track Proceedings (2015)
Lenssen, J.E., Fey, M., Libuschewski, P.: Group equivariant capsule networks. In: Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, Montréal, Canada, 3–8 December 2018, pp. 8858–8867 (2018)
Rajasegaran, J., Jayasundara, V., Jayasekara, S., Jayasekara, H., Seneviratne, S., Rodrigo, R.: DeepCaps: going deeper with capsule networks. CoRR abs/1904.09546 (2019)
Sabour, S., Frosst, N., Hinton, G.E.: Dynamic routing between capsules. In: Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, Long Beach, CA, USA, 4–9 December 2017, pp. 3859–3869 (2017)
Wang, D., Liu, Q.: An optimization view on dynamic routing between capsules. In: 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, 30 April–3 May 2018, Workshop Track Proceedings (2018)
Xie, S., Girshick, R.B., Dollár, P., Tu, Z., He, K.: Aggregated residual transformations for deep neural networks. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, 21–26 July 2017, pp. 5987–5995 (2017)
Zagoruyko, S., Komodakis, N.: Wide residual networks. In: Proceedings of the British Machine Vision Conference 2016, BMVC 2016, York, UK, 19–22 September 2016 (2016)
Zhang, S., Zhou, Q., Wu, X.: Fast dynamic routing based on weighted kernel density estimation. In: Lu, H. (ed.) ISAIR 2018. SCI, vol. 810, pp. 301–309. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-04946-1_30
Zhong, Z., Zheng, L., Kang, G., Li, S., Yang, Y.: Random erasing data augmentation. CoRR abs/1708.04896 (2017)
Acknowledgements
This work was supported by National Natural Science Foundation of China under Grant 61703039 and Beijing Natural Science Foundation under Grant 4174095.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Yu, ZX., He, Y., Zhu, C., Tian, S., Yin, XC. (2019). CARNet: Densely Connected Capsules with Capsule-Wise Attention Routing. In: Ning, H. (eds) Cyberspace Data and Intelligence, and Cyber-Living, Syndrome, and Health. CyberDI CyberLife 2019 2019. Communications in Computer and Information Science, vol 1137. Springer, Singapore. https://doi.org/10.1007/978-981-15-1922-2_22
Download citation
DOI: https://doi.org/10.1007/978-981-15-1922-2_22
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-15-1921-5
Online ISBN: 978-981-15-1922-2
eBook Packages: Computer ScienceComputer Science (R0)