Skip to main content
SpringerLink
Log in

Orthogonalizing Weights in Capsule Network Architecture

  • Conference paper
  • First Online:
ICT Analysis and Applications

Part of the book series: Lecture Notes in Networks and Systems ((LNNS,volume 93))

Abstract

Scalar neural network algorithms are limited in their ability to understand scale, rotational, or affine transformations within images and resort to average or max-pooling techniques which result in translational invariance. In an attempt to overcome these limitations, Hinton et al. introduced vectorized capsule network frameworks which support equivariance while capturing spatial relationships between data points, thus enhancing predictive capabilities of networks. However, experimenting with activation functions, hyperparameters, and optimizers have proven faster convergence and orthogonalizing weights within the layers of capsules enhance performance by slashing associated average error rates.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 169.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 219.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Sabour S, Frosst N, Hinton G (2017) Dynamic routing between capsules. In: Thirty first conference on neural information processing systems. arXiv:1710.09829

  2. Hinton G, Sabour S, Frosst N (2018) Matrix capsules with EM routing. In: International conference on learning representations

    Google Scholar 

  3. Pascanu R, Mikolov T, Bengio Y (2013) On the difficulty of training recurrent neural networks. In: International conference on machine learning, pp 1310–1318

    Google Scholar 

  4. Dauphin Y et al (2014) Identifying and attacking the saddle point problem in high dimensional non convex optimization. In: Advances in neural information processing systems, pp 2933–2941

    Google Scholar 

  5. Ioffe S et al (2015) Batch normalization: accelerating deep network training by reducing internal covariance shift. In: International conference on machine learning, pp 448–456

    Google Scholar 

  6. Bansal N, et al (2018) Can we gain more from orthogonality regularizations in training Deep CNN’s? In: Advances in neural information processing systems. arXiv:1810.09102

  7. Balestriero R et al (2018) Mad max: affine spline insights into deep learning. arXiv:1805.06576

  8. Zhu Z et al (2019) A convolutional neural network based on a capsule network with strong generalisation for bearing fault diagnosis. Neurocomputing 323: 62–75

    Google Scholar 

  9. Lin Z, Lu C, Li H (2015) Optimized projections for compressed sensing via direct mutual coherence minimization. arXiv:1508.03117

  10. Zhong C (2019) Residual capsule networks. In: 11th annual undergraduate research symposium. Poster session. University of Wisconsin, Milwaukee

    Google Scholar 

  11. Jain R (2019) Improving performance and inference on audio classification tasks using capsule networks. arXiv:1902.05069

  12. He S, Shin K (2019) Spatio-temporal capsule based reinforcement learning for mobility-on-demand network coordination. In: Association for computing machinery. ISBN: 978-1-4503-6674-8/19/05

    Google Scholar 

  13. Zhaopeng Xu, Wei Lu, Zhang Qqui, Yeung Yuileong, Chen Xin (2019) Gait recognition based on capsule network. J Vis Commun Image Represent 59:159–167

    Article  Google Scholar 

  14. Yao H et al (2019) Capsule network assisted IoT traffic classification mechanism for smart cities. IEEE Internet Things J

    Google Scholar 

  15. Kim Y, Wang P, Zhu Y, Mihaylova L (2018) A capsule network for traffic speed prediction in complex road networks. In: Sensor data fusion: trends, solutions, applications. IEEE, Bonn, Germany. https://doi.org/10.1109/sdf.2018.8547068

  16. Du Y, Zhao X, He M, Guo W (2019) A novel capsule based hybrid Neural network for sentiment classification. IEEE Access 7: 39321–39328. https://doi.org/10.1109/access.2019.2906398

  17. Qiao K, Zhang C, Wang L, Yan B, Chen J, Zeng L, Tong L (2018) Accurate reconstruction of image stimuli from human fMRI based on the decoding model with capsule network architecture. arXiv:1801.00602

  18. Mobiny A, Nguyer H (2018) Fast Capsnet for lung cancer screening. In: Frangi A, Schnabel J, Davatzikos C, Alberola Lopez C, Fichtinger G (eds) Medical image computing and computer assisted intervention (MICCAI). Lecture notes in Computer Science, vol 11071. Springer, Cham

    Google Scholar 

  19. Glorot X, Bengio Y (2010) Understanding the difficulty of training deep feed forward neural networks. In: Proceedings of thirteenth international conference on artificial intelligence and statistics, pp 249–256

    Google Scholar 

  20. He K, Zhang X, Ren S, Sun J (2015) Delving deep into rectifiers: surpassing human-level performance on image classification. In: Proceedings of IEEE international conference on Computer Vision, pp 1026–1034

    Google Scholar 

  21. Vorontsov E, Trabelsi C, Kadoury S, Pal C (2017) On orthogonality and learning recurrent networks with long term dependencies. arXiv:1702.00071

  22. Saxe A et al (2013) Exact solutions to the non linear dynamics of learning in deep linear neural networks. arXiv:1312.6120

  23. Rodriguez P, et al (2016) Regularizing CNN’s with locally constrained decorrelations. arXiv:1611.01967

  24. Xiao H, Rasul K, Vollagraf R (2017) Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms. arXiv:1708.07747

  25. Mass AL et al (2013) Rectified non linearities improve neural network acoustic models. In: International conference on machine learning, vol 30

    Google Scholar 

  26. Gagana B, Natarajan S (2019) Hyperparameter optimizations for capsule networks. In: EAI endorsed transactions on cloud systems

    Google Scholar 

  27. Gagana B, Athri U, Natarajan S (2018) Activation function optimizations for capsule networks. In: IEEE international conference on advances in computing, communications and informatics. ISBN: 978-1-5386-5314-2

    Google Scholar 

  28. Le QV et al (2017) Swish: a self gated activation function. In: Neural and evolutionary computing, computer vision and pattern recognition. arXiv:1710.05941

  29. Liang M et al (2015) Recurrent convolutional neural network for object recognition. In: Conference on computer vision and pattern recognition

    Google Scholar 

  30. Kingma D, Ba J (2015) Adam: a method for stochastic optimisation. arXiv:1412.6980

  31. Luo L, Xiong Y, Liu Y, Sun X (2019) Adaptive gradient methods with dynamic bound of learning rate. In: International conference on learning representations

    Google Scholar 

  32. Katebi R (2019) Examining extreme nuclear variability in the galaxies that host Active Galactic Nuclei. American Astronomical Society

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Madhav Institute of Technology, Gwalior, India

    Shubhranil Kundu

  2. PES University, Bangalore, India

    B. Gagana

Authors
  1. Shubhranil Kundu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. B. Gagana
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shubhranil Kundu .

Editor information

Editors and Affiliations

  1. University of Macau, Macau, China

    Simon Fong

  2. Techno India College of Engineering, Kolkata, India

    Nilanjan Dey

  3. Ahmedabad, India

    Amit Joshi

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Kundu, S., Gagana, B. (2020). Orthogonalizing Weights in Capsule Network Architecture. In: Fong, S., Dey, N., Joshi, A. (eds) ICT Analysis and Applications. Lecture Notes in Networks and Systems, vol 93. Springer, Singapore. https://doi.org/10.1007/978-981-15-0630-7_8

Download citation

  • DOI: https://doi.org/10.1007/978-981-15-0630-7_8

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-15-0629-1

  • Online ISBN: 978-981-15-0630-7

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation