Avoiding Implementation Pitfalls of “Matrix Capsules with EM Routing” by Hinton et al.
The recent progress on capsule networks by Hinton et al. has generated considerable excitement in the machine learning community. The idea behind a capsule is inspired by a cortical minicolumn in the brain, whereby a vertically organised group of around 100 neurons receive common inputs, have common outputs, are interconnected, and may well constitute a fundamental computation unit of the cerebral cortex. However, Hinton’s paper on “Matrix Capsule with EM Routing” was unfortunately not accompanied by a release of source code, which left interested researchers attempting to implement the architecture and reproduce the benchmarks on their own. This has certainly slowed the progress of research building on this work. While writing our own implementation, we noticed several common mistakes in other open source implementations that we came across. In this paper we share some of these learnings, specifically focusing on three implementation pitfalls and how to avoid them: (1) parent capsules with only one child; (2) normalising the amount of data assigned to parent capsules; (3) parent capsules at different positions compete for child capsules. While our implementation is a considerable improvement over currently available implementations, it still falls slightly short of the performance reported by Hinton et al. (2018). The source code for this implementation is available on GitHub at the following URL: https://github.com/IBM/matrix-capsules-with-em-routing.
KeywordsCapsules EM routing Hinton CNN
- 1.Calvano, G.S.: Some clarification on the convolution topology? July 2018. https://openreview.net/forum?id=HJWLfGWRb¬eId=BJgX7Iy04m
- 3.Hinton, G., Krizhevsky, A., Jaitly, N., Tieleman, T., Tang, Y.: Does the brain do inverse graphics? In: Brain and Cognitive Sciences Fall Colloquium, vol. 2 (2012)Google Scholar
- 5.Hinton, G.E., Sabour, S., Frosst, N.: Lambda and margin, July 2018. https://openreview.net/forum?id=HJWLfGWRb¬eId=BkelcSxC47
- 6.Hinton, G.E., Sabour, S., Frosst, N.: Matrix capsules with EM routing. In: International Conference on Learning Representations (2018). https://openreview.net/forum?id=HJWLfGWRb
- 7.Hinton, G.E., Sabour, S., Frosst, N.: Regularization and learning rate? October 2018. https://openreview.net/forum?id=HJWLfGWRb¬eId=rJeQnSsE3X
- 8.Huang, Y.: Capsules. GitHub, April 2019. https://github.com/Officium/Capsules. Commit: e1f02d3
- 9.LeCun, Y., Cortes, C., Burges, C.J.: The MNIST database of handwritten digits (1998)Google Scholar
- 10.LeCun, Y., Huang, F.J., Bottou, L., et al.: Learning methods for generic object recognition with invariance to pose and lighting. In: CVPR, vol. 2, pp. 97–104. Citeseer (2004)Google Scholar
- 11.Lei, J.Y.: Matrix-capsules-em-pytorch. GitHub, March 2019. https://github.com/yl-1993/Matrix-Capsules-EM-PyTorch. Commit: c4547bf
- 12.Sabour, S.: Code for capsule model used in dynamic routing between capsules. GitHub, January 2018. https://github.com/Sarasra/models/tree/master/research/ capsules. Commit: cac8804
- 13.Sabour, S., Frosst, N., Hinton, G.E.: Dynamic routing between capsules. In: Advances in Neural Information Processing Systems, pp. 3856–3866 (2017)Google Scholar
- 14.Zhang, S.: Matrix-capsules-em-tensorflow. GitHub, February 2018. https://github.com/www0wwwjs1/Matrix-Capsules-EM-Tensorow. Commit: 0196ead