Computational Visual Media

, Volume 3, Issue 2, pp 189–198 | Cite as

Robust camera pose estimation by viewpoint classification using deep learning

  • Yoshikatsu NakajimaEmail author
  • Hideo Saito
Open Access
Research Article


Camera pose estimation with respect to target scenes is an important technology for superimposing virtual information in augmented reality (AR). However, it is difficult to estimate the camera pose for all possible view angles because feature descriptors such as SIFT are not completely invariant from every perspective. We propose a novel method of robust camera pose estimation using multiple feature descriptor databases generated for each partitioned viewpoint, in which the feature descriptor of each keypoint is almost invariant. Our method estimates the viewpoint class for each input image using deep learning based on a set of training images prepared for each viewpoint class. We give two ways to prepare these images for deep learning and generating databases. In the first method, images are generated using a projection matrix to ensure robust learning in a range of environments with changing backgrounds. The second method uses real images to learn a given environment around a planar pattern. Our evaluation results confirm that our approach increases the number of correct matches and the accuracy of camera pose estimation compared to the conventional method.


pose estimation augmented reality (AR) deep learning convolutional neural network 


  1. [1]
    Kato, H.; Billinghurst, M. Marker tracking and HMD calibration for a video-based augmented reality conferencing system. In: Proceedings of the 2nd IEEE and ACM International Workshop on Augmented Reality, 85–94, 1999.CrossRefGoogle Scholar
  2. [2]
    Lee, T.; Hollerer, T. Hybrid feature tracking and user interaction for markerless augmented reality. In: Proceedings of IEEE Virtual Reality Conference, 145–152, 2008.Google Scholar
  3. [3]
    Maidi, M.; Preda, M.; Le, V. H. Markerless tracking for mobile augmented reality. In: Proceedings of IEEE International Conference on Signal and Image Processing Applications, 301–306, 2011.Google Scholar
  4. [4]
    Lowe, D. G. Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision Vol. 60, No. 2, 91–110, 2004.CrossRefGoogle Scholar
  5. [5]
    Mikolajczyk, K.; Schmid, C. A performance evaluation of local descriptors. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 27, No. 10, 1615–1630, 2005.CrossRefGoogle Scholar
  6. [6]
    Lepetit, V.; Fua, P. Keypoint recognition using randomized trees. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 28, No. 9, 1465–1479, 2006.CrossRefGoogle Scholar
  7. [7]
    Breiman, L. Random forests. Machine Learning Vol. 45, No. 1, 5–32, 2001.CrossRefzbMATHGoogle Scholar
  8. [8]
    Yoshida, T.; Saito, H.; Shimizu, M.; Taguchi, A. Stable keypoint recognition using viewpoint generative learning. In: Proceedings of the International Conference on Computer Vision Theory and Applications, Vol. 2, 310–315, 2013.Google Scholar
  9. [9]
    Hartigan, J. A.; Wong, M. A. Algorithm AS 136: A k-means clustering algorithm. Journal of the Royal Statistical Society. Series C (Applied Statistics) Vol. 28, No. 1, 100–108, 1979.zbMATHGoogle Scholar
  10. [10]
    Fukushima, K.; Miyake, S. Neocognitron: A new algorithm for pattern recognition tolerant of deformations and shifts in position. Pattern Recognition Vol. 15, No. 6, 455–469, 1982.CrossRefGoogle Scholar
  11. [11]
    Hubel, D. H.; Wiesel, T. N. Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex. The Journal of Physiology Vol. 160, No. 1, 106–154, 1962.CrossRefGoogle Scholar
  12. [12]
    LeCun, Y.; Boser, B.; Denker, J. S.; Henderson, D.; Howard, R. E.; Hubbard, W.; Jackel, L. D. Backpropagation applied to handwritten zip code recognition. Neural Computation Vol. 1, No. 4, 541–551, 1989.CrossRefGoogle Scholar
  13. [13]
    Russakovsky, O.; Deng, J.; Su, H.; Krause, J.; Satheesh, S.; Ma, S.; Huang, Z.; Karpathy, A.; Khosla, A.; Bernstein, M.; Berg, A. C.; Fei-Fei, L. ImageNet large scale visual recognition challenge. International Journal of Computer Vision Vol. 115, No. 3, 211–252, 2015.MathSciNetCrossRefGoogle Scholar
  14. [14]
    Agrawal, P.; Carreira, J.; Malik, J. Learning to see by moving. In: Proceedings of IEEE International Conference on Computer Vision, 37–45, 2015.Google Scholar
  15. [15]
    Rumelhart, D. E.; Hintont, G. E.; Williams, R. J. Learning representations by back-propagating errors. Nature Vol. 323, 533–536, 1986.CrossRefzbMATHGoogle Scholar
  16. [16]
    Hinton, G. E.; Srivastava, N.; Krizhevsky, A.; Sutskever, I.; Salakhutdinov, R. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580, 2012.zbMATHGoogle Scholar
  17. [17]
    Krizhevsky, A.; Sutskever, I.; Hinton, G. E. ImageNet classification with deep convolutional neural network. In: Proceedings of Advances in Neural Information Processing Systems, 1097–1105, 2012.Google Scholar
  18. [18]
    Mikolajczyk, K.; Tuytelaars, T.; Schmid, C.; Zisserman, A.; Matas, J.; Schaffalitzky, F.; Kadir, T.; GooL, L. V. A comparison of affine region detectors. International Journal of Computer Vision Vol. 65, No. 1, 43–72, 2005.CrossRefGoogle Scholar
  19. [19]
    Fischler, M. A.; Bolles, R. C. Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM Vol. 24, No. 6, 381–395, 1981.MathSciNetCrossRefGoogle Scholar
  20. [20]
    Yu, G.; Morel, J.-M. ASIFT: An algorithm for fully affine invariant comparison. Image Processing On Line Vol. 1, 1–28, 2011.Google Scholar
  21. [21]
    Ozuysal, M.; Calonder, M.; Lepetit, V.; Fua, P. Fast keypoint recognition using random ferns. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 32, No. 3, 448–461, 2009.CrossRefGoogle Scholar
  22. [22]
    Tokui, S.; Oono, K.; Hido, S.; Clayton, J. Chainer: A next-generation open source framework for deep learning. In: Proceedings of Workshop on Machine Learning Systems (LearningSys) in the 29th Annual Conference on Neural Information Processing Systems, 2015.Google Scholar
  23. [23]
    Lin, M.; Chen, Q.; Yan, S. Network in network. arXiv preprint arXiv:1312.4400, 2013.Google Scholar
  24. [24]
    Alcantarilla, P. F.; Nuevo, J.; Bartoli, A. Fast explicit diffusion for accelerated features in nonlinear scale spaces. In: Proceedings of British Machine Vision Conference, 13.1–13.11, 2013.Google Scholar

Copyright information

© The Author(s) 2016

Open Access The articles published in this journal are distributed under the terms of the Creative Commons Attribution 4.0 International License (, which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Authors and Affiliations

  1. 1.Department of Science and TechnologyKeio UniversityKeioJapan

Personalised recommendations