Mixed Reality Navigation on a Tablet Computer for Supporting Machine Maintenance in Wide-Area Indoor Environment

  • Koji MakitaEmail author
  • Thomas Vincent
  • Soichi Ebisuno
  • Masakatsu Kourogi
  • Tomoya Ishikawa
  • Takashi Okuma
  • Minoru Yoshida
  • Laurence Nigay
  • Takeshi Kurata
Conference paper


This paper describes a maintenance service support system for wide-area indoor environment, such as a factory and a hospital. In maintenance services, operators often have to check a map to find out a way to a target machine, and also have to refer documents to get information about check-up and repair of the machine. In order to reduce working load of operators, information technology can help operators carry out additional but important operations during maintenance, such as referring documents and maps, recording maintenance logs and so on. In this paper, we propose mixed reality navigation on a tablet computer composed of augmented virtuality mode and augmented reality mode. Augmented virtuality mode performs map-based navigation shows positions of the user and the target machine. Augmented reality mode performs intuitive visualization of information about the machine by overlaying annotations on camera images. The proposed system is based on a hybrid localization technique realized with pedestrian dead reckoning (PDR) and 3D model-based image processing for the purpose of covering wide-area indoor environment. Experimental results using our prototype with a mock-up model of a machine are also described for showing feasibility of our concept in the paper.


Maintenance support Human navigation Mixed reality Mobile computing 



This work was supported by Strategic Japanese-French Cooperative Program on Information and Communication Technology Including Computer Science (ANR in France and JST in Japan).


  1. 1.
    Henderson SJ, Feiner S (2009) Evaluating the benefits of augmented reality for task localization in maintenance of an armored personnel carrier turret. In: Proceedings of the IEEE international symposium on mixed and augmented reality 2009, Science and technology, pp 135–144Google Scholar
  2. 2.
    Platonov J, Heibe H, Meier P, Grollmann B (2006) A mobile markerless AR system for maintenance and repair. In: Proceedings of the 5th IEEE and ACM international symposium on mixed and augmented reality, pp 105–108Google Scholar
  3. 3.
    Klein G, Murray D (2009) Parallel tracking and mapping on a camera phone. In: Proceedings of the international symposium on mixed and augmented reality, pp 83–86Google Scholar
  4. 4.
    Reitmayr G, Drummond T (2007) Initialisation for visual tracking in urban environments. In: Proceedings of the international symposium on mixed and augmented reality, pp 161–172Google Scholar
  5. 5.
    Höllerer T, Feiner S, Terauchi T, Rashid G, Hallaway D (1999) Exploring MARS: developing indoor and outdooruser interfaces to a mobile augmented reality system. Comput Graph 23(6):779–785CrossRefGoogle Scholar
  6. 6.
    Saito S, Hiyama A, Tanikawa T, Hirose M (2007) Indoor marker-based localization using coded seamless pattern for interior decoration. In: Proceedings of the IEEE virtual reality 2007, pp 67–74Google Scholar
  7. 7.
    Tenmoku R, Kanbara M, Yokoya N (2003) A wearable augmented reality system using positioning infrastructures and a pedometer. In: Proceedings of the IEEE international symposium on wearable computers, pp 110–117Google Scholar
  8. 8.
    Yoshida H, Ito S, Kawaguchi N (2006) Evaluation of pre- acquisition methods for position estimation system using wireless LAN. The third International Conference on Mobile Computing and Ubiquitous networking (ICMU 2006), pp 148–155Google Scholar
  9. 9.
    Cipolla R, Robertson D, Tordoff B (2004) Image-based localization. In: Proceedings of the international conference on Virtual System and Multimedia (VSMM2004), pp 22–29Google Scholar
  10. 10.
    Kourogi M, Kurata T, Sakaue K (2001) A panorama-based method of personal positioning and orientation and its realtime applications for wearable computers. In: Proceedings of the fifth International Symposium on Wearable Computers (ISWC’01), pp 107–114Google Scholar
  11. 11.
    Torr P, Zisserman A (1999) Feature based methods for structure and motion estimation. International workshop on vision algorithms, pp 278–295Google Scholar
  12. 12.
    Bleser G, Wuest H, Stricker D (2006) Online camera pose estimation in partially known and dynamic scenes. In: Proceedings of the international symposium on mixed and augmented reality, pp 56–65Google Scholar
  13. 13.
    Simon G, Berger M (2002) Pose estimation from planar structures. IEEE Comput Graph Appl 22(6):46–53CrossRefGoogle Scholar
  14. 14.
    Hähnel D, Burgard W, Thrun S (2003) Learning compact 3D models of indoor and outdoor environments with a mobile robot. Robot Auton Syst 44(1):15–27CrossRefGoogle Scholar
  15. 15.
    Jensen B, Weingarten J, Kolski S, Siegwart R (2005) Laser range imaging using mobile robots: from pose estimation to 3D-models. In: Proceedings of the of 1st range imaging research day, pp 129–144Google Scholar
  16. 16.
    Neubert J, Pretlove J, Drummond T (2007) Semi-autonomous generation of appearance-based edge models from image sequences. In: Proceedings of the international symposium on mixed and augmented reality, pp 79–89Google Scholar
  17. 17.
  18. 18.
    Ishikawa T, Thangamani K, Kourogi M, Gee AP, Mayol W, Jung K, Kurata T (2009) In-situ 3D indoor modeler with a camera and self-contained sensors. In: Proceedings of the HCII2009, LNCS 5622, pp 454–464Google Scholar
  19. 19.
    Langer M, BenHimane S (2010) An interactive vision-based 3D reconstruction workflow for industrial AR applications, online. In: Proceedings of the ISMAR 2010 workshop on augmented reality super modelsGoogle Scholar
  20. 20.
    Arth C, Wagner D, Klopschitz M, Irschara A, Schmalstieg D (2009) Wide area localization on mobile phones. In: Proceedings of the international symposium on mixed and augmented reality, pp 73–82Google Scholar
  21. 21.
    Klein G, Murray D (2008) Improving the agility of keyframe-based SLAM. In: Proceedings of the 10th European Conference on Computer Vision (ECCV’08)Google Scholar
  22. 22.
    Irschara A, Zach C, Frahm JM, Bischof H (2009) From structure-from-motion point clouds to fast location recognition. Conference on Computer Vision and Pattern Recognition (CVPR)Google Scholar
  23. 23.
    Kourogi M, Ishikawa T, Kurata T (2010) A method of pedestrian dead reckoning using action recognition. In: Proceedings of the IEEE/ION Position, Location and Navigation Symposium (PLANS), pp 85–89Google Scholar
  24. 24.
    Makita K et al (2013) Photo-shoot localization of a mobile camera based on registered frame data of virtualized reality models. In: Proceedings of the S&T ISMAR2013, pp 273–274Google Scholar
  25. 25.
    Bay H, Tuytelaars T, Gool LV (2006) SURF: speeded up robust features. In: Proceedings of the 9th European Conference on Computer Vision (ECCV’2006), Springer LNCS volume 3951, part 1, pp 404–417Google Scholar
  26. 26.
    Tomasi C, Kanade T (1991) Detection and tracking of point features. Technical Report CMU-CS-91-132, Carnegie Mellon University, April 1991Google Scholar
  27. 27.
    Vincent T, Nigay L, Kurata T (2013) Precise pointing techniques for handheld augmented reality. In: Proceedings of the INTERACT 2013, IFIP-Springer (2013), pp 122–139Google Scholar
  28. 28.
    CAUTIC (Conception Assistée par l’Usage pour les Technologies l’Innovation et le Changement).
  29. 29.
    Kano N (1984) Attractive quality and must-be quality [in Japanese]. J Jpn Soc Qual Control 14(2):147–156MathSciNetGoogle Scholar
  30. 30.
    Sauerwein E, Bailon F, Matzler K, Hinterhuber H (1996) The Kano model: how to delight your customers. In: Preprints volume I of the IX: international working seminar on production economics, pp 313–327Google Scholar

Copyright information

© Springer Japan 2016

Authors and Affiliations

  • Koji Makita
    • 1
    Email author
  • Thomas Vincent
    • 2
  • Soichi Ebisuno
    • 3
  • Masakatsu Kourogi
    • 1
    • 3
  • Tomoya Ishikawa
    • 3
  • Takashi Okuma
    • 1
  • Minoru Yoshida
    • 3
  • Laurence Nigay
    • 2
  • Takeshi Kurata
    • 1
  1. 1.Center for Service ResearchNational Institute of Advanced Industrial Science and TechnologyUmezono, TsukubaJapan
  2. 2.Université Joseph FourierSaint-Martin-d’HèresFrance
  3. 3.KODO Lab IncTsukubaJapan

Personalised recommendations