Information Overlay for Camera Phones in Indoor Environments

  • Harlan Hile
  • Gaetano Borriello
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4718)


Increasingly, cell phones are used to browse for information while location systems assist in gathering information that is most appropriate to the user’s current location. We seek to take this one step further and actually overlay information on to the physical world using the cell phone’s camera and thereby minimize a user’s cognitive effort. This “magic lens” approach has many applications of which we are exploring two: indoor building navigation and dynamic directory assistance. In essence, we match “landmarks” identified in the camera image with those stored in a building database. We use two different types of features – floor corners that can be matched against a floorplan and SIFT features that can be matched to a database constructed from other images. The camera’s pose can be determined exactly from a match and information can be properly aligned so that it can overlay directly onto the phone’s image display. In this paper, we present early results that demonstrate it is possible to realize this capability for a variety of indoor environments. Latency is shown to already be reasonable and likely to be improved by further optimizations. Our goal is to further explore the computational tradeoff between the server and phone client so as to achieve an acceptable latency of a few seconds.


Augmented Reality Indoor Environment Image Space Sift Feature Camera Phone 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Liu, A.L., Hile, H., Kautz, H., Borriello, G., Brown, P.A., Harniss, M., Johnson, K.: Indoor wayfinding: developing a functional interface for individuals with cognitive impairments. In: Assets 2006: Proceedings of the 8th international ACM SIGACCESS conference on Computers and accessibility, pp. 95–102. ACM Press, New York, USA (2006)CrossRefGoogle Scholar
  2. 2.
    Ekahau Wi-Fi positioning engine, a commercial location product,
  3. 3.
    Ferris, B., Haehnel, D., Fox, D.: Gaussian processes for signal strength-based location estimation. In: Proceedings of Robotics: Science and Systems, Philadelphia, USA (August 2006)Google Scholar
  4. 4.
    Snavely, N., Seitz, S.M., Szeliski, R.: Photo tourism: exploring photo collections in 3d. ACM Trans. Graph 25(3), 835–846 (2006)CrossRefGoogle Scholar
  5. 5.
    Lowe, D.G.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vision 60(2), 91–110 (2004)CrossRefGoogle Scholar
  6. 6.
    Kosaka, A., Pan, J.: Purdue experiments in model-based vision for hallway navigation. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) 1, 87–96 (1995)Google Scholar
  7. 7.
    Karlsson, N., di Bernardo, E., Ostrowski, J., Goncalves, L., Pirjanian, P., Munich, M.: The vslam algorithm for robust localization and mapping. In: Proceedings of the 2005 IEEE International Conference on Robotics and Automation, pp. 24–29 (April 2005)Google Scholar
  8. 8.
    Dellaert, F., Burgard, W., Fox, D., Thrun, S.: Using the condensation algorithm for robust, vision-based mobile robot localization. In: Proc. of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (1999)Google Scholar
  9. 9.
    Wolf, J., Burgard, W., Burkhardt, H.: Robust vision-based localization for mobile robots using an image retrieval system based on invariant features. In: Proceedings of the IEEE International Conference on Robotics and Automation (2002)Google Scholar
  10. 10.
    Fritz, G., Seifert, C., Paletta, L.: A mobile vision system for urban detection with informative local descriptors. In: ICVS 2006: Proceedings of the Fourth IEEE Intl Conference on Computer Vision Systems, IEEE Computer Society, Los Alamitos (2006)Google Scholar
  11. 11.
    Rekimoto, J., Ayatsuka, Y.: Cybercode: Designing augmented reality environments with visual tags. In: Proceedings of DARE (2000)Google Scholar
  12. 12.
    Hile, H., Kim, J., Borriello, G.: Microbiology tray and pipette tracking as a proactive tangible user interface. In: Ferscha, A., Mattern, F. (eds.) PERVASIVE 2004. LNCS, vol. 3001, pp. 323–339. Springer, Heidelberg (2004)Google Scholar
  13. 13.
    Caudell, T., Mizell, D.: Augmented reality: an application of heads-up display technology tomanual manufacturing processes. In: Proceedings of Hawaii International Conference on System Sciences, vol. 2, pp. 659–669 (1992)Google Scholar
  14. 14.
    Köhler, M., Patel, S., Summet, J., Stuntebeck, E., Abowd, G.: Tracksense: Infrastructure free precise indoor positioning using projected patterns. In: Pervasive Computing (2007)Google Scholar
  15. 15.
    Hecht, B., Rohs, M., Schöning, J., Krüger, A.: Wikeye - using magic lenses to explore georeferenced wikipedia content. In: 3rd International Workshop on Pervasive Mobile Interaction Devices (PERMID 2007) (2007)Google Scholar
  16. 16.
    Comaniciu, D., Meer, P.: Mean shift analysis and applications. In: ICCV 1999: Proceedings of the International Conference on Computer Vision, vol. 2, IEEE Computer Society, Los Alamitos (1999)Google Scholar
  17. 17.
    Guru, D., Dinesh, R., Nagabhushan, P.: Boundary based corner detection and localization using new ’cornerity’ index: A robust approach. In: CRV 2004: Proceedings of the 1st Canadian Conference on Computer and Robot Vision (CRV 2004), pp. 417–423. IEEE Computer Society, Los Alamitos (2004)CrossRefGoogle Scholar
  18. 18.
    Rous, M., Lupschen, H., Kraiss, K.: Vision-based indoor scene analysis for natural landmark detection. In: IEEE International Conference on Robotics and Automation, pp. 4642–4647 (April 2005)Google Scholar
  19. 19.
    Nokia mobile augmented reality applications (MARA),
  20. 20.
    Muller, H.J., Schoning, J., Kruger, A.: Mobile map interaction - evaluation in an indoor scenario. In: Workshop,Mobile and Embedded Interactive Systems (MEIS 2006), part of Informatik (October 2006)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2007

Authors and Affiliations

  • Harlan Hile
    • 1
  • Gaetano Borriello
    • 1
  1. 1.Dept. of Computer Science and Engineering, University of Washington, Box 352350, Seattle, WA 98195-2350USA

Personalised recommendations