Personal and Ubiquitous Computing

, Volume 9, Issue 3, pp 169–185 | Cite as

A rapid prototyping software infrastructure for user interfaces in ubiquitous augmented reality

Original Article

Abstract

Recent user interface concepts, such as multimedia, multimodal, wearable, ubiquitous, tangible, or augmented-reality-based (AR) interfaces, each cover different approaches that are all needed to support complex human–computer interaction. Increasingly, an overarching approach towards building what we call ubiquitous augmented reality (UAR) user interfaces that include all of the just mentioned concepts will be required. To this end, we present a user interface architecture that can form a sound basis for combining several of these concepts into complex systems. We explain in this paper the fundamentals of DWARF’s user interface framework (DWARF standing for distributed wearable augmented reality framework) and an implementation of this architecture. Finally, we present several examples that show how the framework can form the basis of prototypical applications.

Keywords

Augmented reality Ubiquitous computing Tangible user interfaces Multimodality Software architectures Frameworks Mobile systems 

References

  1. 1.
    Anderson D, Marks J, Agarwala A, Beardsley P, Leigh D, Sullivan E, Yedidia J, Frankel J, Hodgins JK , Ryall K (2000) Tangible interaction and graphical interpretation: a new approach to intuitive 3D modeling. In: Proceedings of the 27th ACM annual conference on computer graphics (SIGGRAPH 2000), New Orleans, Louisiana, July 2000Google Scholar
  2. 2.
    R Azuma (1997) A survey of augmented reality. Presence–Teleop Virt 6(4):355–385Google Scholar
  3. 3.
    Bauer M, Bruegge B, Klinker G, MacWilliams A, Reicher T, Riss S, Sandor C, Wagner M (2001) Design of a component-based augmented reality framework. In: Proceedings of the IEEE and ACM international symposium on augmented reality (ISAR 2001), New York City, New York, October 2001, pp 45–53Google Scholar
  4. 4.
    Bauer M, Bruegge B, Klinker G, MacWilliams A, Reicher T, Sandor C, Wagner M (2002) An architecture concept for ubiquitous computing aware wearable computers. In: Proceedings of the 2nd international workshop on smart appliances and wearable computing (IWSAWC 2002), Vienna, Austria, July 2002Google Scholar
  5. 5.
    Bauer M, Hilliges O, MacWilliams A, Sandor C, Wagner M, Newman J, Reitmayr G, Fahmy T, Klinker G, Pintaric T, Schmalstieg D (2003) Integrating Studierstube and DWARF. In: Proceedings of the international workshop on software technology for augmented reality systems (STARS 2003), Tokyo, Japan, October 2003Google Scholar
  6. 6.
    Blasko G, Feiner S (2002) A menu interface for wearable computing. In: Proceedings of the 6th international symposium on wearable computers (ISWC 2002), Seattle, Washington, October 2002, pp 164–165Google Scholar
  7. 7.
    Broll W, Meier E, Schardt T (2000) The virtual round table—a collaborative augmented multi-user environment. In: Proceedings of the 3rd international conference on collaborative virtual environments (CVE 2000), San Francisco, California, September 2000, pp 39–45Google Scholar
  8. 8.
    W Buxton (1991) The three mirrors of interaction: a holistic approach to user interfaces. In: Proceedings of the Friend21 international symposium on next generation human interfaces, Tokyo, Japan, November 1991Google Scholar
  9. 9.
    Caroll JM (2002) Human–computer interaction in the new millenium. Addison-Wesley, Reading, MassachusettsGoogle Scholar
  10. 10.
    Davies JM (1998) An ambient computing system. Masters thesis, Department of Electrical Engineering and Computer Science, University of Kansas, KansasGoogle Scholar
  11. 11.
    Echtler F, Sturm F, Kindermann K, Klinker G, Stilla J, Trilk J, Najafi H (2003) The intelligent welding gun: augmented reality for experimental vehicle construction. In: Ong S, Nee A (eds) Virtual and augmented reality applications in manufacturing, chap 17. Springer, Berlin Heidelberg New YorkGoogle Scholar
  12. 12.
    Feiner S, MacIntyre B, Haupt M, Solomon E (2003) Windows on the world: 2D windows for 3D augmented reality. In: Proceedings of the 6th annual ACM symposium on user interface software and technology (UIST’93), Atlanta, Georgia, November 1993. ACM Press, New York, pp 145–155Google Scholar
  13. 13.
    Gamma E, Helm R, Johnson R, Vlissides J (1995) Design patterns: elements of reusable object-oriented software. Addison-Wesley, Reading, MassachusettsGoogle Scholar
  14. 14.
    Garlan D, Siewiorek D, Smailagic A, Steenkiste P (2002) Project Aura: toward distraction-free pervasive computing. IEEE Pervas Comput 1(2):22–31CrossRefGoogle Scholar
  15. 15.
    Jacob RJK, Deligiannidis L, Morrision S (1999) A software model and specification language for non-WIMP user interfaces. ACM Trans Comput–Hum Interact 6:1–46Google Scholar
  16. 16.
    Jensen K, Rozenberg G (1991) High-level Petri nets: theory and applications. Springer, Berlin Heidelberg New York, ISBN 3-540-54125 XGoogle Scholar
  17. 17.
    Johanson B, Fox A (2004) Extending tuplespaces for coordination in interactive workspaces. J Syst Softw 69:243–266CrossRefGoogle Scholar
  18. 18.
    Johanson B, Fox A, Winograd T (2002) The interactive workspaces project: experiences with ubiquitous computing rooms. IEEE Pervas Comput 1:67–74CrossRefGoogle Scholar
  19. 19.
    Kaiser E, Olwal A, McGee D, Benko H, Corradini A, Li X, Feiner S, Cohen P (2003) Mutual disambiguation of 3D multimodal interaction in augmented and virtual reality. In: Proceedings of the 5th international conference on multimodal interfaces (ICMI 2003), Vancouver, British Columbia, November 2003. ACM Press, New York, pp 12–19Google Scholar
  20. 20.
    Kato H, Billinghurst M, Poupyrev I, Tetsutani N (2001) Tangible augmented reality for human–computer interaction. In: Proceedings of the 17th Japanese conference of computer graphics (NICOGRAPH 2001), Nagoya, Japan, November 2001Google Scholar
  21. 21.
    Klinker G, Reicher T, Bruegge B (2000) Distributed tracking concepts for augmented reality applications. In: Proceedings of the IEEE and ACM international symposium on augmented reality (ISAR 2000), Munich, Germany, October 2000Google Scholar
  22. 22.
    Klinker G, Stricker D, Reiners D (1999) An optically based direct manipulation interface for human–computer interaction in an augmented world. In: Proceedings of the 5th EUROGRAPHICS workshop on virtual environments (EGVE’99), Vienna, Austria, June 1999Google Scholar
  23. 23.
    Kortuem G, Schneider J (2001) An application platform for mobile ad-hoc networks. In: Proceedings of the workshop on application models and programming tools for ubiquitous computing (UbiTools 2001), Atlanta, Georgia, September 2001Google Scholar
  24. 24.
    Kulas C (2003) Usability engineering for ubiquitous computing. Masters thesis, Technische Universität München, Munich, GermanyGoogle Scholar
  25. 25.
    MacWilliams A, Sandor C, Wagner M, Bauer M, Klinker G, Bruegge B (2003) Herding SHEEP: live development of a distributed augmented reality system. In: Proceedings of the 2nd IEEE and ACM international symposium on mixed and augmented reality (ISMAR 2003), Tokyo, Japan, October 2003Google Scholar
  26. 26.
    Mattern F (2001) Pervasive/ubiquitous computing. Informatik-Spektrum 24:145–147CrossRefGoogle Scholar
  27. 27.
    Maybury M, Whalster W (eds) (1998) Readings in intelligent user interfaces. Morgan Kaufmann, San Mateo, CaliforniaGoogle Scholar
  28. 28.
    Milgram P, Kishino F (1994) A taxonomy of mixed reality visual displays. IEICE Trans Inform Syst E77-D(12): 1321–1329Google Scholar
  29. 29.
    Nigay L, Coutaz J (1993) A design space for multimodal systems: concurrent processing and data fusion. In: Proceedings of the joint conference of ACM SIGCHI and INTERACT (InterCHI’93), Amsterdam, The Netherlands, April 1993. IOS Press, Amsterdam, The Netherlands, pp 172–178Google Scholar
  30. 30.
    Novak V (2004) Attentive user interfaces for DWARF. Masters thesis, Technische Universität München, Munich, GermanyGoogle Scholar
  31. 31.
    Olsen D (1992) User interface management systems: models and algorithms. Morgan Kaufmann, San Mateo, CaliforniaGoogle Scholar
  32. 32.
    Olwal A (2002) Unit—a modular framework for interaction technique design, development and implementation. Masters thesis, Royal Institute of Technology (KTH), Stockholm, SwedenGoogle Scholar
  33. 33.
    Oviatt SL (1999) Ten myths of multimodal interaction. Commun ACM 42:74–81CrossRefGoogle Scholar
  34. 34.
    Oviatt SL (1999) Mutual disambiguation of recognition errors in a multimodel architecture. In: Proceedings of the ACM SIGCHI conference on human factors in computing systems (CHI’99), Pittsburgh, Pennsylvania, May 1999. ACM Press, New York, pp 576–583Google Scholar
  35. 35.
    Oviatt SL (2000) Multimodal interface research: a science without borders. In: Proceedings of the 6th international conference on spoken language processing (ICSLP 2000), Beijing, China, October 2000Google Scholar
  36. 36.
    Perlin K, Fox D (1993) Pad: an alternative approach to the computer interface. Comput Graph 27:57–72Google Scholar
  37. 37.
    Poupyrev I, Billinghurst M, Weghorst S, Ichikawa T (1996) The go-go interaction technique: non-linear mapping for direct manipulation in VR. In: Proceedings of the 9th annual ACM symposium on user interface software and technology (UIST’96), Seattle, Washington, November 1996. ACM Press, New York, pp 79–80Google Scholar
  38. 38.
    Rauterberg M, Fjeld M, Krueger H, Bichsel M, Leonhardt U, Meier M (1998) BUILD-IT: a planning tool for construction and design. In: Proceedings of the ACM SIGCHI conference on human factors in computing systems (CHI’98), Los Angeles, California, April 1998. ACM Press, New York, pp 177–178Google Scholar
  39. 39.
    Regenbrecht H, Wagner M (2002) Interaction in a collaborative augmented reality environment. In: Proceedings of the ACM SIGCHI conference on human factors in computing systems (CHI 2002), Minneapolis, Minnesota, April 2002Google Scholar
  40. 40.
    Reicher T, MacWilliams A, Bruegge B, Klinker G (2003) Results of a study on software architectures for augmented reality systems. In: Proceedings of the 2nd IEEE and ACM international symposium on mixed and augmented reality (ISMAR 2003), Tokyo, Japan, October 2003Google Scholar
  41. 41.
    Reitmayr G, Schmalstieg D (2001) OpenTracker—an open software architecture for reconfigurable tracking based on XML. In: Proceedings of the IEEE virtual reality conference (VR 2001), Yokohama, Japan, March 2001, pp 285–286Google Scholar
  42. 42.
    Rekimoto J (1997) Pick-and-drop: a direct manipulation technique for multiple computer environments. In: Proceedings of the 10th annual ACM symposium on user interface software and technology (UIST’97), Banff, Alberta, Canada, October 1997. ACM Press, New York, pp 31–39Google Scholar
  43. 43.
    Sandor C, MacWilliams A, Wagner M, Bauer M, Klinker G (2002) SHEEP: the shared environment entertainment pasture. In: Demonstration at the IEEE and ACM international symposium on mixed and augmented reality (ISMAR 2002), Darmstadt, Germany, September/October 2002Google Scholar
  44. 44.
    Schmidt A, Gellersen H-W, Beigl M, Thate O (2000) Developing user interfaces for wearable computers—don’t stop to point and click. In: Proceedings of the international workshop on interactive applications of mobile computing (IMC 2000), Rostock, Germany, November 2000Google Scholar
  45. 45.
    Shneiderman B (1997) Designing the user interface. Addison-Wesley, Reading, MassachusettsGoogle Scholar
  46. 46.
    Song D, Norman M (1993) Nonlinear interactive motion control techniques for virtual space navigation. In: Proceedings of the IEEE virtual reality annual international symposium (VRAIS’93), Seattle, Washington, September 1993, pp 111–117Google Scholar
  47. 47.
    Strauss P, Carey R (1992) An object-oriented 3D graphics toolkit. In: Proceedings of the 19th ACM annual conference on computer graphics (SIGGRAPH 1992), Chicago, Illinois, July 1992, vol 26, issue 2, pp 341–349Google Scholar
  48. 48.
    Stricker D, Klinker G, Reiners D (1998) A fast and robust line-based optical tracker for augmented reality applications. In: Proceedings of the 1st IEEE international workshop on augmented reality (IWAR’98), San Francisco, California, November 1998. AK Peters, Wellesley, Massachusetts, pp 129–145Google Scholar
  49. 49.
    Turk M, Robertson G (2000) Perceptual user interfaces (introduction). Commun ACM 43:32–34CrossRefGoogle Scholar
  50. 50.
    Ullmer B, Ishii H (1997) The metaDESK: models and prototypes for tangible user interfaces. In: Proceedings of the 10th annual ACM symposium on user interface software and technology (UIST’97), Banff, Alberta, Canada, October 1997. ACM Press, New York, pp 223–232Google Scholar
  51. 51.
    Ullmer B, Ishii H (2000) Emerging frameworks for tangible user interfaces. IBM Syst J 39(3–4):915–931Google Scholar
  52. 52.
    Underkoffler J, Ishii H (1999) Urp: a luminous-tangible workbench for urban planning and design. In: Proceedings of the ACM SIGCHI conference on human factors in computing systems (CHI’99), Pittsburgh, Pennsylvania, May 1999. ACM Press, New York, pp 386–393Google Scholar
  53. 53.
    Vertegaal R (2003) Attentive user interfaces. Commun ACM 46(3):40–46Google Scholar
  54. 54.
    Waibel A, Vo MT, Duchnowski P, Manke S (1995) Multimodal interfaces. Artif Intell Rev 10:299–319CrossRefGoogle Scholar
  55. 55.
    Weiser M (1993) Hot topics: ubiquitous computing. IEEE Comput 26:71–72Google Scholar

Copyright information

© Springer-Verlag London Limited 2005

Authors and Affiliations

  1. 1.Institut für InformatikTechnische Universität München MunichGermany

Personalised recommendations