Skip to main content
SpringerLink
Log in

A rapid prototyping software infrastructure for user interfaces in ubiquitous augmented reality

  • Original Article
  • Published:
Personal and Ubiquitous Computing Aims and scope Submit manuscript

Abstract

Recent user interface concepts, such as multimedia, multimodal, wearable, ubiquitous, tangible, or augmented-reality-based (AR) interfaces, each cover different approaches that are all needed to support complex human–computer interaction. Increasingly, an overarching approach towards building what we call ubiquitous augmented reality (UAR) user interfaces that include all of the just mentioned concepts will be required. To this end, we present a user interface architecture that can form a sound basis for combining several of these concepts into complex systems. We explain in this paper the fundamentals of DWARF’s user interface framework (DWARF standing for distributed wearable augmented reality framework) and an implementation of this architecture. Finally, we present several examples that show how the framework can form the basis of prototypical applications.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5a, b
Fig. 6a, b
Fig. 7a, b
Fig. 8
Fig. 9a–e
Fig. 10
Fig. 11a–c
Fig. 12a, b
Fig. 13a–c

Similar content being viewed by others

References

  1. Anderson D, Marks J, Agarwala A, Beardsley P, Leigh D, Sullivan E, Yedidia J, Frankel J, Hodgins JK , Ryall K (2000) Tangible interaction and graphical interpretation: a new approach to intuitive 3D modeling. In: Proceedings of the 27th ACM annual conference on computer graphics (SIGGRAPH 2000), New Orleans, Louisiana, July 2000

  2. R Azuma (1997) A survey of augmented reality. Presence–Teleop Virt 6(4):355–385

    Google Scholar 

  3. Bauer M, Bruegge B, Klinker G, MacWilliams A, Reicher T, Riss S, Sandor C, Wagner M (2001) Design of a component-based augmented reality framework. In: Proceedings of the IEEE and ACM international symposium on augmented reality (ISAR 2001), New York City, New York, October 2001, pp 45–53

  4. Bauer M, Bruegge B, Klinker G, MacWilliams A, Reicher T, Sandor C, Wagner M (2002) An architecture concept for ubiquitous computing aware wearable computers. In: Proceedings of the 2nd international workshop on smart appliances and wearable computing (IWSAWC 2002), Vienna, Austria, July 2002

  5. Bauer M, Hilliges O, MacWilliams A, Sandor C, Wagner M, Newman J, Reitmayr G, Fahmy T, Klinker G, Pintaric T, Schmalstieg D (2003) Integrating Studierstube and DWARF. In: Proceedings of the international workshop on software technology for augmented reality systems (STARS 2003), Tokyo, Japan, October 2003

  6. Blasko G, Feiner S (2002) A menu interface for wearable computing. In: Proceedings of the 6th international symposium on wearable computers (ISWC 2002), Seattle, Washington, October 2002, pp 164–165

  7. Broll W, Meier E, Schardt T (2000) The virtual round table—a collaborative augmented multi-user environment. In: Proceedings of the 3rd international conference on collaborative virtual environments (CVE 2000), San Francisco, California, September 2000, pp 39–45

  8. W Buxton (1991) The three mirrors of interaction: a holistic approach to user interfaces. In: Proceedings of the Friend21 international symposium on next generation human interfaces, Tokyo, Japan, November 1991

  9. Caroll JM (2002) Human–computer interaction in the new millenium. Addison-Wesley, Reading, Massachusetts

    Google Scholar 

  10. Davies JM (1998) An ambient computing system. Masters thesis, Department of Electrical Engineering and Computer Science, University of Kansas, Kansas

  11. Echtler F, Sturm F, Kindermann K, Klinker G, Stilla J, Trilk J, Najafi H (2003) The intelligent welding gun: augmented reality for experimental vehicle construction. In: Ong S, Nee A (eds) Virtual and augmented reality applications in manufacturing, chap 17. Springer, Berlin Heidelberg New York

  12. Feiner S, MacIntyre B, Haupt M, Solomon E (2003) Windows on the world: 2D windows for 3D augmented reality. In: Proceedings of the 6th annual ACM symposium on user interface software and technology (UIST’93), Atlanta, Georgia, November 1993. ACM Press, New York, pp 145–155

  13. Gamma E, Helm R, Johnson R, Vlissides J (1995) Design patterns: elements of reusable object-oriented software. Addison-Wesley, Reading, Massachusetts

    Google Scholar 

  14. Garlan D, Siewiorek D, Smailagic A, Steenkiste P (2002) Project Aura: toward distraction-free pervasive computing. IEEE Pervas Comput 1(2):22–31

    Article  Google Scholar 

  15. Jacob RJK, Deligiannidis L, Morrision S (1999) A software model and specification language for non-WIMP user interfaces. ACM Trans Comput–Hum Interact 6:1–46

    Google Scholar 

  16. Jensen K, Rozenberg G (1991) High-level Petri nets: theory and applications. Springer, Berlin Heidelberg New York, ISBN 3-540-54125 X

  17. Johanson B, Fox A (2004) Extending tuplespaces for coordination in interactive workspaces. J Syst Softw 69:243–266

    Article  Google Scholar 

  18. Johanson B, Fox A, Winograd T (2002) The interactive workspaces project: experiences with ubiquitous computing rooms. IEEE Pervas Comput 1:67–74

    Article  Google Scholar 

  19. Kaiser E, Olwal A, McGee D, Benko H, Corradini A, Li X, Feiner S, Cohen P (2003) Mutual disambiguation of 3D multimodal interaction in augmented and virtual reality. In: Proceedings of the 5th international conference on multimodal interfaces (ICMI 2003), Vancouver, British Columbia, November 2003. ACM Press, New York, pp 12–19

  20. Kato H, Billinghurst M, Poupyrev I, Tetsutani N (2001) Tangible augmented reality for human–computer interaction. In: Proceedings of the 17th Japanese conference of computer graphics (NICOGRAPH 2001), Nagoya, Japan, November 2001

  21. Klinker G, Reicher T, Bruegge B (2000) Distributed tracking concepts for augmented reality applications. In: Proceedings of the IEEE and ACM international symposium on augmented reality (ISAR 2000), Munich, Germany, October 2000

  22. Klinker G, Stricker D, Reiners D (1999) An optically based direct manipulation interface for human–computer interaction in an augmented world. In: Proceedings of the 5th EUROGRAPHICS workshop on virtual environments (EGVE’99), Vienna, Austria, June 1999

  23. Kortuem G, Schneider J (2001) An application platform for mobile ad-hoc networks. In: Proceedings of the workshop on application models and programming tools for ubiquitous computing (UbiTools 2001), Atlanta, Georgia, September 2001

  24. Kulas C (2003) Usability engineering for ubiquitous computing. Masters thesis, Technische Universität München, Munich, Germany

  25. MacWilliams A, Sandor C, Wagner M, Bauer M, Klinker G, Bruegge B (2003) Herding SHEEP: live development of a distributed augmented reality system. In: Proceedings of the 2nd IEEE and ACM international symposium on mixed and augmented reality (ISMAR 2003), Tokyo, Japan, October 2003

  26. Mattern F (2001) Pervasive/ubiquitous computing. Informatik-Spektrum 24:145–147

    Article  Google Scholar 

  27. Maybury M, Whalster W (eds) (1998) Readings in intelligent user interfaces. Morgan Kaufmann, San Mateo, California

  28. Milgram P, Kishino F (1994) A taxonomy of mixed reality visual displays. IEICE Trans Inform Syst E77-D(12): 1321–1329

    Google Scholar 

  29. Nigay L, Coutaz J (1993) A design space for multimodal systems: concurrent processing and data fusion. In: Proceedings of the joint conference of ACM SIGCHI and INTERACT (InterCHI’93), Amsterdam, The Netherlands, April 1993. IOS Press, Amsterdam, The Netherlands, pp 172–178

  30. Novak V (2004) Attentive user interfaces for DWARF. Masters thesis, Technische Universität München, Munich, Germany

  31. Olsen D (1992) User interface management systems: models and algorithms. Morgan Kaufmann, San Mateo, California

    Google Scholar 

  32. Olwal A (2002) Unit—a modular framework for interaction technique design, development and implementation. Masters thesis, Royal Institute of Technology (KTH), Stockholm, Sweden

  33. Oviatt SL (1999) Ten myths of multimodal interaction. Commun ACM 42:74–81

    Article  Google Scholar 

  34. Oviatt SL (1999) Mutual disambiguation of recognition errors in a multimodel architecture. In: Proceedings of the ACM SIGCHI conference on human factors in computing systems (CHI’99), Pittsburgh, Pennsylvania, May 1999. ACM Press, New York, pp 576–583

  35. Oviatt SL (2000) Multimodal interface research: a science without borders. In: Proceedings of the 6th international conference on spoken language processing (ICSLP 2000), Beijing, China, October 2000

  36. Perlin K, Fox D (1993) Pad: an alternative approach to the computer interface. Comput Graph 27:57–72

    Google Scholar 

  37. Poupyrev I, Billinghurst M, Weghorst S, Ichikawa T (1996) The go-go interaction technique: non-linear mapping for direct manipulation in VR. In: Proceedings of the 9th annual ACM symposium on user interface software and technology (UIST’96), Seattle, Washington, November 1996. ACM Press, New York, pp 79–80

  38. Rauterberg M, Fjeld M, Krueger H, Bichsel M, Leonhardt U, Meier M (1998) BUILD-IT: a planning tool for construction and design. In: Proceedings of the ACM SIGCHI conference on human factors in computing systems (CHI’98), Los Angeles, California, April 1998. ACM Press, New York, pp 177–178

  39. Regenbrecht H, Wagner M (2002) Interaction in a collaborative augmented reality environment. In: Proceedings of the ACM SIGCHI conference on human factors in computing systems (CHI 2002), Minneapolis, Minnesota, April 2002

  40. Reicher T, MacWilliams A, Bruegge B, Klinker G (2003) Results of a study on software architectures for augmented reality systems. In: Proceedings of the 2nd IEEE and ACM international symposium on mixed and augmented reality (ISMAR 2003), Tokyo, Japan, October 2003

  41. Reitmayr G, Schmalstieg D (2001) OpenTracker—an open software architecture for reconfigurable tracking based on XML. In: Proceedings of the IEEE virtual reality conference (VR 2001), Yokohama, Japan, March 2001, pp 285–286

  42. Rekimoto J (1997) Pick-and-drop: a direct manipulation technique for multiple computer environments. In: Proceedings of the 10th annual ACM symposium on user interface software and technology (UIST’97), Banff, Alberta, Canada, October 1997. ACM Press, New York, pp 31–39

  43. Sandor C, MacWilliams A, Wagner M, Bauer M, Klinker G (2002) SHEEP: the shared environment entertainment pasture. In: Demonstration at the IEEE and ACM international symposium on mixed and augmented reality (ISMAR 2002), Darmstadt, Germany, September/October 2002

  44. Schmidt A, Gellersen H-W, Beigl M, Thate O (2000) Developing user interfaces for wearable computers—don’t stop to point and click. In: Proceedings of the international workshop on interactive applications of mobile computing (IMC 2000), Rostock, Germany, November 2000

  45. Shneiderman B (1997) Designing the user interface. Addison-Wesley, Reading, Massachusetts

  46. Song D, Norman M (1993) Nonlinear interactive motion control techniques for virtual space navigation. In: Proceedings of the IEEE virtual reality annual international symposium (VRAIS’93), Seattle, Washington, September 1993, pp 111–117

  47. Strauss P, Carey R (1992) An object-oriented 3D graphics toolkit. In: Proceedings of the 19th ACM annual conference on computer graphics (SIGGRAPH 1992), Chicago, Illinois, July 1992, vol 26, issue 2, pp 341–349

  48. Stricker D, Klinker G, Reiners D (1998) A fast and robust line-based optical tracker for augmented reality applications. In: Proceedings of the 1st IEEE international workshop on augmented reality (IWAR’98), San Francisco, California, November 1998. AK Peters, Wellesley, Massachusetts, pp 129–145

  49. Turk M, Robertson G (2000) Perceptual user interfaces (introduction). Commun ACM 43:32–34

    Article  Google Scholar 

  50. Ullmer B, Ishii H (1997) The metaDESK: models and prototypes for tangible user interfaces. In: Proceedings of the 10th annual ACM symposium on user interface software and technology (UIST’97), Banff, Alberta, Canada, October 1997. ACM Press, New York, pp 223–232

  51. Ullmer B, Ishii H (2000) Emerging frameworks for tangible user interfaces. IBM Syst J 39(3–4):915–931

    Google Scholar 

  52. Underkoffler J, Ishii H (1999) Urp: a luminous-tangible workbench for urban planning and design. In: Proceedings of the ACM SIGCHI conference on human factors in computing systems (CHI’99), Pittsburgh, Pennsylvania, May 1999. ACM Press, New York, pp 386–393

  53. Vertegaal R (2003) Attentive user interfaces. Commun ACM 46(3):40–46

    Google Scholar 

  54. Waibel A, Vo MT, Duchnowski P, Manke S (1995) Multimodal interfaces. Artif Intell Rev 10:299–319

    Article  Google Scholar 

  55. Weiser M (1993) Hot topics: ubiquitous computing. IEEE Comput 26:71–72

    Google Scholar 

Download references

Acknowledgements

Special thanks go to our students Daniel Pustka, Franz Strasser, Gerrit Hillebrand, Marco Feuerstein, Ming-Ju Lee, Otmar Hilliges, Chris Kulas, Manja Kurzak, Felix Loew, Marcus Tönnis, Johannes Wöhler, and Bernhard Zaun for developing many useful components and tools; without these, this work would not have been possible. The Ph.D. students who designed and developed most parts of DWARF are: Thomas Reicher, Martin Bauer, Martin Wagner, and Asa MacWilliams. The Columbia University’s Computer Graphics and User Interfaces Lab provided us with two prototypes of the TouchGlove; we would like to thank Steven Feiner and Gabor Blasko for their support. The tracking system used for SHEEP was partially on loan from BMW (TI-360). This work was partially supported by the High-Tech-Offensive of the Bayerische Staatskanzlei, Germany (Prof. Brügge).

Author information

Authors and Affiliations

  1. Institut für Informatik, Technische Universität München, Munich, Germany

    Christian Sandor & Gudrun Klinker

Authors
  1. Christian Sandor
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Gudrun Klinker
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Christian Sandor.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Sandor, C., Klinker, G. A rapid prototyping software infrastructure for user interfaces in ubiquitous augmented reality. Pers Ubiquit Comput 9, 169–185 (2005). https://doi.org/10.1007/s00779-004-0328-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00779-004-0328-1

Keywords

Search

Navigation