Abstract
Recent user interface concepts, such as multimedia, multimodal, wearable, ubiquitous, tangible, or augmented-reality-based (AR) interfaces, each cover different approaches that are all needed to support complex human–computer interaction. Increasingly, an overarching approach towards building what we call ubiquitous augmented reality (UAR) user interfaces that include all of the just mentioned concepts will be required. To this end, we present a user interface architecture that can form a sound basis for combining several of these concepts into complex systems. We explain in this paper the fundamentals of DWARF’s user interface framework (DWARF standing for distributed wearable augmented reality framework) and an implementation of this architecture. Finally, we present several examples that show how the framework can form the basis of prototypical applications.
Similar content being viewed by others
References
Anderson D, Marks J, Agarwala A, Beardsley P, Leigh D, Sullivan E, Yedidia J, Frankel J, Hodgins JK , Ryall K (2000) Tangible interaction and graphical interpretation: a new approach to intuitive 3D modeling. In: Proceedings of the 27th ACM annual conference on computer graphics (SIGGRAPH 2000), New Orleans, Louisiana, July 2000
R Azuma (1997) A survey of augmented reality. Presence–Teleop Virt 6(4):355–385
Bauer M, Bruegge B, Klinker G, MacWilliams A, Reicher T, Riss S, Sandor C, Wagner M (2001) Design of a component-based augmented reality framework. In: Proceedings of the IEEE and ACM international symposium on augmented reality (ISAR 2001), New York City, New York, October 2001, pp 45–53
Bauer M, Bruegge B, Klinker G, MacWilliams A, Reicher T, Sandor C, Wagner M (2002) An architecture concept for ubiquitous computing aware wearable computers. In: Proceedings of the 2nd international workshop on smart appliances and wearable computing (IWSAWC 2002), Vienna, Austria, July 2002
Bauer M, Hilliges O, MacWilliams A, Sandor C, Wagner M, Newman J, Reitmayr G, Fahmy T, Klinker G, Pintaric T, Schmalstieg D (2003) Integrating Studierstube and DWARF. In: Proceedings of the international workshop on software technology for augmented reality systems (STARS 2003), Tokyo, Japan, October 2003
Blasko G, Feiner S (2002) A menu interface for wearable computing. In: Proceedings of the 6th international symposium on wearable computers (ISWC 2002), Seattle, Washington, October 2002, pp 164–165
Broll W, Meier E, Schardt T (2000) The virtual round table—a collaborative augmented multi-user environment. In: Proceedings of the 3rd international conference on collaborative virtual environments (CVE 2000), San Francisco, California, September 2000, pp 39–45
W Buxton (1991) The three mirrors of interaction: a holistic approach to user interfaces. In: Proceedings of the Friend21 international symposium on next generation human interfaces, Tokyo, Japan, November 1991
Caroll JM (2002) Human–computer interaction in the new millenium. Addison-Wesley, Reading, Massachusetts
Davies JM (1998) An ambient computing system. Masters thesis, Department of Electrical Engineering and Computer Science, University of Kansas, Kansas
Echtler F, Sturm F, Kindermann K, Klinker G, Stilla J, Trilk J, Najafi H (2003) The intelligent welding gun: augmented reality for experimental vehicle construction. In: Ong S, Nee A (eds) Virtual and augmented reality applications in manufacturing, chap 17. Springer, Berlin Heidelberg New York
Feiner S, MacIntyre B, Haupt M, Solomon E (2003) Windows on the world: 2D windows for 3D augmented reality. In: Proceedings of the 6th annual ACM symposium on user interface software and technology (UIST’93), Atlanta, Georgia, November 1993. ACM Press, New York, pp 145–155
Gamma E, Helm R, Johnson R, Vlissides J (1995) Design patterns: elements of reusable object-oriented software. Addison-Wesley, Reading, Massachusetts
Garlan D, Siewiorek D, Smailagic A, Steenkiste P (2002) Project Aura: toward distraction-free pervasive computing. IEEE Pervas Comput 1(2):22–31
Jacob RJK, Deligiannidis L, Morrision S (1999) A software model and specification language for non-WIMP user interfaces. ACM Trans Comput–Hum Interact 6:1–46
Jensen K, Rozenberg G (1991) High-level Petri nets: theory and applications. Springer, Berlin Heidelberg New York, ISBN 3-540-54125 X
Johanson B, Fox A (2004) Extending tuplespaces for coordination in interactive workspaces. J Syst Softw 69:243–266
Johanson B, Fox A, Winograd T (2002) The interactive workspaces project: experiences with ubiquitous computing rooms. IEEE Pervas Comput 1:67–74
Kaiser E, Olwal A, McGee D, Benko H, Corradini A, Li X, Feiner S, Cohen P (2003) Mutual disambiguation of 3D multimodal interaction in augmented and virtual reality. In: Proceedings of the 5th international conference on multimodal interfaces (ICMI 2003), Vancouver, British Columbia, November 2003. ACM Press, New York, pp 12–19
Kato H, Billinghurst M, Poupyrev I, Tetsutani N (2001) Tangible augmented reality for human–computer interaction. In: Proceedings of the 17th Japanese conference of computer graphics (NICOGRAPH 2001), Nagoya, Japan, November 2001
Klinker G, Reicher T, Bruegge B (2000) Distributed tracking concepts for augmented reality applications. In: Proceedings of the IEEE and ACM international symposium on augmented reality (ISAR 2000), Munich, Germany, October 2000
Klinker G, Stricker D, Reiners D (1999) An optically based direct manipulation interface for human–computer interaction in an augmented world. In: Proceedings of the 5th EUROGRAPHICS workshop on virtual environments (EGVE’99), Vienna, Austria, June 1999
Kortuem G, Schneider J (2001) An application platform for mobile ad-hoc networks. In: Proceedings of the workshop on application models and programming tools for ubiquitous computing (UbiTools 2001), Atlanta, Georgia, September 2001
Kulas C (2003) Usability engineering for ubiquitous computing. Masters thesis, Technische Universität München, Munich, Germany
MacWilliams A, Sandor C, Wagner M, Bauer M, Klinker G, Bruegge B (2003) Herding SHEEP: live development of a distributed augmented reality system. In: Proceedings of the 2nd IEEE and ACM international symposium on mixed and augmented reality (ISMAR 2003), Tokyo, Japan, October 2003
Mattern F (2001) Pervasive/ubiquitous computing. Informatik-Spektrum 24:145–147
Maybury M, Whalster W (eds) (1998) Readings in intelligent user interfaces. Morgan Kaufmann, San Mateo, California
Milgram P, Kishino F (1994) A taxonomy of mixed reality visual displays. IEICE Trans Inform Syst E77-D(12): 1321–1329
Nigay L, Coutaz J (1993) A design space for multimodal systems: concurrent processing and data fusion. In: Proceedings of the joint conference of ACM SIGCHI and INTERACT (InterCHI’93), Amsterdam, The Netherlands, April 1993. IOS Press, Amsterdam, The Netherlands, pp 172–178
Novak V (2004) Attentive user interfaces for DWARF. Masters thesis, Technische Universität München, Munich, Germany
Olsen D (1992) User interface management systems: models and algorithms. Morgan Kaufmann, San Mateo, California
Olwal A (2002) Unit—a modular framework for interaction technique design, development and implementation. Masters thesis, Royal Institute of Technology (KTH), Stockholm, Sweden
Oviatt SL (1999) Ten myths of multimodal interaction. Commun ACM 42:74–81
Oviatt SL (1999) Mutual disambiguation of recognition errors in a multimodel architecture. In: Proceedings of the ACM SIGCHI conference on human factors in computing systems (CHI’99), Pittsburgh, Pennsylvania, May 1999. ACM Press, New York, pp 576–583
Oviatt SL (2000) Multimodal interface research: a science without borders. In: Proceedings of the 6th international conference on spoken language processing (ICSLP 2000), Beijing, China, October 2000
Perlin K, Fox D (1993) Pad: an alternative approach to the computer interface. Comput Graph 27:57–72
Poupyrev I, Billinghurst M, Weghorst S, Ichikawa T (1996) The go-go interaction technique: non-linear mapping for direct manipulation in VR. In: Proceedings of the 9th annual ACM symposium on user interface software and technology (UIST’96), Seattle, Washington, November 1996. ACM Press, New York, pp 79–80
Rauterberg M, Fjeld M, Krueger H, Bichsel M, Leonhardt U, Meier M (1998) BUILD-IT: a planning tool for construction and design. In: Proceedings of the ACM SIGCHI conference on human factors in computing systems (CHI’98), Los Angeles, California, April 1998. ACM Press, New York, pp 177–178
Regenbrecht H, Wagner M (2002) Interaction in a collaborative augmented reality environment. In: Proceedings of the ACM SIGCHI conference on human factors in computing systems (CHI 2002), Minneapolis, Minnesota, April 2002
Reicher T, MacWilliams A, Bruegge B, Klinker G (2003) Results of a study on software architectures for augmented reality systems. In: Proceedings of the 2nd IEEE and ACM international symposium on mixed and augmented reality (ISMAR 2003), Tokyo, Japan, October 2003
Reitmayr G, Schmalstieg D (2001) OpenTracker—an open software architecture for reconfigurable tracking based on XML. In: Proceedings of the IEEE virtual reality conference (VR 2001), Yokohama, Japan, March 2001, pp 285–286
Rekimoto J (1997) Pick-and-drop: a direct manipulation technique for multiple computer environments. In: Proceedings of the 10th annual ACM symposium on user interface software and technology (UIST’97), Banff, Alberta, Canada, October 1997. ACM Press, New York, pp 31–39
Sandor C, MacWilliams A, Wagner M, Bauer M, Klinker G (2002) SHEEP: the shared environment entertainment pasture. In: Demonstration at the IEEE and ACM international symposium on mixed and augmented reality (ISMAR 2002), Darmstadt, Germany, September/October 2002
Schmidt A, Gellersen H-W, Beigl M, Thate O (2000) Developing user interfaces for wearable computers—don’t stop to point and click. In: Proceedings of the international workshop on interactive applications of mobile computing (IMC 2000), Rostock, Germany, November 2000
Shneiderman B (1997) Designing the user interface. Addison-Wesley, Reading, Massachusetts
Song D, Norman M (1993) Nonlinear interactive motion control techniques for virtual space navigation. In: Proceedings of the IEEE virtual reality annual international symposium (VRAIS’93), Seattle, Washington, September 1993, pp 111–117
Strauss P, Carey R (1992) An object-oriented 3D graphics toolkit. In: Proceedings of the 19th ACM annual conference on computer graphics (SIGGRAPH 1992), Chicago, Illinois, July 1992, vol 26, issue 2, pp 341–349
Stricker D, Klinker G, Reiners D (1998) A fast and robust line-based optical tracker for augmented reality applications. In: Proceedings of the 1st IEEE international workshop on augmented reality (IWAR’98), San Francisco, California, November 1998. AK Peters, Wellesley, Massachusetts, pp 129–145
Turk M, Robertson G (2000) Perceptual user interfaces (introduction). Commun ACM 43:32–34
Ullmer B, Ishii H (1997) The metaDESK: models and prototypes for tangible user interfaces. In: Proceedings of the 10th annual ACM symposium on user interface software and technology (UIST’97), Banff, Alberta, Canada, October 1997. ACM Press, New York, pp 223–232
Ullmer B, Ishii H (2000) Emerging frameworks for tangible user interfaces. IBM Syst J 39(3–4):915–931
Underkoffler J, Ishii H (1999) Urp: a luminous-tangible workbench for urban planning and design. In: Proceedings of the ACM SIGCHI conference on human factors in computing systems (CHI’99), Pittsburgh, Pennsylvania, May 1999. ACM Press, New York, pp 386–393
Vertegaal R (2003) Attentive user interfaces. Commun ACM 46(3):40–46
Waibel A, Vo MT, Duchnowski P, Manke S (1995) Multimodal interfaces. Artif Intell Rev 10:299–319
Weiser M (1993) Hot topics: ubiquitous computing. IEEE Comput 26:71–72
Acknowledgements
Special thanks go to our students Daniel Pustka, Franz Strasser, Gerrit Hillebrand, Marco Feuerstein, Ming-Ju Lee, Otmar Hilliges, Chris Kulas, Manja Kurzak, Felix Loew, Marcus Tönnis, Johannes Wöhler, and Bernhard Zaun for developing many useful components and tools; without these, this work would not have been possible. The Ph.D. students who designed and developed most parts of DWARF are: Thomas Reicher, Martin Bauer, Martin Wagner, and Asa MacWilliams. The Columbia University’s Computer Graphics and User Interfaces Lab provided us with two prototypes of the TouchGlove; we would like to thank Steven Feiner and Gabor Blasko for their support. The tracking system used for SHEEP was partially on loan from BMW (TI-360). This work was partially supported by the High-Tech-Offensive of the Bayerische Staatskanzlei, Germany (Prof. Brügge).
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Sandor, C., Klinker, G. A rapid prototyping software infrastructure for user interfaces in ubiquitous augmented reality. Pers Ubiquit Comput 9, 169–185 (2005). https://doi.org/10.1007/s00779-004-0328-1
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00779-004-0328-1