Skip to main content
SpringerLink
Log in

Blended Interaction: understanding natural human–computer interaction in post-WIMP interactive spaces

  • Original Article
  • Published:
Personal and Ubiquitous Computing Aims and scope Submit manuscript

Abstract

We introduce Blended Interaction, a new conceptual framework that helps to explain when users perceive user interfaces as “natural” or not. Based on recent findings from embodied cognition and cognitive linguistics, Blended Interaction provides a novel and more accurate description of the nature of human–computer interaction (HCI). In particular, it introduces the notion of conceptual blends to explain how users rely on familiar and real-world concepts whenever they learn to use new digital technologies. We apply Blended Interaction in the context of post-“Windows Icons Menu Pointer” interactive spaces. These spaces are ubiquitous computing environments for computer-supported collaboration of multiple users in a physical space or room, e.g., meeting rooms, design studios, or libraries, augmented with novel interactive technologies and digital computation, e.g., multi-touch walls, tabletops, and tablets. Ideally, in these spaces, the virtues of the familiar physical and social world are combined with that of the digital realm in a considered manner so that desired properties of each are preserved and a seemingly “natural” HCI is achieved. To support designers in this goal, we explain how the users’ conceptual systems use blends to tie together familiar concepts with the novel powers of digital computation. Furthermore, we introduce four domains of design to structure the underlying problem and design space: individual and social interaction, workflow, and physical environment. We introduce our framework by discussing related work, e.g., metaphors, mental models, direct manipulation, image schemas, reality-based interaction, and illustrate Blended Interaction using design decisions we made in recent projects.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Notes

  1. Post-WIMP = Post-“Windows Icons Menus Pointer”.

  2. Blackwell provides an extensive review on the role and history of user interface metaphors as a design tool in HCI in [13].

  3. Please note that, unlike HCI literature, Fauconnier and Turner do not refer to the desktop UI as the “desktop metaphor”. In their terminology, the desktop UI is a blend and not a metaphor, as is also discussed by Imaz and Benyon: “When we speak of the desktop metaphor now, we are really referring to a large blend […]. It is usually considered to be a metaphor because most of the traditional functionalities of ordinary work have been maintained as expressions in interface tasks […]. But when observed in detail, it is evident that we are dealing with a blend rather than a metaphor—the blend being based on the metaphor” [5: p. 52].

  4. Blended Interactions user experience design studio at Rochester Institute of Technology. http://blendedinteractions.com/about/ (Last accessed Jul 19, 2013).

  5. http://hci.uni-konstanz.de/dcis/.

  6. http://hci.uni-konstanz.de/blendedinteraction2013/.

References

  1. Ju W, Leifer L (2008) The design of implicit interactions: making interactive systems less obnoxious. Des Issues 24(3):72–84. doi:10.1162/desi.2008.24.3.72

    Article  Google Scholar 

  2. Oulasvirta A (2008) When users “do” the Ubicomp. Interactions 15:6. doi:10.1145/1340961.1340963

    Article  Google Scholar 

  3. Weiser M (1991) The computer for the 21st century. Sci Am 3:94–104. doi:10.1145/329124.329126

    Article  Google Scholar 

  4. Whittaker S, Terveen L, Nardi BA (2000) A reference task agenda for HCI. In: Carroll JM (ed) Human–computer interaction in the new Millenium. Addison-Wesley, New York, pp 167–190

    Google Scholar 

  5. Byrne MD (2003) Cognitive architecture. In: Julie AJ, Andrew S (eds) The human–computer interaction handbook. L. Erlbaum Associates Inc., Hillsdale, NJ, pp 97–117

    Google Scholar 

  6. Kieras D (2003) Model-based evaluation. In: Julie AJ, Andrew S (eds) The human–computer interaction handbook. L. Erlbaum Associates Inc., Hillsdale, NJ, pp 1139–1151

    Google Scholar 

  7. Blackwell AF (1998) Metaphor in Diagrams. PhD Thesis, University of Cambridge, Cambridge

  8. Jetter H-C, Leifert S, Gerken J, Schubert S, Reiterer H (2012) Does (multi-)touch aid users’ spatial memory and navigation in ‘panning’ and in ‘zooming and panning’ UIs? In: Proceedings of International Working Conference on Advanced Visual Interfaces (AVI’12). ACM, New York, NY, pp 83–90

  9. Hutchins EL, Hollan JD, Norman DA (1985) Direct manipulation interfaces. Hum Comput Interact 1:311–338

    Article  Google Scholar 

  10. Ravasio P, Tscherter V (2007) User’s theories of the desktop metaphor or why we should seek metaphor-free interfaces. In: Kaptelinin V, Czerwinski M (eds) Beyond the desktop metaphor: designing integrated digital work environments. MIT Press, Cambridge, MA, pp 265–294

    Google Scholar 

  11. Constantine L, Lockwood L (1999) Software for use: a practical guide to the models and methods of usage-centered design. Addison Wesley, Reading, MA

    Google Scholar 

  12. Imaz M, Benyon D (2007) Designing with Blends: conceptual Foundations of Human-Computer Interaction and Software Engineering. MIT Press. doi:http://mitpress.mit.edu/books/designing-blends

    Google Scholar 

  13. Gibbs RW (2006) Embodiment and cognitive science. Cambridge University Press, Cambridge. doi:10.2277/0521811740

    Google Scholar 

  14. Fauconnier G, Turner M (2002) The way we think: conceptual blending and the mind’s hidden complexities. EUA Basic Books, New York

    Google Scholar 

  15. Dourish P (2004) Where the action is: the foundations of embodied interaction. First MIT Press Paperback Edition edn. MIT Press

  16. Lakoff G, Johnson M (1999) Philosophy in the flesh: the embodied mind and its challenge to Western thought. Basic Books, New York

    Google Scholar 

  17. Lakoff G, Johnson M (1980) Metaphors we live by. University of Chicago Press, Chicago

    Google Scholar 

  18. Jacob RJK, Girouard A, Hirshfield LM, Horn MS, Shaer O, Solovey ET, Zigelbaum J (2008) Reality-based interaction: a framework for post-WIMP interfaces. In: Proceedings of the SIGCHI conference on human factors in computing systems (CHI ‘08). ACM, New York, NY, pp 201–210

  19. Hurtienne J, Israel JH (2007) Image schemas and their metaphorical extensions: intuitive patterns for tangible interaction. In: Proceedings of the 1st international conference on tangible and embedded interaction (TEI ‘07). ACM, New York, NY, pp 127–134

  20. Hurtienne J, Israel JH, Weber K (2008) Cooking up real world business applications combining physicality, digitality, and image schemas. In: Proceedings of the 2nd international conference on Tangible and embedded interaction (TEI ‘08). ACM, New York, NY pp 239–246

  21. Blackwell AF (2006) The reification of metaphor as a design tool. ACM Trans Comput–Hum Interact 13(4):490–530. doi:10.1145/1188816.1188820

    Article  Google Scholar 

  22. Ba Nardi, Zarmer CL (1993) Beyond models and metaphors: visual formalisms in user interface design. J Vis Lang Comput 4:5–33. doi:10.1109/HICSS.1991.184010

    Article  Google Scholar 

  23. Bederson BB, Hollan JD, Perlin K, Meyer J, Bacon D, Furnas GW (1996) Pad++: a zoomable graphical sketchpad for exploring alternate interface physics. J Vis Lang Comput 7:3–32

    Article  Google Scholar 

  24. Jacob RJK, Girouard A, Hirshfield LM, Horn MS, Shaer O, Solovey ET, Zigelbaum J (2007) Reality-based interaction: unifying the new generation of interaction styles. In: CHI ‘07 extended abstracts on human factors in computing systems (CHI EA ‘07). ACM, New York, NY pp 2465–2470

  25. Boden MA (1994) Précis of the creative mind: myths and mechanisms. Behav Brain Sci 17(03):519–531. doi:10.1017/S0140525X0003569X

    Article  Google Scholar 

  26. Constantine L (1998) Use and misuse of metaphor. http://www.foruse.com/articles/metaphor.pdf. Accessed Feb 5, 2013

  27. Norman DA (2002) The design of everyday things. Basic Books, New York

    Google Scholar 

  28. Andrews C, Endert A, North C (2010) Space to think: large high-resolution displays for sensemaking. In: Proceedings of the SIGCHI conference on human factors in computing systems (CHI ‘10). ACM, New York, NY, pp 55–64

  29. Kirsh D (1995) The intelligent use of space. Artif Intell 73(1–2):31–68. doi:10.1016/0004-3702(94)00017-u

    Article  Google Scholar 

  30. Scott SD, Carpendale S, Inkpen KM (2004) Territoriality in collaborative tabletop workspaces. In: Proceedings of the 2004 ACM conference on computer-supported cooperative work (CSCW ‘04). ACM, New York, NY, pp 294–303

  31. Jetter H-C, Gerken J, Zöllner M, Reiterer H, Milic-Frayling N (2011) Materializing the query with facet-streams: a hybrid surface for collaborative search on tabletops. In: Proceedings of the SIGCHI conference on human factors in computing systems (CHI ‘11). ACM, New York, NY, pp 3013–3022

  32. Gerken J, Jetter H-C, Schmidt T (2010) Can “touch” get annoying? In ACM international conference on interactive tabletops and surfaces (ITS ‘10). ACM, New York, NY, pp 257–258

  33. Rosson MB, Carroll JM (2002) Usability engineering: scenario-based development of human–computer interaction. Morgan Kaufmann series in interactive technologies, 1st edn. Academic Press, San Fancisco

  34. Geyer F, Budzinski J, Reiterer H (2012) IdeaVis: a hybrid workspace and interactive visualization for paper-based collaborative sketching sessions. In: Proceedings of the 7th Nordic conference on human–computer interaction: Making Sense Through Design (NordiCHI ‘12). ACM, New York, NY, pp 331–340

  35. Collins D (1995) Designing object-oriented user interfaces. Benjamin Cummings, Redwood City, CA

    Google Scholar 

  36. Robinson M (1993) Design for unanticipated use. In: Proceedings of the third conference on European conference on computer-supported cooperative work (ECSCW’93). Kluwer Academic Publishers, Norwell, MA, pp 187–202

  37. O’Keefe B, Slutsky B, Iuliucci N, Nalbandian A, Thanedar A, Mokey S, Mival O (2013) Mobile experiences for tourism: brick city tours. In: CHI ‘13 Extended abstracts on human factors in computing systems (CHI EA ‘13). ACM, New York, NY, pp 1413–1418

  38. Streitz N, Tandler P, Müller-Tomfelde C, Konomi S (2001) Roomware: towards the next generation of human–computer interaction based on an integrated design of real and virtual worlds. In: Carroll JM (ed) Human–Computer Interaction in the New Millenium. Addison Wesley, Reading, MA, pp 551–576. doi:https://www.zotero.org/mikekuniavsky/items/itemKey/DC2Z7CWH

    Google Scholar 

  39. Geyer F, Pfeil U, Höchtl A, Budzinski J, Reiterer H (2011) Designing reality-based interfaces for creative group work. In: Proceedings of the 8th ACM conference on creativity and cognition (C&C ‘11). ACM, New York, NY, pp 165–174

  40. Beyer H, Holtzblatt K (1998) Contextual design: defining customer-centered systems. Morgan Kaufman, San Francisco

    Google Scholar 

  41. Rogers Y (2012) HCI theory: classical, modern, and contemporary, vol Lecture #14. Synthesis Lectures on Human-Centered Informatics. Morgan and Claypool

Download references

Acknowledgments

The authors would like to thank Johannes Zagermann and Daniel Klinkhammer for providing some of the illustrations and figures for this article. Furthermore, the authors would like to thank all the organizers and participants of our DCIS 2012 workshop “Designing Collaborative Interactive Spaces” at AVI 2012Footnote 5 and our workshop “Blended Interaction: Envisioning Future Collaborative Interactive Spaces” at CHI 2013Footnote 6 for the valuable feedback on and productive discussion of Blended Interaction that has helped to further develop our framework.

Author information

Authors and Affiliations

  1. Intel ICRI Cities, University College London, Gower Street, London, WC1E 6BT, UK

    Hans-Christian Jetter

  2. Human–Computer Interaction Group, University of Konstanz, Universitaetsstrasse 10, Box D73, 78457, Konstanz, Germany

    Harald Reiterer & Florian Geyer

Authors
  1. Hans-Christian Jetter
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Harald Reiterer
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Florian Geyer
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hans-Christian Jetter.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Jetter, HC., Reiterer, H. & Geyer, F. Blended Interaction: understanding natural human–computer interaction in post-WIMP interactive spaces. Pers Ubiquit Comput 18, 1139–1158 (2014). https://doi.org/10.1007/s00779-013-0725-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00779-013-0725-4

Keywords

Search

Navigation