Personal and Ubiquitous Computing

, Volume 18, Issue 5, pp 1139–1158 | Cite as

Blended Interaction: understanding natural human–computer interaction in post-WIMP interactive spaces

  • Hans-Christian Jetter
  • Harald Reiterer
  • Florian Geyer
Original Article


We introduce Blended Interaction, a new conceptual framework that helps to explain when users perceive user interfaces as “natural” or not. Based on recent findings from embodied cognition and cognitive linguistics, Blended Interaction provides a novel and more accurate description of the nature of human–computer interaction (HCI). In particular, it introduces the notion of conceptual blends to explain how users rely on familiar and real-world concepts whenever they learn to use new digital technologies. We apply Blended Interaction in the context of post-“Windows Icons Menu Pointer” interactive spaces. These spaces are ubiquitous computing environments for computer-supported collaboration of multiple users in a physical space or room, e.g., meeting rooms, design studios, or libraries, augmented with novel interactive technologies and digital computation, e.g., multi-touch walls, tabletops, and tablets. Ideally, in these spaces, the virtues of the familiar physical and social world are combined with that of the digital realm in a considered manner so that desired properties of each are preserved and a seemingly “natural” HCI is achieved. To support designers in this goal, we explain how the users’ conceptual systems use blends to tie together familiar concepts with the novel powers of digital computation. Furthermore, we introduce four domains of design to structure the underlying problem and design space: individual and social interaction, workflow, and physical environment. We introduce our framework by discussing related work, e.g., metaphors, mental models, direct manipulation, image schemas, reality-based interaction, and illustrate Blended Interaction using design decisions we made in recent projects.


Blended Interaction Post-WIMP Natural user interfaces Ubiquitous computing Computer-supported cooperative work Blends 



The authors would like to thank Johannes Zagermann and Daniel Klinkhammer for providing some of the illustrations and figures for this article. Furthermore, the authors would like to thank all the organizers and participants of our DCIS 2012 workshop “Designing Collaborative Interactive Spaces” at AVI 20125 and our workshop “Blended Interaction: Envisioning Future Collaborative Interactive Spaces” at CHI 20136 for the valuable feedback on and productive discussion of Blended Interaction that has helped to further develop our framework.


  1. 1.
    Ju W, Leifer L (2008) The design of implicit interactions: making interactive systems less obnoxious. Des Issues 24(3):72–84. doi: 10.1162/desi.2008.24.3.72 CrossRefGoogle Scholar
  2. 2.
    Oulasvirta A (2008) When users “do” the Ubicomp. Interactions 15:6. doi: 10.1145/1340961.1340963 CrossRefGoogle Scholar
  3. 3.
    Weiser M (1991) The computer for the 21st century. Sci Am 3:94–104. doi: 10.1145/329124.329126 CrossRefGoogle Scholar
  4. 4.
    Whittaker S, Terveen L, Nardi BA (2000) A reference task agenda for HCI. In: Carroll JM (ed) Human–computer interaction in the new Millenium. Addison-Wesley, New York, pp 167–190Google Scholar
  5. 5.
    Byrne MD (2003) Cognitive architecture. In: Julie AJ, Andrew S (eds) The human–computer interaction handbook. L. Erlbaum Associates Inc., Hillsdale, NJ, pp 97–117Google Scholar
  6. 6.
    Kieras D (2003) Model-based evaluation. In: Julie AJ, Andrew S (eds) The human–computer interaction handbook. L. Erlbaum Associates Inc., Hillsdale, NJ, pp 1139–1151Google Scholar
  7. 7.
    Blackwell AF (1998) Metaphor in Diagrams. PhD Thesis, University of Cambridge, CambridgeGoogle Scholar
  8. 8.
    Jetter H-C, Leifert S, Gerken J, Schubert S, Reiterer H (2012) Does (multi-)touch aid users’ spatial memory and navigation in ‘panning’ and in ‘zooming and panning’ UIs? In: Proceedings of International Working Conference on Advanced Visual Interfaces (AVI’12). ACM, New York, NY, pp 83–90Google Scholar
  9. 9.
    Hutchins EL, Hollan JD, Norman DA (1985) Direct manipulation interfaces. Hum Comput Interact 1:311–338CrossRefGoogle Scholar
  10. 10.
    Ravasio P, Tscherter V (2007) User’s theories of the desktop metaphor or why we should seek metaphor-free interfaces. In: Kaptelinin V, Czerwinski M (eds) Beyond the desktop metaphor: designing integrated digital work environments. MIT Press, Cambridge, MA, pp 265–294Google Scholar
  11. 11.
    Constantine L, Lockwood L (1999) Software for use: a practical guide to the models and methods of usage-centered design. Addison Wesley, Reading, MAGoogle Scholar
  12. 12.
    Imaz M, Benyon D (2007) Designing with Blends: conceptual Foundations of Human-Computer Interaction and Software Engineering. MIT Press. doi: Google Scholar
  13. 13.
    Gibbs RW (2006) Embodiment and cognitive science. Cambridge University Press, Cambridge. doi: 10.2277/0521811740 Google Scholar
  14. 14.
    Fauconnier G, Turner M (2002) The way we think: conceptual blending and the mind’s hidden complexities. EUA Basic Books, New YorkGoogle Scholar
  15. 15.
    Dourish P (2004) Where the action is: the foundations of embodied interaction. First MIT Press Paperback Edition edn. MIT PressGoogle Scholar
  16. 16.
    Lakoff G, Johnson M (1999) Philosophy in the flesh: the embodied mind and its challenge to Western thought. Basic Books, New YorkGoogle Scholar
  17. 17.
    Lakoff G, Johnson M (1980) Metaphors we live by. University of Chicago Press, ChicagoGoogle Scholar
  18. 18.
    Jacob RJK, Girouard A, Hirshfield LM, Horn MS, Shaer O, Solovey ET, Zigelbaum J (2008) Reality-based interaction: a framework for post-WIMP interfaces. In: Proceedings of the SIGCHI conference on human factors in computing systems (CHI ‘08). ACM, New York, NY, pp 201–210Google Scholar
  19. 19.
    Hurtienne J, Israel JH (2007) Image schemas and their metaphorical extensions: intuitive patterns for tangible interaction. In: Proceedings of the 1st international conference on tangible and embedded interaction (TEI ‘07). ACM, New York, NY, pp 127–134Google Scholar
  20. 20.
    Hurtienne J, Israel JH, Weber K (2008) Cooking up real world business applications combining physicality, digitality, and image schemas. In: Proceedings of the 2nd international conference on Tangible and embedded interaction (TEI ‘08). ACM, New York, NY pp 239–246Google Scholar
  21. 21.
    Blackwell AF (2006) The reification of metaphor as a design tool. ACM Trans Comput–Hum Interact 13(4):490–530. doi: 10.1145/1188816.1188820 CrossRefGoogle Scholar
  22. 22.
    Ba Nardi, Zarmer CL (1993) Beyond models and metaphors: visual formalisms in user interface design. J Vis Lang Comput 4:5–33. doi: 10.1109/HICSS.1991.184010 CrossRefGoogle Scholar
  23. 23.
    Bederson BB, Hollan JD, Perlin K, Meyer J, Bacon D, Furnas GW (1996) Pad++: a zoomable graphical sketchpad for exploring alternate interface physics. J Vis Lang Comput 7:3–32CrossRefGoogle Scholar
  24. 24.
    Jacob RJK, Girouard A, Hirshfield LM, Horn MS, Shaer O, Solovey ET, Zigelbaum J (2007) Reality-based interaction: unifying the new generation of interaction styles. In: CHI ‘07 extended abstracts on human factors in computing systems (CHI EA ‘07). ACM, New York, NY pp 2465–2470Google Scholar
  25. 25.
    Boden MA (1994) Précis of the creative mind: myths and mechanisms. Behav Brain Sci 17(03):519–531. doi: 10.1017/S0140525X0003569X CrossRefGoogle Scholar
  26. 26.
    Constantine L (1998) Use and misuse of metaphor. Accessed Feb 5, 2013
  27. 27.
    Norman DA (2002) The design of everyday things. Basic Books, New YorkGoogle Scholar
  28. 28.
    Andrews C, Endert A, North C (2010) Space to think: large high-resolution displays for sensemaking. In: Proceedings of the SIGCHI conference on human factors in computing systems (CHI ‘10). ACM, New York, NY, pp 55–64Google Scholar
  29. 29.
    Kirsh D (1995) The intelligent use of space. Artif Intell 73(1–2):31–68. doi: 10.1016/0004-3702(94)00017-u CrossRefGoogle Scholar
  30. 30.
    Scott SD, Carpendale S, Inkpen KM (2004) Territoriality in collaborative tabletop workspaces. In: Proceedings of the 2004 ACM conference on computer-supported cooperative work (CSCW ‘04). ACM, New York, NY, pp 294–303Google Scholar
  31. 31.
    Jetter H-C, Gerken J, Zöllner M, Reiterer H, Milic-Frayling N (2011) Materializing the query with facet-streams: a hybrid surface for collaborative search on tabletops. In: Proceedings of the SIGCHI conference on human factors in computing systems (CHI ‘11). ACM, New York, NY, pp 3013–3022Google Scholar
  32. 32.
    Gerken J, Jetter H-C, Schmidt T (2010) Can “touch” get annoying? In ACM international conference on interactive tabletops and surfaces (ITS ‘10). ACM, New York, NY, pp 257–258Google Scholar
  33. 33.
    Rosson MB, Carroll JM (2002) Usability engineering: scenario-based development of human–computer interaction. Morgan Kaufmann series in interactive technologies, 1st edn. Academic Press, San FanciscoGoogle Scholar
  34. 34.
    Geyer F, Budzinski J, Reiterer H (2012) IdeaVis: a hybrid workspace and interactive visualization for paper-based collaborative sketching sessions. In: Proceedings of the 7th Nordic conference on human–computer interaction: Making Sense Through Design (NordiCHI ‘12). ACM, New York, NY, pp 331–340Google Scholar
  35. 35.
    Collins D (1995) Designing object-oriented user interfaces. Benjamin Cummings, Redwood City, CAGoogle Scholar
  36. 36.
    Robinson M (1993) Design for unanticipated use. In: Proceedings of the third conference on European conference on computer-supported cooperative work (ECSCW’93). Kluwer Academic Publishers, Norwell, MA, pp 187–202Google Scholar
  37. 37.
    O’Keefe B, Slutsky B, Iuliucci N, Nalbandian A, Thanedar A, Mokey S, Mival O (2013) Mobile experiences for tourism: brick city tours. In: CHI ‘13 Extended abstracts on human factors in computing systems (CHI EA ‘13). ACM, New York, NY, pp 1413–1418Google Scholar
  38. 38.
    Streitz N, Tandler P, Müller-Tomfelde C, Konomi S (2001) Roomware: towards the next generation of human–computer interaction based on an integrated design of real and virtual worlds. In: Carroll JM (ed) Human–Computer Interaction in the New Millenium. Addison Wesley, Reading, MA, pp 551–576. doi: Google Scholar
  39. 39.
    Geyer F, Pfeil U, Höchtl A, Budzinski J, Reiterer H (2011) Designing reality-based interfaces for creative group work. In: Proceedings of the 8th ACM conference on creativity and cognition (C&C ‘11). ACM, New York, NY, pp 165–174Google Scholar
  40. 40.
    Beyer H, Holtzblatt K (1998) Contextual design: defining customer-centered systems. Morgan Kaufman, San FranciscoGoogle Scholar
  41. 41.
    Rogers Y (2012) HCI theory: classical, modern, and contemporary, vol Lecture #14. Synthesis Lectures on Human-Centered Informatics. Morgan and ClaypoolGoogle Scholar

Copyright information

© Springer-Verlag London 2013

Authors and Affiliations

  • Hans-Christian Jetter
    • 1
  • Harald Reiterer
    • 2
  • Florian Geyer
    • 2
  1. 1.Intel ICRI CitiesUniversity College LondonLondonUK
  2. 2.Human–Computer Interaction GroupUniversity of KonstanzKonstanzGermany

Personalised recommendations