Journal on Multimodal User Interfaces

, Volume 9, Issue 3, pp 231–252 | Cite as

Evaluating metaphor reification in tangible interfaces

  • Augusto Celentano
  • Emmanuel DuboisEmail author
Original Paper


Metaphors are a powerful conceptual device to reason about human actions. As such, they have been heavily used in designing and describing human computer interaction. Since they can address scripted text, verbal expression, imaging, sound, and gestures, they can also be considered in the design and analysis of multimodal interfaces. In this paper we discuss the description and evaluation of the relations between metaphors and their implementation in human computer interaction with a focus on tangible user interfaces (TUIs), a form of multimodal interface. The objective of this paper is to define how metaphors appear in a tangible context in order to support their evaluation. Relying on matching entities and operations between the domain of interaction and the domain of the digital application, we propose a conceptual framework based on three components: a structured representation of the mappings holding between the metaphor source, the metaphor target, the interface and the digital system; a conceptual model for describing metaphorical TUIs; three relevant properties, coherence, coverage and compliance, which define at what extent the implementation of a metaphorical tangible interface matches the metaphor. The conceptual framework is then validated and applied on a tangible prototype in an educational application.


Human computer interaction Metaphor Multimodality Tangible interface 



We thank Fabio Pittarello for fruitful discussions about his experiment, that we have used in Sect. 4.2 to introduce our framework. Figures 6 and 7 have been previously used in [22]. The authors wish to thank the Museum of Toulouse for providing the case study and the IJART journal, published by Inderscience, for allowing the reuse of this material.


  1. 1.
    Alty JL, Knott RP (1999) Metaphor and human–computer interaction: a model based approach. In: Nehaniv CL (ed) Computation for metaphors, analogy, and agents. Springer, Berlin, pp 307–321CrossRefGoogle Scholar
  2. 2.
    Alty JL, Knott RP, Anderson B, Smyth M (2000) A framework for engineering metaphor at the user interface. Interact Comput 13(2):301–322CrossRefGoogle Scholar
  3. 3.
    Andersen P (1992) Computer semiotics. Scand J Inf Syst 4:3–30Google Scholar
  4. 4.
    Anderson B, Smyth M, Knott R, Bergan M, Bergan J, Alty J (1994) Minimising conceptual baggage: making choices about metaphor. In: People and computers IX, proceedings of HCI ’94. Cambridge University Press, New York, pp 179–194Google Scholar
  5. 5.
    Antle AN, Corness G, Bakker S, Droumeva M, Van Den Hoven E, Bevans A (2009) Designing to support reasoned imagination through embodied metaphor. In: Proceeding of the seventh ACM conference on creativity and cognition, CC 09, p 275Google Scholar
  6. 6.
    Averbukh V, Bakhterev M, Baydalin A, Ismagilov D, Trushenkova P (2007) Interface and visualization metaphors. In: Jacko J (ed) Human–computer interaction. Interaction platforms and techniques, lecture notes in computer science, vol 4551. Springer, Berlin, pp 13–22. doi: 10.1007/978-3-540-73107-8-2 CrossRefGoogle Scholar
  7. 7.
    Averbukh V et al (2008) Searching and analysis of interface and visualization metaphors. In: Asai K (ed) Human computer interaction: new developments. InTech, New YorkGoogle Scholar
  8. 8.
    Bakker S, Antle A, Van Den Hoven E (2012) Embodied metaphors in tangible interaction design. Personal Ubiquitous Comput 16(4):433–449CrossRefGoogle Scholar
  9. 9.
    Barr P (2003) User-interface metaphor in theory and practice. Master’s thesis, Victoria University of Wellington, Department of Mathematical and Computing SciencesGoogle Scholar
  10. 10.
    Barr P, Biddle R, Noble J (2002) A taxonomy of user interface metaphors. In: Proc. of SIGCHI-NZ Symposium On Computer-Human InteractionGoogle Scholar
  11. 11.
    Barr P, Biddle R, Noble J (2003) A semiotic model of user-interface metaphor. In: 6th International Workshop on Organisational SemioticsGoogle Scholar
  12. 12.
    Beaudouin-Lafon M (2000) Instrumental interaction. In: Proceedings of the SIGCHI conference on human factors in computing systems—CHI ’00, pp 446–453Google Scholar
  13. 13.
    Beaudouin-Lafon M (2004) Designing interaction, not interfaces. In: Proceedings of the working conference on advanced visual interfaces. ACM, New York, AVI ’04, pp 15–22Google Scholar
  14. 14.
    Black M (1954) Metaphor. Proc Aristot Soc N Ser 55:273–294Google Scholar
  15. 15.
    Blackwell AF (2006) The reification of metaphor as a design tool. ACM Trans Comput Hum Interact 13(4):490–530CrossRefGoogle Scholar
  16. 16.
    Bortolaso C, Dubois E, Duranthon F, Bach C (2011) Co-design of interactive museographic exhibits: the MIME case study. In: Ciolfi L, Scott K, Barbieri S (eds) Re-thinking technology in museums. University of Limerick, Limerick, pp 37–48Google Scholar
  17. 17.
    Carroll JM, Mack RL (1985) Metaphor, computing systems, and active learning. Int J Man Mach Stud 22(1):39–57CrossRefGoogle Scholar
  18. 18.
    Celentano A, Dubois E (2012) Metaphor modelling for tangible interfaces evaluation. In: Proceedings of the international working conference on advanced visual interfaces. ACM, New York, AVI ’12, pp 78–81Google Scholar
  19. 19.
    Coutaz J, Nigay L, Salber D, Blandford A, May J, Young R (1995) Four easy pieces for assessing the usability of multimodal interaction: the CARE properties. In: Nordby K, Helmersen PH, Gilmore DJ, Arnesen SA (eds) Proceedings of INTERACT ’95, IFIP TC13 interantional conference on human–computer interaction. Chapman & Hall, Lillehammer, pp 115–120Google Scholar
  20. 20.
    Coutrix C, Nigay L (2006) Mixed reality: a model of mixed interaction. In: Proceedings of the working conference on Advanced visual interfaces, AVI2006. ACM Press, New York, pp 43–50Google Scholar
  21. 21.
    Dubois E, Gray P (2007) A design-oriented information-flow refinement of the ASUR interaction model. In: Gulliksen J, Harning M, Palanque P, van der Veer G, Wesson J (eds) Proceedings of the international conference on engineering interactive systems EIS’07, vol 4940. Springer, Berlin, pp 465–482Google Scholar
  22. 22.
    Dubois E, Bortolaso C, Bach C, Duranthon F, Blanquer-Maumont A (2011) Design and evaluation of mixed interactive museographic exhibits. Int J Arts Technol 4(4):408–441CrossRefGoogle Scholar
  23. 23.
    Fishkin K (2004) A taxonomy for and analysis of tangible interfaces. Pers Ubiquit Comput 8(5):347–358CrossRefGoogle Scholar
  24. 24.
    Forceville CJ, Urios-Aparisi E (2009) Chapter 1. Introduction. In: Forceville CJ, Urios-Aparisi E (eds) Multimodal metaphor. Mouton de Gruyter, Berlin, pp 3–17CrossRefGoogle Scholar
  25. 25.
    Gero J (2004) The situated function–behaviour–structure framework. Des Stud 25(4):373–391CrossRefGoogle Scholar
  26. 26.
    Hartshorne C, Weiss P (eds) (1936) Collected papers of Charles Sanders Peirce, vol 1–6. Harvard University Press, CambridgeGoogle Scholar
  27. 27.
    Hennig W (1950) Grundzüge einer Theorie der Phylogenetischen Systematik. Deutscher Zentralverlag, BerlinGoogle Scholar
  28. 28.
    Hochmair H, Luttich K (2006) An analysis of the navigation metaphor-and why it works for the world wide web. Spatial Cognit Comput 6(3):235–278CrossRefGoogle Scholar
  29. 29.
    Hornbaek K, Frokjaer E (2002) Evaluating user interfaces with metaphors of human thinking. In: Carbonell N, Stephanidis C (eds) Proceedings of the User interfaces for all 7th international conference on Universal access: theoretical perspectives, practice, and experience (ERCIM ’02). Springer, Berlin, pp 486–507Google Scholar
  30. 30.
    Hornecker E (2012) Beyond affordance. In: Proceedings of the sixth international conference on tangible, embedded and embodied interaction, TEI ’12, New York, New York, USA. ACM Press, New York, p 175Google Scholar
  31. 31.
    Hornecker E, Buur J (z2006) Getting a grip on tangible interaction: a framework on physical space and social interaction. In: Proc. CHI 2006. ACM Press, New York, pp 436–446Google Scholar
  32. 32.
    Hurtienne J (2009) Image schemas and design for intuitive use. Exploring new guidance for user interface design. PhD thesis, Technische Universitat BerlinGoogle Scholar
  33. 33.
    Hurtienne J, Blessing L (2007) Design for intuitive use—testing image schema theory for user interface design. In: ICED ’07, 16th international conference on engineering design, Paris, France, pp 1–12Google Scholar
  34. 34.
    Hurtienne J, Israel JH (2007) Image schemas and their metaphorical extensions: intuitive patterns for tangible interaction. In: Proceedings of the 1st international conference on Tangible and embedded interaction. ACM Press, New York, pp 127–134Google Scholar
  35. 35.
    Hurtienne J, Stößel C, Sturm C, Maus A, Rötting M, Langdon P, Clarkson J (2010) Physical gestures for abstract concepts: inclusive design with primary metaphors. Interact Comput 22(6):475–484CrossRefGoogle Scholar
  36. 36.
    Indurkhya B (1986) Constrained semantic transference: a formal theory of metaphors. Synthese 68(3):515–551MathSciNetCrossRefGoogle Scholar
  37. 37.
    Ishii H (2008) Tangible bits: beyond pixels. In: Proceedings of the 2nd international conference on Tangible and embedded interaction. ACM, New York, TEI ’08, pp xv–xxvGoogle Scholar
  38. 38.
    Ishii H (2008) The tangible user interface and its evolution. Commun ACM 51(6):32–36CrossRefGoogle Scholar
  39. 39.
    Ishii H, Ben-Joseph E, Underkoffler J, Yeung L, Chak D, Kanji Z, Piper B (2002) Augmented urban planning workbench: overlaying drawings, physical models and digital simulation. In: ISMAR ’02, proceedings of the 1st international symposium on mixed and augmented reality. IEEE Computer Society, Washington, DC, pp 203–211Google Scholar
  40. 40.
    Jacob RJK, Girouard A, Hirshfield LM, Horn MS, Shaer O, Solovey ET, Zigelbaum J (2008) Reality-based interaction. In: Proceeding of the twenty sixth annual CHI conference on human factors in computing systems, CHI 08. ACM Press, New York, p 201Google Scholar
  41. 41.
    Johnson M (1987) The body in the mind. University of Chicago Press, ChicagoGoogle Scholar
  42. 42.
    Jordà S, Geiger G, Alonso M, Kaltenbrunner M (2007) The reacTable: exploring the synergy between live music performance and tabletop tangible interfaces. In: Proceedings of the 1st international conference on tangible and embedded interaction. ACM, New York, pp 139–146Google Scholar
  43. 43.
    Jourde F, Laurillau Y, Morán AL, Nigay L (2008) Towards specifying multimodal collaborative user interfaces: a comparison of collaboration notations. In: Interactive systems. Design, specification, and verification, 15th international workshop, DSV-IS, Kingston, Canada, pp 281–286Google Scholar
  44. 44.
    Kay AC (1972) A personal computer for children of all ages. In: Proceedings of the ACM annual conference, vol 1. ACM, New YorkGoogle Scholar
  45. 45.
    Kintsch W (2000) Metaphor comprehension: a computational theory. Psychonom Bull Rev 7(2):257–266CrossRefGoogle Scholar
  46. 46.
    Koleva B, Benford S, Ng KH, Rodden T (2003) A framework for tangible user interfaces. In: In workshop proceedings on real world user interfaces, mobile HCI conference 03, pp 257–264Google Scholar
  47. 47.
    Lakoff G (1987) Women, fire, and dangerous things: what categories reveal about the mind. University of Chicago Press, ChicagoCrossRefGoogle Scholar
  48. 48.
    Lakoff G (1993) The contemporary theory of metaphor. Metaphor and thought, pp 202–251Google Scholar
  49. 49.
    Lakoff G, Johnson M (1980) Metaphors we live by. University of Chicago Press, ChicagoGoogle Scholar
  50. 50.
    Lamata P, Ali W, Cano A, Cornella J, Declerck J, Elle OJ, Freudenthal A, Furtado H, Kalkofen D, Naerum E, Samset E, Sánchez-Gonzalez P, Sánchez-Margallo FM, Schmalstieg D, Sette M, Stüdeli T, Sloten JV, Gómez EJ (2010) Augmented reality for minimally invasive surgery: overview and some recent advances. In: Maad S (ed) Augmented reality. InTech, CroatiaGoogle Scholar
  51. 51.
    Macaranas A, Antle AN, Riecke BE (2012) Bridging the gap. In: Proceedings of the sixth international conference on tangible, embedded and embodied interaction, TEI ’12. ACM, New York, p 161Google Scholar
  52. 52.
    Mackay WE, Fayard AL, Frobert L, Médini L (1998) Reinventing the familiar: exploring an augmented reality design space for air traffic control. In: Proceedings of the SIGCHI conference on human factors in computing systems. ACM Press/Addison-Wesley Publishing Co., New York, CHI ’98, pp 558–565Google Scholar
  53. 53.
    Maquil V, Ras E, Zephir O (2011) Understanding the characteristics of metaphors in tangible user interfaces. Mensch Comput 2011:1–6Google Scholar
  54. 54.
    Maquil V, Zephir O, Ras E (2012) Creating metaphors for tangible user interfaces in collaborative urban planning: questions for designers and developers. In: Dugdale J, Masclet C, Grasso MA, Boujut JF, Hassanaly P (eds) From research to practice in the design of cooperative systems: results and open challenges. Springer, London, pp 137–151CrossRefGoogle Scholar
  55. 55.
    Marcus A (1994) Managing metaphors for advanced user interfaces. In: Proceedings of the workshop on advanced visual interfaces. ACM, New York, AVI ’94, pp 12–18.
  56. 56.
    McGee D, Cohen PR (2001) Creating tangible interfaces by augmenting physical objects with multimodal language. In: Proceedings of the 6th international conference on Intelligent user interfaces. ACM, New York, IUI ’01, pp 113–119Google Scholar
  57. 57.
    McNerney TS (2004) From turtles to tangible programming bricks: explorations in physical language design. Pers Ubiquitous Comput 8(5):326–337. doi: 10.1007/s00779-004-0295-6 CrossRefGoogle Scholar
  58. 58.
    Norman DA (2002) The design of everyday things. Basic Books, New YorkGoogle Scholar
  59. 59.
    Oppl S, Stary C (2011) Towards informed metaphor selection for TUIs. In: Proceedings of the 3rd ACM SIGCHI symposium on engineering interactive computing systems, EICS ’11. ACM, New York, pp 247–252Google Scholar
  60. 60.
    Perlman R (1976) Using computer technology to provide a creative learning environment for preschool children. Tech. rep., MIT AI Lab memo 360, Logo memo 24Google Scholar
  61. 61.
    Pittarello F, Stecca R (2010) Querying and navigating a database of images with the magical objects of the wizard Zurlino. In: IDC 2010, 9th international conference on interaction design and children. ACM Press, New York, pp 250–253Google Scholar
  62. 62.
    Pittarello F, Stecca R (2011) Mapping physical objects to digital functions: a tangible interface for querying and navigating a multimedia database. In: DEXA ’11, proceedings of the 22nd international workshop on database and expert systems applications. IEEE Computer Society, New York, pp 134–138Google Scholar
  63. 63.
    Poynor R (1995) The hand that rocks the cradle. ID the international design magazine, pp 60–65Google Scholar
  64. 64.
    Praggeljaz Group (2007) MIP: a method for identifying metaphorically used words in discourse. Metaphor Symbol 22(1):1–39Google Scholar
  65. 65.
    Price S, Jewitt C (2013) A multimodal approach to examining ’embodiment’ in tangible learning environments. In: Proceedings of the 7th international conference on tangible, embedded and embodied interaction. ACM, New York, TEI ’13, pp 43–50Google Scholar
  66. 66.
    Sajaniemi J, Stützle T (2007) Lightweight techniques for structural evaluation of animated metaphors. Interact Comput 19(4):457–471CrossRefGoogle Scholar
  67. 67.
    Shaer O, Hornecker E (2010) Tangible user interfaces: past, present, and future directions. Found Trends Hum Comput Interact 3(1–2):1–137Google Scholar
  68. 68.
    Sickenius de Souza C (1993) The semiotic engineering of user interface languages. Int J Man Mach Stud 39(5):753–773CrossRefGoogle Scholar
  69. 69.
    Steinhart E (2005) Generating & interpreting metaphors with NETMET. APA Newslett 4(2):3–7Google Scholar
  70. 70.
    Suzuki H, Kato H (1995) Interaction-level support for collaborative learning: algoblock—an open programming language. In: The first international conference on computer support for collaborative learning. L. Erlbaum Associates Inc., Hillsdale, CSCL ’95, pp 349–355. doi: 10.3115/222020.222828
  71. 71.
    Svanaes D, Verplank W (2000) In search of metaphors for tangible user intefaces. In: Proceedings of DARE 2000 on designing augmented reality environments (DARE ’00), pp 121–129Google Scholar
  72. 72.
    Ullmer B, Ishii H (2000) Emerging frameworks for tangible user interfaces. IBM Syst J 39(3–4):915–931CrossRefGoogle Scholar
  73. 73.
    Ullmer B, Ishii H, Jacob R (2005) Token+constraint systems for tangible interaction with digital information. ACM Trans Comput Hum Interact 12(1):81–118CrossRefGoogle Scholar
  74. 74.
    Underkoffler J, Ishii H (1999) URP: a luminous-tangible workbench for urban planning and design. In: Proceedings of the SIGCHI conference on Human factors in computing systems, CHI ’99. ACM Press, New York, pp 386–393Google Scholar
  75. 75.
    Vaananen K, Schmidt J (1994) User interfaces for hypermedia: how to find good metaphors? In: Conference companion on Human factors in computing systems (CHI ’94), pp 263–264Google Scholar
  76. 76.
    Van Hees K, Engelen J (2013) Equivalent representations of multimodal user interfaces. Univers Access Inf Soc 12(4):339–368CrossRefGoogle Scholar
  77. 77.
    Wagensberg J (2005) The “total” museum, a tool for social change. História, Ciências, Saúde Manguinhos 12(supplement):309–321Google Scholar
  78. 78.
    Zhou CL, Yang Y, Huang XX (2007) Computational mechanisms for metaphor in languages: a survey. J Comput Sci Technol 22(2):308–319MathSciNetCrossRefGoogle Scholar

Copyright information

© OpenInterface Association 2015

Authors and Affiliations

  1. 1.DAISUniversità Ca’ Foscari VeneziaVeniceItaly
  2. 2.IRITUniversité Paul Sabatier, CNRSToulouseFrance

Personalised recommendations