Evaluating metaphor reification in tangible interfaces


Metaphors are a powerful conceptual device to reason about human actions. As such, they have been heavily used in designing and describing human computer interaction. Since they can address scripted text, verbal expression, imaging, sound, and gestures, they can also be considered in the design and analysis of multimodal interfaces. In this paper we discuss the description and evaluation of the relations between metaphors and their implementation in human computer interaction with a focus on tangible user interfaces (TUIs), a form of multimodal interface. The objective of this paper is to define how metaphors appear in a tangible context in order to support their evaluation. Relying on matching entities and operations between the domain of interaction and the domain of the digital application, we propose a conceptual framework based on three components: a structured representation of the mappings holding between the metaphor source, the metaphor target, the interface and the digital system; a conceptual model for describing metaphorical TUIs; three relevant properties, coherence, coverage and compliance, which define at what extent the implementation of a metaphorical tangible interface matches the metaphor. The conceptual framework is then validated and applied on a tangible prototype in an educational application.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7


  1. 1.

    This section and part of Sect. 5 have been published in a preliminary form in [18].

  2. 2.


  3. 3.

    We consider only differences relevant in terms of feedback to the user; e.g., a different computed result not returned to the user for subsequent interaction is not considered relevant in the metaphor analysis.

  4. 4.

    Cladograms are usually represented with the root at the bottom.


  1. 1.

    Alty JL, Knott RP (1999) Metaphor and human–computer interaction: a model based approach. In: Nehaniv CL (ed) Computation for metaphors, analogy, and agents. Springer, Berlin, pp 307–321

    Google Scholar 

  2. 2.

    Alty JL, Knott RP, Anderson B, Smyth M (2000) A framework for engineering metaphor at the user interface. Interact Comput 13(2):301–322

    Article  Google Scholar 

  3. 3.

    Andersen P (1992) Computer semiotics. Scand J Inf Syst 4:3–30

    Google Scholar 

  4. 4.

    Anderson B, Smyth M, Knott R, Bergan M, Bergan J, Alty J (1994) Minimising conceptual baggage: making choices about metaphor. In: People and computers IX, proceedings of HCI ’94. Cambridge University Press, New York, pp 179–194

  5. 5.

    Antle AN, Corness G, Bakker S, Droumeva M, Van Den Hoven E, Bevans A (2009) Designing to support reasoned imagination through embodied metaphor. In: Proceeding of the seventh ACM conference on creativity and cognition, CC 09, p 275

  6. 6.

    Averbukh V, Bakhterev M, Baydalin A, Ismagilov D, Trushenkova P (2007) Interface and visualization metaphors. In: Jacko J (ed) Human–computer interaction. Interaction platforms and techniques, lecture notes in computer science, vol 4551. Springer, Berlin, pp 13–22. doi:10.1007/978-3-540-73107-8-2

    Google Scholar 

  7. 7.

    Averbukh V et al (2008) Searching and analysis of interface and visualization metaphors. In: Asai K (ed) Human computer interaction: new developments. InTech, New York

    Google Scholar 

  8. 8.

    Bakker S, Antle A, Van Den Hoven E (2012) Embodied metaphors in tangible interaction design. Personal Ubiquitous Comput 16(4):433–449

    Article  Google Scholar 

  9. 9.

    Barr P (2003) User-interface metaphor in theory and practice. Master’s thesis, Victoria University of Wellington, Department of Mathematical and Computing Sciences

  10. 10.

    Barr P, Biddle R, Noble J (2002) A taxonomy of user interface metaphors. In: Proc. of SIGCHI-NZ Symposium On Computer-Human Interaction

  11. 11.

    Barr P, Biddle R, Noble J (2003) A semiotic model of user-interface metaphor. In: 6th International Workshop on Organisational Semiotics

  12. 12.

    Beaudouin-Lafon M (2000) Instrumental interaction. In: Proceedings of the SIGCHI conference on human factors in computing systems—CHI ’00, pp 446–453

  13. 13.

    Beaudouin-Lafon M (2004) Designing interaction, not interfaces. In: Proceedings of the working conference on advanced visual interfaces. ACM, New York, AVI ’04, pp 15–22

  14. 14.

    Black M (1954) Metaphor. Proc Aristot Soc N Ser 55:273–294

    Google Scholar 

  15. 15.

    Blackwell AF (2006) The reification of metaphor as a design tool. ACM Trans Comput Hum Interact 13(4):490–530

    Article  Google Scholar 

  16. 16.

    Bortolaso C, Dubois E, Duranthon F, Bach C (2011) Co-design of interactive museographic exhibits: the MIME case study. In: Ciolfi L, Scott K, Barbieri S (eds) Re-thinking technology in museums. University of Limerick, Limerick, pp 37–48

    Google Scholar 

  17. 17.

    Carroll JM, Mack RL (1985) Metaphor, computing systems, and active learning. Int J Man Mach Stud 22(1):39–57

    Article  Google Scholar 

  18. 18.

    Celentano A, Dubois E (2012) Metaphor modelling for tangible interfaces evaluation. In: Proceedings of the international working conference on advanced visual interfaces. ACM, New York, AVI ’12, pp 78–81

  19. 19.

    Coutaz J, Nigay L, Salber D, Blandford A, May J, Young R (1995) Four easy pieces for assessing the usability of multimodal interaction: the CARE properties. In: Nordby K, Helmersen PH, Gilmore DJ, Arnesen SA (eds) Proceedings of INTERACT ’95, IFIP TC13 interantional conference on human–computer interaction. Chapman & Hall, Lillehammer, pp 115–120

  20. 20.

    Coutrix C, Nigay L (2006) Mixed reality: a model of mixed interaction. In: Proceedings of the working conference on Advanced visual interfaces, AVI2006. ACM Press, New York, pp 43–50

  21. 21.

    Dubois E, Gray P (2007) A design-oriented information-flow refinement of the ASUR interaction model. In: Gulliksen J, Harning M, Palanque P, van der Veer G, Wesson J (eds) Proceedings of the international conference on engineering interactive systems EIS’07, vol 4940. Springer, Berlin, pp 465–482

  22. 22.

    Dubois E, Bortolaso C, Bach C, Duranthon F, Blanquer-Maumont A (2011) Design and evaluation of mixed interactive museographic exhibits. Int J Arts Technol 4(4):408–441

    Article  Google Scholar 

  23. 23.

    Fishkin K (2004) A taxonomy for and analysis of tangible interfaces. Pers Ubiquit Comput 8(5):347–358

    Article  Google Scholar 

  24. 24.

    Forceville CJ, Urios-Aparisi E (2009) Chapter 1. Introduction. In: Forceville CJ, Urios-Aparisi E (eds) Multimodal metaphor. Mouton de Gruyter, Berlin, pp 3–17

    Google Scholar 

  25. 25.

    Gero J (2004) The situated function–behaviour–structure framework. Des Stud 25(4):373–391

    Article  Google Scholar 

  26. 26.

    Hartshorne C, Weiss P (eds) (1936) Collected papers of Charles Sanders Peirce, vol 1–6. Harvard University Press, Cambridge

  27. 27.

    Hennig W (1950) Grundzüge einer Theorie der Phylogenetischen Systematik. Deutscher Zentralverlag, Berlin

    Google Scholar 

  28. 28.

    Hochmair H, Luttich K (2006) An analysis of the navigation metaphor-and why it works for the world wide web. Spatial Cognit Comput 6(3):235–278

    Article  Google Scholar 

  29. 29.

    Hornbaek K, Frokjaer E (2002) Evaluating user interfaces with metaphors of human thinking. In: Carbonell N, Stephanidis C (eds) Proceedings of the User interfaces for all 7th international conference on Universal access: theoretical perspectives, practice, and experience (ERCIM ’02). Springer, Berlin, pp 486–507

  30. 30.

    Hornecker E (2012) Beyond affordance. In: Proceedings of the sixth international conference on tangible, embedded and embodied interaction, TEI ’12, New York, New York, USA. ACM Press, New York, p 175

  31. 31.

    Hornecker E, Buur J (z2006) Getting a grip on tangible interaction: a framework on physical space and social interaction. In: Proc. CHI 2006. ACM Press, New York, pp 436–446

  32. 32.

    Hurtienne J (2009) Image schemas and design for intuitive use. Exploring new guidance for user interface design. PhD thesis, Technische Universitat Berlin

  33. 33.

    Hurtienne J, Blessing L (2007) Design for intuitive use—testing image schema theory for user interface design. In: ICED ’07, 16th international conference on engineering design, Paris, France, pp 1–12

  34. 34.

    Hurtienne J, Israel JH (2007) Image schemas and their metaphorical extensions: intuitive patterns for tangible interaction. In: Proceedings of the 1st international conference on Tangible and embedded interaction. ACM Press, New York, pp 127–134

  35. 35.

    Hurtienne J, Stößel C, Sturm C, Maus A, Rötting M, Langdon P, Clarkson J (2010) Physical gestures for abstract concepts: inclusive design with primary metaphors. Interact Comput 22(6):475–484

    Article  Google Scholar 

  36. 36.

    Indurkhya B (1986) Constrained semantic transference: a formal theory of metaphors. Synthese 68(3):515–551

    MathSciNet  Article  Google Scholar 

  37. 37.

    Ishii H (2008) Tangible bits: beyond pixels. In: Proceedings of the 2nd international conference on Tangible and embedded interaction. ACM, New York, TEI ’08, pp xv–xxv

  38. 38.

    Ishii H (2008) The tangible user interface and its evolution. Commun ACM 51(6):32–36

    Article  Google Scholar 

  39. 39.

    Ishii H, Ben-Joseph E, Underkoffler J, Yeung L, Chak D, Kanji Z, Piper B (2002) Augmented urban planning workbench: overlaying drawings, physical models and digital simulation. In: ISMAR ’02, proceedings of the 1st international symposium on mixed and augmented reality. IEEE Computer Society, Washington, DC, pp 203–211

  40. 40.

    Jacob RJK, Girouard A, Hirshfield LM, Horn MS, Shaer O, Solovey ET, Zigelbaum J (2008) Reality-based interaction. In: Proceeding of the twenty sixth annual CHI conference on human factors in computing systems, CHI 08. ACM Press, New York, p 201

  41. 41.

    Johnson M (1987) The body in the mind. University of Chicago Press, Chicago

    Google Scholar 

  42. 42.

    Jordà S, Geiger G, Alonso M, Kaltenbrunner M (2007) The reacTable: exploring the synergy between live music performance and tabletop tangible interfaces. In: Proceedings of the 1st international conference on tangible and embedded interaction. ACM, New York, pp 139–146

  43. 43.

    Jourde F, Laurillau Y, Morán AL, Nigay L (2008) Towards specifying multimodal collaborative user interfaces: a comparison of collaboration notations. In: Interactive systems. Design, specification, and verification, 15th international workshop, DSV-IS, Kingston, Canada, pp 281–286

  44. 44.

    Kay AC (1972) A personal computer for children of all ages. In: Proceedings of the ACM annual conference, vol 1. ACM, New York

  45. 45.

    Kintsch W (2000) Metaphor comprehension: a computational theory. Psychonom Bull Rev 7(2):257–266

    Article  Google Scholar 

  46. 46.

    Koleva B, Benford S, Ng KH, Rodden T (2003) A framework for tangible user interfaces. In: In workshop proceedings on real world user interfaces, mobile HCI conference 03, pp 257–264

  47. 47.

    Lakoff G (1987) Women, fire, and dangerous things: what categories reveal about the mind. University of Chicago Press, Chicago

    Google Scholar 

  48. 48.

    Lakoff G (1993) The contemporary theory of metaphor. Metaphor and thought, pp 202–251

  49. 49.

    Lakoff G, Johnson M (1980) Metaphors we live by. University of Chicago Press, Chicago

    Google Scholar 

  50. 50.

    Lamata P, Ali W, Cano A, Cornella J, Declerck J, Elle OJ, Freudenthal A, Furtado H, Kalkofen D, Naerum E, Samset E, Sánchez-Gonzalez P, Sánchez-Margallo FM, Schmalstieg D, Sette M, Stüdeli T, Sloten JV, Gómez EJ (2010) Augmented reality for minimally invasive surgery: overview and some recent advances. In: Maad S (ed) Augmented reality. InTech, Croatia

    Google Scholar 

  51. 51.

    Macaranas A, Antle AN, Riecke BE (2012) Bridging the gap. In: Proceedings of the sixth international conference on tangible, embedded and embodied interaction, TEI ’12. ACM, New York, p 161

  52. 52.

    Mackay WE, Fayard AL, Frobert L, Médini L (1998) Reinventing the familiar: exploring an augmented reality design space for air traffic control. In: Proceedings of the SIGCHI conference on human factors in computing systems. ACM Press/Addison-Wesley Publishing Co., New York, CHI ’98, pp 558–565

  53. 53.

    Maquil V, Ras E, Zephir O (2011) Understanding the characteristics of metaphors in tangible user interfaces. Mensch Comput 2011:1–6

    Google Scholar 

  54. 54.

    Maquil V, Zephir O, Ras E (2012) Creating metaphors for tangible user interfaces in collaborative urban planning: questions for designers and developers. In: Dugdale J, Masclet C, Grasso MA, Boujut JF, Hassanaly P (eds) From research to practice in the design of cooperative systems: results and open challenges. Springer, London, pp 137–151

    Google Scholar 

  55. 55.

    Marcus A (1994) Managing metaphors for advanced user interfaces. In: Proceedings of the workshop on advanced visual interfaces. ACM, New York, AVI ’94, pp 12–18. http://doi.acm.org/10.1145/192309.192317

  56. 56.

    McGee D, Cohen PR (2001) Creating tangible interfaces by augmenting physical objects with multimodal language. In: Proceedings of the 6th international conference on Intelligent user interfaces. ACM, New York, IUI ’01, pp 113–119

  57. 57.

    McNerney TS (2004) From turtles to tangible programming bricks: explorations in physical language design. Pers Ubiquitous Comput 8(5):326–337. doi:10.1007/s00779-004-0295-6

    Article  Google Scholar 

  58. 58.

    Norman DA (2002) The design of everyday things. Basic Books, New York

    Google Scholar 

  59. 59.

    Oppl S, Stary C (2011) Towards informed metaphor selection for TUIs. In: Proceedings of the 3rd ACM SIGCHI symposium on engineering interactive computing systems, EICS ’11. ACM, New York, pp 247–252

  60. 60.

    Perlman R (1976) Using computer technology to provide a creative learning environment for preschool children. Tech. rep., MIT AI Lab memo 360, Logo memo 24

  61. 61.

    Pittarello F, Stecca R (2010) Querying and navigating a database of images with the magical objects of the wizard Zurlino. In: IDC 2010, 9th international conference on interaction design and children. ACM Press, New York, pp 250–253

  62. 62.

    Pittarello F, Stecca R (2011) Mapping physical objects to digital functions: a tangible interface for querying and navigating a multimedia database. In: DEXA ’11, proceedings of the 22nd international workshop on database and expert systems applications. IEEE Computer Society, New York, pp 134–138

  63. 63.

    Poynor R (1995) The hand that rocks the cradle. ID the international design magazine, pp 60–65

  64. 64.

    Praggeljaz Group (2007) MIP: a method for identifying metaphorically used words in discourse. Metaphor Symbol 22(1):1–39

  65. 65.

    Price S, Jewitt C (2013) A multimodal approach to examining ’embodiment’ in tangible learning environments. In: Proceedings of the 7th international conference on tangible, embedded and embodied interaction. ACM, New York, TEI ’13, pp 43–50

  66. 66.

    Sajaniemi J, Stützle T (2007) Lightweight techniques for structural evaluation of animated metaphors. Interact Comput 19(4):457–471

    Article  Google Scholar 

  67. 67.

    Shaer O, Hornecker E (2010) Tangible user interfaces: past, present, and future directions. Found Trends Hum Comput Interact 3(1–2):1–137

    Google Scholar 

  68. 68.

    Sickenius de Souza C (1993) The semiotic engineering of user interface languages. Int J Man Mach Stud 39(5):753–773

    Article  Google Scholar 

  69. 69.

    Steinhart E (2005) Generating & interpreting metaphors with NETMET. APA Newslett 4(2):3–7

    Google Scholar 

  70. 70.

    Suzuki H, Kato H (1995) Interaction-level support for collaborative learning: algoblock—an open programming language. In: The first international conference on computer support for collaborative learning. L. Erlbaum Associates Inc., Hillsdale, CSCL ’95, pp 349–355. doi:10.3115/222020.222828

  71. 71.

    Svanaes D, Verplank W (2000) In search of metaphors for tangible user intefaces. In: Proceedings of DARE 2000 on designing augmented reality environments (DARE ’00), pp 121–129

  72. 72.

    Ullmer B, Ishii H (2000) Emerging frameworks for tangible user interfaces. IBM Syst J 39(3–4):915–931

    Article  Google Scholar 

  73. 73.

    Ullmer B, Ishii H, Jacob R (2005) Token+constraint systems for tangible interaction with digital information. ACM Trans Comput Hum Interact 12(1):81–118

    Article  Google Scholar 

  74. 74.

    Underkoffler J, Ishii H (1999) URP: a luminous-tangible workbench for urban planning and design. In: Proceedings of the SIGCHI conference on Human factors in computing systems, CHI ’99. ACM Press, New York, pp 386–393

  75. 75.

    Vaananen K, Schmidt J (1994) User interfaces for hypermedia: how to find good metaphors? In: Conference companion on Human factors in computing systems (CHI ’94), pp 263–264

  76. 76.

    Van Hees K, Engelen J (2013) Equivalent representations of multimodal user interfaces. Univers Access Inf Soc 12(4):339–368

    Article  Google Scholar 

  77. 77.

    Wagensberg J (2005) The “total” museum, a tool for social change. História, Ciências, Saúde Manguinhos 12(supplement):309–321

    Google Scholar 

  78. 78.

    Zhou CL, Yang Y, Huang XX (2007) Computational mechanisms for metaphor in languages: a survey. J Comput Sci Technol 22(2):308–319

    MathSciNet  Article  Google Scholar 

Download references


We thank Fabio Pittarello for fruitful discussions about his experiment, that we have used in Sect. 4.2 to introduce our framework. Figures 6 and 7 have been previously used in [22]. The authors wish to thank the Museum of Toulouse for providing the case study and the IJART journal, published by Inderscience, for allowing the reuse of this material.

Author information



Corresponding author

Correspondence to Emmanuel Dubois.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Celentano, A., Dubois, E. Evaluating metaphor reification in tangible interfaces. J Multimodal User Interfaces 9, 231–252 (2015). https://doi.org/10.1007/s12193-015-0198-z

Download citation


  • Human computer interaction
  • Metaphor
  • Multimodality
  • Tangible interface