Measuring the Effect of Tangible Interaction on Design Cognition

  • Mary Lou MaherEmail author
  • John Gero
  • Lina Lee
  • Rongrong Yu
  • Tim Clausner
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9743)


Recent developments in interaction design provide gesture and tangible interaction as an alternative or complement to mouse, keyboard, and touch interaction. Tangible user interfaces provide affordances that encourage and facilitate specific actions on physical objects. There is evidence that gesture and action affect cognition, and therefore it is hypothesized that the affordances of tangible interaction will affect design cognition. In this paper we report on the analysis of experimental data in which participants are asked to make word combinations from a set of six nouns and give them meaning. The task is presented as a design task with references to function, behavior, and structure of the word combination meanings. The participants performed the task in two conditions: one in which grasping the words was afforded and one in which pointing at the words was afforded. We segmented and coded the verbal data using the function-behavior-structure coding scheme to compare the participants’ references to design issues across the two conditions. The results show that the two conditions differ in the phase in which they search for word combinations and the phase in which they described new meanings.


Tangible interaction Cognition Creativity 



This research was funded by NFS grant no. IIS-1218160 to M.L. Maher, T. Clausner, and A. Druin. The author contributions of this paper are: Clausner led the design of the experiment, which yielded data for coding and analysis both by cognitive scientific methods (in preparation), and the FBS coding and analysis (this paper) were led by Maher and Gero, with assistance by Lee and Yu.


  1. 1.
    Carlson, S.M., White, R.E.: Executive function, pretend play, and imagination. In: Taylor, M. (ed.) The Oxford Handbook of the Development of Imagination, pp. 161–174. Oxford University Press, New York (2013)Google Scholar
  2. 2.
    Smith, L.H., White, A.J., Callahan, C.M., Hartman, R.K., Westberg, K.L.: Scales for Rating the Behavioral Characteristics of Superior Students. Creative Learning Press, Mansfield Center (1976)Google Scholar
  3. 3.
    Fitzmaurice, G.W., Ishii, H., Buxton, W.A.S.: Bricks: laying the foundations for graspable user interfaces. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 442–449. ACM Press, New York (1995)Google Scholar
  4. 4.
    Ishii, H., Ullmer, B.: Tangible bits: towards seamless interfaces between people, bits and atoms. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 234–241. ACM Press, New York (1997)Google Scholar
  5. 5.
    Fjeld, M., Bichsel, M., Rauterberg, M.: BUILD-IT: an intuitive design tool based on direct object manipulation. In: Wachsmut, I., Frölich, M. (eds.) Gesture and Sign Language in Human-Computer Interaction, pp. 297–308. Springer, Heidelberg (1998)CrossRefGoogle Scholar
  6. 6.
    Lee, C., Ma, Y., Jeng, T.: A spatially-aware tangible user interface for computer-aided design. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 960–961. ACM Press, New York (2003)Google Scholar
  7. 7.
    Kim, M.J., Maher, M.L.: The impact of tangible user interfaces on designers’ spatial cognition. HCI 23(2), 101–137 (2008)Google Scholar
  8. 8.
    Brereton, M., McGarry, B.: An observational study of how objects support engineering design thinking and communication: implications for the design of tangible media. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 217–224. ACM Press, New York (2000)Google Scholar
  9. 9.
    Vega, M.D., Marschark, M., Intons-Peterson, M.J., Johnson-Laird, P.N., Denis, M.: Representations of visuospatial cognition: a discussion. In: De Vega, M.M.M., Intons-Peterson, M.J., Johnson-Laird, P.N., Denis, M. (eds.) Models of Visuospatial Cognition, pp. 198–226. Oxford University Press, New York (1996)Google Scholar
  10. 10.
    Cook, S.W., Mitchell, Z., Goldin-Meadow, S.: Gesturing makes learning last. Cognition 106(2), 1047–1058 (2008)CrossRefGoogle Scholar
  11. 11.
    Kessell, A., Tversky, B.: Using diagrams and gestures to think and talk about insight problems. In: Proceedings of the Meeting of the Cognitive Science Society, pp. 2528–2537. Erlbaum, Mahwah (2006)Google Scholar
  12. 12.
    Trofatter, C., Kontra, C., Beilock, S., Goldin-Meadow, S.: Gesturing has a larger impact on problem-solving than action, even when action is accompanied by words. Lang. Cogn. Neurosci. 30(3), 251–260 (2014)CrossRefGoogle Scholar
  13. 13.
    Alibali, M.W., DiRusso, A.A.: The function of gesture in learning to count: more than keeping track. Cogn. Dev. 14(1), 37–56 (1999)CrossRefGoogle Scholar
  14. 14.
    Carlson, R.A., Avraamides, M.N., Cary, M., Strasberg, S.: What do the hands externalize in simple arithmetic? J. Exp. Psychol.: Learn. Mem. Cogn. 33(4), 747–756 (2007)Google Scholar
  15. 15.
    Goldin-Meadow, S., Cook, S.W., Mitchell, Z.A.: Gesturing gives children new ideas about math. Psychol. Sci. 20, 267–272 (2009)CrossRefGoogle Scholar
  16. 16.
    Cook, S.W., Goldin-Meadow, S.: The role of gesture in learning: do children use their hands to change their minds? J. Cogn. Dev. 7(2), 211–232 (2006)CrossRefGoogle Scholar
  17. 17.
    Goldin-Meadow, S., Beilock, S.: Action’s influence on thought: the case of gesture. Perspect. Psychol. Sci.: J. Assoc. Psychol. Sci. 5(6), 664–674 (2010)CrossRefGoogle Scholar
  18. 18.
    Boden, M.: The Creative Mind: Myths and Mechanisms, 2nd edn. Routledge, London and New York (2003)Google Scholar
  19. 19.
    Gero, J.S.: Computational models of innovative and creative design processes. Technol. Forecast. Soc. Chang. 64, 183–196 (2000)CrossRefGoogle Scholar
  20. 20.
    Clausner, T.C., Maher, M.L., Gonzolez, A.: Conceptual combination modulated by action using tangible computers. In: Poster Presented at the 37th Annual Meeting of the Cognitive Science Society, Pasadena, CA (2015)Google Scholar
  21. 21.
    Maher, M.L.: Evaluating creativity in humans, computers, and collectively intelligent systems. In: Proceedings of DESIRE 2010: Creativity and Innovation in Design, pp. 22–28. Aurhus, Denmark (2010)Google Scholar
  22. 22.
    Gero, J.S.: Design prototypes: a knowledge representation schema for design. AI Mag. 11(4), 26–36 (1990)Google Scholar
  23. 23.
    Wisniewski, E.J., Gentner, D.: On the combinatorial semantics of noun pairs: minor and major adjustments to meaning. In: Simpson, G.B. (ed.) Understanding Word and Sentence, pp. 241–284. North Holland, Amsterdam (1991)CrossRefGoogle Scholar
  24. 24.
    Csikszentmihalyi, M., Wolfe, R.: New conceptions and research approaches to creativity: implications of a systems perspective for creativity in education. In: Heller, K. (ed.) The International Handbook of Giftedness and Talent, 2nd edn, pp. 81–94. Elsevier, Philadelphia (2000)Google Scholar
  25. 25.
    Amabile, T.: Social psychology of creativity: a consensual assessment technique. J. Pers. Soc. Psychol. 43(5), 997–1013 (1982)CrossRefGoogle Scholar
  26. 26.
    Maher, M.L., Gero, J.S., Lee, L.N., Clausner, T.: Characterizing tangible interaction during a creative combination task. In: Gero, J.S. (ed.) Design Computing and Cognition 2016. Springer (2016)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  • Mary Lou Maher
    • 1
    Email author
  • John Gero
    • 1
  • Lina Lee
    • 1
  • Rongrong Yu
    • 2
  • Tim Clausner
    • 3
  1. 1.University of North Carolina at CharlotteCharlotteUSA
  2. 2.University of NewcastleNewcastleAustralia
  3. 3.University of MarylandCollege ParkUSA

Personalised recommendations