Quantitative Evaluation of the Effectiveness of Idea Generation in the Wild

  • Lassi A. Liikkanen
  • Matti M. Hämäläinen
  • Anders Häggman
  • Tua Björklund
  • Mikko P. Koskinen
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6776)


New ideas are the primary building blocks in attempts to produce novel interactive technology. Numerous idea generation methods such as Brainstorming have been introduced to support this process, but there is mixed evidence regarding their effectiveness. In this paper we describe an experimental, quantitative methodology from the domain of product design research for evaluating different idea generation methods. We present prominent results from relevant literature and new data from a study of idea generation in the wild. The study focused on the effects of the physical environment, or in other words, the physical context, on designers’ capacity to produce ideas. 25 students working in small groups took part in an experiment with two design tasks. Moving from an office environment to the actual surroundings of the intended use, we discovered that the change in resulting ideas was surprisingly small. Of the measured dimensions, the real-world context influenced only the feasibility of ideas, leaving quantity, novelty, utility and level of detail unaffected. This finding questions the value of diving into the context as a design idea generation practice.


Design methods idea generation creativity psychology 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Osborn, A.F.: Applied imagination: principles and procedures of creative problem-solving. Revised edition. Scribner, New York (1957)Google Scholar
  2. 2.
    Shneiderman, B.: Creativity Support Tools. Communications of the ACM 50, 20–32 (2007)CrossRefGoogle Scholar
  3. 3.
    Diehl, M., Stroebe, W.: Productivity loss in brainstorming groups: Toward the solution of a riddle. J. Pers. Soc. Psychol. 53, 497–509 (1987)CrossRefGoogle Scholar
  4. 4.
    Diehl, M., Stroebe, W.: Productivity loss in idea-generating groups: Tracking down the blocking effect. J. Pers. Soc. Psychol. 61, 392–403 (1991)CrossRefGoogle Scholar
  5. 5.
    Gallupe, R.B., Bastianutti, L.M., Cooper, W.H.: Unblocking brainstorms. J. Appl. Psychol. 76, 137–142 (1991)CrossRefGoogle Scholar
  6. 6.
    Wang, H.-C., Fussell, S.R., Setlock, L.R.: Cultural Difference and Adaptation of Communication Styles in Computer-Mediated Group Brainstorming. In: SIGCHI Conference on Human Factors in Computing Systems (CHI 2009), Boston, MA (2009)Google Scholar
  7. 7.
    Carroll, E.A., Latulipe, C., Fung, R., Terry, M.: Creativity factor evaluation: towards a standardized survey metric for creativity support. In: ACM Creativity and Cognition (C&C 2009), pp. 127–136. ACM Press, Berkeley (2009)Google Scholar
  8. 8.
    Torrance, P.E.: Torrance Tests of Creative Thinking. Norms-Technical Manual. Ginn and Company, Lexington (1974)Google Scholar
  9. 9.
    Chrysikou, E.G., Weisberg, R.W.: Following the wrong footsteps: Fixation effects of pictorial examples in a design problem-solving task. J. Exp. Psychol.-Learn. Mem. Cogn. 31, 1134–1148 (2005)CrossRefGoogle Scholar
  10. 10.
    Nijstad, B.A., Stroebe, W.: How the group affects the mind: A cognitive model of idea generation in groups. Pers. Soc. Psychol. Rev. 10, 186–213 (2006)CrossRefGoogle Scholar
  11. 11.
    Volkema, R.J.: Problem Formulation In Planning And Design. Manage. Sci. 29, 639–652 (1983)CrossRefGoogle Scholar
  12. 12.
    Shah, J.J., Vargas Hernandez, N., Smith, S.M.: Metrics for measuring ideation effectiveness. Design Stud. 24, 111–134 (2003)CrossRefGoogle Scholar
  13. 13.
    Perttula, M., Sipilä, P.: The idea exposure paradigm in design idea generation. J. Eng. Design 18, 93–102 (2007)CrossRefGoogle Scholar
  14. 14.
    Kudrowitz, B.M., Wallace, D.R.: Assessing the Quality of Ideas from Prolific, Early-Stage Product Ideation. In: ASME DETC 2010, Montreal (2010)Google Scholar
  15. 15.
    Visser, W.: Design: one, but in different forms. Design Stud. 30, 187–223 (2009)CrossRefGoogle Scholar
  16. 16.
    Vargas Hernandez, N., Shah, J.J., Smith, S.M.: Cognitive Models of Design Ideation. In: Design Engineering Technical Conferences (ASME DETC 2007), Las Vegas, NV (2007)Google Scholar
  17. 17.
    Liikkanen, L.A., Perttula, M.: Inspiring design idea generation: Insights from a memory-search perspective. J. Eng. Design 21, 545–560 (2010)CrossRefGoogle Scholar
  18. 18.
    Shah, J.J., Kulkarni, S.V., Vargas Hernandez, N.: Evaluation of Idea Generation Methods for Conceptual Design: Effectiveness Metrics and Design of Experiments. J. Mech. Design 122, 377–384 (2000)CrossRefGoogle Scholar
  19. 19.
    Cross, N.: Expertise in design: an overview. Design Stud. 25, 427–441 (2004)CrossRefGoogle Scholar
  20. 20.
    Dow, S.P., Fortuna, J., Schwartz, D., Altringer, B., Schwartz, D.L., Klemmer, S.R.: Prototyping Dynamics: Sharing Multiple Designs Improves Exploration, Group Rapport, and Results. In: CHI 2011, Vancouver, BC (2011) (in press)Google Scholar
  21. 21.
    Mayer, R.E.: Fifty Years of Creativity Research. In: Sternberg, R.J. (ed.) Handbook of Creativity, pp. 449–460. Cambridge University Press, Cambridge (1999)Google Scholar
  22. 22.
    Srivathsavai, R., Genco, N., Hölttä-Otto, K.: Study OF Existing Metrics Used in Measurement of Ideation Effectiveness. In: ASME DETC 2010, Montreal (2010)Google Scholar
  23. 23.
    Yang, M.C.: Observations on concept generation and sketching in engineering design. Research in Engineering Design 20, 1–11 (2009)CrossRefGoogle Scholar
  24. 24.
    Reinig, B.A., Briggs, R.O., Nunamaker, J.F.: On the measurement of ideation quality. J. Manage. Inf. Syst. 23, 143–161 (2007)CrossRefGoogle Scholar
  25. 25.
    Shepherd, M.M., Briggs, R.O., Reinig, B.A., Yen, J., Nunamaker, J.F.: Invoking social comparison to improve electronic brainstorming: beyond anonymity. J. Manage. Inf. Syst. 12, 155–170 (1995)CrossRefGoogle Scholar
  26. 26.
    Stroebe, W., Diehl, M., Abakoumkin, G.: The Illusion of Group Effectivity. Pers. Soc. Psychol. Bull. 18, 643–650 (1992)CrossRefGoogle Scholar
  27. 27.
    Dugosh, K.L., Paulus, P.B., Roland, E.J., Yang, H.C.: Cognitive stimulation in brainstorming. J. Pers. Soc. Psychol. 79, 722–735 (2000)CrossRefGoogle Scholar
  28. 28.
    Jansson, D.G., Smith, S.M.: Design fixation. Design Stud. 12, 3–11 (1991)CrossRefGoogle Scholar
  29. 29.
    Purcell, A.T., Gero, J.S.: Design and other types of fixation. Design Stud. 17, 363–383 (1996)CrossRefGoogle Scholar
  30. 30.
    Smith, S.M., Ward, T.B., Schumacher, J.S.: Constraining Effects Of Examples In A Creative Generation Task. Mem. Cogn. 21, 837–845 (1993)CrossRefGoogle Scholar
  31. 31.
    Liikkanen, L.A., Björklund, T., Hämälainen, M.M., Koskinen, M.: Time Constraints in Design Idea Generation. In: International Conference on Engineering Design, ICED 2009, Palo Alto, CA (2009)Google Scholar
  32. 32.
    Oulasvirta, A., Kurvinen, E., Kankainen, T.: Understanding contexts by being there: case studies in bodystorming. Pers. Ubiquit. Comput. 7, 125–134 (2003)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Lassi A. Liikkanen
    • 1
    • 2
  • Matti M. Hämäläinen
    • 2
    • 3
  • Anders Häggman
    • 4
  • Tua Björklund
    • 2
  • Mikko P. Koskinen
    • 2
    • 3
  1. 1.Helsinki Institute for Information Technology HIITAalto University and University of HelsinkiAaltoFinland
  2. 2.Aalto Design FactoryAalto UniversityFinland
  3. 3.Engineering Design and ProductionAalto UniversityFinland
  4. 4.Dept. of Mechanical EngineeringMassachusetts Institute of TechnologyCambridgeUSA

Personalised recommendations