Skip to main content

A Standardised Procedure for Evaluating Creative Systems: Computational Creativity Evaluation Based on What it is to be Creative

Cognitive Computation volume 4, pages 246–279 (2012)Cite this article

Abstract

Computational creativity is a flourishing research area, with a variety of creative systems being produced and developed. Creativity evaluation has not kept pace with system development with an evident lack of systematic evaluation of the creativity of these systems in the literature. This is partially due to difficulties in defining what it means for a computer to be creative; indeed, there is no consensus on this for human creativity, let alone its computational equivalent. This paper proposes a Standardised Procedure for Evaluating Creative Systems (SPECS). SPECS is a three-step process: stating what it means for a particular computational system to be creative, deriving and performing tests based on these statements. To assist this process, the paper offers a collection of key components of creativity, identified empirically from discussions of human and computational creativity. Using this approach, the SPECS methodology is demonstrated through a comparative case study evaluating computational creativity systems that improvise music.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Notes

  1. The components of creativity are strongly recommended as a basis for this definition.

  2. In the context of music generation systems, which often contribute to computational creativity literature.

  3. Both Bundy [5] and Pearce et al. [6] stress that specific goals and value may vary across types of systems; this assertion is upheld in this paper.

  4. Definition taken from the computationalcreativity.net website [13], which hosts relevant information about the field, including details of research events and of the steering committee who act to shape the general directions that computational creativity research takes.

  5. Knowledge-Based Systems 2006: 19(7), New Generation Computing 2006: 24(3), AI Magazine 2009: 30(3), Minds and Machines 2010: 20(4).

  6. Computational Creativity workshops have been held in conjunction with several AI conferences (AISB’99, AISB’00, AISB’01, ECAI’02, AISB’02, IJCAI’03, AISB’03, IJCAI’05, ECAI’06) case-based reasoning conferences (ICCBR’01, ECCBR’04) and linguistics conferences (LREC’04, NAACL’09).

  7. Autonomous workshops grew out of the International Joint Workshop on Computational Creativity series (2004–2008), which started through the coming together of communities from AI and from Cognitive Science, to hold joint research events on computational creativity. Separate symposiums have also been held, in Stanford, California (twice).

  8. The pre-2004 workshops typically contained 10–15 papers, with programme committee sizes around 5–15 depending on the event. This has grown to an average of 33 accepted papers and an average of 42 programme committee members over the 2010–2012 conferences.

  9. Existing evaluation methodologies for computational creativity are examined later in this section of the paper.

  10. This paper will return later to these discussions at ICCC’11.

  11. This view was also expressed in [19].

  12. The combination of novelty and originality is often used as a reductionist definition of creativity [2, 20–25]. Definitional issues shall be returned to later in this paper.

  13. For ICCC’11, this phrasing appeared with a qualifier: “quasi-formal approaches that, for example, argue for recognition without definition or that define the absence of creativity may have interesting implications for computational creativity”. This was probably in response to just such a evaluation framework offered by Colton [3], which quickly became adopted more often than more formally stated predecessors such as [2, 20], as shall be shown later in this paper.

  14. See for example the proceedings for ICCC’11, ICCC’10 or IJWCC’07 [27–29] where a position paper is distinguishable from a full technical paper only by its number of pages.

  15. Whether creativity is contained in the creative process, or in the output generated by a system, or in both (and other aspects besides), is a debate which shall be returned to in greater depth later in this paper, in “The Product/Process Debate” section. At this stage of the paper, Ritchie’s product-focussed perspective on this debate is highlighted; it will be argued in “The Product/Process Debate” section that this can lead to disregarding of crucial evidence of creativity, and is a somewhat misunderstood interpretation of creativity

  16. Here Colton makes an important distinction; rather than positing the creative tripod qualities as necessary components of a creative system, he argues that the system merely needs to be perceived to have these qualities. In other words, the challenge is to engineer a system that appears to be creative to its audience, rather than engineering a system that possesses a level of creativity existing independently of an audience’s perception.

  17. Proceedings from annual events in 2007–2010 were included in the survey, which was conducted in late 2010-early 2011. Proceedings from creativity research events prior to 2007 are not readily available in an online format, making them difficult to locate for this survey and also less likely to have influence on researchers today unless they were one of the relatively few people who attended that workshop (in comparison with attendances of such events in more recent years).

  18. The case study reported below demonstrates the incorporation of value judgements and considerations of domain competence into creativity evaluation.

  19. A similar variety of points of views was acknowledged during discussions on evaluation at the 2009 computational creativity seminar at Dagstuhl, including the perspectives of ‘viewer/experiencer’, ‘creator’ and ‘interactive participant [38, p. 1].

  20. All comments in this section are anonymised.

  21. Another issue mentioned during this part of the discussion was that it was often difficult to obtain up-to-date, maintained and fully working materials to use for evaluation, such as the system’s source code or products.

  22. The evaluation survey looks at 75 papers describing such systems.

  23. ERI-designer, mentioned above, could perhaps be compared to architectural design systems or game design systems.

  24. The work in the “Key Components of Creativity” section will adopt a confluence-style approach, seeking to capture a wider disciplinary spectrum of perspectives on creativity that has previously been attempted [58, 78, 79].

  25. What does appear when reviewing legal research are interesting discussions of whether or not computational creativity can be legally recognised [83, 88–90].

  26. Conceptual art is where the concepts and motivations behind the artistic process form a significant contribution of the artwork.

  27. Colton’s solution is to report systems in high-level terms only, rather than giving details of the program [3, p. 8].

  28. This set of components is pictured in Fig. 1.

  29. Some non-musicians were included in the questionnaire as they had experience of listening to musical improvisation and were therefore able to give a slightly different perspective. The questionnaire distribution was however weighted towards professional musicians and improvisers.

  30. This was partly due to the demographics of the participants, whose nationalities ranged from British to Brazilian, though the majority of participants were recruited from UK-based contacts.

  31. An additional two judges were used in pilot studies, but the data provided in these pilot studies is excluded from the evaluation results presented in this paper.

  32. Judges were restricted to evaluating two of the four systems rather than all four due to practical restrictions on time.

  33. Judges could use ratings of x.5 out of 10 if they specifically asked to. Hence the rating scale was effectively a 21-point numeric scale, with 5 as the midpoint between the two extremes of 0 and 10.

  34. The FACE/IDEA models [4], published after the evaluations in this case study had been performed, will also be applied in the near future and results will be published in [126].

  35. Definitions of creativity are often supplied as a list of components or contributory aspects of creativity [54, 58, 65, 77]. The Fig. 1 components are offered as a working definition because the empirical methods used to derive them are based around writings from the time period 1950–2009; as creativity changes over time, the components may need to be updated in the future, but for the present time, they are derived from writings from the last sixty years of research on creativity.

  36. Principle Component Analysis (PCA) is another dimensionality reduction technique. PCA identifies the minimal representation of data by combining and merging components, using eigenvectors and eigenvalues for dimensionality reduction. As PCA therefore does not keep the components distinct and examine the importance of components individually, this has an adverse affect on how evaluative results can be used as formative feedback, as the correlations between component and results have been lost,

  37. Additionally, in the context of the specific domain investigated in the case study (musical improvisation), respondents to the questionnaire about creativity in musical improvisation collectively mentioned each component at some point (to varying degrees).

  38. Perhaps this issue was replaced with the issue of whether the judges understood each component well enough—a possibility despite careful attention paid to describing the components to judges.

References

  1. Cardoso A, Veale T, Wiggins GA. Converging on the divergent: the history (and future) of the international joint workshops in computational creativity. AI Mag. 2009;30(3):15–22.

    Google Scholar 

  2. Ritchie G. Some empirical criteria for attributing creativity to a computer program. Minds Mach. 2007;17:67–99.

    Article  Google Scholar 

  3. Colton S. Creativity versus the perception of creativity in computational systems. In: Proceedings of AAAI symposium on creative systems; 2008. p. 14–20.

  4. Pease A, Colton S. On impact and evaluation in computational creativity: a discussion of the turing test and an alternative proposal. In: Proceedings of the AISB’11 convention. York, UK: AISB; 2011. p. 1–8.

  5. Bundy A. What kind of field is AI? In: Partridge D, Wilks Y, editors. The foundations of artificial intelligence. Cambridge: Cambridge University Press; 1990. p. 215–222.

    Chapter  Google Scholar 

  6. Pearce MT, Meredith D, Wiggins GA. Motivations and methodologies for automation of the compositional process. Musicae Scientae. 2002;6(2):119–147.

    Google Scholar 

  7. Jordanous A. A fitness function for creativity in Jazz improvisation and beyond. In: Proceedings of the international conference on computational creativity. Lisbon, Portugal; 2010. p. 223–227.

  8. Biles JA. Improvising with genetic algorithms: GenJam. In: Miranda ER, Biles JA, editors. Evolutionary computer music. London: Springer; 2007. p. 137–169.

    Chapter  Google Scholar 

  9. Gillick J, Tang K, Keller RM. Machine learning of jazz grammars. Compu Music J. 2010;34(3):56–66.

    Article  Google Scholar 

  10. Lewis GE. Too many notes: computers, complexity and culture in Voyager. Leonardo Music J. 2000;10:33–39.

    Article  Google Scholar 

  11. Ritchie G. Assessing creativity. In: Proceedings of the AISB symposium on AI and creativity in arts and science. York; 2001. p. 3–11.

  12. Colton S, Pease A, Ritchie G. The effect of input knowledge on creativity. In: Proceedings of workshop program of ICCBR-Creative systems: approaches to creativity in AI and cognitive science; 2001. p. 1–6.

  13. Computational Creativity. The computational creativity website; Last accessed 25th May 2012. URL:http://www.computationalcreativity.net.

  14. Wiggins GA. A preliminary framework for description, analysis and comparison of creative systems. Knowl Based Syst. 2006;19(7):449–458.

    Article  Google Scholar 

  15. Widmer G, Flossmann S, Grachten M. YQX plays chopin. AI Mag. 2009;30(3):35–48.

    Google Scholar 

  16. León C, Gervás P. The role of evaluation-driven rejection in the successful exploration of a conceptual space of stories. Minds Mach. 2010;20(4):615–634.

    Article  Google Scholar 

  17. Pérez y Pérez R. MEXICA: a computer model of creativity in writing, Ph.D. thesis. Brighton: University of Sussex; 1999.

  18. Colton S, de Mataras RL, Stock O. Computational creativity: coming of age. AI Mag. 2009;30(3):11–14.

    Google Scholar 

  19. Wiggins GA. Closing the loop: computational creativity from a model of music cognition. COGS research seminar, School of Informatics, University of Sussex; 2008 (October 2008).

  20. Pease A, Winterstein D, Colton S. Evaluating machine creativity. In: Proceedings of ICCBR workshop on approaches to creativity; 2001. p. 129–137.

  21. Peinado F, Gervas P. Evaluation of automatic generation of basic stories. New Generation Comput. 2006;24(3):289–302.

    Article  Google Scholar 

  22. Pereira FC, Cardoso A. Experiments with free concept generation in Divago. Knowl Based Syst. 2006;19(7):459 – 470.

    Article  Google Scholar 

  23. Alvarado Lopez J, Pérez y Pérez R. A computer model for the generation of monophonic musical melodies. In: Proceedings of the 5th international joint workshop on computational creativity. Madrid, Spain; 2008. p. 117–126.

  24. Brown D. Computational artistic creativity and its evaluation. In: Computational creativity: an interdisciplinary approach. No. 09291 in Dagstuhl Seminar proceedings. Dagstuhl, Germany; 2009. p. 1–8.

  25. Chordia P, Rae A. Tabla Gyan: an artificial Tabla improviser. In: Proceedings of the international conference on computational creativity. Lisbon, Portugal; 2010. p. 155–164.

  26. Pease A. Personal communications; 2012. In conversation.

  27. ICCC’11. Proceedings of the international conference for computational creativity, Mexico City, Mexico; 2011. URL:http://iccc11.cua.uam.mx/proceedings (last accessed 25th May 2012).

  28. ICCC’10. Proceedings of the international conference for computational creativity, Lisbon, Portugal; 2010. URL:http://eden.dei.uc.pt/~amilcar/ftp/e-Proceedings_ICCC-X.pdf (last accessed 25th May 2012).

  29. IJWCC’07. Proceedings of the international joint workshop for computational creativity, London, UK; 2007. URL:http://doc.gold.ac.uk/isms/CC07/CC07Proceedings.pdf (last accessed 25th May 2012).

  30. Ventura D. A Reductio Ad Absurdum experiment in sufficiency for evaluating (computational) creative systems. In: Proceedings of the 5th international joint workshop on computational creativity. Madrid, Spain; 2008. p. 11–19.

  31. Tearse B, Mawhorter P, Mateas M, Wardrip-Fonin N. Experimental results from a rational reconstruction of MINSTREL. In: Proceedings of the 2nd international conference on computational creativity. Mexico City, Mexico; 2011. p. 54–59.

  32. Gervás P. Exploring quantitative evaluations of the creativity of automatic poets. In: Proceedings of the 2nd. Workshop on creative systems, approaches to creativity in artificial intelligence and cognitive science (ECAI 2002); 2002. p. 1–8.

  33. Pereira FC, Mendes M, Gervás P, Cardoso A. Experiments with assessment of creative systems: an application of Ritchie’s criteria. In: Proceedings of the workshop on computational creativity (IJCAI 05); 2005. p. 1–8.

  34. Colton S. Automated theory formation in pure mathematics. Distinguished dissertations. Springer, London, UK, 2002.

  35. Colton S, Charnley J, Pease A. Computational creativity theory: the FACE and IDEA descriptive models. In: Proceedings of the 2nd international conference on computational creativity. Mexico City, Mexico; 2011. p. 90–95.

  36. Pease A, Colton S. Computational creativity theory: inspirations behind the FACE and the IDEA models. In: Proceedings of the 2nd international conference on computational creativity. Mexico City, Mexico; 2011. p. 72–77.

  37. Pearce M, Wiggins G. Towards a framework for the evaluation of machine compositions. In: Proceedings of the AISB symposium on AI and creativity in arts and science. York, UK; 2001. p. 1–12.

  38. Brown P. Autonomy, Signature and creativity. In: Computational creativity: an interdisciplinary approach. No. 09291 in Dagstuhl Seminar proceedings. Dagstuhl, Germany; 2009. p. 1–7.

  39. Gervas P. Computational approaches to storytelling and creativity. AI Mag. 2009;30(3):49–62.

    Google Scholar 

  40. Meehan J. Tale-Spin. In: Schank RC, Riesbeck CK, editors. Inside computer understanding: five programs plus minatures. Hillside, NJ: Lawrence Erlbaum Associates; 1981. p. 197–227.

  41. Turner SR. The creative process: a computer model of storytelling and creativity. Hillside, NJ: Erlbaum; 1994.

    Google Scholar 

  42. Bringsjord S. Artificial intelligence and literary creativity: inside the mind of BRUTUS. London, UK: Lawrence Erlbaum Associates; 2000.

    Google Scholar 

  43. Pérez y Pérez R, Sharples M. Three computer-based models of storytelling: BRUTUS, MINSTREL and MEXICA. Knowl Based Syst. 2004;17(1):15–29.

    Article  Google Scholar 

  44. Peinado F, Francisco V, Hervás R, Gervás P. Assessing the novelty of computer-generated narratives using empirical metrics. Minds Mach. 2010;20(4):565–588.

    Article  Google Scholar 

  45. Pérez y Pérez R, Aguilar A, Negrete S. The ERI-designer: a computer model for the arrangement of furniture. Minds Mach. 2010;20(4):533–564.

    Article  Google Scholar 

  46. Aguilar A, Hernandez D, Pérez y Pérez R, Rojas M, Zambrano MdL. A computer model for novel arrangements of furniture. In: Proceedings of the 5th international joint workshop on computational creativity. Madrid, Spain; 2008. p. 157–162.

  47. Whorley RP, Wiggins GA, Pearce MT. Systematic evaluation and improvement of statistical models of harmony. In: Proceedings of the 4th international joint workshop on computational creativity. London, UK; 2007. p. 81–88.

  48. Whorley R, Wiggins G, Rhodes C, Pearce M. Development of techniques for the computational modelling of harmony. In: Proceedings of the international conference on computational creativity. Lisbon, Portugal; 2010. p. 11–15.

  49. Gervás P, Perez y Perez R. On the fly collaborative story-telling: revising contributions to match a shared partial story line. In: Proceedings of the 4th international joint workshop on computational creativity. London, UK; 2007. p. 13–20.

  50. Montfort N, Pérez y Pérez R. Integrating a plot generator and an automatic narrator to create and tell stories. In: Proceedings of the 5th international joint workshop on computational creativity. Madrid, Spain; 2008. p. 61–70.

  51. Pérez y Pérez R, Negrete S, Penãlosa E, Ávila R, Castellanos V, Lemaitre C. MEXICA-Impro: a computational model for narrative improvisation. In: Proceedings of the international conference on computational creativity. Lisbon, Portugal; 2010. p. 90–99.

  52. Plucker JA. Beware of simple conclusions: the case for content generality of creativity. Creativ Res J. 1998;11(2):179–182.

    Article  Google Scholar 

  53. Baer J. The case for domain specificity of creativity. Creativ Res J. 1998;11(2):173–177.

    Article  Google Scholar 

  54. Plucker JA, Beghetto RA. Why creativity is domain general, why it looks domain specific, and why the distinction doesn’t matter. In: Sternberg RJ, Grigorenko EL, Singer JL, editors. Creativity: from potential to realization. Washington, DC: American Psychological Association; 2004. p. 153–167.

    Chapter  Google Scholar 

  55. Baer J. Is Creativity domain-specific? In: Kaufman JC, Sternberg RJ, editors. The Cambridge handbook of creativity. New York, NY: Cambridge University Press; 2010. p. 321–341.

    Google Scholar 

  56. Rhodes M. An analysis of creativity. Phi Delta Kappan. 1961;42(7):305–310.

    Google Scholar 

  57. Plucker JA, Beghetto RA, Dow GT. Why isn’t creativity more important to educational psychologists? Potentials, pitfalls, and future directions in creativity research. Educ Psychol. 2004;39(2):83–96.

    Article  Google Scholar 

  58. Sternberg RJ, Lubart TI. The concept of creativity: prospects and paradigms. In: Sternberg RJ, editor. Handbook of creativity. Cambridge, UK: Cambridge University Press; 1999. p. 3–15.

    Google Scholar 

  59. Veale T, Gervás P, Pease A. Understanding creativity: a computational perspective. New Generation Comput. 2006;24(3):203–207.

    Article  Google Scholar 

  60. Newell A, Shaw JG, Simon HA. The process of creative thinking. In: Gruber HE, Terrell G, Wertheimer E, editors. Contemporary approaches to creative thinking. New York: Atherton; 1963. p. 63–119.

    Google Scholar 

  61. McCarthy J. Ascribing mental qualities to machines. In: Ringle M, editor. Philosophical perspectives in artificial intelligence. Atlantic Highlands, NJ: Humanities Press; 1979. p. 161–195.

    Google Scholar 

  62. Wiggins GA. Searching for computational creativity. New Generation Comput. 2006;24(3):209–222.

    Article  Google Scholar 

  63. Jennings KE. Search strategies and the creative process. In: Proceedings of the international conference on computational creativity. Lisbon, Portugal; 2010. p. 130–139.

  64. Ventura D. No free lunch in the search for creativity. In: Proceedings of the 2nd international conference on computational creativity. Mexico City, Mexico; 2011. p. 108–110.

  65. Boden MA. The creative mind: Myths and mechanisms. 2nd ed. London, UK: Routledge; 2004.

    Google Scholar 

  66. Ritchie G. Uninformed resource creation for humour simulation. In: Proceedings of the 5th international joint workshop on computational creativity. Madrid, Spain; 2008. p. 147–150.

  67. Cohen LM. A Review of: “Expanding visions of creative intelligence: an interdisciplinary exploration by Don Ambrose”. Creativi Res J. 2009;21(2–3):307–308.

    Article  Google Scholar 

  68. Williams F. The mystique of unconscious creation. In: Kagan J, editor. Creativity and learning. Boston: Beacon Press; 1967. p. 142–152.

    Google Scholar 

  69. Albert RS, Runco MA. A history of research on creativity. In: Sternberg RJ, editor. Handbook of creativity. Cambridge, MA: Cambridge University Press; 1999. p. 16–31.

    Google Scholar 

  70. Kaufman JC. Creativity 101. The Psych 101 series. New York: Springer; 2009.

    Google Scholar 

  71. Poincaré H. Mathematical creation. In: The foundations of science: science and hypothesis, the value of science, science and method, vol. Science and method (Original French version published 1908, Authorized translation by George Bruce Halsted). New York: The Science Press; 1929. p. 383–394.

  72. Wallas G. The art of thought. abridged ed. London, UK: C. A. Watts & Co; 1945.

    Google Scholar 

  73. Csikszentmihalyi M. Society, culture, and person: a systems view of creativity. In: Sternberg RJ, editor. The nature of creativity. Cambridge, UK: Cambridge University Press; 1988. p. 325–339.

    Google Scholar 

  74. Resnick M. Sowing the seeds for a more creative society. Learn Leadi Technol. 2007;35(4).

  75. Guilford JP. Creativity. Am Psychol. 1950;5:444–454.

    PubMed  Article  CAS  Google Scholar 

  76. Mednick SA. The remote associates test. Boston: Houghton Mifflin Company; 1967.

    Google Scholar 

  77. Torrance EP. The nature of creativity as manifest in its testing. In: Sternberg RJ, editor. The nature of creativity. Cambridge, UK: Cambridge University Press; 1988. p. 43–75.

    Google Scholar 

  78. Mayer RE. Fifty years of creativity research. In: Sternberg RJ, editor. Handbook of creativity. Cambridge, UK: Cambridge University Press; 1999. p. 449–460.

    Google Scholar 

  79. Ivcevic Z. Creativity map: toward the next generation of theories of creativity. Psychol Aesthet Creativ Arts. 2009;3(1):17–21.

    Article  Google Scholar 

  80. Stein MI. A transactional approach to creativity. In: Taylor CW, Barron F, editors. Scientific creativity: its recognition and development. New York: Wiley; 1963. p. 217–227.

    Google Scholar 

  81. MacKinnon DW. Creativity: a multi-faceted phenomenon. In: Roslansky JD, editor. Creativity: a discussion at the nobel conference. Amsterdam, The Netherlands: North-Holland Publishing Company; 1970. p. 17–32.

    Google Scholar 

  82. Odena O, Welch G. A generative model of teachers’ thinking on musical creativity. Psychol Music. 2009;37(4):416–442.

    Article  Google Scholar 

  83. Clifford RD. Random numbers, Chaos theory, and cogitation: a search for the minimal creativity standard in copyright law. Denver Univ Law Rev. 2004;82(2):259–299.

    Google Scholar 

  84. Mandel GN. To promote the creative process: intellectual property law and the psychology of creativity. Notre Dame Law Rev. 2011;86(1):1999–2024 (online).

  85. Feist. Feist Publications, Inc. v. Rural Telephone Service Co. 499 US 340. 1991;111 S. Ct 1282, 113 L. Ed. 2d 358(Supreme Court).

  86. Karjala DS. Copyright and creativity. UCLA Entertain Law Rev. 2008;15:169–201.

    Google Scholar 

  87. Copyright, Designs and Patents Act. UK Government Legislation; 1988. Ch. 48/1988.

  88. Noll AM. The beginnings of computer art in the United States: a memoir. Leonardo. 1994;27(1):39–44.

    Article  Google Scholar 

  89. Holmes N. The automation of originality: when originality is automated, what becomes of personality? Computer. 2009;98–100 (March 2009).

  90. Warner J. The absence of creativity in Feist and the computational process. J Am Soc Inform Sci Technol. 2010;61(11):2324–2336.

    Article  Google Scholar 

  91. Hennessey BA, Amabile TM. Creativity. Ann Rev Psychol. 2010;61:569–598.

    PubMed  Article  Google Scholar 

  92. Sternberg RJ. A three-facet model of creativity. In: Sternberg RJ, editor. The nature of creativity. Cambridge, UK: Cambridge University Press; 1988. p. 125–147.

    Google Scholar 

  93. Weisberg RW. Problem solving and creativity. In: Sternberg RJ, editor. The nature of creativity. Cambridge, UK: Cambridge University Press; 1988. p. 148–176.

    Google Scholar 

  94. Bryan-Kinns N. Everyday creativity. In: Proceedings of the 7th ACM conference on creativity and cognition. Berkeley, California; 2009. p. 1.

  95. Torrance EP. Torrance tests of creative thinking. Bensenville, IL: Scholastic Testing Service; 1974.

  96. Kraft U. Unleashing creativity. Scientific American Mind. 2005;April.

  97. Runco MA, Dow G, Smith WR. Information, experience, and divergent thinking: an empirical test. Creativ Res J. 2006;18(3):267–277.

    Google Scholar 

  98. Kaufman JC, Kaufman SB, Lichtenberger EO. Finding creative potential on intelligence tests via divergent production finding creative potential on intelligence tests via divergent production finding creative potential on intelligence tests via divergent production. Can J School Psychol. 2011;26(2):83–106.

    Google Scholar 

  99. Boden MA, editor. Dimensions of creativity. Cambridge, MA: MIT Press; 1994.

    Google Scholar 

  100. Seth AK. Explanatory correlates of consciousness: theoretical and computational challenges. Cogn Comput. 2009;1:50–63.

    Article  Google Scholar 

  101. McCrae RR, Costa Jr PT. A five-factor theory of personality. In: Pervin LA, John OP, editors. Handbook of personality: theory and research. 2nd ed. New York: The Guilford Press; 1999. p. 139–153.

    Google Scholar 

  102. Romero P, Calvillo-Gamez E. Towards an embodied view of flow. In: Proceedings of the 2nd international workshop on user models for motivational systems: the affective and the rational routes to persuasion (UMMS 2011). Girona, Spain; 2011. p. 100–105.

  103. Huron D. Tone and voice: a derivation of the rules of voice-leading from perceptual principles. Music Percep. 2001;19(1):1–64.

    Article  Google Scholar 

  104. Evans V, Green M. Cognitive linguistics: an introduction. Edinburgh, UK: Edinburgh University Press; 2006.

    Google Scholar 

  105. Oakes MP. Statistics for corpus linguistics. Edinburgh, UK: Edinburgh University Press; 1998.

    Google Scholar 

  106. Kilgarriff A. Comparing corpora. Int J Corpus Linguist. 2001;6(1):97–133.

    Article  Google Scholar 

  107. Kilgarriff A. Where to go if you would like to find out more about a word than the dictionary tells you. Macmillan English Dictionary Mag. 2006;Issue 35 (Jan–Feb).

  108. Lin D. An information-theoretic definition of similarity. In: Proceedings of the 15th international conference on machine learning. Madison, WI; 1998. p. 296–304.

  109. Biemann C. Chinese Whispers: an efficient graph clustering algorithm and its application to natural language processing problems. In: Proceedings of textGraphs: the first workshop on graph based methods for natural language processing. Morristown, NJ: Association for Computational Linguistics; 2006. p. 73–80.

  110. Jordanous A. Evaluating evaluation: assessing progress in computational creativity research. In: Proceedings of the second international conference on computational creativity (ICCC-11). Mexico City, Mexico; 2011. p. 102–107.

  111. Boden MA. What is creativity? In: Boden MA, editor. Dimensions of creativity. Cambridge, MA: MIT Press; 1994. p. 75–117.

    Google Scholar 

  112. Moffat DC, Kelly M. An investigation into people’s bias against computational creativity in music composition. In: Proceedings of the 3rd international joint workshop on computational creativity (ECAI06 Workshop). Riva del Garda, Italy; 2006. p. 1–8.

  113. Haenen J, Rauchas S. Investigating artificial creativity by generating melodies, using connectionist knowledge representation. In: Proceedings of the 3rd international joint workshop on computational creativity (ECAI06 Workshop). Riva del Garda, Italy; 2006. p. 33–38.

  114. Pearce MT, Wiggins GA. Evaluating cognitive models of musical composition. In: Proceedings of the 4th international joint workshop on computational creativity. London, UK; 2007. p. 73–80.

  115. Lewis GE. Improvising with creative machines: reflections on human-machine interaction (keynote talk). In: Proceedings of the 2nd international conference on computational creativity. Mexico City, Mexico; 2011. p. xii–xiii.

  116. Parker E. Drifting on a reed; 2011. Keynote presentation at The Improvised Space. London, UK: Techniques, Traditions and Technologies.

  117. Csikszentmihalyi M. The creative person and the creative system (keynote address). In: Proceeding of the seventh ACM conference on creativity and cognition. Berkeley, California; 2009. p. 5–6.

  118. Friis-Olivarius M, Wallentin M, Vuust P. Improvisation—the neural foundation for creativity (poster). In: Proceedings of the 7th ACM creativity and cognition conference. Berkeley, California; 2009. p. 411–412.

  119. Berkowitz AL, Ansari D. Expertise-related deactivation of the right temporoparietal junction during musical improvisation. NeuroImage. 2010;49(1):712–719.

    PubMed  Article  Google Scholar 

  120. Berliner PF. Thinking in jazz: the infinite art of improvisation. Chicago studies in ethnomusicology. Chicago, IL: The University of Chicago Press; 1994.

    Google Scholar 

  121. Gibbs L. Evaluating Creative (jazz) Improvisation: Distinguishing Invention and Creativity. In: Proceedings of leeds international Jazz conference 2010: Improvisation—jazz in the creative moment. Leeds, UK; 2010. p. 1–4.

  122. Bailey D. Improvisation: its nature and practice in music. New York: Da Capo Press; 1993.

    Google Scholar 

  123. Biles JA. GenJam: a genetic algorithm for generating Jazz Solos. In: Proceedings of the international computer music conference. Denmark; 1994. p. 131–137.

  124. Jordanous A. Defining creativity: finding keywords for creativity using corpus linguistics techniques. In: Proceedings of the international conference on computational creativity. Lisbon, Portugal; 2010. p. 278–287.

  125. BNC Consortium. The British National Corpus, version 3 (BNC XML Edition); 2007. Distributed by Oxford University Computing Services on behalf of the BNC Consortium. URL: http://www.natcorp.ox.ac.uk (last accessed 25th May 2012).

  126. Jordanous A. Evaluating computational creativity: a Standardised Procedure for Evaluating Creative Systems and its application, Ph.D. thesis. University of Sussex. Brighton, UK; forthcoming.

  127. Zhu X, Xu Z, Khot T. How creative is your writing? A linguistic creativity measure from computer science and cognitive psychology perspectives. In: Proceedings of NAACL HLT workshop on computational approaches to linguistic creativity (ACL). Boulder, Colorado; 2009. p. 87–93.

  128. Sauro J, Kindlund E. A Method to standardize usability metrics into a single score. In: Proceedings of the CHI’05 conference on human factors in computing systems. Portland, OR; 2005. p. 401–409.

  129. Temperley N, Wollny P. Bach Revival; Last accessed 25th May 2012. Available at Grove Music Online, part of Oxford Music Online (subscription required). URL: http://www.oxfordmusiconline.com/subscriber/article/grove/music/01708.

Download references

Acknowledgments

Thanks to Nick Collins for his support and supervisory input during this work, and to all the participants who took part in the case study. Communications and discussions with Alison Pease and Steve Torrance have also been extremely beneficial, as have the comments by the three anonymous reviewers of this article. The quality of computational linguistics work to derive the components of creativity reported in this paper was greatly enhanced by the collaborative involvement and knowledge of Bill Keller. The contents of this paper are the result of doctoral research conducted at the Department of Informatics, University of Sussex, UK, who provided a doctoral stipend to partially fund this work. Some financial assistance was also received from the Sir Richard Stapley Educational Trust and the Society for the Study of Artificial Intelligence and the Simulation of Behaviour (AISB).

Author information

Authors and Affiliations

  1. Centre for e-Research, Department of Digital Humanities, King’s College London, 26-29 Drury Lane, London, WC2B 5RL, UK

    Anna Jordanous

Authors
  1. Anna Jordanous
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Anna Jordanous.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Jordanous, A. A Standardised Procedure for Evaluating Creative Systems: Computational Creativity Evaluation Based on What it is to be Creative. Cogn Comput 4, 246–279 (2012). https://doi.org/10.1007/s12559-012-9156-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12559-012-9156-1

Keywords

  • Computational creativity
  • Creativity evaluation
  • Cognitively inspired evaluation
  • Methodology
  • Creativity
  • Musical creativity