Skip to main content
SpringerLink
Log in

Assessing the Novelty of Computer-Generated Narratives Using Empirical Metrics

  • Published:
Minds and Machines Aims and scope Submit manuscript

Abstract

Novelty is a key concept to understand creativity. Evaluating a piece of artwork or other creation in terms of novelty requires comparisons to other works and considerations about the elements that have been reused in the creative process. Human beings perform this analysis intuitively, but in order to simulate it using computers, the objects to be compared and the similarity metrics to be used should be formalized and explicitly implemented. In this paper we present a study on relevant elements for the assessment of novelty in computer-generated narratives. We focus on the domain of folk-tales, working with simple plots and basic narrative elements: events, characters, props and scenarios. Based on the empirical results of this study we propose a set of computational metrics for the automatic assessment of novelty. Although oriented to the implementation of our own story generation system, the measurement methodology we propose can be easily generalized to other creative systems.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Notes

  1. Similar considerations apply to all the weights that appear in the rest of the formulas presented in this section, though the fact will not be repeatedly mentioned to avoid redundancy.

References

  • Boden, M. A. (1990). The creative mind: Myths and mechanisms. London: Weidenfeld & Nicholson.

    Google Scholar 

  • Gomes, P., Bento, C., & Gago, P. (1998). A function for novelty evaluation in design. Paper presented at the 4th international round-table conference on computational and cognitive models of creative design, Sydney.

  • Loke, K. S. (2005). Notes on defining novelty for computational creativity. Paper presented at the 19th international joint conference on artificial intelligence, Edinburgh.

  • Macedo, L., & Cardoso, A. (2002). Assessing creativity: The importance of unexpected novelty. Paper presented at the 2nd workshop on creative systems: Approaches to creativity in artificial intelligence and cognitive science, from the 15th European conference on artificial intelligence, Lyon.

  • McGraw, G., & Hofstadter, D. R. (1993). Letter spirit: An architecture for creativity in a microdomain. Paper presented at the 3rd congress of the Italian association for artificial intelligence, Torino.

  • Pease, A., Winterstein, D., & Colton, S. (2001). Evaluating machine creativity. Paper presented at the 1st workshop on creative systems: Approaches to creativity in artificial intelligence and cognitive science, from the 4th international conference on case-based reasoning, Washington.

  • Peinado, F., & Gervás, P. (2006). Evaluation of automatic generation of basic stories. New Generation Computing, computational paradigms and computational intelligence, special issue: Computational creativity, 24(3), 289–302.

  • Peinado, F. (2008). Un Armazón para el Desarrollo de Aplicaciones de Narración Automática basado en Componentes Ontológicos Reutilizables. Ph.D. Dissertation (in Spanish), Universidad Complutense de Madrid.

  • Pérez y Pérez, R., & Sharples, M. (2001). Mexica: A computer model of a cognitive account of creative writing. Journal of Experimental and Theoretical Artificial Intelligence, 13(2), 119–139.

    Article  Google Scholar 

  • Pérez y Pérez, R., & Sharples, M. (2004). Three computer-based models of storytelling: Brutus, Minstrel and Mexica. International Journal on Knowledge Based Systems, 17(1), 15–29.

    Article  Google Scholar 

  • Propp, V. (1968). Morphology of the Folktale. Austin: University of Texas Press.

    Google Scholar 

  • Ritchie, G. (2001). Assessing creativity. Paper presented at the 3rd AISB symposium on artificial intelligence and creativity in arts and science, New York.

  • Ritchie, G. (2007). Some empirical criteria for attributing creativity to a computer program. Minds & Machines, 17, 67–99.

    Article  Google Scholar 

  • Rosenman, M. A., & Gero, J. S. (1993). Creativity in design using a design prototype approach. In J. S. Gero, & M. L. Maher (Ed.), Modeling creativity and knowledge-based creative design (pp. 111–138). Hillsdale: Lawrence Erlbaum.

    Google Scholar 

  • Sharples, M. (1999). How we write: Writing as creative design. London: Routledge.

    Google Scholar 

  • Turner, S. R. (1992). Minstrel: A computer model of creativity and storytelling. Ph.D. Dissertation, University of California.

  • Wiggins, G. A. (2006). A preliminary framework for description, analysis and comparison of creative systems. International Journal of Knowledge Based Systems, 19(7), 449–458.

    Article  Google Scholar 

Download references

Acknowledgments

This research is funded by the Spanish Ministry of Education and Science (TIN2009-14659-C03-01 Project), Universidad Complutense de Madrid and Banco Santander Central Hispano (GR58/08 Research Group Grant). We are very grateful to our evaluators: Javier Arroyo, Susana Bautista, Jorge Carrillo, Ángela Francisco, Jesús Herrera, Carlos León, Juanma Martín, Gonzalo Méndez, Pablo Moreno, Laura Plaza, Toñi Torreño, Miguel Vázquez and Salvador de la Puente.

Author information

Authors and Affiliations

  1. Departamento de Ingeniería del Software e Inteligencia Artificial, Facultad de Informática, Universidad Complutense de Madrid, Madrid, Spain

    Federico Peinado, Virginia Francisco & Raquel Hervás

  2. Instituto de Tecnología del Conocimiento, Universidad Complutense de Madrid, Madrid, Spain

    Pablo Gervás

Authors
  1. Federico Peinado
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Virginia Francisco
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Raquel Hervás
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Pablo Gervás
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Federico Peinado.

Appendix A: The Survey: Subjective Evaluation of Novelty in Folk-Tales

Appendix A: The Survey: Subjective Evaluation of Novelty in Folk-Tales

[QUESTIONNAIRE NUMBER AND DATE]

The goal of this survey is to obtain information about how you perceive the novelty of a story. Firstly, you will read a number of simplified plots, in the style of classic folk-tales, that have been generated using different algorithmic processes. Then, you will have to value, using a numerical scale, how different they are from each other, sometimes considering additional information provided, e.g. the elements that have been used in the generation process.

A.1 Part I

Please, fill the following information about yourself.

  1. a)

    Are you male or female?

  2. b)

    How old are you?

  3. c)

    What is your level of academic studies?

  4. d)

    How many folk-tales do you remember?

  5. e)

    How many plots have you written?

  6. f)

    What is your level of literature studies?

  7. g)

    What is your level of English?

A.2 Part II

First of all, spend a couple of minutes thinking about what is a “typical folk-tale”. Describe your ideas in the answer box below.

  1. a)

    What kind of things appear in the “typical folk-tale” and in what order?

[FOR EACH OF THE PLOTS, IN THE GIVEN ORDER, THE FOLLOWING BLOCK OF QUESTIONS WAS PRESENTED]

Now, read the next plot (Plot X) carefully and fill the following questions.

These questions require an answer consisting on an integer number between “0” and “10”, both included. Numbers represent all the possible degrees of agreement with what is proposed in the question, where “0” means “Complete disagreement” and “10” means “Complete agreement”. When the question asks about the similarity between two plots, “0” means “Complete dissimilarity” (i.e. plots have absolutely nothing in common) and “10” means “Complete similarity” (i.e. plots are absolutely identical).

[TEXT OF THE CORRESPONDING PLOT WAS PRESENTED HERE]

  1. a)

    What is the topic of the tale?

  2. b)

    Is this a novel (i.e. original) plot? [0–10]

  3. c)

    How similar is this plot to the “typical folk-tale” you have described before? [0–10]

  4. d)

    How similar is this plot to a classic folk-tale that you remember? [0–10 and the folk-tale that you remember]

It is especially interesting for us if you explain your criteria to decide whether to write a number lower or greater than 5, as well as commenting how different you consider the topics of the folk-tales you are comparing.

Finally, fill in these questions about similarity between the precedent plots. Note that it is assumed that similarity between two plots is exactly the same no matter what plot you consider first (e.g. Plot A is as similar to Plot B as Plot B is to Plot A).

  1. a)

    How similar is Plot A to Plot B? [0–10]

  2. b)

    How similar is Plot A to Plot C? [0–10]

  3. c)

    ...

[EQUIVALENT QUESTIONS WERE INCLUDED FOR EACH PAIR OF PLOTS]

It is especially interesting for us if you explain your criteria to decide whether to write a number lower or greater than 5, as well as commenting what are the differences or similarities between the plots.

A.3 Part III

In this part we provide you with basic information about how the plots of Part II have been created. Then, we ask you again about the similarity between the previous plots, but this time we want you to do it from the point of view of the computational creation process, considering all the information that is revealed to you.

Basic information about the creation of the plots

  • Plot F is a simplified version of the classic folk-tale called “Cinderella”.

  • Plot E is exclusively based on Plot F.

  • Plot D is exclusively based on Plot E.

  • Plot C is exclusively based on Plot D.

  • Plot B is exclusively based on Plot C.

  • Plot A has been created from scratch, i.e. not based in any other plot or classic folk-tale.

Considering this new information, fill again the questions about similarity between the plots.

[SAME QUESTIONS AS AT THE END OF PART II]

In this part it is very important, in case you change the similarity with respect to the previous part, that you comment why you made that change.

A.4 Part IV

In this part we provide you with information about the elements of the plots of Part II. Then, we ask you again about the similarity between the previous plots. You should answer from the point of view of the computational creation process, considering all the information that has been revealed to you.

Information about the elements of the plotsAll the plots are created using the same limited set of elements. These are the narrative events, characters, props and scenarios used to create the plots (Table 4).

Table 4 Narrative elements used to create the plots
Full size table

Considering this new information, fill again the questions about similarity between the plots.

[SAME QUESTIONS AS AT THE END OF PART II]

It is especially interesting for us if you explain your criteria to decide the similarity and if there are aspects (characters, props, scenarios or events) more important than others for you for deciding the similarity.

A.5 Part V

Please, fill these final questions about your comprehension of the text.

  1. a)

    Have you understood the English of this questionnaire? [0–5]

  2. b)

    Do you want to receive the results of this research project by e-mail?

Rights and permissions

Reprints and permissions

About this article

Cite this article

Peinado, F., Francisco, V., Hervás, R. et al. Assessing the Novelty of Computer-Generated Narratives Using Empirical Metrics. Minds & Machines 20, 565–588 (2010). https://doi.org/10.1007/s11023-010-9209-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11023-010-9209-8

Keywords

Search

Navigation