A New Dataset and Method for Creativity Assessment Using the Alternate Uses Task

Abstract

Creativity ratings by humans for the alternate uses task (AUT) tend to be subjective and inefficient. To automate the scoring process of the AUT, previous literature suggested using semantic distance from non-contextual models. In this paper, we extend this line of research by including contextual semantic models and more importantly, exploring the feasibility of predicting creativity ratings with supervised discriminative machine learning models. Based on a newly collected dataset, our results show that supervised models can successfully classify between creative and non-creative responses even with unbalanced data, and can generalise well to out-of-domain unseen prompts.

Appendices

A A The Instructions Used for the AUT

General instruction: For the next four questions, there will be a time limit. For each task, please read the instructions and enter each possible answer separately by pressing the enter key after each one. If you run out of answers you may move on by pressing the next button, otherwise your question will automatically change after the allocated time.

Each task requires you to come up with as many different answers as possible. Try to be creative as there is no right or wrong answer.

Prompt 1: List as many different uses of a bowl as you can think of.

Prompt 2: Think of many different uses of a paperclip.

B B Detailed CFA Results

Table 6. Latent correlations between human creativity ratings and semantic distance factors (Model$_{\textbf {non-contextual}}$ and Model$_{\textbf {contextual}}$) on the Cambridge AUT dataset.

Fig. 1.

CFA diagram of Model$_{\textbf {non-contextual}}$ on the Cambridge AUT dataset. r1-3: rater1-3; glv: GloVe; w2v: Word2vec; fst: fastText; HCR: human creativity rating factor, NSD: non-contextual semantic distance factor.

Fig. 2.

CFA diagram of Model$_{\textbf {contextual}}$ on the Cambridge AUT datseta. r1-3: rater1-3; uni: Universal Sentence Encoder; sen: Sentence-Transformers; rbt: RoBERTa; gpt: GPT-3; HCR: human creativity rating factor, CSD: contextual semantic distance factor.

C C Cross Validation Results

Table 7. Fine-tuned BERT cross validation results on the Cambridge AUT training sets. P: precision; R: recall.

Table 8. Fine-tuned RoBERTa cross validation results on the Cambridge AUT training sets. P: precision; R: recall.

Table 9. Fine-tuned GPT-3 babbage cross validation results on the Cambridge AUT training sets. P: precision; R: recall.

Table 10. Prediction performance on the dataset from [6]. P: precision; R: recall.

D D Model Performance on the Dataset from [6]

E E ChatGPT Prompt

You are a judge in the alternate uses task, where respondents are asked to list different uses for a common object. You will be presented with the object and a response that illustrates one of its uses. Please judge if the response is creative or non-creative. Inappropriate, invalid, irrelevant responses, and responses with common uses are considered non-creative, whereas appropriate, valid, novel and unusual uses are considered creative.

The object is: {prompt}

The response is: {response}

Please give your answer in “creative” or “non-creative”.

Your answer:

F F ChatGPT Classification Results

Table 11. ChatGPT results on the Cambridge AUT dataset. P: precision; R: recall.

Table 12. ChatGPT results on the dataset from [6]. P: precision; R: recall.