Keywords

1 Introduction

Quizzes are established as an assessment tool. Not so common, but remarkable as well are their applications as learning tools [1, 2]. The quiz format can be used regardless of the technical area of the knowledge. In this context we argued for a common technical infrastructure to facilitate quizzes, especially in educational contexts [3]. Furthermore, recently raised proliferation of mobile internet has enabled the ubiquitous and enormous spread of quiz apps (e.g. Quizkampen [4] or QuizUp [5]). In a previous study [6] we found that quiz app players enter the game with an expectation to learn. Additionally, competition with friends has been identified as one main motivation for users of such apps [6]. Thus commercial quiz apps can be considered as potential educational tool.

In 2015 the commercial quiz app QuizUp has been opened for user-defined topics [7]. We used this feature in two engineering university courses. As a result, we found that players can be categorized in two groups: learners – who accomplish their quantum of educational content and then leave the app – and gamers – who get stuck in the game after having fulfilled their learning duties [8]. Further results have not drawn a clear picture. On one hand the integration of the app into the course has been appreciated by students as an enrichment, on the other hand the actual usage of the app – trackable via public ranking lists – has not been overwhelming. This ambivalence provided no answer to the question, to what extent quiz apps can contribute to intrinsic motivation in learning contexts. Furthermore, more detailed insights in usage experiences of such apps within educational settings would be helpful for the purposeful design of effective application scenarios. Therefore, the aim of the following explorative study is to contribute to a set of general conditions, necessities and requirements, which could frame the usage of game-like quiz apps as educational tools.

2 Study Design

The employed app QuizUp uses multiple choice questions with three distractors. Two players can compete in a match of seven questions all belonging to a certain topic. If currently there is no other player available, the system automatically assigns as opponent a bot. A question has to be answered within 10 s. The less time is needed for the correct answer, the more points are earned. A match takes on average three minutes.

As a prerequisite, we have developed a corpus of 57 questions about three areas of basic knowledge in the engineering topic of Ecological Sanitation. (The publicly available topic in QuizUp is called “NASS” – N ew A lternative S anitation S ystems). This corpus contains questions of different difficulty levels: there are questions about simple factual knowledge like abbreviations. On the other hand there are more complex questions which require the comparison of systems. Questions differ in text length, but they cannot exceed a system provided maximum length. During the study the corpus remained unchanged.

As participants 4 students (2 male, 2 female) of engineering courses of study were recruited by means of online pin boards. Prerequisites for participation were being a student, interested in quiz apps and technical topics, the possession of a smartphone and being available for a determined time frame. As a compensation a € 25 online-shopping voucher was promised. Those four persons, who responded most quickly, have been selected. Except for one person they had no prior experience with QuizUp. However, all of them were active players of Quizkampen. Although such a small sample size clearly limits the reliability of the results, we have chosen this approach to enable a familiar environment during the gaming sessions very quickly.

Within 12 days three meetings have been scheduled. One part of each of these meetings has been an attended gaming session of 30 min. Additionally, these meetings have been used to perform semi-structured interviews [9] with the participants. The interviews have been roughly guided by the main categories Content, Motivation and Learning outcome: These topics constitute main pillars of games in educational contexts. In the last session, participants had to answer a questionnaire. After Sessions 1 and 2 they have been provided with extra tasks for the next meeting in order to keep up the engagement with the app over time. In the following we refer to the period between Sessions 1 and 2 as home period 1 and to the period between Sessions 2 and 3 as home period 2.

Session 1.

Here we conducted a semi-structured group interview (and asked additionally for prior knowledge, expectations and experience) and a play session of 30 min. Participant played against each other, they connected to each other in QuizUp and established a chat group with the purpose of coordination. A 20-page brochure containing an overview of the topic NASS [10] was handed out to the participants. The task for home period 1 was to play at least one match a day. Outside of the gaming sessions, players were free to play against other participants or bots The intention of this task was to establish regular play and to point students to a bonus which is issued by QuizUp to reward daily play.

Session 2.

This session took place seven days after the first session. Besides the play session, we interviewed them about their experiences since the last sessions. The task for home period 2 was to reach 25,000 points which is equivalent to 50 matches (if students play each day and are rewarded with the bonus). Intention of this instruction was on one side to investigate, how participants react to becoming the game a task instead of being fun. On the other side, we wanted to know if the habit of daily play has been kept up. This was the case, 3 of 4 participants still declared in Session 3 to have played on a daily base.

Session 3.

In this session, held five days after Session 2, we conducted an interview about their experiences in the last five days. After the play session, a questionnaire has been answered.

There have been no variations of the experimental setting during the study. The only condition we changed was the specific task to accomplish during the home periods.

3 Results

Sessions, interviews and questionnaire revealed various, partially very detailed aspects. In the following we give an overview about selected aspects. Although categorization has been partly ambiguous, we assigned each aspect to one out of three main topics: Content (Knowledge has to be expressed by means of multiple choice questions (MCQs)), Motivational aspects (Quiz apps use game mechanics, therefore fostering of motivation, especially intrinsic motivation is a relevant aspect) and Learning outcomes (Learning is the purpose for employing quiz apps in this context). These were the main categories of the semi-structured interviews. Subcategories have been developed a posteriori according to the found results.

3.1 Content

After having defined three basic learning fields, the according questions corpus has been designed due to the recommendations of Haladyna and Rodriguez [11]. Nevertheless, there are further observations. In the following we describe them and suggest possible improvements.

Text length.

Questions with short texts for question stem and answer options were preferred by participants, as they do not require much reading effort. This is especially critical as there is time pressure and limited space on the display. Long questions require more time for reading and are printed in smaller letters. Both increases the felt difficulty of answering. The allowed time to answer could be adjusted programmatically – e.g. linearly according to the length of the question stem and answer options.

Negation.

Further, although negation in question should be avoided in general [11], one participant gave an example of such a question, which he considered as very informative: Which is not an appropriate usage of treated stormwater? Body care – Washing machine – Toilet flushing – Cleaning. He stated that the correct answer is not a far-fetched appropriate usage, so all options have to be considered carefully, which contributes to learning.

Level design.

Not knowing the correct answer frustrates (One participant uttered often apparently almost in desperation “How should I know that?!”). So there is the need to order questions by complexity and release them stepwise. Questions, which refer to further, not included complex knowledge, can be seen as critical. Figure 1 depicts such a question: the question asks for a specific sanitation system, which fulfills a particular characteristic. Each answer option names such a system. If these systems are not described by other questions, the question pool is not self-contained. External knowledge is required. However, participants have not looked up additional knowledge on their own. Our conclusion is that not self-contained question pools require embedding in a formal educational setting.

Fig. 1.
figure 1

Sample question with references to external knowledge.

Full size image

3.2 Motivational Aspects

Motivation is a key issue for a game based learning app, because motivation encourages engagement with the app and therefore causes indirectly a learning outcome. In general, the app together with its educational content is experienced as a game, as indicated by the results of the Game Experience Questionnaire (GEQ) [12]. We included the GEQ-in-game variant in the questionnaire. Among the results (see Fig. 2) are high values especially for Positive affect (3.63) and Challenge (3.13), whereas Tension (1.88) is rated low. Although the sample size is by far too small for reliable quantitative findings, received values seem to be game-typical, except for the quite high value of Negative affect (2.88).

Fig. 2.
figure 2

Mean value and standard deviation of GEQ categories [12] (n = 4)

Full size image

One participant compared the regular play of the educational topic with the healthy habit of eating fruits: “Sometimes it tastes good, sometimes not”. In the following, further observed aspects are described.

Competition.

Matches against other persons have been indicated as more interesting as matches against bots. However, in many cases a specific bot, which imitates a perfect opponent (answering all questions within the lowest time possible) has been assigned. So there was no chance to win a match. A participant stated that such a bot assignment strategy made him quitting the gaming session. Another aspect of competition is caused by ranking lists: One participant reached more than 35.000 points in the first week. This is equivalent to 150 matches or 7.5 h of gaming. His dedicated intention was to conquer the top rank. Another player experienced a software failure, which prevented him from having a look at the ranking lists. As a consequence, he reported a decreased motivation.

Social Presence.

In the play sessions students competed against each other and teased each other verbally. This was described by the participants as highly motivating.

Direct encouragement.

In the first play session participants reported to be motivated by a textual app feedback after a match. However, this effect has not been mentioned in later sessions.

Focus and engagement.

During a match the participants almost did not talk and seemed to be quite focused, especially in the first session. As an indicator for engagement, they uttered anger about false answers and cursed from time to time. Furthermore, working with the app requires attention: percentage of correct answers lowered, when there was a distraction, e.g. a switched-on TV during the home periods.

App-internal distraction.

Besides our educational topic, QuizUp contains the original entertainment topics. In the first home period, three of four participants played entertainment topics at the same rate as technical topics. In the second home period, the number of matches about entertainment topics even increased.

Challenge.

Especially in the third session the decreased motivation has been mentioned. Participants stated that the game has been receipted as boring, because 90 % of the questions could be answered from memory. Figure 3 mirrors the decrease of motivation over time.

Fig. 3.
figure 3

Course of motivation (self-assessment) over study period on a scale between 0 (unmotivated and 1 (highly motivated)

Full size image

3.3 Learning Outcome

All participants pointed out, that they have experienced learning in quiz apps and especially during this study. The following detail aspects have been found.

Level of knowledge.

Participants stated that mostly factual knowledge can be taught with quiz apps. Simple questions, e.g. naming abbreviations, have been learned easily and could be answered in Session 3 without providing answer options. Questions about more complex topics could be answered after a while. However, especially in the case of complex topics, participants admitted, that they knew the correct answer, but did not gather the meaning of the question and missed an overview of the topic.

Learning curve.

Already in the course of the first session effects of learning could be observed, participants could provide more and more correct answers. In the beginning of Session 1, mostly 1 to 2 questions of a match (seven questions) have been answered correctly. At the end of the first session, after approximately 10 matches, this rate increased to 4 to 5 questions per match.

Intervals of usage.

One participant, which played in the home period 2 only on two days, observed fading memorization of knowledge after longer breaks. The answers could not be retrieved easily from memory, but she had to think again about the possible answers to the questions. There are two possible interpretations for this observation. First, it could be a sign of surface learning, which does not lead to long-term memorization. On the other hand, it could be a learning strategy for deeper learning, as repeated considerations for finding the correct answers could help to think thoroughly about the question.

Situated learning.

During the play sessions, participants started discussions about missed questions and their correct answers. These face-to-face discussions contribute to learning [13].

External knowledge sources.

Additional sources for topic-relevant knowledge have almost not been used. The established chat group did not show discussions about the technical topic. The brochure about the topic which has been handed out to participants has not been used. In the second and third session participants had not questions about the content – despite their statement, that they had not a thorough comprehension of the content of many questions. Although, students indicated that they had a better overview about the topic compared to the beginning of the study, it has to be investigated, if and in how far deeper learning processes can be triggered by a quiz app.

Time pressure.

The limit of ten seconds to answer the questions and its relevance for the reward seems to hinder learning. Participants reported that the urge to answer fast, leads at least partly to superficial reading and to a kind of visual pattern recognition: The shape of question and answer options is recognized, but the question is not read thoroughly. However, this would be necessary for learning.

4 Discussion, Conclusions and Outlook

The most limiting restriction of this study is the low number of participants. Even for an exploratory approach four persons are quite a small sample. However, this small number of participants led to familiar atmosphere during the gaming sessions, which may have caused a more authentic behavior. Nevertheless, all findings have to be valued in consideration of this small sample size. The relative small size of the question corpus may have influenced findings as well. The grade of challenge may have not fallen that fast in the presence of a larger corpus. However, in this case the grade of frustration may have decreased the motivation of participants. Due to such a potential ambiguity, we consider the findings as hints, which require more detailed investigations.

In general, this study confirms learning effects of quiz apps. However, there are multiple aspects still to be discussed. On a more technical level, such aspects are partially related to the limitations and features of the studied app, QuizUp. So, learning algorithms – as implemented in flash card systems [14, 15] – are missing. More statistical data about answered and failed questions has been indicated as potentially being motivational. The role of time in the match process needs to be balanced carefully. Direct time pressure seems to be counterproductive. Further, the foreseen reading time should be adjusted specifically to current text length and question complexity.

In a broader range there is the process of content generation: an important aspect is the kind of knowledge which can be transformed into MCQs. Complex knowledge may have to be conveyed in other ways (e.g. lectures). Questions do not provide sufficient possibilities to teach all kinds of knowledge. An enrichment through graphics and pictures would improve current possibilities. Furthermore, we have found a relatively fast decreasing motivation among participants in this study. As a countermeasure we suggest the embedding of quiz apps into educational settings, which determine e.g. level design, regular play sessions and concrete app-related goals. In general, proven settings still have to be defined. Another important aspect, which has to be discussed is the role of entertainment topics: are they a distraction from or an enrichment for the learning process? Further, there is the basic aspect of type-dependency: the grade of motivation a quiz app can foster in a person depends on their individual preferences. Relevant in an educational context is to reach a sufficient motivation for any learner. Consequentially, a quiz app has to compete with genuine question-based learning tools like Skive [16], StudyBlue [17] or Quizlet [18]. In a further step we currently conduct a comparative study between a quiz app and a dedicated learning app regarding efficacy and acceptance. Commonly, app supported learning – no matter if game-based or not – can offer the advantages of adaptivity, ubiquitous accessibility and –at least basic – collaboration. Therefore, further research should help to mitigate current limitations.