Choice Architecture for Nudges to Support Constructive Learning in Active Video Watching

Abstract

Video-based learning is widely used today in both formal education and informal learning in a variety of contexts. Videos are especially powerful for transferable skills learning (e.g. communicating, negotiating, collaborating), where contextualization in personal experience and ability to see different perspectives are crucial. With the ubiquity of widely available video content, video-based learning is seen as one of the main strategies to provide engaging learning environments. However, numerous studies show that to learn effectively while watching videos, students need to engage actively with video content. We have developed an active video watching platform (AVW-Space) to facilitate engagement with video content by providing means for constructive learning. The initial studies with AVW-Space on presentation skills show that only students who commented on videos and who rated comments written by their peers have improved their understanding of the target transferable skill. In order to foster deeper engagement, we designed a choice architecture and a set of nudges to encourage students to engage deeper. We conducted two studies investigating the effect of nudges. The results provide evidence that the initial set of implemented nudges is effective: the students who received nudges wrote more comments, used different aspects, and there were more students who wrote comments. The nudges were particularly helpful for undergraduate students who were less experienced in self-regulated learning. Future work includes designing additional nudges to enhance student engagement by improving the quality of comments and by encouraging participation in collaborative activities.

Introduction

Learning by watching videos (Yousef, et al., 2014; Vieira et al., 2014; Oleksandra et al., 2018) has become popular in various contexts, such as flipped classrooms (Kurtz et al., 2014; Dodson et al., 2019), online courses and MOOCs (Guo et al., 2014; Koedinger et al., 2015; Soffer & Cohen, 2019), and informal learning (Giannakos et al., 2016). Video-based-learning is widely adopted due to the potential positive effects on learning experiences, such as improved student attention and motivation to learn, flexibility and opportunities for self-control of learning (Sablić et al., 2020; Chatti et al., 2016).

However, watching videos is inherently a passive form of learning (Chi and Wylie, 2014; Chatti et al., 2016), often resulting in a low level of engagement. Providing ways to support students’ engagement during video watching is paramount. Effective use of videos for learning can be enhanced by managing cognitive load of video watching, maximising student engagement with the video content and promoting active learning (Brame, 2016; Fiorella & Mayer, 2018; Risko et al., 2013; van der Meij and Dunkel, 2020). Hence, the support can aim to foster metacognitive skills essential for effective learning, which students often lack (Bannert & Mengelkapm, 2008; Chi, 2000). Moreover, learners can often overlook or may be unable to recognise key details in a video (Chaudhury & Chilana, 2019). Therefore, support should be provided to help students notice key points and link these points to learning goals. Such support is investigated in this paper.

Using videos is especially powerful for learning transferable skills (Conkey et al., 2013; Cronin & Cronin, 1992; Anthony & Garner, 2016), which require contextualization in personal experience and ability to see different perspectives are crucial. Transferable skills (e.g. intercultural awareness, negotiating, reasoning about societal/ethical responsibilities) are widely seen as crucial for employability in the knowledge economy (World Economic Forum, 2016; National Research Council, 2012). Research shows that transferable skills contribute as much as 85% to students’ success (Wats & Wats, 2009). However, it is challenging to teach transferable skills explicitly to tertiary students (Anthony and Garner, 2016), as they are time-consuming and difficult to document. Training students to deliver presentations is mostly based on practical role-based experiences, which is resource intense (Hetzner et al., 2011). Students need to practice under various conditions, receive feedback, reflect on it and do more practice. Teachers typically do not have enough resources to provide such support to each individual student. In this paper, we focus on the ill-defined task of delivering pitch presentations. Pitch presentations are used to present research results, business proposals, public engagement and other areas, and need to be short, sharp and engaging.

We developed an active video watching platform named AVW-Space as a scalable way of assisting learners to improve their knowledge by watching videos. Our goal was to develop a teacher-friendly way of supporting video-based learning; AVW-Space enables teachers to select publicly available videos, rather than having to record and/or edit them. Furthermore, minimal effort is required from the teacher in order to develop ways of interacting with students. To increase engagement with video content, AVW-Space fosters active learning via interactive note taking and by providing micro-scaffolds for reflection. In addition to writing comments, students can also rate comments written by others, thus providing opportunities for social learning.

The findings from early studies with AVW-Space showed that only students who wrote comments and rated comments written by others improved their knowledge. The early studies allowed us to identify differences between various categories of learners, and identify requirements for personalised nudges to promote desired learning behaviour: assist students with noticing important points in videos, linking video snippets to aspects related to transferable skill learning (e.g. recognise key skill elements, contextualise in past experience and plan future presentations) and broadening transferable skill learning portfolio (e.g. notice a variety of skill elements and use various reflection triggers when making notes). On the basis of those findings, we propose a choice architecture for personalised nudges to promote deep engagement with the video content. The proposed choice architecture fosters active video watching, by providing interactive visualisations and nudges tailored to the individual learner’s profile and behaviour. Our approach provides automatic ways of constructing visualisations and personalised prompts from learner-sourced data, using learning analytics approaches. We then explore the impact of nudges on learner engagement with videos in two evaluation studies.

The research reported in this paper has been conducted over five years, during which we conducted ten studies with the AVW-Space platform. We present a short overview of the two initial studies, from which we identified the requirements leading to nudges, going on to the design and implementation of nudges and choice architecture, and finally the evaluation of the effectiveness of the nudges. The main contribution of the paper is the choice architecture for personalised nudges to promote deep engagement with video content. Our choice architecture (i) takes into account the learner’s context and knowledge/experience, (ii) aims to increase the learner’s motivation to engage in active video watching and to improve their knowledge, and (iii) automatically identifies opportunities to support engagement in active video watching to improve learning. Our work is the first attempt to devise a choice architecture for personalised nudges to improve learning from videos. We use a data-driven approach using behaviour of past learners to shape new learning experiences for future learning. The choice architecture has been applied to the AVW-Space, implementing interactive nudges in the form of visualisations and personalised prompts. We have conducted two studies to compare the AVW-Space with and without nudges and examine the effect of nudges on improving domain knowledge (presentation skills in this case) and learning experiences.

The structure of the paper is as follows. We start by reviewing related work. We next introduce the features of AVW-Space, including the pedagogical framework on which our approach is based, the experimental design for Studies 1 and 2, and the requirements derived from the two studies. The next section presents the choice architecture we designed in order to support active video watching, followed by the discussion of how the choice architecture was implemented in AVW-Space. We then present the research questions and findings from two additional studies (Study 3 and Study 4), conducted with the enhanced version of AVW-Space and two different population of students. The final section presents conclusions and avenues for future work.

Parts of the work presented here have been included in earlier publications. The AVW-Space platform, Study 1 and Study 2 have been reported in (Lau et al., 2016; Mitrovic et al., 2016; Dimitrova et al., 2017; Mitrovic et al., 2017). We have summarised the main features of AVW-Space here, and have presented only the findings related to designing interactive nudges. Some aspects of Study 4 have been reported in (Mitrovic et al., 2019), a deeper analysis of the data collected in that study is presented here. Study 3 has not been reported in earlier publications. A preliminary version of the framework for choice architecture for intelligent nudges to encourage active/constructive learning with videos was presented in (Dimitrova et al., 2017). In this paper we provide a significantly improved version which gives more detail and takes into account broader aspects of interacting with videos. More importantly, the framework has been generalised to allow its broader application. The related work section extends similar sections from past publications including recent work in interactive support for video-based learning.

Related Work

Engagement in Video-Based Learning

A number of approaches have investigated ways of integrating interactive elements to increase learner engagement with videos. One of the proven strategies to increase engagement with videos is in-video quizzes (Giannakos, Sampson and Kidziński, 2016; Kleftodimos and Evangelidis, 2016; Kovacs, 2016; Wachtler et al., 2016). It has been shown that in-video quizzes increase engagement with videos but may also lead to learners skipping key points to focus just on answering the quizzes (c.f. Kovacs, 2016). Moreover, the production of in-video quizzes requires substantial effort from the teacher, as quizzes need to be embedded at time of video production. This limits the application of the approach for using existing, freely available, educational videos. In contrast, the approach presented here assumes no additional effort in creating videos and has been applied in scenarios when freely available videos are used. Instead of relying on pre-defined video break-out points, we show how to automatically identify suitable places where interaction features can be included.

Another technique to increase learner engagement with videos is to add explicit prompts while watching the videos. Explicit prompts have been used in computer-based learning environments to support metacognitive activities such as self-explanation (Conati & VanLehn, 2000; Weerasinghe & Mitrovic, 2004) or reflection (Lin & Lehman, 1999; Bannert & Mengelkamp, 2013). Several systems prototype the use of prompts to enhance learner engagements with videos. For example, Kovacs (2015) presents question-focused prompts for navigating and reviewing lecture videos by offering the video segment that is linked to the prompt and a timeline of previously answered questions. An evaluation study with the system shows better engagement and improved recall of key concepts. Shin et al. (2018) explore instructor and learner perceptions of in-video prompts where learners answer reflective questions while watching videos. They found that prompts could be useful checkpoints for reflection, but could also be seen as distraction when learners do not see the benefits for them. It was also found that different prompt formats were appropriate for different people. Chaudhury & Chilana (2019) designed more sophisticated prompts in the form of flashcards that provide expert-curated retrieval exercises that are prompted automatically at predefined intervals or appear on-demand. The evaluation study of the approach showed that learners found it less effortful to engage and were more confident about identifying key concepts. There was preference for using on-demand prompts to allow the learner to personalise their learning. These works inherit the deficiency of in-video quizzes described above, as the production of the prompts requires substantial authoring effort and would not be applicable to a wide range of freely available videos. Furthermore, the evaluation studies indicate the need for personalisation, i.e. adapting to the learner’s context and preferences. The research presented in this paper offers automatic ways for both constructing prompts for video engagement and adapting the content and timing of the prompts to individual learner’s needs.

Visualisations are another technique that is used to increase student engagement with videos – either by providing awareness of students’ interaction with videos or by showing key points presented in the video. Learning analytics approaches can be employed to develop visualisations of students’ viewing and video interaction behaviour in the form of engagement heatmaps (Xia & Wilson, 2018; Chatti et al., 2016) or graphs (Risko et al., 2013). Such visualisations are a popular way to present an open learner model or an open social learner model which can make students aware of their behaviour and the behaviour of others and can promote reflection and improve engagement (Brusilovsky et al., 2015). Visualisations can also show aspects of the video content to promote deeper understanding of what has been presented in the video, facilitate video navigation and thus promote engagement. Concept maps have been used for this by showing key concepts and links between them (Liu et al., 2018; Zhang et al., 2019). While a very powerful technique, the construction of concept maps can be challenging, either from the learners’ perspective (Lui et al., 2018) or from an automation perspective (Zhang et al., 2019). The approach presented here visualises past student engagement with videos; hence, it can be seen as a form of open social student models. To understand the content of the video, we use the comments other students have written while interacting with videos, which is a simpler way to show video content and support navigation. Distinctively, we combine visualisations and personalised prompts to offer a holistic interactive support for video engagement and promote student reflection on experiences related to content shown in the videos.

Video Annotation for Learning

Video annotation offers an elegant way to enhance student engagement with videos without modifying videos. This is done by providing interactive means to support learners to take notes while watching the video (Evi-Colombo et al., 2020). Past research has shown notable benefits of notetaking while watching videos, such as improved search and retrieval, deeper analysis of video content, fostering reflection, engaging in discussions (Khurana & Chandak 2013; Yousef et al., 2014; Pardo et al., 2015). Several attempts have been made to support student engagement in video notetaking. Chatti et al. (2016) utilised mind map-based techniques to support collaborative video annotation and to encourage learners’ interaction around a video lecture. Dodson et al. (2018) present a video player which allows students to write notes and highlights important parts of the video using the video transcript. Chiu et al. (2018) developed a video annotation learning system which supports students to make annotations on the video by highlighting a part of the video and adding a textual comment directly on the video. An approach to support video notetaking by prompting learners to insert annotations on transcripts of video lectures and then reinterpret and synthesise their notes is presented in Liu et al. (2019). Our work complements these existing works by facilitating learners’ notetaking when watching videos, in the form of free-text comments. Our approach provides two levels of support: (i) meta-scaffolds in the form of sentence openers to focus the comments and (ii) interactive nudges to encourage learners to note key points and make comments.

While notetaking has a potential to empower learning, studies have shown the need to better promote and scaffold higher-order cognitive strategies and deeper learning with the use of video annotation software (Pardo et al., 2015). Techniques from learning analytics (Oleksandra et al., 2018) and artificial intelligence (Seo et al., 2020) can be utilised to characterise the learning process and to provide effective support for learning through video notetaking. Existing approaches utilise text mining techniques. For example, Dodson et al. (2018) have utilised text analytics to identify important aspects of the video from the video transcript. Chatti et al. (2016) embed text comparison techniques to define the similarity between annotations in order to facilitate annotation organisation and search. Joksimović et al. (2019) explore linguistic and discourse properties to classify student self-reflections in video-based annotations. Hoppe et al. (2016) apply network text analysis of video comments while students share videos to extract information on the learners’ domain understanding and detect possible misconceptions. Network text analysis is also applied to characterise learner engagement in video watching by analysing the video comments learners have made (Hecking et al., 2017). Personalised support is often identified as highly needed by earlier works but they have not shown how to provide this support. Similarly to these works, we utilise data analytics approaches to get deeper insight into learning behaviour from the digital traces the learners leave when making comments on videos. However, our approach presents a step change in the use of data analytics – not only can we understand engagement, but we have shown how this can then be used to shape automatic ways to offer personalised support in the form of interactive nudges. We present a novel framework for adding interactive nudges to foster active video watching, which are tailored to the individual learner’s behaviour. We utilise the framework in a video notetaking system, which is extended with interactive nudges in the form of visualisations and personalised prompts. We then explore the impact of nudges on learner engagement with videos in two evaluation studies, which show promising results.

Nudges and Choice Architecture to Support Learning

Nudges were introduced in decision support as a form of interventions which influence people’s behaviour (Thaler & Sunstein, 2008). While sharing many features with persuasion (Masthoff & Vassileva, 2015), nudges are more about behaviour changes while persuasion focuses on changing beliefs. Michie et al. (2011) note the key principles of behaviour change: (i) maximise capability to regulate own behaviour; (ii) increase/reduce motivation to engage /discontinue in the desired/undesired behaviour; (iii) maximise opportunity to support self-regulation. We adopt these principles in video-based learning, as follows: (i) capability: take into account the learner’s context and knowledge/experience of the target domain; (ii) motivation: aim to increase the learner’s motivation to engage in active video watching and to improve their knowledge; (iii) opportunity: automatically identify opportunities to support engagement in active video watching to improve learning.

Behaviour change nudges are adopted in educational systems; examples include visual signposting and interactive interventions. Although not explicitly called nudges, open student models can act as signposting nudges to promote reflection and self-awareness (Bull & Kay, 2016; Long & Aleven, 2017), while open social student models can promote social comparisons (Brusilovsky et al., 2015). These approaches focus on the design of effective visualisations, and rely on the students’ abilities to interpret such visualisations. On the other hand, interactive intervention approaches rely on the system’s ability to automatically trigger short interaction scripts to nudge the learners to the desired behaviour. Interactive nudges can be simple reminders of college tasks (Castleman & Page, 2015), prompts for goal setting and reflection (Kravčík & Klamma, 2011), dialogue games to promote reflection (Dimitrova & Brna, 2016), interactions to support articulation of thoughts (Sottilare et al., 2014), or navigation path suggestions (Thakker et al., 2012; Al-Tawil et al., 2020). Piotrkowicz et al. (2020) investigate what kind of nudges are effective in a healthcare e-learning scenario and which learner characteristics will be useful for personalisation of such nudges. None of the current approaches aims at adding nudges to improve the effectiveness of video-based learning, which is the main goal of our approach. While all current utilisations of nudges in educational context offer bespoke implementations suitable for the specific system and learning goals, we present here a generic approach providing a choice architecture for nudges to encourage active video watching.

The kernel for designing nudges is the Choice Architecture, i.e. the method used by the system to help people make choices that lead to better outcomes (paternalism) but in an unobtrusive and non-compulsory manner (libertarian) (Thaler & Sunstein, 2008). Choice Architecture can provide the underlying framework for designing intelligent interactive systems by defining the design processes to nudge people to make choices that are beneficial for them (Jameson et al., 2014). Earlier examples of Choice Architecture for designing interactive systems consider helping with choices related to participation in online communities (Jameson et al., 2014), e-commerce or tourism (Bothos et al., 2015). More recently, examples of Choice Architecture for interactive nudges have been developed, e.g. to help users to change their energy consumption habits (Starke et al., 2020) or to increase the informed consent and privacy awareness of users (Bergram et al., 2020). A ‘holistic’ framework for the design of Choice Architecture for recommender systems is proposed in (Cena et al., 2020), including a broad set of user characteristics related to behaviour habits. To the best of our knowledge, there are no Choice Architecture frameworks for learning environments. Such architecture should not only define interaction adaptation decisions (as in conventional learner-adaptive educational system), but should also specify how the system’s interventions (in the form of nudges) link to learner behaviour (behaviour that has to be corrected/avoided and desirable behaviour that brings educational benefits). While there is research that considers the interaction behaviour with learning system, there is no systematic way how to shape the interaction environment to improve the learners’ behaviour leading to more productive interaction. This paper presents such research within the context of video-based informal learning. We present a unique choice architecture framework which adopts the nudge taxonomy proposed by Münscher et al. (2016) to guide the provision of personalised nudges to promote active video watching for learning. Our work is the first attempt to devise choice architecture for personalised nudges to improve learning. We adopt a data-driven approach, which taps into learner sourcing to inform the design of personalisation nudges - the analysis of interaction behaviour by other users is used to identify areas in a video when interactive interventions can be appropriate and to inform the composition of interactive nudges.

Learner sourcing is a relatively new research stream which explores the adoption of crowdsourcing approaches to advance learning (Kim, 2015). Its popularity is driven by the vast data collected in online learning environments which creates new opportunities to harness data from past learners to improve the learning experiences of new learners (Doroudi et al., 2018). Learner sourcing has been adopted to enable educational content creation, assessment and feedback at scale. Video-based learning is one of the first domains where learner sourcing was adopted; this underpinned data-driven approaches which use large-scale learning interaction data to dynamically improve video content and video interaction. Kim (2015) applied learners’ collective activities to identify points of confusion or importance in a video, extract a video structure, and improve navigation for learning. The approach was applied in MOOCs and freely available how-to YouTube tutorial videos allowing new leaners to benefit from the interaction histories of past learners (Weir et al., 2015). Liu et al. (2018) recruited crowd workers to collaboratively generate a concept map by prompting them to externalise reflections on the video; this helped with generating concept-based video navigation and comprehension which was comparable with expert advice. Similarly to these works, we use a data-driven approach where behaviour of past learners help to shape new learning experiences for future learning. Distinctively, our unique learner sourcing approach informs the design of nudges for active video watching, which provides a new avenue not studied by existing work.

Active Video Watching Platform

The research presented here follows an empirically-driven methodology where we utilise a platform for video-based learning to derive requirements for interactive nudges, which inform our novel framework for choice architecture. We then implement nudges in the platform and examine their effect on learning experience.

Our video-based learning approach builds upon students’ experiences with social media sites for video sharing (e.g. YouTube) and integrates interactive notetaking during video watching to facilitate student engagement and reflective learning. The approach is illustrated with the Active Video Watching (AVW) space (AVW-Space) (Lau et al., 2016; Mitrovic et al., 2016), a controlled video watching learning environment designed for self-study. It can be customised by the teacher who selects a set of videos for a class and defines aspects to serve as micro-scaffolds for learning. Aspects aim to draw the student’s attention to specific points related to the target transferable skill and to trigger reflective experiential learning. The AVW-Space is a fully customisable video learning environment, instantiated in systems with identical functionality hosted by the University of Leeds and the University of Canterbury, respectively. Earlier instantiations were prepared within the EU project ImREAL and included communication skills training in business contexts, medical communication training, and adapting to academic environment in a foreign country (Despotakis et al., 2013). These instantiations were adopted in practical settings but there were no rigorous evaluation studies to assess the pedagogical effectiveness of AVW-Space.

An AVW-Space instantiation was prepared to provide an experimental platform to assess how the system can support transferable skills learning. The instantiation, which was shaped by a research team from the University of Leeds, the University of Canterbury and the University of Adelaide, focuses on training university students to deliver pitch presentations (presentations that pitch an idea to a broad audience) (Lau et al., 2016; Mitrovic et al., 2016, Dimitrova et al., 2017). The AVW-Space for pitch presentations includes eight YouTube videos (Table 1). Four videos are tutorials on giving presentations, while the other four are actual recordings of pitch presentations (two TED talks, and two 3-min PhD pitch presentations). The criteria for selecting the videos were: (i) appropriate content (covering opening, closing, structure, delivery and visual aids; or examples of pitch presentations); (ii) no longer than 10 min; (iii) balance of gender for the presenters; (iv) two popular examples and two not so popular (based on the videos’ YouTube ratings).

Table 1 Videos used in studies

AVW-Space includes two components: Private Space and Social Space. Initially students watch and comment on videos individually in the Private Space (Fig. 1, left). In order to enter a comment, the learner needs to stop the video, type in their thoughts and select an aspect. We specified three reflective aspects for tutorials: “I didn’t realise I wasn’t doing it” (TA2), “I am rather good at this” (TA3), “I did/saw this in the past” (TA4); these aspects stimulate learners to recall and reflect on their own experiences. There was one additional aspect, “I like this point” (TA1), which allows the learner to externalise learning points. For the example videos, the aspects were: “Delivery” (EA1), “Speech” (EA2), “Structure” (EA3), and “Visual aids” (EA4), corresponding to the concepts covered in the tutorials. The system records information about the specific place in the video (i.e. the time elapsed from the start) related to the comment. The student can watch the video multiple times. Once the teacher approves comments for sharing, anonymised comments are available for browsing in the Social Space (Fig. 1, right). A second level of micro-scaffolds is provided where the students are encouraged to rate the comments. The rating categories, which are designed to further promote reflection, are: “This is useful for me”, “I hadn’t thought of this”, “I didn’t notice this”, “I don’t agree with this”, and “I like this point”.

Fig. 1

Screenshots from the Leeds version of AVW-Space: Adding a comment (Personal Space, left); and rating a comment (Social Space, right)

Pedagogical Framework

Engagement is crucial for effective learning but engagement in online learning, and especially learning from videos, is often low (Chi & Wylie, 2014; Morgan & Adams, 2009; Yousef et al., 2014). To understand how AVW-Space supports video-based learning, we operationalised the ICAP framework (Chi & Wylie, 2014), which considers overt actions students perform as the prime source of information about their engagement. ICAP classifies overt learner behaviours into four type of learning modes, corresponding to different levels of cognitive engagement - Interactive, Constructive, Active and Passive. Chi and Wylie (2014) provide evidence that as students become more engaged, starting from the passive mode to the interactive mode, the learning effectiveness increases; i.e. Passive < Active < Constructive < Interactive. Passive learners are simply receiving information, without performing any additional actions; they might be observing a lecture, reading a book or watching a video, but do not engage further. Active learners exhibit additional actions, such as note taking, but those actions simply replicate provided information, such as writing down lecturer’s statements, or rewinding the video to watch important parts multiple times. Constructive learners extend the active mode by generating new information that was not explicitly taught; e.g. summarising points, drawing a concept map, or providing a self-explanation. Interactive learners engage in discussions and collaboration with their peers and/or tutors, where they compare and contrast different opinions and jointly generate solutions to problems.

In AVW-Space, comments provided by the students contain remarks on important events in videos, and contain statements showing reflection and self-explanation. AVW-Space does not currently support collaboration between students, and therefore we do not consider the Interactive mode of ICAP. Hence, operationalising ICAP, we consider the following categories of students:

• Passive learners (PL) log on to the system and watch videos but do not write any comments.

• Active learners (AL) log on to the system, watch videos, and write comments related to the points in the videos.

• Constructive learners (CL) exhibit the same interactive behaviour, i.e. watching videos and writing comments, but their comments add something new – either the comment itself adds something new or the aspect used adds (e.g. when the learners reflect on past experience).

• Active/Constructive learners (A/CL) - in the studies reported in this paper, we do not distinguish between AL and CL, as AVW-Space was not capable of analysing comment text.¹

Experimental Setup

We conducted a series of experimental studies with the AVW-Space instantiation for pitch presentations, and discuss four of those studies in this paper.

• Participants were university students, including undergraduate, Master or PhD students. Each study included a fairly homogeneous group of either undergraduate or postgraduate students from a similar discipline. Participation was always on a voluntary basis.

• The procedure used in the initial two studies (Fig. 2) included two phases. Before and after each phase a corresponding survey was conducted. In Phase 1, the participants were asked to watch the tutorial videos first, and make comments on them. After completing the tutorial videos, they were instructed to critique example videos on the four aspects (structure, visual aids, delivery and speech). There was no specific guidance about what should be included in the comments (apart from the micro-scaffolds which provided implicit guidance on aspects to refer to). In Phase 2, participants were directed to the Social Space where they could see comments from other students (only comments approved by the teacher were seen, all comments were anonymous). The system interaction preserved the free exploration nature of video-based learning, including personal notetaking and reading comments from others.

• Materials included three surveys. Survey 1 collected information for the participant’s profile: demographic information, background experiences, Motivated Strategies for Learning Questionnaire (MSLQ) (Pintrich & De Groot, 1990), and the participants’ knowledge of presentations (conceptual knowledge). The student’s conceptual knowledge was assessed from students’ replies to three timed questions. Participants had one minute per question to write phrases they associated with (i) structure, (ii) delivery and speech, and (iii) visual aids. Survey 2 included the same questions for knowledge of presentations, the NASA-TLX instrument (Hart, 2006) to check participants’ perception of cognitive load when commenting, the Technology Acceptance Model (TAM) (Davis, 1989) to check participants’ perceived usefulness of commenting on videos for learning, and questions on usability related to commenting on videos. Survey 3 was similar to Survey 2 but related to rating others’ comments.

• Data collected from each study included: participant profile (demographic and MSLQ), conceptual knowledge (pre- and post- interaction), AVW-Space interaction log, comments provided, usability scores (TAM & NASA-TLX). To assess students’ responses to the conceptual knowledge questions, we developed an ontology of presentations, consisting of three taxonomies related to the structure, delivery and speech, and visual aids (Abolkasim et al., 2018). Each response was marked by three independent markers, indicating the number of ontology entities associated with the response. The inter-rater reliability was high: the Krippendorff’s alpha was for 0.894 for Study 1, and 0.907 for Study 2. The final scores were confirmed by a fourth marker using the majority vote, or if that was not possible, re-marking the entries. For follow on studies, the assessment was automated applying text analytics methods by using the vocabulary extracted from the ontology (Piotrkowicz et al., 2018).

• Data analysis considered participants’ engagement based on the interaction logs and comments, following the ICAP framework. Comparison between participants was made based on their profiles, conceptual knowledge and usability scores.

Fig. 2

Outline of the experimental set-up for Studies 1 and 2

Findings from Studies 1 and 2 and Requirements for Personalised Nudges

To assess the usefulness of AVW-Space for transferable skills training and elicit requirements for extending the system, we conducted two user studies: Study 1 with postgraduate (PG) students, and Study 2 with undergraduate (UG) university students. Ethical approvals were obtained from both the University of Leeds and the University of Canterbury. In this paper, we focus on how the studies informed the design of personalised nudges to extend the Personal Space. Hence, in this paper we present only an overview of those results and requirements that have directly informed the extension of AVW-Space with nudges. The reader is directed to (Mitrovic et al., 2016; Dimitrova et al., 2017; Mitrovic et al., 2017; Hecking et al., 2017) for more details about the studies and a broader set of results and requirements for active video watching that underpinned other strands of our research.

In Study 1, all participants used the full version of AVW-Space (hosted on the server at the University of Leeds). The study started with 48 participants, recruited on a voluntary basis via advertisement to relevant mailing lists. From them, 40 participants commented on videos, and are classified as Active/Constructive Learners, while two students watched videos passively (PL). The remaining six students have never logged on to AVW-Space and we excluded the data collected about those students from analyses.

In Study 2, we used two versions of AVW-Space, developed at the University of Canterbury. The experimental design used in Study 1 was adapted to include the experimental and control conditions in Study 2 (Fig. 2). The experimental condition used the full version of the system, including the aspects that provided meta-cognitive scaffolds during comment writing, and rated comments. The control condition used the version of AVW-Space without aspects, thus making the platform similar to conventional video-watching social media spaces, such as YouTube. The control condition provided only the Personal Space (the Social Space was not provided). The participants for Study 2 were recruited from undergraduate Engineering courses at the University of Canterbury on a voluntary basis. 37 participants joined the study, and were randomly assigned to either the experimental group (17 males and 2 females) or control group (13 male and 5 female). The majority of participants (83.8%) were aged 18–23. There were 16 participants who completed surveys but have never logged on to AVW-Space. The remaining students watched the videos, including 8 Passive Learners (4 control, 4 experimental), and 13 Active/Constructive Learners (6 control, 7 experimental).

Overall, there were 744 comments and 2706 ratings in Study 1. In Study 2, there were 149 comments written by the control group, while the experimental group participants wrote 90 comments and made 332 ratings. Table 2 presents several example comments the participants made on tutorial videos. Some comments (e.g. C1) are too short; the student did not elaborate on what was good and why. Although C2 was made using the same aspect (I like this point), this comment shows that the participant was reflecting on his/her own experience in giving presentations. In comment C3, the participant elaborated on potential problems with powerpoint slides, again reflecting on their past experience. In comments C4, C5 and C6, the participants are thinking about their future presentations and identify what they need to improve. The last comment (C7) illustrates a situation when a participant is reflecting on his/her past performance, clearly identifying one strength.

Table 2 Example comments on tutorial and example videos

For the four example videos, the comments were almost equally distributed over the aspects, showing that the participants were watching the videos with those aspects in mind. This is evidence that aspects do scaffold participants’ thinking. The students noted both positive points (e.g. C8, C10, C13 in Table 2) and negative points (C9, C11, C12, C14 in Table 2). Comments usually included one sentence, some included several sentences.

The results of Study 1 showed that only Active/Constructive (watching videos, writing comments and rating comments) led to increased conceptual understanding (Mitrovic et al., 2017). In contrast, Passive engagement did not lead to increased conceptual understanding. We also found indicators that using aspects when commenting was beneficial, as well as rating comments written by others. The analyses of responses to the TAM and NASA-TLX questionnaires revealed that writing comments was cognitively demanding, as participants needed to identify appropriate places in the video and to reflect on past experience. At the same time, participants noted that commenting allowed focusing their attention on important parts of videos, kept them alert and active, and reinforced learning. Based on these findings, we formulated the following requirements for enhancing AVW-Space:

• Requirement 1: Further enhance AVW-Space with intelligent support to promote Active/Constructive engagement by encouraging students to write comments while watching videos.

• Requirement 2: In the Personal Space, make it mandatory to specify aspects when writing comments. Include intelligent support to encourage students to use a diverse range of aspects, and give preference to aspects that trigger reflection.

• Requirement 3: Include the frequency of writing comments and the use of different micro-scaffolds in the learner profile, which can be used to personalise the intelligent support in the Personal Space.

• Requirement 4: Add interaction means to aid the commenting process, such as indicating areas in the videos where the students can see interesting points and showing example comments that could trigger reflection on points seen in the videos.

Choice Architecture for Video Engagement Nudges

Following the requirements distilled from Studies 1 and 2, we identify the enhancements of AVW-Space to promote active and constructive behaviour. We propose this to be done by adding intelligent nudges - personalised interventions aimed to influence learners’ behaviour towards active/constructive learning without limiting their personal choices for engaging in AVW-Space. In this Section, we outline a generic framework providing choice architecture for personalised nudges to support video engagement leading to active/constructive learning. The choice architecture for intelligent interventions includes several main points, which are outlined below.

Target Behavior

This is associated with the positive/helpful behaviour that should be achieved (for instance, in driving this could be to reduce speed with the acceptable limit). In the case of video-based learning, this includes overt interaction behaviour linked with active/constructive learning. Following the recommendations in the previous Section, we identified several target behaviour situations that can be associated with active/constructive learning:

• making comments while watching videos;

• noticing important segments in videos (e.g. in AVW-Space this can be identified as making comments in places that have attracted attention by many learners);

• reflecting on points made in the video (e.g. in AVW-Space this can be identified as using reflective aspects or writing reflective comments);

• linking points in the video to personal experience (e.g. in AVW-Space this can be related to using reflective aspects in comments on tutorials, or making explicit references to personal experience in comments on both tutorials and examples).

Observed Behaviour

This is usually associated with negative/unhelpful behaviour that should be avoided (for instance, in driving this could be fast speed that exceeds the allowed limit). In the case of video-based learning, observed behaviour includes situations which indicate that learners are not engaging in active/constructive learning based on overt interaction with the system. To detect these situations, the video-based learning system requires a learner model. Following the studies presented in the previous section, we have identified several observed behaviours where nudges may be needed to foster active/constructive learning:

• not knowing which parts of the video may be important to note;

• watching a video (or several videos) without writing comments;

• not knowing what to put in a comment;

• making shallow comments (e.g. in AVW-Space this can be observed as using mainly ‘I like this point’ aspect for tutorials);

• not linking points in the video with personal experiences (e.g. in AVW-Space this can be observed by the limited use of reflective aspects);

• not noticing different aspects of the skills (e.g. in AVW-Space this can be detected by the lack of diversity in aspects associated with the comments).

Framework for Nudge Design

To define a generic framework for the design of interactive nudges for active video watching we follow (Dimitrova & Brna, 2016), and define each nudge as an interaction game: N = <G,T,I,O >, where G defines the goal of the game (i.e. what observed behaviour we want to change with the nudge), T defines the conditions when the game will be triggered (i.e. situation(s) when the intervention will be generated), I defines the interaction means (we consider two interaction means – visualisations and prompts), and O defines the expected outcome (i.e. target behaviour we want to achieve with the nudge). To design nudges for video-based learning, we will follow the three basic categories of choice architecture intervention techniques defined by Münscher et al. (2016), corresponding to decision making stages: decision information, decision structure, and decision assistance.

Decision information nudges facilitate the perceptual processes of problem representation, formulation, or framing to help people process the available information that can affect their behaviour (Münscher et al., 2016). This includes:

• translate information to make it easily understandable (for example, prompts sent prior logging to the system to indicate that commenting is helpful for learning);

• make information visible, including information about other people’s behaviour (for example, showing ‘high attention’ intervals of the video which attracted comments by previous participants) or learner’s own behaviour (for example, an open learner model of the learner’s engagement behaviour);

• provide social reference point, including reference to a behaviour norm (for example, provide information what a constructive learning behaviour is, or use open social learner model to compare the student’s behaviour with the behaviour of the other students in a selected group).

Table 3 provides example interaction games defining decision information nudges.

Table 3 Example decison information nudges to promote active and constructive learning

Decision structure nudges facilitate assessment and selection of alternatives when a decision is to be made, including range/composition of options and default options behaviour (Münscher et al., 2016). This includes:

• choice defaults, such as no-action default (for example, interrupting the video watching to show a prompt when the learner continuously misses to make comments) or a default prompted choice (for example, prompting learners to summarise the main points at the end of the video and link to their experiences);

• change option-related effort to decrease effort (for example, show high-attention intervals to indicate what parts of the video other students found interesting) or increase effort (for example, encourage the learner to use diverse aspects or to make more reflective comments linking video content to personal experience);

• change range or composition of options, including option order (for example, change order of aspects to make reflective aspects easier to select) or grouping of options (for example, visualise past comments from other learners grouped by the aspects used);

• change option consequences, including connect decision to benefit (for example, refer to benefits to learning when using reflective aspects) or change social consequences of the decision (for example, indicate that linking video content to personal experience can show examples that may be useful for other learners).

Table 4 provides example interaction games defining decision structure nudges.

Table 4 Example decision structure nudges to promote active and constructive learning

Decision assistance nudges foster deliberate commitment and remind people of behavioral options (Münscher et al., 2016). This includes:

• provide reminders (for example, when the learner has watched a video but have not made a comment, send them a reminder to do so; reminders can also be sent when the learner passes a high attention interval but has not made a comment);

• facilitate commitment by providing feedback to ‘reward’ positive behaviour (for example, congratulate the learner on making comments that can be valuable to others).

Table 5 provides example interaction games defining decision assistance nudges.

Table 5 Example decision assistance nudges to promote active and constructive learning

The choice architecture for video-based learning outlined here is generic. We will present next how it has been operationalised in AVW-Space.

Extending AVW-Space with Personalised Interactive Nudges

To design nudges, we adopted a data-driven approach using interaction traces generated by learners in the user studies to help improve the interaction of future learners. The design of nudges requires identifying opportunities for intervention, i.e. to decide when there may be a suitable time for an intervention, and what to include in a nudge. For this, we used interaction traces generated by learners in the user studies. We identified video intervals worthy for attention, and investigated ways to identify good comments to use as examples in the nudges.

High Attention Intervals

An attention interval I is defined as a continuous stretch of video consisting of a set of comments C. The granularity of continuity is determined by how big time gap θ is allowed between adjacent comments. We define an aggregation predicate A(C), which aggregates comments from a given set C, as follows:

$$ A(C)\equiv \forall \left({c}_i\in C\right)\exists \left({c}_j\in C\right)\ \left[\left({c}_i\ne {c}_j\right)\bigwedge distance\left({c}_i,{c}_j\right)\le \theta \right] $$

This allows us to aggregate comments in attention intervals - an interval starts from the start time of the first comment to the end time of the last comment in a set of aggregated comments. The attention intervals indicate areas in a video where users have noted something. Table 6 summarises the output of interval aggregation for the eight videos in the AVW-Space instantiation for pitch presentation skills, based on the comments by the students in the two studies presented earlier. The time distance parameter θ was set to 4“ for tutorials and 6” for examples, and was selected as the maximum number that gives a reasonable interval partitioning (larger values of θ will aggregate almost all comments in one interval). The high attention intervals are used for identifying situations that could trigger interactive prompts.

Table 6 Summary of the interval aggregation in each video. T_i are tutorial videos, and E_i are example videos

As a first step towards more intelligent support for active video watching, we implemented nudging in the form of signposting through interactive visualisations and personalised prompts in AVW-Space.

Context Model

To enable adaptation, we propose a context model C = <U_YT, U_K, U_MSLQ, U_L, V_I  > that includes information about both the user and the video. Explicit profiling obtained from Survey 1 (before interaction with AVW-Space) includes U_YT (the user’s experience in using YouTube videos for learning), U_K (the user’s knowledge and previous experience in the target skill), and U_MSLQ (the MSLQ scales used for generating the clusters: self-efficacy, extrinsic motivation, rehearsal, self-regulation, and organization). Implicit profiling from interaction logs U_L includes the number of comments and frequencies of video aspect usage. The video information aggregates the interaction traces by others where V_I is the set of high attention intervals with detected interaction patterns.

Interactive Visualisations in AVW-Space

Interactive visualisations (shown below the video in Fig. 3) are instantiations of the DI_visible decision information nudge and the DS_grouping design structure nudge discussed in the previous Section. The top visualisation is the comment timeline; it provides signposts in terms of comments written by previous learners. Each comment is represented as a coloured dot, representing the time when the comment was made. The colour of the dot depends on the aspect used, with the legend shown on the side. We selected the best comments from previous studies to use in the comment timeline. The comment timeline also allows the learner to inspect comments written by previous learners. When the mouse is positioned over a particular dot, the student can see the comment (as in Fig. 4). Dots are slightly transparent, so that comments made in temporal proximity to each other can be differentiated. Clicking on a dot begins playing the video from that point.

Fig. 3

A screen shot of AVW-Space, showing interactive visualisations

Fig. 4

A screen shot of interactive nudges in AVW-Space, including interactive visualisations and a prompt. The interactive visualisations are modified to show only comments written with the “I didn’t realize I wasn’t doing this” aspect.

The bottom visualisation is the comment histogram, representing the number of comments written for various segments of the video. This visualisation allows the student to identify quickly important parts of a video, where other students have made many comments. The two visualisations meet two identified needs: (1) providing social reference points so that students can see comments written by their peers, and (2) indicating important parts of a video and what kind of content can be expected in those parts, differentiated by aspect colours.

Personalised Prompts in AVW-Space

Personalised prompts are designed to encourage students to write comments. An example of a personalised prompt is shown in Fig. 4, to the right of the video. We developed four types of prompts:

• No comment reminder is a prompt encouraging the student to make a comment. This nudge is an instantiation of the DS_no-action decision structure nudge. This prompt is offered when the student has watched at least 30% of the video without making any comments, and is currently in a high-attention interval.

• No comment reference point prompt reminds the student to make a comment, and offers an example as stimulus. This prompt is an instantiation of the DI_social decision information nudge. The prompt is only shown if the ‘No comment reminder’ prompt has not resulted in a comment. Such prompts are provided when the student has watched at least 70% of the video without comments, the student is in a high-attention interval, and this type of prompt has not been issued on the current video. The comments used as examples have been manually selected for each video from comments gathered in previous studies.

• Aspect under-utilised: a prompt to make a comment using a particular aspect that the student has used least often (Fig. 4). This prompt is an instantiation of the DS_increase-effort decision structure nudge.

  This type of prompt is provided when the student has made at least one comment on the current video, has watched at least 30% of it and is currently in a high-attention interval. When the prompt is issued, the visualisations change to only show comments made using the under-utilised aspect referred to in the prompt. For each aspect, the text of the nudge changes. For example, for the ‘I am rather good at this’ aspect, the title of the nudge is “Are you good at this?”, and the description is “Are there any techniques in the tutorial that you feel you have already mastered?”

• Diverse Aspects: this prompt provides positive reinforcement, displayed when the student has used all relevant aspects on the current video. This prompt is an instantiation of the DA_commitment decision assistance nudge. The title of the prompt is “Well done!” with the explanatory message “Great job using all aspects to comment on the video!”

Evaluating the Effect of Nudges

We report on two studies with the enhanced version of AVW-Space in order to evaluate the impact of intelligent nudging. In both studies, participants had to specify aspects when commenting. Study 3 was conducted with PG students, who only watched and commented on videos. Therefore, it is possible to compare the commenting phase on Study 1 with Study 3.

Furthermore, we conducted Study 4 in a first-year UG course mandatory for all engineering students at the University of Canterbury (Mitrovic et al., 2019). This allowed us to collect a large number of participants. The students worked on a group project and gave a presentation, during which each student presented for one minute. Due to an already full curriculum, students received no formal training on giving presentations. Instead, they were invited to use AVW-Space for online training. The students who watched at least one video in AVW-Space received 1% of the final course grade. The students were randomly allocated to the control or experimental group. The control group interacted with the original version of AVW-Space (no visualisations and personalised prompts), while the experimental group interacted with the enhanced version (Figs. 3 and 4).

Studies 3 and 4 addressed the following research questions, which examine the effect of nudges.

• Research Question 1: To what extent does the inclusion of interactive visualisations and personalised nudges impact student engagement and learning?

Hypothesis H1: Since intelligent nudges encourage students to write more comments, we expected to see higher engagement (i.e. more comments) in Study 3 in comparison to Study 1, and also when comparing the experimental to the control group in Study 4. .

Hypothesis H2: In terms of learning, we expected more learning in Study 3 than in Study 1, and by the experimental group than the control group in Study 4.

Hypothesis H3: Due to the Aspect-underutilised nudge, we hypothesised that there will be a significant difference in how the participants use aspects.

• Research Question 2: Are nudges effective in encouraging active/constructive behaviour?

Hypothesis H4: We expected to see a higher proportion of A/CL students in Studies 3 and 4, due to intelligent nudging.

• Research Question 3: What features of AVW-Space influence learning? Can we infer causal relationships between the use of AVW-Space’s features and learning?

Hypothesis H5: We anticipate that intelligent nudging will have a positive effect on the number of comments written, which will in turn have a positive effect on learning. Therefore, we anticipate the indirect effect of nudges on learning.

• Research Question 4: Do students in control/experimental group have different opinions about the usefulness of AVW-Space and cognitive load?

Hypothesis H6: We expect that there will be a difference in subjective opinions of participants who had nudges in comparison to those who worked with the original version of AVW-Space, because nudges support students in deciding when and what to comment. This will result in different responses to NASA-TLX and TAM.

Experimental Design

Materials

The videos used were the same ones used in Studies 1 and 2, as was the case with Survey 1. The students watched videos and wrote comments first, and then rated the comments in a separate phase. A difference in the experiment design was that we did not administer Survey 2 in between these two phases. Instead, we administered the final survey, which contained the conceptual knowledge questions, and usability and cognitive load questions for both writing comments and rating comments. The experimental group in Study 4 and the participants in Study 3 also received questions related to interactive visualisations and personalised nudges.

Procedure

The students were invited to participate by email. The invitation also stated that the results from our previous studies showed that students who commented on videos, and rated comments written by other students had increased their presentations skills; this was a decision information nudge. After completing Survey 1, the participants were instructed to log on to AVW-Space, watch the four tutorial videos first and then to proceed to critique the example videos.

Results: Comparing Study 1 and Study 3 (PG Students)

In Study 3, 37 participants completed Survey 1, but three participants never logged on to AVW-Space. The participants from Study 3 on average received 16.38 nudges (sd = 4.23). There were two PL and 32 A/CL (11 males and 23 females; four aged 18–23, nine aged 24–29, ten aged 30–35, three aged 36–41 and eight aged 42+). Table 7 presents results from Studies 1 and 3. The average number of comments written by the participants who interacted with AVW-Space is larger in Study 3, although the difference is not significant (p = .34); therefore hypothesis H1 (expecting that the nudges would result in an increased number of comments) is not supported in Study 3. The table also presents the conceptual knowledge scores from Study 1 (only for the commenting phase) and Study 3. There was no significant difference on the conceptual knowledge scores before participants interacted with AVW-Space (CK1) between the two studies. In Study 1, there was no significant difference between CK1 and the conceptual knowledge score at the end of the study (CK2), while there is a significant improvement in Study 3. There was a significant difference on the gains from the two studies (t = 2.58, p = .01), with Study 3 participants learning more from intelligent nudging. Hence, there is some evidence towards hypothesis H2 (expecting more learning when nudges are given) in Study 3. It should be noted though there could be a difference in the two populations, since students in Study 1 started with a higher CK1. Those students might have learnt less because there was less to learn.

Table 7 Conceptual knowledge scores from Study 1 (AVW-Space without nudges) and Study 3 (AVW-Space with nudges)

The fact that they learned less may have less to do with the difference in the AVW-Space system, and more to do with the fact that they had less *to* learn, since they came in with higher prior knowledge. Additionally, there were some differences in procedure (when the CK2 was performed), which could have impacted the results. Therefore, the authors should add more nuance to their claim that “H2 is supported in Study 3”.

Table 8 shows the number of participants in different categories from the two studies. The Chi square test shows no significant difference in distributions of students over the categories in Studies 1 and 3 (p = .79). Therefore, the nudges do not seem to have changed the behaviours of PG students. This is not surprising, given that most of PG students (95%) from Study 1 who interacted with AVW-Space displayed active/constructive learning. Therefore, there is no support for hypothesis H4 (expecting a higher percentage of A/C students when nudges are given) for PG students.

Table 8 Comparing Study 1 (AVW-space without nudges) and Study 3 (AVW-space with nudges)

Table 9 presents the number of comments written using various aspects in Studies 1 and 3. The chi-square test of homogeneity between studies and aspects revealed a significant difference (Chi-square = 4.463, p < .001). The post-hoc test revealed a significant difference on EA1 and EA4 (p < .005), and also EA5/TA5. Therefore, hypothesis H3 is supported in Study 3. However, the difference in how the aspects were used might also be a consequence of the difference in interface design in those two studies; in Study 3 the participants had to select aspects for comments, while in Study 1 aspects were optional. This limitation of comparing two versions (with and without nudges) when there are also slight differences in the interface has been addressed with the comarison of the control and experimental groups in Study 4, where the same interface is used and the only difference is the addition of nudges.

Table 9 Usage of aspects in Study 1 (AVW-space without nudges) and Study 3 (AVW-space with nudges)

Regarding the scores on the NASA-TLX and TAM instruments, we have found no significant difference on any dimensions between the participants from Studies 1 and 3. Hence, H6 (expecting a difference in subjective opinions) is not confirmed for postgraduate students.

Results: Comparing Control and Experimental Groups in Study 4

Out of 1039 students enrolled in the course, 449 completed Survey 1. Of those, 347 have used AVW-Space. We received 263 responses for Survey 2, but that number included some students who have not interacted with AVW-Space. After removing those responses, we had 119 students from the control and 102 students from the experimental group who completed both surveys and interacted with AVW-Space.

Table 10 reports demographic information for the 347 students who interacted with AVW-Space. As typical for engineering courses, there were more males than females. The majority of participants (79.83%) were native English speakers. Most participants (95.39%) were aged between 18 and 23. The questions related to training on giving presentations, experience in giving presentations, using YouTube and using YouTube for learning were based on the Likert scale from 1 (Low) to 5 (High). There were no significant differences between the two groups on these features, as well as on MSLQ scales, with the exclusion of Task Value (U = 15,066.5, p = .043).

Table 10 Demographic data for the participants in Study 4. Apart from the first three rows, the remaining rows present the mean and standard deviation in parentheses

The total number of comments written by the control group students is 784, while there were 1063 comments in the experimental group. For the experimental group students, the correlation between the number of personalised nudges received and the number of comments is 0.51 (significant at p < .01). Table 11 presents some statistics about student engagement in the two conditions. There were no significant differences between the two groups on either the number of distinct days or the number of sessions with AVW-Space, or the number of videos watched. The only significant difference (U = 17,608, p = .004) was on the number of comments written. We used the Mann-Whitney test as the number of comments was not normally distributed. Fig. 5 shows the number of comments per video for the two groups. The distributions of comments are significantly different for the two groups (U = 51.5, p = .038). Therefore, hypothesis H1 (expecting a larger number of comments when nudges are provided) is confirmed in Study 4.

Table 11 Statistics about the experimental and control groups in Study 4

Fig. 5

Number of comments per video

Table 12 presents the conceptual knowledge scores from the two surveys (CK1 and CK2 respectively) overall, and also for the PL and A/CL in the two groups. We divided the students who completed Survey 1 (post-hoc) into A/CL (Active/Constructive learners) and PL (Passive learners), using the median number of comments written by the class (median = 1). A two-way ANCOVA found no significant interaction between group and category (i.e. PL and A/CL), but there was a significant main effect of Category, F(1, 216) = 3.872, p = .05, partial η² = .018. As in previous studies, A/CL improved their knowledge of presentation skills significantly. Therefore, there was no support for hypothesis H2 (expecting higher learning when nudges are given) in Study 4.

Table 12 Conceptual knowledge scores for the two groups

One of the personalised nudges (Aspect under-utilised) focused the student’s attention on aspects he/she has not used when writing comments on a video. We expected to see the difference on how aspects were used by the two groups (hypothesis H3). Table 13 shows the percentage of comments written using a specific aspect. There is no difference on the usage of aspects for example videos (EA1 to EA4). A chi-square test of homogeneity between groups and aspects on tutorial videos (TA1 to TA4) revealed a significant difference (Chi-square = 15.65, p = .001), with the effect size Phi of .117. The post-hoc analysis using the Bonferroni correction revealed a significant difference between the two groups on the usage of TA1 (‘I like this point’) and TA2 (‘I didn’t realize I wasn’t doing this’), p < .00625. Therefore, hypothesis H3 is confirmed in Study 4.

Table 13 Usage of aspects in Study 4

Table 14 presents the numbers of PL and A/CL in the two groups. We expected to see a higher number of A/CL in the experimental group. A chi-square test of homogeneity between group and behaviour type (i.e. A/CL and PL) revealed a significant difference (Chi-square = 4.463, p = .035), with the effect size Phi of .142. Therefore, hypothesis H4 (expecting more A/C learners when nudges are given) is confirmed in Study 4.

Table 14 Numbers of A/CL and PL in the two groups

Furthermore, the interaction behaviour of both groups was compared. All experimental group students received nudges, but PL have not responded to them. Table 15 contrasts the A/CL and PL from the experimental group. The only significant difference between PL and A/CL on the variables from Survey 1 is on Training (U = 2906.5, p = .04). During interaction with AVW-Space, in addition to a significant difference on the number of comments written, these two subgroups differed significantly on the number of videos watched (t = 4.61, p < .001) and prompts received (t = 2.33, p = .022). Please note that some students watched the same video multiple times, so the average number of videos watched can be higher than 8.

Table 15 Differences between PL and A/CL from the experimental group

We anticipated that the personalised prompts would have an indirect effect on learning (Hypothesis H5), and developed a path diagram shown in Fig. 6. The model was evaluated using IBM SPSS AMOS version 25, and the data collected from the experimental group. CK1 and CK2 are conceptual knowledge scores from the two surveys. The chi-square test (2.55) for this model (df = 2) shows that the model’s predictions are not statistically significantly different from the data (p =   .279). All coefficients are significant at p <   .05. The Comparative Fit Index (CFI) was .988, and the Root Mean Square Error of Approximation (RMSEA) was .052. Therefore, the model is acceptable: CFI is greater than .9 and RMSEA is less than .06 (Hu & Bentler, 1999). The model indicates that the higher CK1 score directly causes a higher CK2 score. Therefore, the effect of the number of comments on CK2 is adjusted for and above and beyond this influence (.2, p =   .024). The number of personalised prompts affects the number of comments (.41, p <   .001). Therefore, hypothesis H5 is confirmed in Study 4.

Fig. 6

Path diagram for the experimental group, with standardized coefficients

The responses to the TAM questionnaire (Davis, 1989) were made using the Likert scale from 1 (highest) to 7 (lowest). We compared the scores on the TAM questions for the two groups, and found two significant differences. For question 8 (My interaction with AVW-Space would be clear and understandable), the average score of the 115 control group participants was 3.52 (sd = 1.69), while the average score of the 100 experimental group participants was better, 3.08 (sd = 1.43); the difference was significant at p < .05. There was also a significant difference on question 9 (I would find AVW-Space easy to use), for which the average score of the control group was 3.30 (sd = 1.68) and for the experimental group it was 2.78 (sd = 1.20), p <   .05. Regarding the scores on the NASA-TLX instrument, there were no significant differences between the two groups on the mental demand, effort, frustration and performance in relation to writing comments. Hence, H6 (expecting a difference in subjective opinions) is partially confirmed for Study 4.

Qualitative feedback from students gave some explanation of the positive usability scores. The experimental group received two additional questions in the second survey, the first of which asked for feedback on the usefulness of interactive visualisations. We received 100 responses, 85 of which were positive, such as “See which parts of the video other people find useful” and “To compare yourself with the rest of the class.” One student wrote “I didn’t understand them till id finished most of the videos.” The other question was related to the usefulness of personalised prompts. We received 91 responses, 62 were positive, and 21 negative. Eight participants have not noticed nudges. Two examples of positive opinions were: “Help me to be engaged”, “To give me a little push in the right direction of what to comment on”. Some participants did not find the prompts useful: “It created subtle pressure to make comments which wasn’t really useful at all” and “They were always the same so not hugely useful.”

Conclusions and Discussion

This paper presents a framework for choice architecture of interactive nudges for video-based learning. We follow an empirically driven methodology utilising the AVW-Space platform for video-based learning targeted at transferable skills. We derived requirements for interactive nudges from the two initial studies with AVW-Space, which inform the framework for choice architecture. AVW-Space is then extended to include interactive nudges utilising the choice architecture. Evaluation studies with AVW-Space allowed us to examine the effect of nudges on learning and the learners’ experience with the system.

Summary of the Main Findings

AVW-Space implements a video-based learning approach that builds on students’ experiences with social media sites for video sharing and integrates interactive notetaking during video watching to facilitate student engagement and reflective learning. The AVW-Space platform allows teachers to develop learning spaces with minimal effort, without requiring video editing. AVW-Space provides micro-scaffolds for reflection, in the form of aspects used when commenting, and also rating categories which learners use when reviewing comments written by their peers. To understand learners’ engagement in AVW-Space, we operationalised the ICAP framework (Chi & Wylie, 2014), which considers four levels of cognitive engagement - Interactive, Constructive, Active and Passive. In AVW-Space we have observed passive and active/constructive, and we also report some results of inactive students. In order to avoid passive video watching, AVW-Space supports active learning via note taking (similar to making comments on social networking sites which are common places for informal learning).

We conducted two initial studies with AVW-Space with postgraduate and undergraduate students in order to observe how different types of students learn on the platform. The reported findings from Studies 1 and 2 allowed us to develop requirements for enhancing AVW-Space. These findings indicated that only students who actively engaged with videos, by writing comments and rating others’ comment, improved their conceptual understanding of presentation skills. In addition, Study 2 showed that the aspects provide an additional benefit, by encouraging students to self-reflect. In both studies we found that there were students who did not engage in active/constructive learning. Based on these findings, we derived requirements for enhancing the interaction in the system to improve learner engagement while watching videos. The requirements from Study 1 and Study 2 informed the design of interactive nudges for video-based learning.

We developed a framework for choice architecture by adopting a generic model for behaviour change nudges. The framework is of general nature, and consists of three categories of nudges. Decision information nudges assist learners to process the available information which can help them to better engage with videos. These nudges include helping understand information, making information visible, and providing social reference points. Decision information nudges were implemented in AVW-Space in the form of interactive visualisations showing high-attention intervals in videos, and by providing social reference points in terms of comments written by past students. Decision structure nudges facilitate assessment and selection of alternatives when a decision related to active video watching. These nudges include showing default options, change effort in selecting options, indicate benefit of options. Decision structure nudges implemented in AVW-Space promote engagement of passive learners, by generating personalised hints to encourage students to write comments and by suggesting aspects to use in comments. Decision assistance nudges foster commitment in active video watching. These nudges include providing reminders and rewarding positive behaviour. Decision assistance nudges implemented in AVW-Space include reminding passive students to write summaries and rewarding active/constructive students for writing comments. The enhanced version of AVW-Space provides four types of personalised prompts, and two visualisations: the comment timeline, showing the chosen good-quality comments on the timeline, and the comment histogram which visualises parts of the video with many comments.

Two evaluation studies of the enhanced AVW-Space compared to the initial version of AVW-Space allowed us to investigate the effect of nudges. Study 3 was conducted with PG students using the enhanced version of AVW-Space, so that we could investigate the effect of nudges in comparison to Study 1 (also conducted with PG students but with the original AVW-Space version). We also conducted Study 4 with UG students, in which the participants were randomly allocated to a control group (using the original version of AVW-Space), and the experimental group (using the enhanced AVW-Space version). Postgraduate students are generally more knowledgeable, experienced and motivated than undergraduate students. Having two studies with UG/PG students therefore allowed us to investigate the effect of nudges on different types of learners. We formulated six hypotheses, which were assessed in the evaluation studies (Table 16 summarises the findings).

Table 16 Research hypotheses checked in both evaluation studies comparing the original AVW-Space version and the enhanced version including interactive nudges

To investigate Hypothesis H1, we compared the number of comments written by participants who interacted with the original and with the enhanced version of AVW-Space. There was no significant difference on the number of comments for PG students, which is not surprising, as most of the PG students in Study 1 were active/constructive students who did not need encouragement to write comments. However, we found a significant difference on the number of comments written by the control and experimental groups from Study 4. Therefore, nudges are helpful for less experienced UG students. However, the implemented nudges are basic in nature; they do not analyse the text of comments to ascertain that comments are of good quality and of reflective nature. Encouraging learners to write more reflective comments will be beneficial for both less experienced and more experienced learners.

To address Hypothesis H2, we compared the conceptual knowledge scores from Survey 1 (before interacting with AVW-Space) and Survey 2 (after using the Personal Space for PG students, and after the study for UG students). For PG students, Study 3 participants achieved a significantly higher gain on the scores in comparison to Study 1 participants. In Study 4, there was no significant difference in learning for the control and experimental groups; therefore hypothesis H2 was not supported for UG students. While nudges stimulated undergraduate students to write comments, they did not necessarily foster deeper cognitive engagement. Therefore, more specific and adaptive nudges are needed to improve the quality of commenting and learning.

Hypothesis H3 stated that those students who received nudges would use more types of aspects, as contrary to the findings in Studies 1 and 2, where most of the students used the “I like this point” aspect. For both PG and UG students, we found a significant difference in distributions of aspects over comments, in comparison to students who did not receive nudges. We note the limitation when comparing the results of Studies 1 and 3, as the use of aspects was optional in Study 1, while in the later studies aspects were mandatory. There was no such limitation when comparing Studies 2 and 4, which supports the hypothesis. This provides further evidence about the impact of meta-scaffolds in AVW-Space and their importance for video-based learning.

Our primary intention when designing nudges was to foster active/constructive behaviour in students. Hypothesis H4 proposed that there would be significantly more active/constructive students when nudges are available in comparison to the students who interacted with the original version of AVW-Space. We found no difference between distributions of participants over categories in Studies 1 and 3, as most PG students wrote comments even without nudges. However, there was a significant difference in the number of PL and A/CL in Study 4. Therefore, the nudges are effective with the learners who do need support to become active learners.

Hypothesis H5 anticipated a positive effect of nudges on the number of comments written. We hypothesised and evaluated a path diagram for Study 4, which showed that nudges had a positive effect on the number of comments written, which in turn had a positive effect on learning. Please note that such analysis was not possible using the data collected with PG students due to the small population size in Study 3.

Hypothesis H6 stated that there would be a difference in participants’ subjective opinions of usability and cognitive load when interacting with AVW-Space. We have found no significant differences in participants’ scores on the NASA-TLX questions, for both PG and UG students. For PG students, we also found no difference on the usability of AVW-Space (using the TAM instrument). However, there were significant differences in the responses of UG students in Study 4 on two TAM questions, related to understandability and ease of use of AVW-Space, where experimental group participants had better scores in comparison to the control group students. Therefore, the implemented nudges were effective for students who need support to identify important parts of the videos. As stated earlier, we are already working on defining new types of nudges for supporting students to write better comments. In future work, we also plan to introduce nudges for the rating phase (in the Social Space), as well as to provide visualisations of the learning process which will enable students to reflect on their learning.

There are several possible explanations of the fact that we did not observe learning in all studies with nudges. For example, some students may have better self-regulation skills, enabling them to engage better with videos. Students who do not like writing comments might benefit less from the implemented nudges. The implemented nudges encourage students to write more without assessing the quality of their comments. Another strand of work we completed investigated the impact of the comment content on learning (Taskin et al., 2019). We found that although nudging resulted in a higher number of comments, it did not increase the number of elaborate comments, which are strongly associated with higher learning. This points out the importance of considering the content of comments, not only the number of comments written. This is a strand of our current/future work, as discussed below.

It should be noted that empirical results have to be interpreted with caution. Data-driven approaches are always data-dependent. The findings depend on the specific conditions in the studies, and further investigation will provide stronger evidence for their validity.

Limitations

Experimental Design

As the reported research was performed over five years, the interface of AVW-Space has changed after Study 1 was completed. In Study 1, the use of aspects was optional, while later they were mandatory. Therefore the alternative explanation for the significant difference in how the aspects were used by the postgraduate students in Studies 1 and 3 might be this interface change. However, the evidence found in the comparison of Studies 2 and 4, in which aspects were mandatory, provides evidence towards our hypothesis. Regarding measuring the learning effect, a more rigorous test would be to have student give actual presentations, and compare the marks provided by the human markers; however, such a test is resource demanding.

Implementation

A limitation of our research is the relatively small scope of user context in the current implementation of the nudges. In the reported studies, context only included the user profile which contained the demographic data and the behavioural data collected during sessions with AVW-Space. The limited user profile provided the foundation for the implemented nudges. In the current work, we are extending AVW-Space with functionality that allows student-generated comments to be classified into several quality categories (Mohammadhassan et al., 2020). User profile will be extended to capture the quality of comments a student has written, which provide the foundation for designing a new set of nudges to improve the comment quality.

Cold Start

In order to fully implement our approach, i.e. generate interactive visualisations and initialise nudges, it is necessary to collect data about user interactions with AVW-Space (e.g. comments written on videos). This builds on the interaction with videos learners are accustomed in social spaces like YouTube. However, this can bring the cold start challenge, as comments will be needed to set up the nudges. We are currently experimenting with using only the video transcripts, complemented with domain knowledge, to identify the interesting episodes in the videos which can be beneficial for learning (Mohammed & Dimitrova, 2020). This will allow us to setup default nudges before user comments are collected.

Choice Architecture and its Implications

The main contribution of this paper is a Choice Architecture for devising nudges in video-based learning systems to support active video watching. The work presented in the paper is a result of a cross-institutional collaboration with AVW-space. This allowed us to derive proxies for helpful/positive behaviour that can lead to improved learning, as well as to identify proxies for unhelpful/negative learner engagement with videos which may hinder learning. In such cases, researchers look for ways to improve the system, and this is what we did as well. We wanted to do this in a systematic way, which ensures that we can target the unhelpful behaviour and at the same time we wanted to preserve the freedom for the learner to make their own choices and take control of their learning. This is important for the context of our research - informal learning, self-regulation and transferable skills. Crucially, we also wanted to have a framework that would offer flexibility allowing gradual extension of the system by adding new features in a systematic way. The Choice Architecture proposed here shows how we have addressed these needs.

Although we have applied the Choice Architecture in AVW-space only, the proposed framework is generic and can be applied in other video-based learning systems. It adopts an established pedagogic framework for learning with multimedia (ICAP) and identifies proxies for the different levels of engagement which allows defining positive/helpful and negative/unhelpful engagement behaviour. The nudges we describe in this paper are aimed at promoting active video watching. In this respect, they are generic and can be applied in other video-based learning scenarios. Furthermore, the dialogue-game based approach for defining nudges offers flexibility how to design new nudges to extend the interaction means to move learners from passive and active levels to a constructive level. Following the Choice Architecture framework, new nudges for video-based learning can be added. Firstly, nudges already defined in the paper can be implemented with a different interface offering variations of the interaction (e.g. different visualisation or different formulation of the prompts). Secondly, nudges can be based on context parameters already presented in the paper and can offer different interaction means to achieve the expected outcome. Finally, following the nudge structure, new nudges can be defined specifying the goal, the trigger, the interaction means and expected outcome. The latter will require further analysis to identify possible goals and triggers. We have shown here a data-driven approach that adopts learner sourcing ideas how to derive the goal, trigger and expected outcome, which can be followed in other systems. There can also be other ways to identify the negative/unhelpful behaviour, e.g. experienced teachers will be able to specify this or some ideas can be given by learners in co-design activities.

In essence, the Choice Architecture is a meta-level which forces system designers to think about the learners’ interaction with the system in a holistic manner so that the system is designed in a way that nudges the user to positive/helpful behaviour. This positive behaviour can be linked to improved learning, motivation, confidence, self-esteem, etc. The Choice Architecture defines interaction elements (called nudges) which are triggered when unhelpful behaviour is detected, so that the user can make choices that avoid this unhelpful behaviour. We follow research in intelligent interactive systems (c.f. Jameson et al., 2014) arguing for a holistic approach that links together the computational part (user modelling and adaptation) and the interaction part (human-computer interaction). This can be provided with a carefully designed Choice Architecture. Similarly to recommender systems (Cena, et al., 2020), where the link between the user interaction with the system and user behaviour is made, we argue that intelligent learning environments need to adopt the Choice Architecture approach to link interaction features with desired behaviour related to the specific learning context.

Choice architecture and nudges provide a paradigm shift in the way interaction in learning systems is designed, namely looking at behaviour and aiming at changing habits. It allows linking learning analytics work that identifies user engagement problems, e.g. gaming the system, mind wandering, confusion, frustration, off-task interaction (Baker et al., 2010; Bosch & D’Mello, 2019; Faber et al., 2018; Hutt et al., 2019; Mills et al., 2020; Paquette & Baker, 2019; Peters et al., 2018) with interaction design to offer choices leading to better engagement with the system to improve learning. Taking a Choice Architecture point of view will allow comparing different interaction features (nudges) and combining them in a flexible way to improve the system.

Future Work

There are multiple avenues for future work to explore the generalisability of the approach, and conducting studies focusing on different skills. AVW-Space does not support the Interactive mode of the ICAP framework. In future work, we plan to add support for interaction between learners. We have recently added a new set of nudges to AVW-Space that encourage students to write better quality comments; the preliminary results of this study show that student who write high-quality comments learn more than those who write low-quality comments (Mohammadhassan et al., under review). We also plan to improve nudging, by suggesting a sequence of video segments tailored to the needs of individual learners. Our initial work on this explores domain ontology to characterise video segments and generate video narratives (Mohammed & Dimitrova, 2020).

As explained earlier, our goal when developing AVW-Space was to support teaching of transferable skills. In order to test the generalisability of our approach, we are currently conducting a study focusing on a different transferable skill: communication in software development meetings. We also plan to investigate the applicability of the presented approach to other skills, such as cultural awareness.

Our research opens a new avenue in developing intelligent learning environments, which adapt established interventions for behaviour change in the form of nudges. A significant feature of our approach is that it is easy to scale up, as AVW-Space does not require teachers to edit instructional videos and the choice architecture can be utilised in a range of learning contexts. AVW-Space provides a simple and elegant approach of turning videos into interactive learning. Nudges are added in the form of an intelligent component that enhances the interaction by encouraging active/constructive behaviour when interacting with videos. Our approach is applicable to a range of domains to foster video-based learning, where one can learn from their experience and that of others.