1 Introduction

Robots are increasingly being used to provide motivating, engaging and personalised support to learners [26]. Robotic tutors have been able to increase problem solving time by providing personalised hints [21] or increase learning gain by personalised problem selection [9]. Yet, they have never been used to assist children in developing self-regulated learning (SRL) skills. SRL is the meta-cognitive process where a student uses self-assessment, goal setting, and the selecting and deploying of strategies to acquire academic skills [44]. The use of SRL strategies are significantly correlated with measures of academic performance [44]. By supporting these skills students may be able to learn more effectively, even outside of the tutoring session.

This paper explores how personalised tutoring by a robot achieved using an open learner model (OLM) promotes SRL processes and how this can impact learning in primary school children (Fig. 1). We describe a study where a robotic tutor provides different levels of personalised SRL scaffolding to primary school children. The autonomous robotic tutor’s behaviour builds upon information provided to a student in an OLM. OLM externalise the model that the system has of the learner in a way that is interpretable by the learner [5]. An OLM can support SRL by promoting reflection to raise awareness of understanding or developing skills, which can help planning and decision-making [6].

To date robots have not aimed to support the development of SRL processes. The benefits of a personalised robotic tutor may motivate and engage students to utilise SRL process in the learning activity. We adopt an OLM as the basis for the personalisation as this is a simple and intuitive way of displaying to the learners their developing skills; an OLM allows us to ensure that the learner has all relevant information on which to base their reflections and SRL processes upon.

We hypothesise that more personalised and adapted scaffolding of SRL processes via OLM will lead to higher learning gain and improvement in SRL processes. Results show that more personalised and adaptive scaffolding lead to a greater indication of SRL processes and higher learning gains.

This paper is organised as follows. First we present the relevant background research on educational robots, SRL, and OLM (Sect. 2). After describing the methodology employed to conduct the study, we present the results based on the domain tests and activity logs of the learning activity. We conclude the paper with a discussion of how a robotic tutor can scaffold SRL (Sect. 5) and how this can impact child learning (Sect. 6).

Fig. 1
figure 1

Robot highlighting OLM to a primary school student

Full size image

2 Related Work

2.1 Educational Robots

There is an increasing amount of research that investigates how robots can be of benefit in an educational context. When comparing the presence of a robotic tutor to a virtual agent or on-screen feedback, there is a preference for the robotic embodiment with reference to social presence [17], enjoyment [43], trust [14], performance [12], and learning gain [22]. A key benefit of a robotic tutor (and its physical embodiment) is that it can motivate students to engage in the learning activity [22].

The social capabilities of the robot play a large part in the interaction and can be quite complex and counterintuitive. For example, it has been shown that a robotic tutor that is too social may negatively impact learning [16]. An alternative approach is to reverse the social dynamic and teaching roles so that the learner is the one to teach a robot peer [9, 40]; this can lead to a strong social bond [40] and motivate the learner to learn more words in a verb-learning game [41].

There is increasing interest in how human-robot interaction (HRI) can personalise or adapt to the learner. A robotic peer that requests help on problems estimated to optimise information gain can lead to greater learning gains [9]. A robotic tutor that personalises hints based on a student’s puzzle solving skills can lead to a more successful interaction with reduced problem solving time and a more motivated learner [21]. Prompts personalised to a specific level of detail based on the ability and performance of the learner can be more effective and less frustrating  [10]. Robot behaviours adapted to a learner’s engagement can increase recall levels [39].

There is increasing interest and amount of proposed research in exploring how personalisation can make HRI more effective by adapting difficulty levels [31], responding to affective states [15, 32], learning styles [7], and help-seeking behaviours [33]. Yet, there is no work looking at how HRI can impact SRL or meta-cognition in an educational context.

2.2 Scaffolding SRL and OLM

Scaffolding is support or feedback that is given in a timely manner to help a learner achieve a goal that they may not have without that support [8, 19]. It is important to encourage or scaffold SRL processes as students may not always be meta-cognitively or motivationally active during the learning process [2]. Scaffolding of SRL can be part of the feedback provided by an intelligent tutoring system (ITS); ITS that support meta-cognition can increase meta-cognition and learning outcomes [18]. Research indicates that real-time monitoring and adaptive or personalised scaffolding of help seeking behaviour within an ITS can improve student’s help seeking behaviour in the system [33, 35].

One of the tools used in an ITS to support SRL is an OLM. An OLM frequently takes the form of a series of skill meters [4, 25, 28]. Previous studies suggest that an OLM can help students better allocate efforts [4] and improve problem selection [29]. OLM used as a basis for reflective self-assessment activities can increase learning outcomes [25, 30].

Teachers are important in support of reflective and meta-cognitive processes [34]. SRL scaffolding from a teacher can take the form of hints, feedback, and motivation [2]. When teachers scaffold SRL with a personalised or adaptive approach it can lead to a learner adopting better SRL skills as compared with conditions where fixed or no scaffolding is offered [3].

3 Method

To investigate how personalised scaffolding via OLM with a robotic tutor impacts learning gain and SRL processes, we conducted an experiment with four different levels of robot personalisation. We hypothesise that more personalised and adapted scaffolding of SRL processes via OLM will lead to higher learning gain and improvement in SRL processes.

3.1 Participants

There were 80 (34 female, 46 male) participants of mixed ability levels, all of the students within the year group were able to take part without exclusion or preference for higher ability students. The learners were aged between 10 and 12 and attended the same primary school in the U.K.

3.2 Scenario

3.2.1 Experimental Setup

The robotic tutor was an Aldebaran Robotics NAO torso and was fully autonomous during the activity. The activity runs on a 27 in. touchscreen laid flat on a desk. The learners were standing up to enable them to comfortably reach all areas of the touchscreen. The robot was positioned on a stand opposite the touchscreen in order for it to be at a similar height to the learner. The setup is shown in Fig. 1.

3.2.2 The Task

The robotic tutor supported individual learners in a geography task, namely map reading. The task enables the learner to exhibit SRL skills and processes, i.e. self-monitoring, goal setting, and help seeking. The activity was designed to test compass reading, map symbol knowledge, and distance measuring competencies. The learner had a choice of activities of varying difficulty that allowed them to practice the competencies; the menu for this is visible in the lower left of Fig. 2. The learner was provided with three tools to assist them with the activity. They had the option to open a map key, use a distance tool, display a compass on screen, and to view previous clues in a scrap book; the buttons to enable these tools are in the lower right of Fig. 2.

3.3 Learner Model and OLM

We build a learner model as the basis for the OLM skill meters and the robotic tutor’s SRL scaffolding behaviour.

The model of the learner’s map reading competencies is created using constraint based modelling. This is an approach whereby competency values are calculated by checking the learner’s actions against a set of relevant constraints [29]. Distance and direction are evaluated based on the learner identifying a point on a map that is at a particular distance and/or direction from a starting point. Symbol knowledge is tested by selecting a particular symbol from a choice on a map. It is possible for the learner to provide a partially correct answer by meeting the distance constraint but breaking the direction and symbol constraint; this is reflected in the model with distance competency increasing and the direction and symbol competency decreasing. To ensure that the competency values are current, we use a weighted average so that recent evidence is given a higher weighting than older evidence in determining the overall level of the competency. Additionally the task gives basic feedback when an answer is given; the area of the task that displays the objectives flashes green if the answer given is correct or red if the answer given is incorrect.

Fig. 2
figure 2

Learning task, OLM (upper-left), activity menu (mid-left), instructions (lower-left), tools (lower-right)

Full size image

An expanded view of the OLM is shown in Fig. 3. The OLM allows the student to see a visualisation of the learner model that the student can understand their developing skills and identify areas where they have strong or weak knowledge. The OLM shows skill meters for each competency and is visible at all times in the top left of the screen. Changes to the skill meters are made visible with animation and there are indicators to show the previous values [25]. The learner can inspect a history of the most recent 10 pieces of evidence for each individual competency by clicking on the corresponding skill meter. For example, if the learner expands the skill meter for distance then they will see evidence broken into north, east, south, west, e.g. they may see that they have met the north and south constraints correctly but not the west and the east constraints. This enables the learner to see exactly in which aspect of the competency their strengths and weaknesses lie. The OLM should enable the student to plan their learning by helping them identify knowledge gaps, based on this they can then fill their knowledge/skill gaps by selecting an appropriate activity or tool.

Fig. 3
figure 3

Expanded OLM, overall compass competency (high), level of the compass competency over the session (0–100%), 5 correct answer attempts for ‘north’ (A value of 1 with green), 3 incorrect answer attempts for ‘west’ (A value of 0 and red), overall distance competency (low), overall symbol competency (medium). (Color figure online)

Full size image

3.4 SRL Scaffolding

The aim of scaffolding SRL skills is to enable a student to develop their skills by reflecting on their current abilities, to identify strengths and weaknesses so that they can effectively plan their learning through selecting appropriate strategies, goals, activities, and using the tools and resources available. The most basic level of scaffolding is to provide access to the OLM. To provide more support, static scaffolding can be provided whereby the learner is prompted to use SRL skills at appropriate points in the activity [3, 18]. The highest level of scaffolding would be adaptive scaffolding where support is provided based upon the learner’s state [3, 18].

To support adaptive SRL scaffolding we have created an idealised SRL model for our learning context. Such an approach has been used in another meta-cognitive tutoring system that focuses only on when the student should ask for help [1]. We base the SRL model and SRL scaffolding procedures on a previous study [13]. We observed that teachers can scaffold SRL skills by drawing attention to the learner’s developing competencies using the OLM, then encouraging reflection on why the competencies are changing and using this as a basis to suggest appropriate tools, goals, and strategies for the learner [13]. Based on our previous study appropriate SRL behaviours in the learning task include:

  • Learners should aim to ‘master’ an activity, this means that they have covered all of the content and are confident in correctly answering the content.

  • Learners should move on to a different activity when they ‘mastered’ an activity.

  • Learners should use an appropriate tool to the problem at hand or use the OLM if not confident or incorrectly answering a questions in an activity.

  • Learners should stop relying on a tool when they have shown evidence of being proficient at that type of question, if the learner is using the compass tool when estimated to be proficient at direction questions then this is deemed inappropriate tool use.

We record the learners’ behaviours in the activity and if a learner is not following the appropriate SRL behaviours outlined above then this is used as a basis for the robotic tutor’s behaviours. The robot uses the OLM to prompt the learner to reflect on their developing skills and to use appropriate task strategies and to work at an activity of an appropriate difficulty level. We detail the scaffolding procedures in Table 1. We also base the robots gestures and speech on recordings from the previous study [13].

Table 1 Robot behaviour and triggers in each condition
Full size table

3.5 Procedure

The study was conducted in a meeting room in the primary school. Each student was brought in to the room, given a overview of the study, and asked to complete a pre-activity domain test. The autonomous robotic tutor introduced the learning task and then explained how the task and tools work. Each student then carried out the activity which was limited to 11 min. Each student was then asked to complete a post-activity domain test and a questionnaire with questions about their perception of the robot and the learning scenario. The students were randomly split across conditions, described in the next section, while keeping a balance between age, sex, class, and ability.

3.6 Hypothesis and Conditions

We have devised four conditions to explore our overall hypothesis. In all cases the robotic tutor is present and gives an introduction to the task and the tools. The robot is fully autonomous. The different robot behaviours and the events that trigger them are summarised by condition in Table 1. There are a number of events that can trigger the robot to execute a behaviour, these are: Answer attempt, when the learner answers a step in the activity; Timeout, when there has been no robot or learner activity in the preceding 15 s; and Tool selection, when the learner selects a tool to use. When one of these events occurs the system evaluates if the robot should execute a behaviour according the condition. To avoid repetition or the robot talking too much we have alternative phrases for the robot behaviours and an utterance is not executed if that utterance has been delivered by the robot recently. For example the “Let’s keep going we have not covered everything”? utterance may be triggered by a timeout, if the learner has not carried out an action for over 15 s, or by an answer attempt, but only if the learner has not mastered the activity, meaning they have not shown evidence of correctly answering each aspect of an activity.

SRL_SCAFFOLD—In this condition the autonomous robotic tutor personalises and adapts its SRL scaffolding based on the learner’s skill levels, task performance, and rules for appropriate SRL behaviour for the current state of the learner encoded in our pedagogical model described in Sect. 3.4. This is considered an adaptive or dynamic SRL scaffold as it provides feedback on meta-cognitive errors such as using an inappropriate tool or continuing with an activity that is too easy or too challenging [18].

SRL_PROMPT—In this condition the autonomous robotic tutor offers static reflective SRL prompts that are triggered by certain actions of the learner. The SRL scaffolding is considered static as it is not dependant on the state of the student’s meta-cognition as it is in the above condition [18]. The feedback is still personalised as feedback is contingent on the learner’s actions.

OLM_ONLY—This control condition contains limited personalised feedback in the form of an OLM. After introducing the activity and tools the robot simply performs idle behaviours. This condition will allow us to investigate the impact of the adaptive and static SRL scaffolding over the OLM feedback.

CONTROL—In this control condition the learner has no OLM and is only informed if the answer that they have provided is correct or incorrect by on-screen feedback. After introducing the activity and tools the robot simply performs idle behaviours.

In all conditions the robot introduces the learning activity, tools, and performs idle motions throughout the session. The robot only uses pointing in the SRL_SCAFFOLD and SRL_PROMPT conditions and only towards the OLM and not at any other time. Therefore we do not believe that this prompts greater engagement or focus in the activity.

Our specific hypotheses are as follows:

Hypothesis 1

(H1) Adaptive SRL scaffolding will lead to a greater increase in learning gain and more appropriate SRL learning behaviour than static SRL scaffolding.

Hypothesis 2

(H2) Adaptive SRL scaffolding will lead to a greater increase in learning gain and more appropriate SRL learning behaviour than an OLM alone.

Hypothesis 3

(H3) Adaptive SRL scaffolding will lead to a greater increase in learning gain and more appropriate SRL learning behaviour than no scaffolding.

Hypothesis 4

(H4) Static SRL scaffolding will lead to a greater increase in learning gain and more appropriate SRL learning behaviour than access to an OLM alone.

Hypothesis 5

(H5) Static SRL scaffolding will lead to a greater increase in learning gain and more appropriate SRL learning behaviour than no scaffolding.

Hypothesis 6

(H6) Access to an OLM alone will lead to a greater increase in learning gain and more appropriate SRL learning behaviour than no scaffolding.

We would expect to see these effects in less able students to a greater degree than in more able students that might already have strong domain knowledge and good SRL skills, which would be similar to findings in OLM research [28].

We have asked a number of questions in the post-activity questionnaire based on the Intrinsic Motivation Inventory (IMI) [27, 36]. We believe that the differences in the robot’s behaviour will impact the perception of the robot and the task.

Hypothesis 7

(H7) The perception of the robotic tutor will differ between activities. The robot’s behaviour will affect the learner’s perception of the robot, the role of the robot, and the learner’s attitude towards the robot.

Hypothesis 8

(H8) The perception of the activity will differ between conditions. The robot’s behaviour will affect the learner’s perception of their competence in the activity, the importance/value/interest in the activity, and the perception of the OLM skill meters.

4 Results

The results presented here are derived from the analysis of the log data and domain pre-activity and post-activity domain tests and questionnaires. We have broken down the analysis to investigate differences between more able and less able students based on if the learner was above or below the mean of the pre-activity domain test score as has been done in other OLM research [28, 30]. The breakdown is presented in Table 2.

Table 2 Participant details
Full size table

Significant differences (lower than .05) between conditions are highlighted with a connecting black line (Fig. 4) in the figures below.

4.1 Learning Gain

Learning gains were calculated using Normalised Learning Gain [11], based on the difference between the pre-activity domain test and the post-activity domain test, the calculation is presented in Fig. 5. In both the pre and post test the learners were asked 14 questions that cover compass reading, distance measurement, and map symbols. A one-way ANOVA was used to determine whether there was any statistically significant difference between the Normalised Learning Gain of the groups.

Fig. 4
figure 4

Normalised Learning gain: all learners (left), more able (centre) and less able students (right)

Full size image

The results in Fig. 4 show that there was a statistically significant difference between groups as determined by one-way ANOVA (F(3,70) = 3.916, p = .012) when considering all students. A Tukey post hoc test revealed that the learning gain in the SRL_SCAFFOLD condition (M = .58, SD = .3) is significantly higher than OLM_ONLY condition (M = .20, SD = .3, p = .009) when considering all students. There were no other statistically significant difference between the groups. We see a general trend when considering all students, more able students, and less able students that learning gain is highest for SRL_SCAFFOLD followed by SRL_PROMPT then CONTROL and finally OLM_ONLY.

4.2 SRL Indicators in Task Performance Data

The indicators we have extracted from the logs aim to measure SRL behaviours. One-way ANOVAs were used to determine whether there was any statistically significant difference between the indicators of the groups.

Learner model final value This is the average of all the skill levels from the learner model at the end of the activity. It is an indicator of how well the student is at the content in the activity that they have attempted to answer. If the learners are using the OLM to reflect and SRL processes are used, then the students should be looking to ensure that their actions lead to an increase in the OLM skill meters. To do this the students should be working on getting answers correct by using the tools rather than guessing and getting lower learner model values. This value is based on the evidence provided, it shows performance on the questions attempted by the learner. It does not consider coverage of the content of the activity. It is possible to have a high learner model value by answering simple questions so it is not an indicator of total level of knowledge or ability. The results in Fig. 6a show that in the CONTROL condition the learner model value is generally lower than all other conditions. However, there are no statistically different results.

Number of questions answered This gives an indication of how long a learner spends on each question; A learner could complete fewer questions because that learner is struggling, distracted, reflecting more, or making use of tools. So we must take this indicator into account with the indicators that follow in this section.

Fig. 5
figure 5

Normalised Learning Gain

Full size image
Fig. 6
figure 6

SRL indicators

Full size image

The results in Fig. 6b show a statistically significant difference between groups as determined by one-way ANOVA when considering all students (F(3,76) = 15.72, p = .000) and more able students (F(3,35) = 12.888, p = .000). A Tukey post hoc test revealed the following statistically significant differences. When considering all students the learning gain SRL_SCAFFOLD (M = 48.16, SD = 12.1) learners complete significantly fewer questions than OLM_ONLY (M = 81.00, SD = 31.9, p = .000) and CONTROL (M = 90.61, SD = 27.8, p = .000) conditions, the SRL_PROMPT (M = 56.70, SD = 20.0) learners complete significantly fewer questions than OLM_ONLY (M = 81.00, SD = 31.9, p = .015) and CONTROL (M = 90.61, SD = 27.8, p = .000) learners. When considering more able students SRL_SCAFFOLD (M = 48.19, SD = 11.8) learners complete significantly fewer questions than OLM_ONLY (M = 105.83, SD = 37.1, p = .000) and CONTROL (M = 88.60, SD = 26.8, p = .001), SRL_PROMPT (M = 63.15, SD = 19.7) learners complete significantly fewer questions than OLM_ONLY (M = 105.83, SD = 37.1, p = .003).

Percentage of questions answered correctly This gives an indication of how deliberately the students are answering questions. If this is high then it shows that the student is getting most question attempts correct, however this may not always be desirable because it can indicate that the student is focusing on questions that they may already know the answer to and are not pushing themselves. The results in Fig. 6c show there are no statistically different results.

Attempts until a successful answer This measures on average how many attempts it takes for a learner to answer successfully. If this is high it is an indication that a student is not thinking carefully enough about how they are answering questions or indicates that the learner is not aware that they need to work on a skill. The results in Fig. 6d do not show statistically significantly different values between conditions, however in the CONTROL condition learners take more attempts to get a correct answer, particularly the less able learners. This may indicate that learners in the control condition are not taking appropriate SRL actions when they are getting questions incorrect.

Tool use This is a count of tool use in the activity. In Fig. 6e we do not see statistically significant differences between conditions, however in the CONTROL condition the tool use is lower than the other conditions. This may indicate that the students do not realise that they have issues or that the tools can help them with address the issues.

When we look at all of the indicators together we can see some general trends between the conditions. In the adaptive scaffolding condition SRL_SCAFFOLD indicates a greater adoption of SRL behaviours. More time is taken over fewer questions, the number of steps to get a correct answer are fewer; however, the percentage of correct answers is lower, which may indicate that the learner is working on more challenging questions. This may be a factor in the higher learning gains for this group.

Fig. 7
figure 7

Questionnaire results

Full size image

In the static scaffolding condition SRL_PROMPT the adoption of SRL behaviours seems similar to SRL_SCAF FOLD, however this does not translate to as high a degree of learning gain.

The OLM_ONLY condition indicates a lesser degree of SRL behaviours, less time is taken over more questions, and we see a slightly higher percentage of questions correct that indicates that the students are spending more time on questions that they find comfortable. So the learners perform well but appear to not push themselves.

The control condition CONTROL appears to have the least degree of SRL behaviours; learners take the least time over the greatest number of questions, they have a lower percentage of questions correct, and take more attempts to get a successful answer, and the tool use is low. The learner model final values are also lower. This indicates that the learners are not aware of where they have issues and do not work to address these issues with the tools available.

4.3 Questionnaire Results

The questions asked in the post-activity questionnaire are based on the Intrinsic Motivation Inventory (IMI) [27, 36]. We ask the questions as we wanted to understand how the differences in robot’s behaviour affected the perception of the robot and the activity. Specifically, we wanted to explore if there were differences in the learner’s enjoyment, engagement with the activity and the robot. We also wanted to see if the learner could perceive the robot’s understanding of the learner. Each question was asked on a 7 point scale ranging from 0, “not at all true”? to 1, “very true”. The mean values of each sub-scale of the IMI and the individual items were analysed by comparing each condition against each other using a Mann-Whitney U test. The significant values (lower than .05) were then further investigated. We also report the reliability of these sub-scale using Cronbach’s alpha.

Fig. 8
figure 8

Questionnaire results

Full size image
Fig. 9
figure 9

Questionnaire results

Full size image

Learner’s perception of the robot This sub-scale consists of questions about the learner’s perception of the robot. The Cronbach’s Alpha for this grouping was .864. We see in Fig. 7a that the value for the perception was significantly higher in the CONTROL condition than the OLM_ONLY condition (U = 66.000; p = .010).

The question that contributes the most to this is the question “the robot was helpful”. We see in Fig. 7b that the robot was rated significantly less helpful in the OLM_ONLY than the CONTROL (U = 72.500; p = .013) and the SRL_PROMPT (U = 85.000; p = .041).

It also appears that the learners did not enjoy the SRL_SCAFFOLD condition as much as the CONTROL condition. We see in Fig. 7c that for the question “I enjoyed interacting with the robot very much”? the SRL_SCAFFOLD is significantly lower than the SRL_PROMPT(U = 110.000; p = .025) and CONTROL(U = 125.000; p = .027).

We see in Fig. 7d that for the question “While I was interacting with the robot I was thinking about how much I enjoyed it”? the SRL_PROMPT condition is lower than the CONTROL condition(U = 130.500; p = .047).

Learner’s perception of the robot’s perception of the learner This sub-scale consists of questions about how the learner felt the robot perceived them. The Cronbach’s Alpha for this grouping was .575, which is a rather low value. We see in Fig. 7e that the OLM_ONLY condition is significantly less than the SRL_SCAFFOLD condition (U = 77.500; p = .028) and CONTROL (U = 53.000; p = .002).

The OLM_ONLY is consistently and significantly lower across the questions than all of the other conditions. These questions were “I feel that the robot understands me”, “The robot was happy for me when I was doing well”, “The robot felt sorry for me when I was having problems”. This indicates that the learners were aware that the robot was not helping them when their issues were highlighted by the OLM.

Role of the robot This sub-scale consists of questions about the learners’ perception of the role of the robot. The learner was asked to mark on a scale from 1 to 7 how much the robot was like a classmate, friend, or teacher. We see in Fig. 8a that the only significant difference between conditions was for the question “I thought the robot was like a friend”?; the CONTROL condition is given a significantly higher value than the OLM_ONLY condition (U = 63.000, p = .005). When we look at the role that was given the highest rating we see that the role of teacher Fig. 8c was the highest rated role in all conditions. The robot is perceived as a teacher more frequently in the SRL_PROMPT condition. The robot is perceived as a friend more frequently in the CONTROL condition. The robot is perceived as a classmate Fig. 8b more frequently in the SRL_SCAFFOLD condition.

Pressure and tension This sub- scale measures the pressure and tension the learner perceives in the activity. The Cronbach’s Alpha for the questions that compose the pressure/tension sub-scale from the IMI activity evaluation questionnaire was .629. We see in Fig. 9a that the pressure sub-scale is significantly higher in the SRL_PROMPT condition than the SRL_SCAFFOLD condition for less able students (U = 9; p = .033).

We see in Fig. 9b that for the question “I felt very tense while doing this activity”? for less able students the SRL_SCAFFOLD is significantly lower than the SRL_PROMPT condition (U = 7; p = .016) and the OLM_ONLY condition (U = 15.5; p = .044).

This indicates that In the SRL_SCAFFOLD and OLM_ONLY conditions the learner is made more aware of issues but has less support from the robotic tutor. The less able students are less able to identify how to solve the problems or may not be as used to engaging in SRL processes that are now required which could explain the increased pressure that they feel. The SRL_PROMPT condition gives the learner specific personalised strategy that can reduce the pressure that the learner feels.

Following the robot This sub-scale consists of questions about whether the learner followed advice from the robot. The Cronbach’s Alpha for this grouping was .861. We see in Fig. 9c that the value for OLM_ONLY is significantly lower than the SRL_SCAFFOLD (U = 78.000; p = .030) and CONTROL (U = 59.500; p = .005) conditions. The OLM_ONLY is consistently and significantly lower across the questions than all of the other conditions. These questions were “The robot helped me decide what to do next”, “The robot helped me choose the right tools”.

Interest in the activity This sub-scale measures how much interest or enjoyment the learner perceives in the activity. The Cronbach’s Alpha for the questions that compose the interest/enjoyment sub-scale from the IMI activity evaluation questionnaire was .793. We see in Fig. 9d that the interest and enjoyment is fairly similar between all conditions. There is higher interest/enjoyment with the CONTROL condition overall (U = 105; p = .047). This might be linked with how the learners in the control condition perceived the role of the robot.

Perceived competence This sub-scale measures the how competent the learner thinks they are at the activity. The Cronbach’s Alpha for the questions that compose the perceived competence sub-scale from the IMI activity evaluation questionnaire was .700. There are no significant differences between the conditions in the sub-scale.

We see in Fig. 9e that one interesting results is that for the question “I think I did pretty well at this activity compared with other students”? for less able students the SRL_SCAFFOLD is significantly lower the SRL_PROMPT (U = 9.000, p = .031) . This indicates that the weaker students are noticing an improvement in their skills based on the feedback from the robot.

Importance and value of activity There are two sub-scales from the IMI activity evaluation questionnaire that measure the how important and valuable the task was to the learner. The Cronbach’s Alpha for the questions that compose the importance sub-scale was .700 and the value sub-scale was .817. There are no significant differences between the conditions in these sub-scales. This indicates that there is no difference in levels of motivation to do the task.

Skill meters This sub-scale measures the learners’ perception of how the skill meters helped them. The Cronbach’s Alpha for this grouping was .859. There are no significant differences between the conditions.

5 Discussion

There is some evidence to support our hypothesis that a more personalised and adapted scaffolding of SRL processes via OLM lead to higher learning gain and improvement in SRL processes.

H1 is not supported, as there are not statistically significant higher learning gains between students in the personalised conditions for adaptive scaffolding SRL_SCAFFOLD and static scaffolding SRL_PROMPT. In terms of SRL indicators there does not appear be a difference.

H2 is supported, as we see that personalised adaptive SRL scaffolding SRL_SCAFFOLD as compared with a personalised OLM_ONLY alone leads to significantly higher learning gains and more time spent on fewer questions. They key difference appears to be that the OLM alone does not prompt the learner to push on to more difficult questions as can be seen with the higher percentage of questions correct.

H3 is supported, as personalised adaptive SRL scaffolding SRL_SCAFFOLD as compared with a the CONTROL condition leads to significantly higher learning gains and more time spent on fewer questions. The learners in the control conditions show the least indication of SRL behaviours; they do not appear to be aware of or able to act on their weaknesses in the activity.

H4 is supported, as we see that personalised static SRL scaffolding SRL_PROMPT as compared with an OLM alone OLM_ONLY leads to significantly higher learning gains when considering all students. As with H2 the key difference appears to be that the OLM alone does not prompt the learner to push on to more difficult questions as can be seen with the higher percentage of questions correct.

H5 is not supported, as we do not see a statistically significant difference in learning gain in the personalised static SRL scaffolding SRL_PROMPT as compared with the CONTROL condition.

H6 is not supported, as learning gain does not differ significantly between the OLM only OLM_ONLY and the CONTROL condition. In fact there appears that there might even be more learning gain in the CONTROL condition.

With regard to the questions to ascertain the learner’s perception of the learning activity and the role of the robot.

H7 is supported as the different conditions appear to have affected the way that learners perceive the robot and if they would listen to the robot in the future. In the SRL_SCAFFOLD condition the robot is perceived mainly as a teacher but is more frequently referred to as a classmate than the other conditions. In the OLM_ONLY condition the robot is perceived the least favourably. In the CONTROL condition the robot was perceived surprisingly positively, the robot did exactly the same behaviour as the OLM_ONLY condition but as the students were not as aware of their difficulties the robot is perceived as a friend.

H8 is supported as the different conditions appear to have affected the way that learners perceive the task. The students in the SRL_PROMPT condition felt most like they have performed better in the task than other students. This could be because they were aware of overcoming problems themselves without much assistance. These students also felt some stress due to the lack of assistance when they were aware of their weaknesses. The students in the OLM_ONLY condition were aware that the robot was not helping them and consequently had a low perception of the robot. These students also felt tense. We see no difference in the importance and value of activity but this may be due to the novelty of the task.

Below we summarise the main results, key findings, and limitations of this study. A higher level of personalisation and adaptive scaffolding of SRL seems to lead to greater adoption of SRL behaviours and an increase in learning gain. Less able students in the SRL conditions appear to have been helped the most. All students should be familiar with the material as it is part of the National Curriculum we see that on average the pre-test domain scores were 6.4 out of 10 (SD.1.83) and post-test domain scores were 7.6 out of 10 (SD. 1.79).

Without any SRL support in the control condition, learners do not appear to engage many SRL processes. The presence of a robot may motivate the students to engage in the learning scenario, in fact the robot in the control condition is the most well perceived in terms of enjoyment, motivation, and being thought of as a friend. However this does not necessarily foster SRL processes, appropriate scaffolding must still be made available.

An OLM on its own does not lead to students engaging in SRL processes. Making the learner aware of their issues via an OLM but not providing specific help can increase the pressure experience by the learner as we can see with the higher levels reported in the SRL_PROMPT and OLM_ONLY conditions. Some pressure and tension is good for learning as it will prompt the learner to take some action, however we would need to be careful as too much pressure could cause the learners to become disengaged. If the robot is present with an OLM it should offer some support to assist the learner. Otherwise the learner will perceive the robot poorly and may not follow its advice in the future.

This study shows the importance of how the robot’s behaviours can be perceived within the context of the activity and the importance of finding a balance between appropriate social support and SRL support to successfully scaffold SRL skills. Social support is essential for reducing pressure and tension and supporting engagement in the activity. It appears from the results that in the case of the control condition that the robot is perceived as a friend due to its behaviour being non judgemental, however, this same behaviour in the OLM condition is seen as unhelpful. This is a new finding in OLM research as no other research uses a pedagogical agent and OLM a similar way.

The different robotic behaviours in the static and adaptive SRL conditions may make the robot seem more like a teacher or classmate but appear to offer enough social and SRL support to reduce pressure and tension but still allow the learner to push themselves and learn. This shows how important social interaction such as encouragement or supportive interaction is to the development of SRL. We can also argue that the personality of the robot must match the role that the robot plays to the learner. If the robot has an overly social personality in a tutoring role then it may in fact harm the performance of the user [16]. This is an example how a social robot tutor could be argued to have as a basis the socio-constructivist approach to learning, where the cognitive development of the individual is supported by social interaction [42]. It is believed that socially assistive robots can support the best practises of socio-constructivist learning theories [7], which we believe could lead to the adoption of SRL skills. For example, adoption of good SRL processes can be influenced by members of a social network in a learning planning application [23].

It may be that SRL scaffolding of a less social nature may have been effective coming from on screen prompts or a virtual agent as with ITS research [18]. We decided not to compare virtual to physical feedback as it has been shown before that a physical embodiment is preferred to a virtual embodiment [22], and learners prefer explanations of a simple OLM via a robotic tutor rather than text based explanations displayed on-screen [14].

It is possible that other forms of scaffolding SRL that are not based upon an OLM would be effective. For example, providing SRL prompts when an OLM was not present. We chose an OLM as it is one of the most effective ways to show the learner their developing skill levels and assist them with reflection. Alternatives might have been to allow other mechanisms for reflection in the activity such as skill diaries [24] or other note taking tool [37].

We would also like to further investigate the SRL indicators described in this study. The indicators that we have investigated here are high level and not granular over the interaction. Unfortunately due to previous familiarity with material, the amount of material we have available in our activity, and the time available for this study we were unable to measure a baseline of a learners SRL indicators or behaviours. In the future a more long-term study may allow us to better understand how SRL indicators may change over time or even perform some within-subject studies.

6 Conclusion

This paper explores how personalised tutoring by a robot achieved using an OLM promotes SRL processes and how this can impact learning in primary school children. We see significant differences between the learning gain in the the adaptive SRL scaffolding and the OLM only and the control conditions. We believe that the differences are due to the support of SRL behaviour in the conditions chosen in the study. The main benefit of the support given by the robot and OLM in the adaptive SRL scaffold condition is to prompt the learner to reflect and to motivate the students to choose appropriate task strategies. We see that for a learner to engage SRL practises they must be aware that they have issues with the task and also motivate them to engage meta-cognitive processes to fix those issues. As we see in the OLM only condition it is not enough to make the learners aware of issues. These differences can be seen in the high level indicators of SRL from the task data between the conditions that appears to support this conclusion.

We see here the possibility of a robotic tutor to motivate the students to engage in SRL processes. We have seen in previous work that the robotic tutor may increase trust, enjoyment, and understanding in explanations of an OLM as compared to on-screen feedback alone [14], which could motivate the students to make more use of the feedback. In this work we see that the robot in the adaptive condition appears to be able to motivate the students to use SRL processes with it well placed suggestions. The robot in the static scaffolding condition appears to raise awareness of issues while adding stress to the learner which does not necessarily help, however this may help the learners in the long run. In the OLM only condition the students are aware that the robot does not help. We see that the robot in the control group is generally engaging and well liked by the students but it does does little to motivate meta-cognitive processes. This indicates that students do look to the robot for and would likely accept assistance.

Further, this study shows the importance of adapting to a learner when scaffolding SRL processes. This reflects the findings from human-human interactions where adaptive scaffolding has led to improved learner understanding compared with fixed or no scaffolding [3]. The need to adapt to student’s SRL skills is highlighted with more able students, at best static and less personalised scaffolding does not provide any greater degree of support to these learners, at worst it could be dangerous to continue to scaffold basic SRL processes as this support could start to become distracting, less effective, and frustrating for the learner [10]. Removing support when it is no longer needed is one of the principles of scaffolding [19]. Fading or removing scaffolding based on an OLM to assist in problem selection can increase the ability for students to select more appropriate problems [28]. Consequently we need to be able to model the students’ SRL skills to be able to decide when to reduce the SRL scaffolding.

There are open questions around adapting to the learners meta-cognitive state, finding appropriate social behaviours for the robotic tutor, and investing the scaffolding of SRL in longer term studies. We aim to investigate how we can better adapt to SRL skills, including identifying the factors that can indicate the level of SRL skills possessed by the learner. Based on previous research the indicators of SRL behaviour are pre test scores [38], ability at problem selection [28], and the frequency of tool or resource use [38]. In this study we have seen how important the robotic tutors social behaviours are to the interaction, which is in line with a review of long-term interactions with robots [20], which recommends that a robot should be able to display an awareness of and respond to the user’s affective state and also adapt to the individual’s preferences in order to build a good social interaction which is essential for long-term support. This type of long term interaction is essential to investigate if this type of SRL scaffolding can lead to long term changes in SRL behaviour, as such changes can be difficult to achieve with an ITS [18]. Examples of more social supportive behaviour would be calling the learner by name, referring back to previous interactions, and commenting on the development of the learner, and other supportive and motivating statements.

In summary, we have found that adaptive SRL scaffolding delivered by a social robotic tutor can lead to greater SRL behaviours and learning gains. However, care must be taken with the delivery of SRL scaffolding as highlighting issues but not providing sufficient level of support can make the learners feel higher levels of stress and pressure, which may cause learners to become disengaged.