Inferring Student Attention with ASQ

  • Vasileios Triglianos
  • Cesare Pautasso
  • Alessandro Bozzon
  • Claudia Hauff
Conference paper

DOI: 10.1007/978-3-319-45153-4_23

Part of the Lecture Notes in Computer Science book series (LNCS, volume 9891)
Cite this paper as:
Triglianos V., Pautasso C., Bozzon A., Hauff C. (2016) Inferring Student Attention with ASQ. In: Verbert K., Sharples M., Klobučar T. (eds) Adaptive and Adaptable Learning. EC-TEL 2016. Lecture Notes in Computer Science, vol 9891. Springer, Cham

Abstract

ASQ is a Web application for broadcasting and tracking interactive presentations, which can be used to support active learning pedagogies during lectures, labs and exercise sessions. Students connect their smartphones, tablets or laptops to receive the current slide as it is being explained by the teacher. Slides can include interactive teaching elements (usually questions of different forms). In contrast to other existing platforms, ASQ does not only collect, aggregate and visualize the answers in real-time, it also supports the data analytics in the classroom paradigm by providing the teacher with a real-time analysis of student behaviour during the entire session. One vital aspect of student behaviour is (in)attention and in this paper we discuss how we infer — in real-time — student attention based on log traces ASQ collects.

1 Introduction

In the traditional post-secondary classroom-based learning, forty-five or ninety minute units of teaching are the norm. Students’ attention during such teaching sessions varies significantly, as shown in a wide range of empirical studies that have either probed students directly for self-reports of attention (or daydreaming and mind wandering) levels [2, 5, 8, 10] or aimed to infer [in]attention based on (i) students’ behaviour (e.g. their patterns of note-taking [9] or physical signs of inattention such as gazing [4]), (ii) physiological measures such as skin temperature [1], or, (iii) students’ levels of knowledge retention [8, 14].

Many of these techniques can only be employed at reasonable cost for a small subset of classes and/or a small subset of students due to their obtrusive nature (examples include physiological markers or minute-by-minute self-reports), issues of scale (e.g., the presence of external observers and the analyses of taken notes), and, the additional cognitive & timely burden placed on students (e.g., through retention tests). Moreover, with few exceptions, e.g. [12], these techniques do not enable lecturers to adapt their teaching on-the-fly, as they are not able to continuously determine students’ attention in real-time; instead students are probed at specific intervals during the lecture or post-lecture data collection and data analyses steps are required.

In this work, we investigate to what extent modern Web technologies can facilitate and enable the continuous, scalable and unobtrusive inference of student attention in real-time. We target the traditional classroom setting – so as to enable lecturers to react in a timely manner to the attention needs of their students – and we focus on Web-mediated teaching and formative assessment activities. We seek answer to the following Research Questions:

RQ1. To what extent can students’ attention be inferred from their interactions with a Web-based platform?

RQ2. Which type of interactions are most correlated with (in)attention?

As common in previous works, we infer attention from students’ retention levels. To this end, we have extended ASQ [16], a Web platform aimed at providing active classroom-based learning pedagogics such as enquiry based learning, problem based learning and collaborative learning. ASQ provides extensive logging capabilities, thus enabling the tracking and recording of real-time students’ interactions during lectures. We deployed ASQ in the context of three ninety minute university-level lectures given by two different instructors, with varying interactivity levels and up to 187 students. Our results show that ASQ can provide fine-grained insights on students’ attention states that relate to previous findings on the subject, thus demonstrating ASQ ’s ability to obtain an accurate view of students’ attention in a classroom setting.

2 Related Work

Measuring and influencing peoples’ state of attention in their workplaces, daily lives and educational settings has been investigated for a number of decades in psychology and pedagogy; in more recent years technological advances have also led to contributions by the human computer interaction and the learning analytics communities [6, 7].

Our research focus is in the measuring of students’ attention in the post-secondary classroom, and thus in this section we narrow our overview to works that have investigated attention in the educational context only. Two important meta-studies [15, 17], published in 2007 and 2013 respectively, not only summarize the current state of knowledge about student attentiveness, but also critically highlight the often contradictory findings — in [17] specifically, the assertion of the 10–15 min attention span of students is tackled in great detail. The contradictions are generally attributed to the nature of the individual experiments, which are typically conducted on a small number of students taking a class of less than one hour, which may have been specifically designed for the experiment. Factors which can explain the observed differences include the inherent variability of students’ academic interests, instructor styles and means of measuring attention, which are usually not controlled for across experiments [15]. Of the many findings, we list here those which have been observed in several experiments1. F1: Students’ attention drops over the class period [5]; as a consequence, in retention tests students tend to perform better on material presented early on in the class [8]. F2: attention breaks occur regularly and increase in frequency as the class progresses [4]. F3: As the class progresses, students tend to take less notes [9]. F4: the percentage of students attentive to the class varies significantly (depending on class topic, the instructor and the pedagogical tool employed). Between 40 % and 70 % of students are attentive at any moment during frontal teaching. Attention rises when interactive elements are introduced (discussions and problem solving) [2]. F5: immediately after interactive teaching elements, the level of distraction is lower than before the start of the interaction [2, 3].

One common denominator of the aforementioned studies is their lack of technologies to determine students’ attention directly or indirectly. Existing technology-based solutions, while enabling real-time insights, are also limited, due to the invasive technologies employed. In [12, 13] EEG signals are recorded to infer students’ attention — while accurate, those studies are restricted to either small classroom or lab settings. Sun et al. [11] rely, among others, on facial expressions to detect attention, which, while technologically feasible raises privacy concerns. Bixler et al. [1] find eye gaze and skin conductance and temperature recordings to be indicative of attention.

In contrast, in our work we explore the use of a non-invasive and scalable technological solution.

3 ASQ: From Low-Level Events to Attention States

ASQ is a Web-based tool for delivering interactive lectures. It builds upon the modern Web technology stack and allows teachers to broadcast HTML slides to students’ devices and on-the-fly to receive and process their reactions and responses. The slides may contain exercises with interactive questions such as “choose one out of five”, “highlight the text”, “classify elements”, “program a JavaScript function”, or “write a SQL query” (Fig. 2) — these question types can be extended for different needs and new question types can easily be added to ASQ due to its modular nature. The answers students submit are available to the instructor for review and discussion in real-time. Moreover, most question types support the automatic aggregation and clustering of the answers, thus reducing the cognitive load of the instructor which in turn enables a quicker (and more accurate) feedback cycle. To reiterate, the main design driver of ASQ was to enable teachers to gather feedback live in the classroom and immediately assess the level of understanding of the entire classroom, by turning student devices from potential distractions into a novel communication channel — Fig. 1 shows an example session of ASQ in the classroom.
Fig. 1.

ASQ in the classroom: most students’ laptops are connected and focused on the slide material being explained.

Fig. 2.

SQLite question from Lecture 1, Advanced SQL. It comprises a text editor (left) to write and execute SQL queries on an in-browser database instance, and a results pane (right) to visualize the query results. (Best viewed in the electronic version.)

Fig. 3.

A text input question from Lecture 3, Web Security.

Low-Level Event Capturing. In order to capture the interactions with the taught material, and to understand how they contribute to the learning process and student attention, ASQ tracks various events (e.g. a user connects to the ASQ presentation, submits an answer or is idle for a number of seconds) generated by each learner’s browser during a live presentation session. Note, that we do not require students to login to ASQ, as long as a student’s browser is connected to the ASQ presentation relevant events will be captured; closing the browser tab that contains the ASQ presentation will disconnect the student. Specifically, in this work we consider the browser events listed in Table 1; events are generated not only when students interact with an ASQ question type, but also when they interact more generally with the browser window containing the ASQ tab.
Table 1.

Overview of web browser events monitored by the ASQ application

Event name

Description

tabhidden

The browser tab that displays the ASQ web app becomes hidden.

tabvisible

The browser tab that displays the ASQ web app becomes visible.

windowfocus

The browser window that displays the ASQ web app receives focus.

windowblur

The browser window that displays the ASQ web app loses focus (blurs in HTML terminology).

exercisefocus

An ASQ exercise HTML element receives focus.

exerciseblur

An ASQ exercise HTML element blurs.

input

There is student input in the browser window that displays ASQ.

questioninput

Some ASQ question types emit this event when there is student input.

exercisesubmit

A student submits the solution to an ASQ exercise.

answersubmit

A student submits an answer for an ASQ question (an exercise can have multiple questions).

idle

Emitted by the browser window that displays the ASQ web app when none of the above events has occurred for 10 seconds.

connected

A student connects to the ASQ server.

disconnected

A student disconnects from the ASQ server.

Recall that our overarching goal is to infer student attention. To this end, based on the introduced low-level events, we define higher-level activity indicators, which denote the activity (or lack thereof) currently performed by a student in a lecture. Subsequently, we use these indicators to infer a basic model of student attention states.

Activity Indicators. Each low-level browser event occurs at a specific point in time; we map sequences of browser events generated by a student to one of six binary activity indicators, which we consider to be natural components of a student’s attention state. These indicators are non-exclusive (i.e. several indicators can be true at the same time) and listed in Table 2: exercise, connected, focus, idle, input and submitted.
Table 2.

Overview of activity indicators based on browser events

Name

Description

exercise

True when the current slide has an exercise.

connected

True when the student browser is connected.

focus

True when the browser has focus on the tab or exercise related to the lecture.

idle

True from the time of an idle event until one of tabhidden, tabvisible, windowfocus, windowblur, focusin, focusout, exercisefocus, exerciseblur, input, questioninput, exercisesubmit and answersubmit occurs.

input

True when an input or questioninput event occurs. This state is valid only on slides that contain exercises.

submitted

True when the student has submitted at least once this exercise (as indicated by an exercisesubmit event). This state is valid only on slides that contain exercises.

Table 3.

Modeling student attention based on activity indicators. Activity indicators are binary, \(\mathbf \checkmark \) represents True, and - represents False.

exercise

connected

focus

idle

input

submitted

Inferred attention state

-

-

-

-

-

-

Disconnected

\(\checkmark \)

-

-

-

-

-

Disconnected

\(\checkmark \)

-

-

-

-

\(\checkmark \)

Disconnected

-

\(\checkmark \)

-

-

-

-

Distracted

-

\(\checkmark \)

-

\(\checkmark \)

-

-

Distracted

\(\checkmark \)

\(\checkmark \)

-

-

-

-

Searching for a solution

\(\checkmark \)

\(\checkmark \)

-

\(\checkmark \)

-

-

Searching for a solution

-

\(\checkmark \)

\(\checkmark \)

-

-

-

Interacting with non-question slide

-

\(\checkmark \)

\(\checkmark \)

\(\checkmark \)

-

-

Following

\(\checkmark \)

\(\checkmark \)

\(\checkmark \)

-

-

-

Thinking

\(\checkmark \)

\(\checkmark \)

\(\checkmark \)

\(\checkmark \)

-

-

Thinking

\(\checkmark \)

\(\checkmark \)

-

-

-

\(\checkmark \)

Bored

\(\checkmark \)

\(\checkmark \)

-

\(\checkmark \)

-

\(\checkmark \)

Bored

\(\checkmark \)

\(\checkmark \)

\(\checkmark \)

-

-

\(\checkmark \)

Waiting

\(\checkmark \)

\(\checkmark \)

\(\checkmark \)

\(\checkmark \)

-

\(\checkmark \)

Waiting

\(\checkmark \)

\(\checkmark \)

\(\checkmark \)

-

\(\checkmark \)

-

Working on an answer

\(\checkmark \)

\(\checkmark \)

\(\checkmark \)

-

\(\checkmark \)

\(\checkmark \)

Reworking answer

Student Attention States. We take a data-driven approach to the exploration of the activity indicators and in Table 3 list all the 17 combinations of indicators that we observed in our data traces (described in detail in Sect. 4). We manually assign ten different semantic labels to each combination. For instance, a student who has submitted an answer to an exercise and is now idle with ASQ in focus is considered to be Waiting (e.g. for the instructor to provide feedback), while a student who also submitted an answer and is neither idle nor having ASQ in focus is considered to be Bored (and occupying himself with other activities on the device). Thus, at each point in time a student is in exactly one of the ten listed attention states. Figure 4 showcases the progression of two students’ attention states across an entire lecture; while Student 1 starts off the lecture at a high level of attention (indicating by the continuous Following state) and later on toggles between the Following and Distracted states, Student 2 starts off the lecture in a Distracted state and only exhibits short bursts of attention shortly before or after some of the interactive exercises.
Fig. 4.

Two example progressions of inferred attention states during the course of a single 90-min lecture (specifically: Web Security). The dark-grey areas represent slides with interactive exercises (6 in total), while the light-grey vertical bars indicate slide transitions. While Student 1 starts off highly attentive, Student 2 is inattentive from the start of the lecture.

Although we are using psychological terms such as Bored, Distracted, Thinking, and the like, these should be not be interpreted beyond the strict definition of Table 3 as our goal is to give a readable representation of the aggregated activity indicators that can be amenable of further analysis and experimentation. In the remainder of this paper we analyze to what extent our definition of inferred attention states is suitable to reproduce findings from the literature.

4 ASQ Deployment and Data Collection

We deployed ASQ in the 2015/16 edition of Web and Database Technology, a compulsory course for \(1^{st}\) year BSc Computer Science and an elective for \(3^{rd}\) year BSc minor students, at the Delft University of Technology. The course was followed by 310 students in total: 260 \(1^{st}\) year and 50 minor students. Across the eight course weeks, fifteen 90-minute lectures were given. We utilized ASQ in three of those sessions, identified as suitable for experimentation: at regular intervals, the lecture material was interspersed with interactive elements consisting of live programming exercises, multiple choice questions, and visual question types.

At the beginning of each ASQ session, students in the lecture hall were instructed (though not compelled) to open the lecture presentation in the browser. Students connected anonymously; a random identifier was assigned to each connection, enabling us to group all interactions made by the same student within one lecture together (identifying markers across lectures were not retained for privacy reasons). The lecture slides were not only visible in the students’ browser but also on the lecture hall screen and thus students who decided not to use ASQ treated the sessions as standard lectures.

We posed questions of three question types that depended on the lecture material and assessment goals of each class: (A) multiple-choice, (B) SQLite programming (Fig. 2), and (C) text-input (Fig. 3). Table 4 summarizes the main characteristics of the three lectures, including the lectures’ topic, the number of students participating through ASQ and the number of questions posed per type. Note that Lecture 1. Advanced SQL Topics has generated almost seven times more browser events than the other two lectures due to its usage of SQLite programming quizzes: not only the large amount of typing contributed to the events generation, but also the question setup which required the students to consult a database schema diagram resulting in a considerable amount of blur/focus events between ASQ and the diagram.
Table 4.

Overview of the three ASQ lecture sessions each given by one of two instructors (identified as I1 and I2). For each session, the number of students participating, the number of exercises (per type) and the number of ASQ low-level browser events logged are listed.

Instr.

Topic

#Students using ASQ

#ASQ events

#Question types

A

B

C

1

I1

Advanced SQL Topics

143

121,062

0

7

0

2

I1

ER Conceptual Design

111

17,460

8

0

0

3

I2

Web Security

187

17,562

4

0

2

5 Analysis

In our exploration of the collected logs, we are guided by our research questions and the five main findings of prior works (identified in Sect. 2) exploring students’ attentiveness in the classroom.

F1: Students’ attention drops over the class period. For all lecture logs, we translated low-level browser events into activity indicators (Fig. 5) and subsequently inferred attention states (Fig. 6). We consider the two activity indicators connected and focused and the union of the states Following/Thinking/Working as well as Distracted/Bored as most suitable representatives of student attention and inattention respectively. To explore how attention changes over time, we correlate the lecture time (in units of 1 second) with the number of students in the specific state(s) or activity setting. If, as expected student attention drops over time, we will observe a decrease in focus over time and an increase in Distracted/Bored students. The results in Table 5 show that this is indeed the case: inattention-oriented activities/states are positively correlated with time while attention-oriented activities/states are negatively correlated with time. Moreover, the high-level inferred attention states achieve higher absolute correlations, indicating that they are more suitable to infer (in)attention than our low-level activity indicators.

We thus posit that based on the events logged in ASQ, we are able to infer in real-time (and live in the classroom) when and to what extent attention drops over time, relying on either the focus activity indicator as a basic measure or a combination of the more high-level attention states Following/Thinking/Working (and their counterparts).
Table 5.

Linear correlation coefficient (significant correlations at the \(p<0.05\) level are marked \(\dagger \)) between time and number of students exhibiting a particular activity indicator (top part) or one of a set of inferred attention states (bottom part).

+++ Activity indicators +++

Lecture

All slides

Slides w/o exercises

Connected

Focus

Connected

Focus

1

Advanced SQL Topics

0.176\(\dagger \)

−0.182\(\dagger \)

0.281\(\dagger \)

−0.059

2

ER Conceptual Design

−0.224\(\dagger \)

−0.569\(\dagger \)

−0.284\(\dagger \)

−0.637\(\dagger \)

3

Web Security

0.263\(\dagger \)

−0.177\(\dagger \)

0.284\(\dagger \)

−0.228\(\dagger \)

+++ Attention states +++

Lecture

All slides

Slides w/o exercises

Distracted/Bored

Following/Thinking/Working

Distracted

Following/Thinking

1

Advanced SQL Topics

0.324\(\dagger \)

−0.274\(\dagger \)

0.450\(\dagger \)

−0.257\(\dagger \)

2

ER Conceptual Design

0.039

−0.549\(\dagger \)

0.230\(\dagger \)

−0.657\(\dagger \)

3

Web Security

0.391\(\dagger \)

−0.262\(\dagger \)

0.458\(\dagger \)

−0.390\(\dagger \)

Fig. 5.

Connected and focused activity indicators for all the sessions

F2: Attention breaks occur regularly and increase in frequency as the class progresses. For each second of the lecture we track the number of attention breaks, that is, the number of students that switch their device from focused on ASQ to some other activity. We also track attention recovery which we define as the number of students whose device switches back to focus on ASQ. The attention focus variation is the net sum of attention recoveries minus the attention breaks observed during the same period (a window of 30 s). For each of the three lectures we present their attention focus variations in Fig. 7. We observe that attention breaks occur regularly but there is no noticeable increase in frequency as the class progresses. We note that although this is in contrast to F2, not all empirical studies in the past observed this increase in attention breaks [15].
Fig. 6.

Inferred student attention state for all the sessions

F3: Attention rises when interactive elements are introduced. Drawing on our analysis of attention focus variation, we observe that whenever there are interactive elements in the slide, in the form of questions, we observe spikes of attention recovery (Fig. 7) and an increase of connected students (Fig. 5). While introducing interactive elements thus captures the attention of the students (positive attention focus variation), shortly thereafter we observe the subsequent loss of focus due to students waiting on each other to answer. Likewise, students might be searching for solutions using their devices, something ASQ cannot distinguish from students simply leaving the application to do something else. As we can observe in the charts of Fig. 6 for all the lectures, whenever there is a slide with a question, the number of students that have their ASQ page out of focus (Searching state) is always lower than in slides without a question (Distracted state). Similarly, the magnitude of attention focus variation is smaller for slides without questions than for slides with questions, which literally appear to send jolts through the collective attention span of the students in the classroom (Fig. 7). Our results thus confirm previous findings of rising attention at interactive elements.
Fig. 7.

Attention Focus Variation: how many students have changed the focus of attention during lecture (moving sum of attention breaks and recoveries using a window of 30 s).

F4: Immediately after interactive teaching elements, the level of distraction is lower than before the start of the interaction. While there is a peak of interest as soon as questions are asked, after students submit their answers, their focus switches to other activities. Hence, as shown in Fig. 6 towards the end of the question, the number of students we infer to be in a Distracted state rises considerably and is almost always higher than right before the interactive teaching element. The effect depends on the length of time students have to wait for other students to complete the exercise (before the instructor moves on in the lecture) and on the type of feedback given either individually or globally on the submitted answer. This result is a clear deviation from prior works and suggests that our attention model, in particular the Distracted state captures more than just students’ distraction.

F5: In retention tests, students tend to perform better on material presented early on in the class. Instead of dedicated retention tests, we rely on the multiple choice (MC) questions as a retention proxy (we restrict ourselves to MC questions as the open question types require manual grading to achieve highly accurate results).

Table 6 lists the accuracy of the student answers for the eight MC questions of Lecture 2. ER Conceptual Design as well as the specific time they were posed in the lecture. Note that shortly after 10:30 am the official 15 min break commenced. We observe that students tend to perform better in the first half of the class than the second. Although a subset of questions from a single lecture do not provide enough evidence to support or reject this finding in the context of ASQ it shows once more ASQ ’s capabilities to provide fine-grained real-time logging and analyses to the instructor.
Table 6.

Correct vs incorrect answers ordered by time of question for Lecture 2

Question

Start time

Correct%

Correct

Incorrect

Total

1

10:25:36

92.31

72

6

78

2

10:27:04

77.22

61

18

79

3

10:28:24

82.50

66

14

80

4

10:30:58

1.43

1

69

70

5

10:32:04

90.79

69

7

76

6

11:17:48

70.77

46

19

65

7

11:20:25

75.41

46

15

61

8

11:23:48

65.00

39

21

60

6 Conclusions and Future Work

ASQ is an interactive Web-based teaching platform that allows capturing browser events to observe and categorise the behavior of its users. ASQ is able to provide real-time data analytics in the classroom, thus providing a lecturer with the capability to observe her students in a data-driven manner. In this paper we have shown how ASQ can be employed to infer student attention, based on either activity indicators and states we aggregate based on low-level browser events. The visualizations presented here enable instructors to observe at a very fine-grained level the behavior of an entire class with hundreds (or potentially thousands) of students. Our analysis confirms existing research findings, whereby: (1) student attention drops during the class period; (2) attention breaks occur regularly as the class progresses; and (3) attention rises when interactive elements are introduced. Additionally, we could observe a drop in attention as soon as the interactive activity is completed by individual students, which should be taken into account when planning to introduce questions and interactive exercises within a lecture.

ASQ can also be used to support adaptive teaching. As future work, we will further exploit ASQ ’s attention level monitoring capabilities to recommend teachers subject-related questions that could used to restore focus, if an attention drop is detected during the presentation of slides. The current version of ASQ is only a first step to a highly sensor- and data-driven classroom. In our future work we plan to complement ASQ ’s data collection and aggregation abilities with additional sensors and technologies (eye-tracking and activity sensors) in order to acquire a more complete picture of the students in the classroom.

Footnotes
1

Once more, it should be noted that for these findings some contradictory evidence exists as well.

 

Acknowledgements

This research was partially supported by the Extension School of the Delft University of Technology.

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  • Vasileios Triglianos
    • 1
  • Cesare Pautasso
    • 1
  • Alessandro Bozzon
    • 2
  • Claudia Hauff
    • 2
  1. 1.Faculty of InformaticsUniversity of LuganoLuganoSwitzerland
  2. 2.Web Information SystemsDelft University of TechnologyDelftThe Netherlands

Personalised recommendations