Keywords

1 Introduction

Learner models capture an abstract representation of a student’s knowledge state. By and large, learner models approximate a student’s knowledge state solely based on their performance on assessment items. As a point of reference, many popular learner models such as Bayesian Knowledge Tracing (BKT) [9], Item Response Theory (IRT) [24], Adaptive Factor Models (AFM) [7], Performance Factor Analysis (PFA) [25], deep knowledge tracing (DKT) [27], collaborative filtering based models [1, 28], Elo-based modes [3, 26], and knowledge tracing machines (KTM) [29] only use responses of students to assessment items and information about them in their modelling. This can probably be attributed to the fact that in many educational learning systems, students are prominently involved in just answering assessment items (e.g., [22]).

In recent years, the use of crowdsourcing in education, often referred to as learnersourcing [21], to support learning at scale has received significant attention. Examples of tasks that have been learnersourced include creating resources [12, 20], creating solutions and explanations [14, 32], rating quality [12, 20], giving feedback [23] and annotating videos [10, 31]. The adoption of learnersourcing approaches is often motivated by learning theories that promote higher-order learning [4] and have been demonstrated to enhance student learning [12, 18].

Building on the growing evidence that learnersourcing practices enhance learning, this paper explores whether information about the learnersourcing contributions of students can be leveraged towards modelling of learners. For this exploration, we make use of the knowledge tracing machines (KTMs) framework [29] for modelling student learning. Commonly, KTMs have been used as a framework for modelling learners based on a single task (attempting assessment items). We present an encoding extension on KTMs so that the framework can capture students’ interactions across multiple types of tasks (multi-tasks). To evaluate our approach, we use two data sets collected from a crowdsourcing adaptive educational system called RiPPLE in which students are engaged with multiple types of tasks within the system. Results suggest that leveraging data associated with learnersourcing contributions of the students on some types of tasks such as content creation and content moderation can be used to improve the predictive performance of the learner model compared to traditional learner models. In addition, in recent years, OLMs have been extensively integrated into various educational tools to help students in monitoring, reflecting, planning, and regulating their learning [2, 5, 6, 8, 15, 16]. In the context of open learner models [6], updating models of the learners based on their crowdsourced contributions can further highlight the link between learnersourcing and learning as well as acknowledging their contribution. This may act as a method of incentivising student engagement with learnersourcing.

2 Approach

To infer learner models that incorporate data from learnersourced contributions alongside student assessment data, we present an encoding extension over the knowledge tracing machine (KTM) framework [29] so that interactions across multiple types of tasks (multi-tasks) can be captured. We denote students by \(u_n \in \{ u_1 \ldots u_N \}\), learning resources (items) by \(q_m \in \{ q_1 \ldots q_M \}\), and knowledge components (concepts) by \(\delta _c \in \{ \delta _1 \ldots \delta _C \}\). Each item can be tagged with one or more concepts. We denote the relationship between items and concepts by \(\omega _{mc} \in \Omega _{M\times C}\), where \(\omega _{mc}\) is 1 if item \(q_m\) is tagged with \(\delta _c\), and 0 otherwise. We further denote \(o_{nc}\) to keep track of the number of opportunities a student \(u_n\) has had on a concept \(\delta _{c}\) at a given time.

Commonly, KTMs have been used as a generic framework for traditional computer-based educational systems where students are only involved in attempting assessment items available in the repository of the system. Therefore, the set of tasks, T, represented in these systems is only limited to one task. We present a simple extension that enables KTMs to capture and encode data on students interacting with more than one task. We denote different types of tasks that students are allowed to perform in relation to items by \(t_k \in \{ t_1 \ldots t_k \}\). Furthermore, we extend \(o_{nc}\) to \(o^k_{nc}\) to represent the number of opportunities a user \(u_n\) has had on a task \(t_k\) on a concept \(\delta _{c}\) at a given time. Our proposed approach encodes and uses data from student, items, concepts, and opportunities on each of the tasks to infer a learner model \(\Lambda _{N \times M}\) that estimate each student \(u_n\)’s knowledge state for correctly attempting learning item \(q_m\).

Figure 1 presents an example of the input file (part (a)) and its one-hot encoding (part (b)) for an educational system with multi-tasks using a chronologically ordered log file with seven observed interactions from an educational system based on two students, three tasks, three items and two concepts into a sparse vector for training KTM (for details, please see [29]).

Fig. 1.
figure 1

(a) An example of a log file with 7 interactions from a crowdsourcing educational system with three types of tasks (b) On-hot encoded of the log for training KTM

Full size image

3 Evaluation

Data Sets. We used two data sets obtained from an adaptive educational system called RiPPLE that recommends learning activities to students based on their knowledge state from a pool of learnersourced learning items [18]. RiPPLE enables students to create, attempt, moderate, rate, and leave comments on a range of items, including worked examples and multiple-choice questions. For this study, we consider three main types of tasks that students are allowed to perform in RiPPLE: (1) attempting items (Attempt), (2) creating new items (Create), and (3) moderating items (Moderate). The data sets are obtained from two courses, namely, ‘Preparation for US Medical Licensing Examination (USMLE) (Medi) and ‘Biological Fate of Drugs’ (Pharm). For each of these two courses, the RiPPLE platform was used for 13 weeks of the semester. Each item in the repository is associated with one or more concepts (KC) covered in the course. Overall information about these data sets are provided in Table 1.

Table 1. overall statistics for data sets (\(\#\) stands for number of).
Full size table

Models for Comparison. We implemented the proposed model based on the encoding of students, items, concepts, opportunities on attempting (\(o^A\)), opportunities on creating (\(o^C\)), and opportunities on moderating items (\(o^M\)) as tasks in KTM. We compare the predictive performance of this model with traditional learner models, including IRT, AFM, and PFA. To provide a fair comparison between all models, we also made two considerations: (1) We implemented a baseline model within KTM based on the encoding of student, items, concepts, and opportunities on attempting items (\(o^A\)); (2) The pairwise interaction between features in KTM is set to zero (d = 0). For all models, we used 80% of data as the train set and predicted the outcomes on the remaining 20% as the test set. For both of the data sets, 400 epochs are used for training KTM.

Results. Table 2 compares the accuracy (ACC), area under the curve (AUC), and negative log-likelihood (NLL) of the model fit statistics related to each model. Our experimental results suggest that the learner model that leverages data related to content creation and content moderation activities (\(m_5\)) can more accurately estimate students’ knowledge state compared to its baseline and the traditional learner models that only rely on students performance on attempting learning items. This outcome is aligned with past studies on learnersourcing [11,12,13, 18, 19, 30] that suggest engaging students in higher-order learning impacts learning.

Table 2. Performance of different feature encoding for modeling learners
Full size table

4 Discussion and Conclusion

In this paper, we presented a learner model that leveraged data from students’ learnersourced contributions alongside traditional item-assessment data towards modeling the knowledge state of students in crowdsourcing educational systems. The results of our empirical studies suggest that incorporating data from students’ contributions on some types of tasks associated with higher-order learning such as content creation and content moderation can be used to improve the predictive performance of the learner model. This, in turn, can improve the provided personalised feedback by the system and its adaptivity functionalities. Our findings can also have implications for learnersourcing systems that incorporate an open learner model (OLMs). Incorporating OLMs in crowdsourcing educational systems can have two benefits:

  1. 1.

    Highlighting the link between learnersourcing and learning. Updating a student’s competency in a concept by creating or moderating resources on that concept can help the student better associate learnersourcing with learning.

  2. 2.

    Acknowledging students’ learningsourcing contributions. Developing models that recognise students’ learningsourcing contributions and associate it with their learning may incentivise students’ engagement with learnersourcing tasks, which has been identified as a challenge in learnersourcing [17, 31].

A major limitation of the presented evaluation is that the study was conducted on small data sets. Future directions include replicating this study across different disciplines with a larger number of students to evaluate the generalisability of our current findings.