Comparing student and expert-based tagging of recorded lectures

Abstract

In this paper we analyse the way students tag recorded lectures. We compare their tagging strategy and the tags that they create with tagging done by an expert. We look at the quality of the tags students add, and we introduce a method of measuring how similar the tags are, using vector space modelling and cosine similarity. We show that the quality of tagging by students is high enough to be useful. We also show that there is no generic vocabulary gap between the expert and the students. Our study shows no statistically significant correlation between the tag similarity and the indicated interest in the course, the perceived importance of the course, the number of lectures attended, the indicated difficulty of the course, the number of recorded lectures viewed, the indicated ease of finding the needed parts of a recorded lecture, or the number of tags used by the student.

Appendices

Appendix 1 Tagging protocol

To determine where to place the tags, we used signals, or signposts, provided by the structure of the lecture. Exley and Dennick (2004) describe a number of possible lecture structures a lecturer can choose from. They distinguish a number of different types of statements a lecturer can use to inform students about the lecture organization: signposts, frames, foci, and links. Besides these statements by the lecturer, the structure of the slides also provides a signal for the lecture organization. The slides contain titles, lists of important topics, schemas with the structure of the topics, etc. Based on these signals, we created the following tagging protocol for our experiments:

• 1. Examine the lecture structure (see Table 6). This gives an indication of the sort of possible tag indicators that signal useful tags.

  Table 6 Lecture structures

• 2. Playback the recorded lecture, and while playing, listen to oral signals by the lecturer that indicate signposts, frames, foci, or links.

• 3. Mark potential tags. Pause the recording, write down the time code along with potential tag title and a short description of the tag.

• 4. After completion of the tagging process, the tags, descriptions, and time codes were added to the tagged player system.

• 5. Always add a tag at 00:00:00, indicating the beginning of the recording. This gives the student an easy way to return to the beginning of the recording.

Appendix 2 Vector space modelling

To analyze the similarity of the tags added by the students and the expert, we converted the tags into vectors. We will use an example to explain the steps taken to create the vectors during the analysis. Table 7 shows an example for a (short) video where four students (student 1 through student 4) have assigned a total of five different multi-word tags to that video:

Table 7 Raw frequencies for the tags used by the students

The table shows that student 1 used the tags “Intra cellular” and “Cells” once each and that he/she did not use any of the other tags. It also shows that student 3 used “Intra cellular” and “Function of a cell” once and “Homeostasis” twice, etc. To calculate the similarity between the tags assigned by the students, we use a method from the information retrieval and natural language processing domain (Manning and Schütze 1999). The foundation of this method is that the tags are translated into vectors. This translation results in a vector space where the tags are the axes of the space and the students who added the tags are the points or vectors in this space. We can then compare those vectors to calculate a score for the similarity of those vectors. We will use the example in Table 1 to clarify the steps taken.

The term frequency tf _t,s of a tag is defined as the number of times that t has been used by students s. So in our example, tf _1,1 = 1, tf _2,3 = 0 and tf _5,4 = 2. We can represent this table as a set of four vectors, one for each student, where each vector has five dimensions (one for each tag). For example, the vector for student 1 and student 4 would look like this:

$$ \overrightarrow{s1}=\left[\begin{array}{c}\hfill 1\hfill \\ {}\hfill 0\hfill \\ {}\hfill 0\hfill \\ {}\hfill 0\hfill \\ {}\hfill 1\hfill \end{array}\right]\overrightarrow{s}4=\left[\begin{array}{c}\hfill 0\hfill \\ {}\hfill 1\hfill \\ {}\hfill 0\hfill \\ {}\hfill 0\hfill \\ {}\hfill 2\hfill \end{array}\right] $$

Student 1 and student 4 have one tag in common, but student 4 used it (“Cells”) twice. This tag has more importance to student 4, and we want to take that into consideration when we calculate the similarity of tag usage between the two students. However, we use the log frequency weight, and not the raw frequency of the term, to achieve a less than linear relevance indication:

$$ {w}_{t,s}=\left\{\begin{array}{ll}1+{ \log}_{10}{\mathrm{tf}}_{t,s}\hfill & if\kern0.5em {\mathrm{tf}}_{t,s}>0\hfill \\ {}0\hfill & otherwise\hfill \end{array}\right. $$

To calculate the similarity of the tag usage, we calculate the similarity of the vectors. We cannot simply take the Euclidian distance between the vectors. We need to calculate the cosine similarity of the vectors of students. To do that, we first have to length-normalize the vectors by dividing each of its components by its length. For that we use the L₂ norm:

$$ {\Vert \overrightarrow{x}\Vert}_2=\sqrt{{\displaystyle \sum}_i{x}_i^2} $$

So if student 1 is represented by the vector $\overrightarrow{s1}$ and student 2 is represented by the vector $\overrightarrow{s2}$, then the cosine similarity for the tags used by the two students, taking into account length normalization of the vectors, is:

$$ \cos \left(\overrightarrow{s1},\overrightarrow{s2}\right)=\frac{{{\displaystyle \sum}}_{i=1}^{\left|V\right|}s{1}_is{2}_i}{\sqrt{{{\displaystyle \sum}}_{i=1}^{\left|V\right|}s{1}_i^2}\sqrt{{{\displaystyle \sum}}_{i=1}^{\left|V\right|}s{2}_i^2}} $$

If we apply both the weighting factor and the normalization to the vectors and represent it in our original table again, it looks like Table 8:

Table 8 Normalized vectors

We can now calculate the similarity between the tag usages of the students by multiplying the components of the vectors. In our example, the similarity of the tag usage of student 1 and student 2 can be calculated as: (0.7071 * 0.5) + (0 * 0) + (0 * 0.5) + (0 * 0.5) + (0.7071 * 0.5) = 0.7071

This then results in a 4 × 4 table for the mutual similarities between the four students given the example set of five tags (see Table 9).

Table 9 Similarity scores for the students

The diagonal cells in Table 4 are one, indicating that if we compare a vector with itself (e.g., vector $\overrightarrow{s1}$ with vector $\overrightarrow{s1}$, or compare the tags used by student 1 with student 1), there is a perfect match. Also, each value is present twice in the table, because comparing vector $\overrightarrow{s1}$ with vector $\overrightarrow{s2}$ gives the same similarity score as when we compare vector $\overrightarrow{s2}$ with vector $\overrightarrow{s1}$. Table 4 shows that student 3 and student 2 have the highest similarity in tags (similarity score is 0.8589), while student 3 and student 4 have no tag similarity (similarity score is 0). If we remove the duplicate information from the table, the resulting table looks like Table 10.

Table 10 Similarity scores for the students (optimized)