Student online communication skills: Enhancing the comparability of self-reports among different groups of students

Abstract

With the current emphasis placed on ICT skills development in education, accurate information about how well students master these skills becomes invaluable. Despite the wide-spread use of self-report measures of ICT skills, their accuracy has been questioned. An analysis, on a large sample, of the heterogeneity in reporting behavior in the domains of ICT competencies is, as far as we know, missing; we fill this gap. We investigate the (in)comparability of self-reports of online communication skills (e.g., the using of social networks, data sharing) among two contrasting groups of students (a) elite, high-performing grammar schools and (b) economics schools (total N = 1,070 students, 17 secondary schools). Using the anchoring vignette method, we identify scale usage differences among respondents and adjust their self-reports for these differences. We show that grammar school students significantly underestimate their skills. Before the adjustment, grammar school students report significantly lower levels of online communication skills. After the adjustment, grammar school students have non-significantly higher levels of these skills. Differential academic demands thus might be a relevant factor in students’ self-assessment of online communication skills. In practice, students’ under-/over-estimation of skills might impact their access to ICT-related jobs and the effectiveness of educational decision-making in the ICT domain. We also show the potential of the anchoring vignette method to explain paradoxical negative relationships between self-reported skills and results on the achievement tests identified in the literature. Further research could explore this phenomenon in other domains of digital competence and among other student populations.

Appendices

Appendix A

Specification of the CHOPIT model

The text is based on the description provided by Vonkova, Bendl, and Papajoanu (2017) but adapted to reflect our domain of online communication skills, explanatory variables, and our use of more than one anchoring vignette.

The CHOPIT model consists of two parts: the self-assessment part and the anchoring vignette part. First, we model the self-reported online communication skills, reflecting the ordinal nature of responses (we use a 7-point scale). For each student $i\left ( {i = 1, \ldots I} \right)$ we define a latent online communication skills variable $Y_{si}^{*}$ as follows:

$$Y_{si}^{*} = {\varvec{X}}_{i}^{\prime } {\varvec{\beta}} + \varepsilon_{si} ,$$

where ${\varvec{X}}_{i}$ is a vector of explanatory variables. We use the following dummy variables: Grammar school (1 = YES, 0 = NO), where the reference group are students from economics schools; Male (1 = YES, 0 = NO), where the reference group are females; Home population > 50,000 (1 = YES, 0 = NO), where the reference group are students living in a location with a population of less than 50,000 inhabitants; Household books-201 or more (1 = YES, 0 = NO), where the reference group are students having 200 or less books in their household. ${\varvec{\beta}}$ is a vector of unknown parameters and $\varepsilon_{si}$ is the error term assumed to be normally distributed with mean 0 and variance 1 and independent of ${\varvec{X}}_{i}$. The reported online communication skills $Y_{si}$ is an ordinal variable based upon the latent online communication skills variable:

$$Y_{si} = k \Leftrightarrow \tau_{i}^{k - 1} < Y_{si}^{*} \le \tau_{i}^{k} k = 1, \ldots , 7,$$

where $\tau_{i}^{k} ,k = 1, \ldots , 6$ are the thresholds $\left( {\tau_{i}^{0} = - \infty , \tau_{i}^{7} = \infty } \right)$ which are allowed to differ by the explanatory variables of the students.

$$\begin{aligned} & \tau_{i}^{1} = {\varvec{X}}_{i}^{\prime } {\varvec{\gamma}}^{1} , \\ & \tau_{i}^{k} = \tau_{i}^{k - 1} + e^{{{\varvec{X}}_{i}^{^{\prime}} {\varvec{\gamma}}^{k} }} \;k = 2, 3, \ldots ,6. \\ \end{aligned}$$

${\varvec{\gamma}}^{k} ,k = 1, \ldots ,6$ are vectors of unknown parameters. In our study they capture differences in the reporting behavior of the students with respect to different characteristics (type of school, gender, the grade they attend, the population of the location that the student lives in, the number of books in their household). If the thresholds are the same for all students, i.e. no heterogeneity in reporting behavior $\left( {\tau_{i}^{k} = \tau^{k} } \right)$ is assumed, we get a common ordered probit model.

If we use only the self-reported online communication skills, the parameters ${\varvec{\beta}}$ and ${\varvec{\gamma}}^{1}$ are not separately identified (only their difference is identified). In other words, we cannot distinguish between the real online communication skill levels and the differences in reporting styles if we only use the self-reports. Therefore, we need additional information to identify the parameters. The anchoring vignette evaluations serve as the source of this information. In the second part of the CHOPIT model, we model the responses to the anchoring vignettes as follows:

$$\begin{aligned} & Y_{vi}^{*} = \theta_{v} + \varepsilon_{vi} \;v = 1,2,3, \\ & Y_{vi} = k \Leftrightarrow \tau_{i}^{k - 1} < Y_{vi}^{*} \le \tau_{i}^{k} \;k = 1, \ldots , 7, \\ \end{aligned}$$

where $\theta_{v}$ corresponds to the level of the online communication skills of the hypothetical person in the vignette, $\varepsilon_{vi}$ is the error term assumed to follow the normal distribution with mean 0 and variance $\sigma_{v}^{2}$ (assumed to be equal for all vignettes) and independent of $\varepsilon_{si}$, and of ${\varvec{X}}_{i}$. $Y_{vi}$ are the vignette evaluations of student $i$. The thresholds between response categories for the vignettes are modeled the same way as the thresholds between response categories for the self-reports. It enables the identification of ${\varvec{\gamma}}^{1}$ (i.e. the parameters reflecting differences in response styles) and subsequently the identification of ${\varvec{\beta}}$ (the parameters reflecting real differences in online communication skills).

Appendix B

The set of estimated parameters for version 2 of the ordered probit and the CHOPIT model, including the threshold parameters γ for the CHOPIT model

See Table 5

Table 5 The sets of parameter estimates for the (unadjusted) ordered probit and the (adjusted) CHOPIT-versions 2

.