Keywords

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

In previous chapters, we described several associations between students’ computer and information literacy (CIL) and selected variables such as gender and home background. Our aim in this chapter is to investigate the combined influence of a number of variables on variations in CIL, including individual (student-level) as well as contextual (school-level) variables. The ICILS research questions that we address in this chapter are the following:

  • Research Question 2: What aspects of schools and education systems are related to student achievement in computer and information literacy?

  • Research Question 3: What characteristics of students’ levels of access to, familiarity with, and self-reported proficiency in using computers are related to student achievement in computer and information literacy?

  • Research Question 4: What aspects of students’ personal and social backgrounds (such as gender, socioeconomic background, and language background) are related to computer and information literacy?

We used multilevel models to review the extent to which different factors at the student and school level are associated with variations in CIL. Factors of interest include those related to access to, use of, and familiarity with information and communication technology (ICT) as well as other variables reflecting students’ personal and social backgrounds.

A model for explaining variation in CIL

When developing this model, we drew on research literature as well as the contextual framework for ICILS (Fraillon, Schulz, & Ainley, 2013) to determine which predictors of variation in CIL to include in our multivariate analyses.

Prior to ICILS, research into CIL learning outcomes and factors influencing student knowledge in this area was generally limited to national studies. Sample surveys carried out as part of the Australian National Assessment Program (NAP) for ICT Literacy showed that students’ gender (female), socioeconomic background, and experience with and current use of computers were positive predictors of ICT literacy (Australian Curriculum, Assessment and Reporting Authority, 2012; Ministerial Council for Education, Early Childhood Development and Youth Affairs, 2010; Ministerial Council on Education, Employment, Training and Youth Affairs, 2007).

The Chilean national assessment program SIMCE TIC also assessed ICT literacy. Multilevel analyses of this body of data illustrated considerable variation among schools as well as effects of cultural background, socioeconomic status, and school characteristics (private/public, subsidies) on digital competencies (Román & Murrillo, 2013). Further analyses also provided evidence of strong effects of prior achievement in reading and mathematics on digital competence (San Martin, Claro, Cabello, & Preiss, 2013).

As part of its Programme in International Student Assessment (PISA), the OECD assessed the performance of 15-year-old students in digital reading across 16 countries (OECD, 2011). Although this international study assessed reading competences in a digital environment, it also reflected CIL-based skills. Study results showed that socioeconomic background as well as computer use had statistically significant effects on students’ digital reading skills. However, no clear association was found between these skills and computer use at school.

The ICILS contextual framework (Fraillon et al., 2013) postulated that students’ CIL is influenced by context variables located at different levels (wider community, schools/classrooms, individual learner, and home), with these levels featuring antecedent as well as process-related factors. When conducting the analysis of CIL presented in this chapter, we included variables pertaining to the school/classroom context, the context of the individual learner, and the home context.

Another distinction, one that we introduced into the analyses in this report, can be made between variables associated with (1) ICT and learning about CIL, and (2) personal and social background factors in addition to the ICT-related variables. If we use only the first group of variables in a multivariate model (i.e., Model 1), we obtain results that indicate the effects of the ICT-related variables by themselves. Contrasting these results with those from a second model (Model 2), which contains all predictor variables, including those reflecting social and personal background factors, provides us with an indication of the net effects of the ICT-related variables as well as the net effects of background.

The models we chose for our analyses included several predictors that we classified into the following broad categories:

  • ICT resources and use at home: These predictors were ICT resources at home, personal experience with ICT, students’ use of ICT at home and school, and students’ experiences with learning about ICT at school. We included these variables at the student level in Models 1 and 2.

  • ICT resources and use at school: ICILS 2013 collected information on schools’ ICT resources through its ICT-coordinator and teacher questionnaires. The school’s CIL learning context includes experience at school with using ICT in teaching and learning, the extent to which students at school are regular users of computers, and students’ perceptions of their having learned CIL skills at school. We included these variables at the school level in Models 1 and 2.

  • Personal and social background: Previous research and results from other analyses conducted during ICILS (see Chapter 4) illustrate the extent to which gender, students’ expectations of their own educational attainment, and parental socioeconomic status are associated with students’ CIL. We included these variables at the student level in Model 2.

  • Social context of schools: At the school level, the average socioeconomic status of the student body is a factor that, as numerous studies show, is associated with many different learning outcomes. We included this variable at the school level in Model 2.

We used the following variables to indicate home ICT resources:

  • Internet access at home: For the purpose of our analysis, we coded students who reported having internet access at home as 1 and all others as 0.

  • Number of computers at home: We coded the indicator variable resulting from students’ reports of the number of desktop and portable computers in their homes as 0 (no computer), 1 (one computer), 2 (two computers), or 3 (three or more computers).

This next batch of variables relates to students’ individual learning contexts.

  • Experience with computers: This variable reflected how long each ICILS student had been using computers. We coded it in approximate years (with values of 0, 2, 4, and 6) so that the regression coefficient would reflect the change in CIL score points for one additional year of experience.

  • Weekly use of computers at home: This variable reflected the frequency with which the students were using computers at home and was coded 1 for at least weekly use and 0 for less frequent use. This meant that the regression coefficient would reflect the change in CIL score points between students with at least weekly use of a computer at home and students with less frequent use after we had controlled for all other variables in the model.

  • Weekly use of computers at school: This variable reflected the frequency with which students were using computers at school. We coded it 1 for at least weekly use and 0 for less frequent use so that the regression coefficient would reflect the change in CIL score points between students with at least weekly use of a computer at home and students with less frequent use after we had controlled for all other variables in the model.

  • Students’ reports on learning CIL tasks at school: We based this index on a set of eight items that required the ICILS students to indicate whether they had learned about different CIL tasks at school.Footnote 1 The values were IRT (item response theory) scores, which we standardized for our analyses within each country to have a mean of 0 and a standard deviation of 1. We centered these values on the school averages so that the individual values would indicate the difference from the average index score in each school.

The following school-level predictors reflect ICT resources at school but from different perspectives:

  • Availability of ICT resources for teaching and learning: This measure, based on responses from the ICT-coordinators, was computed using ICILS questionnaire data on the availability of nine different computer and ICT resources.Footnote 2 We coded the items dichotomously (1 = available, 0 = not available) and then estimated the IRT scale scores. The higher values indicate more ICT resources at school.

  • ICT resource limitations for teaching and learning: This index reflected the extent to which the ICILS teachers thought their schools had insufficient ICT resources.Footnote 3 We based the IRT scale scores on teacher survey data aggregated at the school level and standardized them for this analysis to have a mean of 0 and a standard deviation of 1 across weighted schools in each education system.

The following school-level predictors reflect the school learning context:

  • School experience with using ICT for teaching and learning: School ICT-coordinators reported on the amount of time their school had been using computers for teaching and learning. We coded the four response categories as 0 for “not using computers,” 2.5 for “fewer than 5 years,” 7.5 for “at least 5 but fewer than 10 years,” and 12.5 for “10 years or more” so that the regression coefficients would reflect the approximate increase per year of computer experience.

  • Percentage of students reporting at least weekly use of computers at home: This index reflected the extent to which students were in a home context where computers were commonly used. At schools where majorities of students tend to use computers at home, we can expect that individual student learning will be fostered by an environment where exchanging ideas about ICT is common.

  • School average of students who said they had learned CIL tasks at school: This measure, derived as the average student score on perceptions of having learned CIL tasks at schools, provided a school-level measure of the extent to which CIL-related content was being used at the school. We standardized the school-level index so that 0 was the mean and 1 the standard deviation of weighted school averages within the participating education systems.

The personal and social student background characteristics included in our analyses were:

  • Students’ gender: We coded this variable as 1 for females and 0 for males.

  • Students’ expected educational attainment: Although this variable is more than a simple background factor, it does reflect home-based expectations regarding students’ ongoing education as well as students’ educational aspirations with respect to fields beyond the domain of the (in this case, ICILS) assessment. For the present analyses, this factor was reflected in three indicator variables of expected highest educational attainment, namely, lower-secondary, post-secondary nonuniversity, and university education (each coded as 1 = expected or 0 = not expected). Expectation of attaining an upper-secondary qualification served as a reference category.

  • Students’ socioeconomic background: This variable was a composite index that we standardized to have a mean of 0 and a standard deviation of 1 within each country and centered on school averages so that it would indicate the effect of socioeconomic background within schools. The index consisted of factor scores derived from a principal component analysis of:

    • highest parental occupation (ISEI scores);

    • highest parental education (categorical variable with 0 = lower-secondary or below, 1 = upper-secondary, 2 = post-secondary nonuniversity education, and 3 = university education); and

    • number of books at home (categorical variable with 0 =0–10 books, 1 =11–25 books, 2 =26–100 books, and 3 = more than 100 books).

We used the following variable to measure the schools’ “social intake”:

  • School socioeconomic context: This variable reflected the average of student scores on the composite index of socioeconomic background. It indicated the social (student) intake of schools and the social context in which the ICILS students were learning. We standardized the index to have a mean of 0 and a standard deviation of 1 across weighted schools within each participating education system.

During multivariate analyses, any issues relating to missing data tend to become more prevalent than in other forms of analysis because of the simultaneous inclusion of numerous variables. To address the missing data issue, we first excluded from the analyses the small proportion of students for whom there were no student questionnaire data. We were able to take this approach because only small proportions of students had missing data for the student-level variables.

Because there were higher proportions of missing data for the variables derived from the ICT-coordinator questionnaire (ICT resources at school and ICT experience at school) and the ICILS teacher survey, we needed to treat these by setting the missing values to national mean or median values, respectively, and then adding a missing indicator variable for missing school data and another one for missing teacher data. We chose this approach (see Cohen & Cohen, 1975) because of its simplicity and because of the relatively limited number of missing values.

On average, data from about 97 percent of tested students were included in the analysis. The only country where this proportion was somewhat lower, at 93 percent, was Germany. The ICILS technical report (Fraillon, Schulz, Friedman, Ainley, & Gebhardt, forthcoming) provides detailed information on the multilevel modeling and treatment of missing data.

The hierarchical nature of the data lent itself to multivariate multilevel regression analysis (see Raudenbush & Bryk, 2002). We estimated, for each national sample, two-level hierarchical models, with students nested within schools. We used the software package MPlus (Version 7; see Muthén & Muthén, 2012) to carry out the analyses and obtained estimates after applying sampling weights at the student and school levels.

We excluded from the analyses some countries and benchmarking participants that had insufficient data. The extremely low participation rates for the teacher survey in the City of Buenos Aires (Argentina) and Switzerland led to the exclusion of their data, while data from the Netherlands had to be excluded because of the missing information on parental occupation that was needed to derive the composite index of students’ socioeconomic background.

When interpreting results from a multilevel analysis, it is important to be aware that first-level (i.e., student-level) variables have a different meaning from those in a single-level regression analysis. This is because student-level coefficients reflect the effect a variable has within schools. Consequently, with respect to ICILS, effects at this level may differ from the findings that emerged from the bivariate analyses reported in previous chapters.

Multilevel analysis also allows estimation of not only random effects models, where within-school effects vary across schools, but also interaction effects between school-level predictors and the slopes of student-level predictors within schools. However, in these first analyses of ICILS data focused on factors influencing CIL, we estimated all student-level effects as fixed effects that varied little across schools.

When conducting the multilevel analysis of CIL, we estimated three different models:

  • Model 0 (the “null model”), which included no predictor variables other than school intercepts;

  • Model 1, which included, as student-level and school-level predictors, only variables related to ICT;

  • Model 2, which, added to the above variables, reflected the personal and social background of students as well as the average socioeconomic background of schools’ student intakes.

Because Model 0 provided estimates of the variance at each level (within and between schools) before the inclusion of predictors, it established the point from which we could determine how much the subsequent models explained the variance. Model 1 included only those predictors directly related to ICT (resources, familiarity, learning context), while Model 2 provided information about how much of the variance over and above the Model 1 predictors was explained when students’ personal and social backgrounds were taken into account.

Influences on variation in CIL

Student-level influences

Table 8.1 shows the unstandardized regression coefficients for student-level variables from both analysis models for the ICILS 2013 participating countries and benchmarking participants.Footnote 4 The coefficients reflect the effect of each ICT-related factor within schools before and after we controlled for personal and social background. The overall results for countries meeting sample participation requirements in ICILS 2013 should be interpreted with some caution, however, as they reflect average regression coefficients that are only meaningful for factors that have consistently positive or negative effects across countries.

Table 8.1 Student-level results: ICT-related context factors
Full size table

For Model 1, the number of computers at home had statistically significant associations with CIL in about half of the participating education systems. The effects ranged from 3.7 CIL score points (per additional computer) in the Czech Republic to 16.5 such points in Newfoundland and Labrador (Canada). However, after controlling for personal and social background (Model 2), we observed statistically significant effects only in Thailand (with 6.8 CIL score points) and Newfoundland and Labrador (10.6 score points). This outcome seems plausible given that we can expect computer acquisition to be highly correlated with socioeconomic background.

Internet access was positively associated with CIL in a number of countries. In Model 1, this factor was associated with increases in score points ranging from 12 in Turkey to almost 97 in Korea. In all but two countries (Russian Federation and Turkey), the (within-school) effects remained statistically significant after we had controlled for personal and social background (in Model 2).

Years of computer experience was consistently and positively associated with CIL in all but two countries (Germany and Hong Kong SAR). In Model 1, on average across the ICILS countries, one year of additional computer experience was associated with about five CIL score points, with the range extending from 1.5 in the Czech Republic to 8.3 in Poland. Model 2 results show that even after we had controlled for other background variables, the estimated effect was only slightly smaller and remained statistically significant across countries.

In many countries, students’ weekly use of computers at home was also positively associated with CIL. In Model 1, statistically significant effects ranged from 14.3 CIL score points (as the estimated difference between students who used home computers at least weekly and others) in the Russian Federation to 42.3 in Lithuania. These effects remained statistically significant for all countries (with the exception of Poland) after we had controlled for personal and social background factors (in Model 2); in some countries, slightly larger effects were recorded. Weekly use of school computers had statistically significant associations with CIL in only five countries—Croatia, Lithuania, the Russian Federation, Thailand, and Hong Kong SAR. These associations were of similar size in both models.

In Model 1, student reports on having learned about ICT at school had statistically significant positive effects in eight education systems (Australia, Croatia, Korea, Slovenia, Turkey, Hong Kong SAR, and the two Canadian provinces), with the effects ranging in strength from 3.8 CIL score points (per national standard deviation) in Slovenia to 9.8 in Newfoundland and Labrador (Canada). Except for Slovenia, these effects remained statistically significant after we had controlled for personal and social background variables (in Model 2).

School-level influences

Table 8.2 records the effects for ICT-related school-level factors for both models. The availability of ICT resources (as reported by the ICT-coordinators) had a statistically significant effect only in the Russian Federation, an outcome that remained unchanged after we controlled for background variables (in Model 2).

Table 8.2 School-level results: ICT-related factors
Full size table

When estimating Model 1, we found teachers’ perceptions of ICT resource limitations for teaching at their school had statistically significant negative effects on CIL in four countries—Australia, Korea, Poland, and the Russian Federation. The effects ranged from −4.7 CIL points (per national standard deviation) in Australia to −10.2 and −10.3 CIL points respectively in Korea and the Russian Federation. However, these effects remained statistically significant only in Korea after we controlled for schools’ socioeconomic context.

For Model 1, students’ school-based experience with ICT was recorded as a statistically significant predictor in Chile and Turkey only (estimated respectively as effects of 12.3 and 15.8 CIL score points per year of experience). However, these effects were no longer significant in these countries after we had controlled for the socioeconomic background of the student cohort in the school (in Model 2).

In six countries we recorded statistically significant context effects for the percentages of students who said they used computers at home at least once a week. In Model 1, these effects ranged from 1.4 CIL score points (per percentage point) in Australia to 3.3 points in the Slovak Republic. In five of six countries, these effects remained significant after we controlled for personal and social background variables. In Germany, however, the effect was no longer statistically significant.

In Model 1, aggregate scores of the index reflecting student reports on having learned about ICT tasks at school had statistically significant positive effects in four education systems (Australia, Poland, Hong Kong SAR, and Newfoundland and Labrador), and a significant negative effect in the Russian Federation. After controlling for the socioeconomic context of schools in Model 2, we observed statistically positive effects in Australia, Chile, Slovenia, Hong Kong SAR, and Newfoundland and Labrador. This finding suggests that school education related to CIL can affect students’ achievement in this area beyond the influence of the socioeconomic context.

Student-level and school-level background influences

Table 8.3 shows the regression coefficients for indicators of students’ personal and social backgrounds as well as the social context of the schools, as measured by the average index of students’ socioeconomic background. These indicators were included in Model 2 only.

Table 8.3 Student and school-level results: personal and social background
Full size table

Female gender was a statistically significant positive predictor in a majority of countries. On average, after controlling for other variables, we found female students scoring about 12 CIL points higher than male students, with effects ranging from 7.5 in the Czech Republic to 35.7 points in Korea.

Expected educational attainment, which is likely to be associated with previous academic performance as well as parental background, was also significantly associated with CIL in all participating countries. While students who expected to attain educational qualifications no higher than lower-secondary tended to have lower CIL scores than those expecting to complete upper-secondary education (the reference category), students in several countries who expected to gain a post-secondary nonuniversity qualification had significantly higher CIL scores than those expecting to go no further than upper-secondary education.

Expected university education was consistently and significantly associated with CIL. After we had controlled for other factors, we observed that, on average across the ICILS countries, the achievement of students in this category was 36 CIL points higher than the score of students expecting to secure only upper-secondary qualifications. The statistically significant within-school effects ranged from 11.2 points in Germany to 61.0 in Croatia.

Within schools, students’ individual socioeconomic background had statistically significant positive effects in a majority of countries, with the effects ranging from 4.1 score points in the Russian Federation to 12.1 in both Norway and Newfoundland and Labrador (Canada). The average socioeconomic background of schools was also a statistically significant predictor in all but three ICILS countries (Lithuania, the Russian Federation, and Turkey). Statistically significant positive effects ranged from 7.8 score points (per national standard deviation across schools) in Slovenia to almost 40 points in Germany. These results possibly reflect the varying degrees of differentiation across study programs or school types within the different ICILS education systems.

Summary of influences on CIL

Table 8.4 provides a summary of the results from our comparison of the two models. It shows the number of statistically significant positive or negative effects for each indicator in both models. Although the variables reflecting students’ ICT familiarity emerged as statistically significant predictors in many countries in both models, the effects of home ICT resources were often no longer significant once we had taken the social background of families into account. This finding is a plausible one given that families with higher socioeconomic status tend to be in a better position to acquire ICT equipment.

Table 8.4 Summary of statistically significant effects across countries
Full size table

In Model 1, school-level indicators of ICT resources and experience with computers at school had significant effects in only a few countries. After we had controlled for the socioeconomic context, we found that these effects were generally no longer significant, a result which suggests that schools with students from higher income strata tend to be better resourced than schools with students from lower strata. However, this finding does not necessarily mean that resource indicators have no impact on student learning of CIL. Rather, it shows that socioeconomic context is a powerful explanatory variable reflecting a range of conditions (e.g., resources, climate, peer support) that positively influence student learning.

It is interesting to note that, in some countries, student context variables, such as the percentage of students who reported frequent computer use or the percentage of students who said they learned about ICT at school, remained significant predictors after we had controlled for the social context. This finding suggests that what schools teach regarding ICT use has an influence on CIL. As such, the finding is worth further investigation.

Table 8.5 shows the variance estimates for each country overall and at each level. The table also shows the extent to which Model 1 (ICT-related factors) and Model 2 (ICT-related factors and personal/social background factors) explained the variance in CIL scores. This information is displayed as a bar chart in the table. The longer bars reflect larger overall variance. Note that each bar’s position relative to the vertical axis indicates whether more variance was found within schools (left-hand side of the axis) or between schools (right-hand side). Shading with darker colors at each side of the vertical axis indicates how much of the variance Model 1 explained (darkest color) and how much additional variance Model 2 explained (darkest and second-darkest colors). The lighter shaded sections of the bars show the variance that remained unexplained by the models.

Table 8.5 Total and explained variance in computer and information literacy
Full size table

As is evident in Table 8.5, the overall variance explained varied considerably across countries. The proportions of variance between schools (in the fourth column) also varied substantially among countries, from 11 percent in Norway and Slovenia to 53 percent in Germany (with an average of 30 percent and an inter-quartile range of 18 to 38 percent).

In line with results from other international studies of educational achievement, countries with comprehensive education systems, such as Norway, Denmark, and Slovenia, tended to have lower proportions of variance in CIL across schools. The education systems with differentiated provision through distinct study programs, such as Germany and the Slovak Republic, or with higher levels of social segregation, such as Chile, Thailand, and Turkey, recorded higher proportions of CIL variance across schools.

Model 1 explained, on average crossnationally, seven percent of the variance in CIL, with the highest proportion of variance explained (12%) recorded in Croatia. School- level predictors explained 37 percent of the variation in CIL, with the range extending from eight percent in Slovenia to 63 percent in Australia.

After we had controlled for personal and social background as well as schools’ socioeconomic intake, Model 2 explained, on average, 17 percent of the student-level and 58 percent of the school-level variance in CIL. In Australia, Chile, Germany, and Poland, the ICT-related variables and personal and social background factors explained more than two thirds of the variation across schools.

Conclusion

Our results show that students’ experience with computers as well as regular use of computers at home had significant positive effects on CIL achievement in many of the ICILS countries even after we had controlled for the influence of personal and social context. This pattern suggests that familiarity with ICT, reflecting what students do and have done, contributes to students’ CIL achievement.

The availability of ICT resources at home, measured as the number of computers and having access to internet, was associated with CIL achievement. However, ICT resources, in particular the number of computers at home, had hardly any effect after socioeconomic background had been taken into account (although internet access remained significant in five of the 14 countries that satisfied sampling requirements). The probable reason behind this finding is that level of ICT resources in homes is associated with socioeconomic background.

We observed statistically significant effects of ICT-related school-level factors on CIL achievement in only a few countries. In a number of education systems, we recorded evidence of limited effects on CIL of the school average of students’ computer use (at home) and the extent to which students reported learning about ICT-related tasks at school. Because ICILS represents an initial exploration into the influences of school-level and student-level factors on CIL learning, these findings deserve further analysis in future research. The notion that school learning is an important aspect of developing CIL is a particularly important consideration and therefore worth investigating in greater detail.

Some of the effects of ICT-related factors that were no longer significant after we had controlled for the socioeconomic context of school could be considered proxies for other variables (resources, school climate, peer influences). In some countries, these effects may also reflect differences between school types and study programs.