Introduction

Abstract

The International Computer and Information Literacy Study 2013 (ICILS 2013) investigated the ways in which young people develop computer and information literacy (CIL) to support their capacity to participate in the digital age. Computer and information literacy is defined as “an individual’s ability to use computers to investigate, create and communicate in order to participate effectively at home, at school, in the workplace and in society” (Fraillon, Schulz, & Ainley, 2013, p. 17).

The International Computer and Information Literacy Study 2013 (ICILS 2013) investigated the ways in which young people develop computer and information literacy (CIL) to support their capacity to participate in the digital age. Computer and information literacy is defined as “an individual’s ability to use computers to investigate, create and communicate in order to participate effectively at home, at school, in the workplace and in society” (Fraillon, Schulz, & Ainley, 2013, p. 17).

Computer-based assessments of discipline-specific learning (such as reading, mathematics, and science) have viewed the computer as a tool that students use to express their discipline-specific knowledge, understanding, and skills. In contrast, ICILS aimed to measure students’ ability to use computers to gather, manage, and communicate information. The study assessed student CIL achievement through a computer-based assessment administered to students in their eighth year of schooling. It examined differences across countries in student CIL achievement and explored how these differences related to student characteristics and students’ use of computer technologies in and out of school. The study also investigated the home, school, and national contexts in which CIL develops.

Within the context of international comparative research, ICILS is the first study to investigate students’ acquisition of CIL. It is also the first crossnational study commissioned by the International Association for the Evaluation of Educational Achievement (IEA) to collect student achievement data via computer. It is a response to the increasing use of information and communication technology (ICT) in modern society and the need for citizens to develop relevant skills in order to participate effectively in the digital age. The study furthermore addressed the need for policymakers and education systems to have a better understanding of the contexts and outcomes of CIL-related education programs in their countries.

The ICILS research team systematically investigated differences in CIL outcomes across the participating countries. The team also explored how these countries were providing CIL-related education and looked at differences within and across the countries with respect to associations between CIL-education outcomes and student characteristics and school contexts. In addition, participating countries provided detailed information on the national contexts in which their CIL education takes place. This information included policies, resourcing, curriculum, and assessment.

ICILS researchers gathered data from almost 60,000 Grade 8 (or equivalent) students in more than 3,300 schools from 21 countries or education systems within countries. ICILS used purpose-designed software for the computer-based student assessment (and questionnaire), which was administered primarily using USB drives attached to school computers. These student data were augmented by data from almost 35,000 teachers in those schools and by contextual data collected from school ICT-coordinators, principals, and the ICILS national research centers.

Background

Recent decades have witnessed the development and pervasive implementation of computer and other information technologies throughout societies around the world. The use of information technologies is now embedded in societies and in schooling. Information technologies provide the tools for creating, collecting, storing, and using knowledge as well as for communication and collaboration (Kozma, 2003a). The development of these technologies has changed not only the environment in which students develop skills for life but also the basis of many occupations and the ways in which various social transactions take place. Knowing about, understanding, and using information technologies has thus become an important component of life in modern society.

Today, many education systems assess these skills as part of their monitoring of student achievement. Since the late 1980s, this area of education has been a feature of IEA’s international comparative research agenda. IEA’s Computers in Education Study (COMPED), conducted in two stages in 1989 and 1992 (Pelgrum, Reinen, & Plomp, 1993), focused on computer availability and use in schools. It also estimated the impact of school-based computer use on student achievement. Twenty-one education systems participated in Stage 1, and 12 in Stage 2 of the study (Pelgrum & Plomp, 1991).

In 1998/1999, IEA’s Second Information Technology in Education Study (SITES) Module 1 collected data from 27 education systems (Pelgrum & Anderson, 1999). SITES Module 2, a qualitative study based on 174 case studies from 28 countries (Kozma, 2003a) and conducted during 2001/2002, investigated pedagogical innovations that employed information technology. SITES 2006 surveyed the use of ICT by Grade 8 mathematics and science teachers in 22 education systems (Law, Pelgrum, & Plomp, 2008).

The SITES studies also collected information on the resourcing and use of ICT in schools. Module 1 looked at the support on hand for teachers to use ICT in their teaching in schools, Module 2 focused on pedagogical innovations using ICT, and SITES 2006 explored the role of ICT in teaching mathematics and science in Grade 8 classrooms (Kozma, 2003a; Pelgrum & Anderson, 2001).

During the early 2000s, the OECD commissioned a study designed to examine the feasibility of including an ICT literacy assessment as part of its Programme for International Student Assessment (PISA). Although the OECD decided not to include ICT literacy in its suite of PISA assessments, the feasibility study prompted development of a framework for ICT literacy applicable within the crossnational context (Educational Testing Service, 2002). Since then, the OECD has included computer-based assessments of digital reading in its PISA assessments (2009 and 2012), and in 2015 it intends to implement a computer-based assessment of collaborative problem-solving.

The OECD Programme for the International Assessment of Adult Competencies (PIAAC) also includes computer-based assessments of digital reading and problem-solving in technology-rich environments (OECD, 2014a). IEA’s ongoing Trends in International Mathematics and Science Study (TIMSS) and Progress in Reading Literacy Study (PIRLS) investigate the role of ICT use in the learning of mathematics, science, and reading (see, for example, Martin, Mullis, Foy, & Stanco, 2012; Mullis, Martin, Foy, & Arora, 2012; Mullis, Martin, Foy, & Drucker, 2012).

These initiatives over the past 25 years illustrate the interest in crossnational assessment of a range of achievement constructs related to the use of ICT not only by school students but also by adults. In addition, there is a general impetus within and across countries to deliver assessment content on computers rather than on paper as previously. The OECD is currently implementing this practice in its PISA assessments.

IEA’s PIRLS 2016 will include an electronic reading assessment option (ePIRLS) featuring multi-layered digital texts. An assessment of electronic reading such as ePIRLS focuses on reading constructs that we can regard as “building blocks” enabling development of CIL. Such assessments do not, however, address the richness and depth of the CIL construct. ICILS is unique and groundbreaking within international large-scale assessment research not only because of the nature of the achievement construct being measured but also because of the innovative, authentic, computer-based assessment tasks designed to measure students’ CIL.

The importance that ICT-related education and training has for providing citizens with the skills they need to access information and participate in transactions through these technologies is widely recognized worldwide (Kozma, 2008). Evidence of this recognition in recent years can be found in major policy statements, research studies, and other initiatives.

For example, according to the authors of a report on E-learning Nordic, a study that explored the impact of ICT on education in Nordic countries, “ICT is … an essential cultural technique which can significantly improve the quality of education” (Pedersen et al., 2006, p. 114). In 2007, the United Kingdom’s Qualifications and Curriculum Authority positioned ICT as “an essential skill for life and enables learners to participate in a rapidly changing world” (para. 1).

In 2008, under its i2010 strategy, the European Commission reported on 470 digital literacy initiatives in Europe and suggested that digital literacy is “increasingly becoming an essential life competence and the inability to access or use ICT has effectively become a barrier to social integration and personal development” (European Commission, 2008, p. 4). The successor to the i2010 strategy, the Digital Agenda for Europe, included “enhancing digital literacy, inclusion and skills” as one of seven priority areas for action (European Commission, 2013, para 1) and led to the establishment of a conceptual framework for “benchmarking digital Europe” (European Commission, 2009a).

In December 2011, under its Lifelong Learning Programme, the European Commission elucidated the knowledge, skills, and attitudes that people need in order to be deemed digitally competent. The commission had earlier identified digital competence as one of its eight identified key competences in education and argued that this competence goes beyond the use of purely functional ICT skills because it embeds the critical, collaborative, creative use of new technologies for employability and societal inclusion (European Commission, 2006).

As a first step toward developing a digital competence framework, the commission provided an in-depth description of what it perceived to be the various components of digital competence. The description covers 21 subcompetences structured according to five main competences—information management, collaboration, communication and sharing, creation of content, and problem-solving (European Commission Joint Research Center-IPTS, 2013). Each of the 21 subcompetences is briefly defined and accompanied by descriptors of three proficiency levels as well as examples of the requisite knowledge, skills, and attitudes.

European Union (EU) member states were closely involved in the framework’s development, and some have already begun implementing it in national contexts. Work is continuing under Erasmus+, an EU program that focuses on formal and informal learning across EU borders. The next version of EUROPASS, another EU initiative that helps Europeans communicate their qualifications and skills across EU member states, will include a set of questions that learners can use to self-assess their digital competency. By the end of 2014, the three proficiency levels will have been extended to eight in order to correspond with the eight levels of the European Qualification Framework (EUROPASS, 2014).

For Ferrari (2012), digital competence is “both a requirement and a right of citizens, if they are to be functional in today’s society” (p. 3). She identified from her analysis of existing digital competence frameworks, seven key areas of competence: information management, collaboration, communication and sharing, creation of content and knowledge, ethics and responsibility, evaluation and problem-solving, and technical operations.

In 2011, a European Commission study collected data from over 190,000 students, teachers, and head teachers across 27 EU (and four non-EU) countries in Europe. The study investigated “educational technology in schools: from infrastructure provision to use, confidence and attitudes” (European Commission, 2013, p. 9).

The United States has in place widespread and varied policies designed to encourage the use of ICT in schools (Anderson & Dexter, 2009). In endeavoring to shape their curricula and assessments according to the policy directives, states have generally followed the National Educational Technology Standards established by the International Society for Technology in Education (2007). The US National Education Technology Plan implicitly and explicitly exhorts the development of skills that enable participation in the digital age. Goal 1.1 of the plan stresses that, regardless of the learning domain, “states should continue to consider the integration of 21st-century competencies and expertise, such as critical thinking, complex problem solving, collaboration, multimedia communication, and technological competencies demonstrated by professionals in various disciplines” (Office of Educational Technology, US Department of Education, 2010, p. xvi).

In the United States, the start of the 2014/2015 school year marked inclusion of an assessment of technology competency (which has ICT as one of its three areas) in the country’s Assessment of Educational Progress (WestEd, 2010). The assessment covers proficiency with computers and software learning tools, networking systems and protocols, hand-held digital devices, and other technologies that enable users to access, create, and communicate information and engage in creative expression. The assessment also identifies five subareas of competence: construction and exchange of ideas and solutions, information research, investigation of problems, acknowledgement of ideas and information, and selection and use of digital tools (Institute of Education Sciences, National Center for Education Statistics, 2012).

Over recent years, a number of countries in Latin America have increased their focus on the use of ICT in classrooms and also introduced one computer to every student in schools (commonly referred to as one-to-one resourcing). Argentina, Brazil, Chile, Peru, and Uruguay are some of the countries that have implemented one-to-one computer policies (see, for example, Ministry of Education of the City of Buenos Aires, 2013; Ministry of Education of Uruguay, 2013; Severin & Capota, 2011; Severin, Santiago, Ibarrarán, Thompson, & Cueto, 2011).

One-to-one resourcing is also evident in Thailand. In line with its one tablet computer per child program, the government distributed over 800,000 tablet computers to Grade 1 students in 2012. The computers were preloaded with content for the core subjects of science, mathematics, social studies, Thai, and English (UNESCO, 2013).

As early as 1996, Korea established a comprehensive plan for education informatization. The republic has since conducted an ongoing four-phased implementation process: deployment of infrastructure and resources, promotion of ICT use and e-learning, transitioning from e-learning to ubiquitous learning (u-learning), and development of ICT-based creative human resources (Korea Education and Research Information Service, 2013).

Despite increasing international recognition of the importance of ICT-related literacies (Blurton, 1999; Kozma, 2003a), there is considerable variation among (and even within) countries with regard to explicit ICT curricula, resources, and teaching approaches (Educational Testing Service, 2002; Kozma, 2008; OECD, 2005; Sturman & Sizmur, 2011). In addition to questions stemming from the variety of approaches in which ICT curricula are conceptualized and delivered, there are questions about the nature of the role that schools and education systems play in supporting the development of ICT-related literacies among young people.

In some countries, young people claim that they learn more about using computers out of school than they do in school (see, for example, Thomson & De Bortoli, 2007), while adults regard the new generation of young people as “digital natives” (Prensky, 2001) who have developed “sophisticated knowledge of and skills with information technologies” as well as learning styles that differ from those of previous generations (Bennett, Maton, & Kervin, 2008, p. 777).

However, various commentators express concern about the value of labeling the new generation this way. They challenge, in particular, assumptions about the knowledge and skills that these assumed digital natives acquire (see, for example, van den Beemt, 2010). In addition to identifying and discussing the “myths” associated with the notion of digital native, Koutropoulos (2011, p. 531) questions assumptions of homogeneity and pervasiveness, arguing that if we look “at the research … we see that there is no one, monolithic group that we can point to and say that those are digital natives. As a matter of fact, the individuals who would fit the stereotype of the digital native appear to be in the minority of the population” (para 36, emphasis original).

Questions are also being raised about the types of ICT use and consequent learning that young people experience, especially when they are away from school. Some scholars query if young people are indeed developing through their ICT use the types of ICT-related knowledge, skills, and understandings that can be of significant value in later life. Crook (2008) characterizes the majority of young people’s communicative exchanges as “low bandwidth,” where the focus is on role allocation and cooperation rather than on genuine collaboration. Selwyn (2009) similarly challenges suppositions about the quality and value of much of young people’s self-directed ICT learning, observing that “if anything young people’s use of the internet can be described most accurately as involving the passive consumption of knowledge rather than the active creation of content” (p. 372).

Today, the research community and policymakers continue to grapple with issues revolving around the development of digital literacies in young people. Although there is consistent rhetoric about the value of emergent digital literacies in providing positive life outcomes, just how school education can and should contribute to this process remains unclear. For ICILS, a primary aim has been to bring greater clarity to these matters through the study’s systematic investigation of CIL in young people and the ways in which this form of literacy is developed.

Research questions

The research questions underpinning ICILS concern students’ acquisition of CIL. The publication elaborating the ICILS assessment framework (Fraillon et al., 2013) describes the development of and provides additional details pertinent to these questions. The publication also outlines the variables that researchers need to consider when conducting analyses of data relevant to the questions.

• RQ 1: What variations exist between countries, and within countries, in student computer and information literacy?

  This research question concerns the distribution of CIL outcomes across participating countries (at the country level) and within these countries. Analyses that address this question focus on the distribution of CIL test data and involve single- and multi-level perspectives.

• RQ 2: What aspects of schools and education systems are related to student achievement in computer and information literacy with respect to the following subquestions?

  • (a) The general approach to computer and information literacy education.

    ICILS collected data at the national level on curriculum and programs as well as at the school level through teacher, ICT-coordinator, and principal questionnaires. Analyses of these data also took into account contextual information about CIL-related learning at the country level as well as more detailed information from schools and classrooms.

  • (b) School and teaching practices regarding the use of technologies in computer and information literacy.

    ICILS collected information from schools, teachers, and students in order to ascertain student perceptions of and teacher reports on instructional practices regarding CIL-related teaching and learning processes.

  • (c) Teacher attitudes to and proficiency in using computers.

    Teachers reported on their experiences of, attitudes toward, and confidence in using computers. They also reported on their use of computers as tools to support their teaching of content related to their own main subject and with respect to aspects of CIL.

  • (d) Access to ICT in schools.

    Students, teachers, ICT-coordinators, and principals reported on their use of and access to ICT in schools.

  • (e) Teacher professional development and within-school delivery of computer and information literacy programs.

    Teachers, ICT-coordinators, and principals reported on teachers’ access to and use of a range of professional learning opportunities.

• RQ 3: What characteristics of students’ levels of access to, familiarity with, and self-reported proficiency in using computers are related to student achievement in computer and information literacy?

  • (a) How do these characteristics differ among and within countries?

    ICILS collected information from students on how long they had been using computers and how often they used computers for a range of recreational and school-related purposes. Information was also sought on student confidence in completing a range of tasks on computer. These data were collected in order to enable descriptions of students’ use of computers and were analyzed with respect to their associations with students’ CIL.

  • (b) To what extent do the strengths of the associations between these characteristics and measured computer and information literacy differ among countries?

    ICILS conducted analyses directed toward determining associations between student access to, familiarity with, and self-reported proficiency in using computers and computer and information literacy within and across countries.

• RQ 4: What aspects of students’ personal and social backgrounds (such as gender, socioeconomic background, and language background) are related to computer and information literacy?

  ICILS examined information about student background and home environment in an effort to explain variation in student’s CIL. The instrument used to gather this information was the student questionnaire.

Participating countries, population, and sample design

Twenty-one countries¹ participated in ICILS. They were Australia, the City of Buenos Aires (Argentina), Chile, Croatia, the Czech Republic, Denmark, Germany, Hong Kong SAR, Korea, Lithuania, the Netherlands, Norway (Grade 9), Newfoundland and Labrador (Canada), Ontario (Canada), Poland, the Russian Federation, the Slovak Republic, Slovenia, Switzerland, Thailand, and Turkey. Three of these education systems—the City of Buenos Aires (Argentina), Newfoundland and Labrador (Canada), and Ontario (Canada)—took part as benchmarking participants.

Population definitions

The ICILS student population was defined as students in Grade 8 (typically around 14 years of age in most countries), provided that the average age of students in this grade was at least 13.5 at the time of the assessment. If the average age of students in Grade 8 was below 13.5 years, Grade 9 became the target population.

The population for the ICILS teacher survey was defined as all teachers teaching regular school subjects to the students in the target grade at each sampled school. It included only those teachers who were teaching the target grade during the testing period and who had been employed at school since the beginning of the school year. ICILS also administered separate questionnaires to principals and nominated ICT-coordinators in each school.

Sample design

The samples were designed as two-stage cluster samples. During the first stage of sampling, PPS procedures (probability proportional to size as measured by the number of students enrolled in a school) were used to sample schools within each country. The numbers required in the sample to achieve the necessary precision were estimated on the basis of national characteristics. However, as a guide, each country was instructed to plan for a minimum sample size of 150 schools. The sampling of schools constituted the first stage of sampling both students and teachers.

The sample of schools ranged in number between 138 and 318 across countries. Twenty students were then randomly sampled from all students enrolled in the target grade in each sampled school. In schools with fewer than 20 students, all students were invited to participate. Appendix A of this report documents the achieved samples for each country.

Up to 15 teachers were selected at random from all teachers teaching the target grade at each sampled school. In schools with 20 or fewer such teachers, all teachers were invited to participate. In schools with 21 or more such teachers, 15 teachers were sampled at random. Because of the intention that teacher information should not be linked to individual students, all teachers of the target grade were eligible to be sampled regardless of the subjects they taught.

The participation rates required for each country were 85 percent of the selected schools and 85 percent of the selected students within the participating schools, or a weighted overall participation rate of 75 percent. The same criteria were applied to the teacher sample, but the coverage was judged independently of the student sample. In the tables in this report, we use annotations to identify those countries that met these response rates only after the inclusion of replacement schools. Education systems that took part as benchmarking participants also appear in a separate section of the tables in this report. Countries or benchmarking participants that did not meet the response rates, even after replacement, are also reported separately, in this instance below the main section of each table.

The ICILS assessment framework

The assessment framework provided the conceptual underpinning of the ICILS international instrumentation (Fraillon et al., 2013). The assessment framework has two parts:

• (1) The computer and information literacy framework: This outlines the outcome measures addressed through the student achievement test.

• (2) The contextual framework: This maps the context factors potentially influencing CIL and explaining variation.

The computer and information literacy framework

The CIL construct has two elements:

• (1) Strand: This refers to the overarching conceptual category used to frame the skills and knowledge addressed by the CIL instruments.

• (2) Aspect: This refers to the specific content category within a strand.

Strand 1 of the framework, collecting and managing information, focuses on the receptive and organizational elements of information processing and management and consists of the following three aspects:

• (a) Knowing about and understanding computer use refers to a person’s declarative and procedural knowledge of the generic characteristics and functions of computers. It focuses on the basic technical knowledge and skills he or she needs in order to use computers to work with information.

• (b) Accessing and evaluating information refers to the investigative processes that enable a person to find, retrieve, and make judgments about the relevance, integrity, and usefulness of computer-based information.

• (c) Managing information refers to individuals’ capacity to work with computer-based information. The process includes ability to adopt and adapt information classification and organization schemes in order to arrange and store information so that it can be used or reused efficiently.

Strand 2 of the framework, producing and exchanging information, focuses on using computers as productive tools for thinking, creating, and communicating. The strand has four aspects:

• (a) Transforming information refers to a person’s ability to use computers to change how information is presented so that it is clearer for specific audiences and purposes.

• (b) Creating information refers to a person’s ability to use computers to design and generate information products for specified purposes and audiences. These original products may be entirely new or may build upon a given set of information and thereby generate new understandings.

• (c) Sharing information refers to a person’s understanding of how computers are and can be used as well as his or her ability to use computers to communicate and exchange information with others.

• (d) Using information safely and securely refers to a person’s understanding of the legal and ethical issues of computer-based communication from the perspectives of both the generator and the consumer of that information.

A detailed discussion of the contents of each of the strands and aspects of the computer and information literacy framework can be found in the IEA publication detailing the ICILS assessment framework (Fraillon et al., 2013).

The ICILS contextual framework

When studying student outcomes related to CIL, it is important to set these in the context of the different influences on CIL development. Students acquire competence in this area through a variety of activities and experiences at the different levels of their education and through different processes in school and out of school. It is also likely, as Ainley, Enger, and Searle (2009) argue, that students’ out-of-school experiences of using ICT influence their learning approaches in school. Contextual variables can also be classified according to their measurement characteristics, namely, factual (e.g., age), attitudinal (e.g., enjoyment of computer use), and behavioral (e.g., frequency of computer use).

Different conceptual frameworks for analyzing educational outcomes frequently point out the multilevel structure inherent in the processes that influence student learning (see, for example, Scheerens, 1990; Scheerens & Bosker, 1997; Schulz, Fraillon, Ainley, Losito, & Kerr, 2008; Travers, Garden, & Rosier, 1989; Travers & Westbury, 1989). The learning of individual students is set in the overlapping contexts of school learning and out-of-school learning, both of which are embedded in the context of the wider community that comprises local, national, supranational, and international contexts. The contextual framework of ICILS therefore distinguishes the following levels:

• The individual: This context includes the characteristics of the learner, the processes of learning, and the learner’s level of CIL.

• Home environment: This context relates to a student’s background characteristics, especially in terms of the learning processes associated with family, home, and other immediate out-of-school contexts.

• Schools and classrooms: This context encompasses all school-related factors. Given the crosscurricular nature of CIL learning, distinguishing between classroom level and school level is not useful.

• Wider community: This level describes the wider context in which CIL learning takes places. It comprises local community contexts (e.g., remoteness and access to internet facilities) as well as characteristics of the education system and country. It also encompasses the global context, a factor widely enhanced by access to the world wide web.

The status of contextual factors within the learning process is also important. Factors can be classified as either antecedents or processes:

• Antecedents are exogenous factors that condition the ways in which CIL learning takes place and are therefore not directly influenced by learning-process variables or outcomes. It is important to recognize that antecedent variables are level-specific and may be influenced by antecedents and processes found at higher levels. Variables such as the socioeconomic status of the student’s family and the school intake along with home resources fall into this category.

• Processes are those factors that directly influence CIL learning. They are constrained by antecedent factors and factors found at higher levels. This category contains variables such as opportunities for CIL learning during class, teacher attitudes toward using ICT for study tasks, and students’ use of computers at home.

Both antecedents and processes need to be taken into account when explaining variation in CIL learning outcomes. Whereas antecedent factors shape and constrain the development of CIL, the level of (existing) CIL learning can influence process factors. For example, the level and scope of classroom exercises using ICT generally depend on students’ existing CIL-related proficiency.

Figure 1.1 illustrates this basic classification of antecedent and process-related contextual factors and their relationship with CIL outcomes located at the different levels. Examples of variables that have the potential to influence learning processes and outcomes accompany each type of factor at each level. The double arrow in the figure between the process-related factors and outcomes emphasizes the possibility of feedback between learning process and learning outcome. The single-headed arrow between antecedents and processes, in turn, indicates the assumption within the ICILS contextual framework of a unidirectional association at each contextual level.

Figure 1.1

Contexts for CIL learning and learning outcomes

Reference to this general conceptual framework enables us to locate potential contextual factors on a two-by-four grid where antecedents and processes constitute the columns and the four levels the rows. Table 1.1 shows examples in each of these cells of the contextual variables collected by the ICILS instruments. The student questionnaire collected data on contextual factors pertaining to the level of the individual student and his or her home context. The teacher, school principal, and ICT-coordinator questionnaires were designed to locate contextual factors associated with the school/classroom level, while the national contexts survey and other available sources (e.g., published statistics) were used to gather contextual data at the level of the wider community.

Table 1.1 Mapping of ICILS context variables to framework grid

The wider community level

Contextual levels and variables

The different levels of this context all have the potential to affect student learning at school or at home. Conceptually, this context has several levels:

• Local communities, where remoteness and lack of stable and fast internet connections may affect conditions for ICT use;

• Regional and national contexts, where communication infrastructure, educational structures, curricula, and general economic/social factors may be of importance; and

• Supranational or even international contexts, where a long-term perspective brings in, for example, factors such as the general advance of ICT globally.

ICILS collected information about the contexts of education systems from published sources as well as through the national contexts survey. Typically, the published sources provided information about antecedent country-context variables while the national contexts survey delivered data on antecedent and process variables at the level of and with respect to the education system. The national contexts survey collected data on, for example, the following:

• Education policy and practice in CIL education (including curriculum approaches to CIL);

• Policies and practices for developing teachers’ CIL expertise; and

• Current debates on and reforms to the implementation of digital technology in schools (including approaches to the assessment of CIL and the provision of ICT resources in schools).

Antecedent variables

International comparative research shows relatively strong associations between the general socioeconomic development of countries and student learning outcomes. ICILS therefore selected national and, where appropriate, subnational indicators related to general human development status regularly reported by the United Nations Development Programme (UNDP, 2009). The range of data relating to human development and ICT infrastructure that ICILS collected included measures of mobile phone and broadband connectivity, economic development (such as gross domestic product, income distribution, percentage of public expenditure on education), and ICT development. The latter drew on the ICT Development Index (IDI), which combines 11 indicators into a single measure that can be used as an index of ICT development in 154 countries. Alternatively, each indicator can be used separately.

Data on a range of other wider-community characteristics of the education systems participating in ICILS were also collected. System-level variables related to this aspect include length of schooling, age-grade profiles, educational finance, and structure of school education (e.g., study programs, public/private management), as well as the autonomy of educational providers.

The national (system) level

Process-related variables

The process-related variables on CIL-related education policy collected by the national contexts survey included:

• The definition of and the priority that each country gives to CIL education in its educational policy and provision;

• The name and national or official definition given to CIL education;

• The place of CIL education in educational reforms;

• The main aims and goals of CIL education; and

• The influence of different institutions or groups on decisions relating to these goals and aims.

Because the ICILS contextual framework references policies and practices developed as outcomes of earlier large-scale surveys of ICT in education, ICILS also considered process-related data in these studies’ reports and databases. The studies examined included IEA SITES (Plomp, Anderson, Law, & Quale, 2009), the European Commission’s Indicators of ICT in Primary and Secondary Education (European Commission, 2009b), and the International Experiences with Technology in Education survey, which covered policies and experiences in 21 countries (Bakia, Murphy, Anderson, & Trinidad, 2011).

The ICILS national contexts survey was used to collect data on:

• The model for including CIL education in the curriculum (i.e., as a separate subject, integrated into different subjects, or crosscurricular);

• The nomenclature for CIL-related curriculum subjects and whether they were compulsory or optional in each program of study; and

• The extent of emphasis in the curriculum on and the amount of instructional time given to CIL education at the target grade.

Another important process-related variable at the system level is the development of teacher expertise in CIL (Charalambos & Glass, 2007; Law et al., 2008). Teacher education programs often provide aspiring teachers with opportunities to develop CIL-related competencies. In ICILS, the national contexts survey and, where appropriate, the teacher, ICT-coordinator, and principal questionnaires were used to collect data on:

• The requirements for becoming a teacher;

• Licensing or certification procedures for teachers;

• The backgrounds of CIL teachers (as a definable class of teacher);

• The extent to which CIL education is part of preservice or initial teacher education;

• The availability of inservice or continuing professional development for CIL education;

• The personnel providing these professional learning activities; and

• The expectations for teachers’ ongoing learning about developments in CIL education.

School/classroom level

Any study of students’ acquisition of CIL must acknowledge the key role of school and classroom contexts in that acquisition. ICT use is becoming standard practice in education and employment. Helping students gain CIL is therefore an increasingly important part of the work that schools do to prepare young people for participation in modern society.

Factors associated with the school and classroom context were collected through the teacher, school principal, and ICT-coordinator questionnaires. The student questionnaire also included several questions gauging student perceptions about classroom practices related to ICT. Although ICILS did not attempt to investigate the relationship between ICT use in schools or classrooms and achievement in academic learning areas such as language, mathematics, and science, there is suggestion of positive associations in the results of a meta-analysis conducted by Tamin, Bernard, Borokhovski, Abrami, and Schmid (2011).

Antecedent variables

In line with the need to take school characteristics into account when investigating variations in CIL, the questionnaire given to each school principal collected information on student enrolment, teachers, the range of grades, and the location of each participating school. This questionnaire also collected information relating to school management (public or private), including details on who held responsibility for acquiring ICT resources.

The SITES 2006 findings indicated that school principals’ views about the pedagogical value of ICT, as well as the ICT-related support teachers had at hand, influenced science teachers’ and mathematics teachers’ ICT use (Law et al., 2008). Findings also indicated that ICT-related teaching and learning was constrained or facilitated by the school’s stated curriculum and its policies with regard to ICT. The ICILS principal questionnaire therefore collected data on the following factors:

• The extent to which the school had policies and procedures relating to ICT use;

• The extent to which the school prioritized ICT acquisition and resourcing;

• The principal’s perception of the importance ascribed to ICT use in teaching at the school;

• The school-level expectations for teachers’ knowledge of and skills in using ICT; and

• The extent to which teachers were participating in ICT-related professional development.

The ICILS questionnaire for each school’s ICT-coordinator included questions on the availability of school-owned computing devices at school, their location within the school, how many students had access to them, which computer operating system the school mainly used, and the number of years the school had been using ICT. The instrument also collected data on the support (in terms of personnel and technology or software resources) the school provided for ICT use in teaching and learning. An additional question measured the coordinator’s perceptions of the adequacy of the ICT on hand for learning and teaching at school.

Teachers’ backgrounds and experiences have the potential to influence the acquisition of student CIL. Results from SITES 2006 indicated that teachers were more likely to use ICT in their teaching when they had higher levels of self-confidence in using ICT in general (Law et al., 2008). SITES 2006 also indicated that, in most of the participating countries, ICT was more frequently used in science teaching than in mathematics teaching.

The ICILS teacher questionnaire therefore included questions on the general professional background of teaching staff (such as age, gender, subject taught at school) and on their ICT experience (number of years using ICT for teaching purposes, general use of computers at different locations, participation in ICT-related professional development activities, and perceived self-confidence in using ICT for different tasks). Teachers were also asked to give their views on the positive and negative consequences of using ICT for teaching and learning, and to identify any factors that they thought impeded using ICT for teaching and learning at their school.

Process-related variables

Researchers and commentators have for some time seen ICT in school education as having the potential to influence teaching and learning processes by enabling wider access to a range of resources, allowing greater power to analyze and transform information, and providing enhanced capacities to present information in different forms. However, some scholars have questioned the degree to which the ideal of ICT use in education has been reflected in classroom practice. Burbules (2007), for example, has argued that although e-learning technologies have the potential to bring transformative effects to classrooms, their implementation has been, for various reasons, surprisingly limited (see also Cuban, 2001).

In order to collect data on specific ICT-related teaching practices, the teachers participating in ICILS were asked to consider one of their classes (specified in the questionnaire) and to identify (where applicable) the types of ICT applications used in that class, the type of and extent to which ICT was used as part of teaching practices and for particular learning activities in that class, and the emphasis placed on developing ICT-based student capabilities. The questionnaire also asked teachers to give their perceptions of whether and how ICT was being used as part of collaborative teaching and learning at their school.

Actual student use of ICT in the learning process is another important factor. A segment of the teacher questionnaire therefore asked teachers to report on student involvement in different learning activities involving ICT use. The student questionnaire also asked students to report on how often they used computers at school, their use of computers for different school-related purposes, and the frequency with which they used ICT in their learning of different subjects.

Home level

Antecedent variables

ICILS collected data from students relating to a range of home background factors known from academic literature to relate to student learning outcomes in general and of specific relevance to consideration of CIL-related learning. These factors included:

• Parental (and student) socioeconomic status, measured through parental occupational status (Ganzeboom, de Graaf, & Treiman, 1992);

• Parental educational attainment;

• Home literacy resources;

• Language used at home;

• Whether or not students and their parents had an immigrant background; and

• Student access at home to digital resources, such as computers and other ICT devices.

Process-related variables

Home environment factors that potentially influence the learning process include the use of ICT in the home context and learning through interaction with family members. The student questionnaire therefore included questions about the extent to which students had learned about different aspects of ICT use from family and/or friends and how often they used computers at home in general.

Individual level

Antecedent variables

Antecedent variables at the level of the individual student consist of basic background characteristics that may influence students’ CIL-related knowledge and skills. In this category, students provided data on their age, gender, and educational aspirations (i.e., the highest level of education they expected to complete).

Process-related variables

Applying ICT for different purposes on a regular basis has considerable potential to increase knowledge and skills in this area (see, for example, Australian Curriculum, Assessment and Reporting Authority, 2012; Fletcher, Schaffhauser, & Levin, 2012). The ICILS student questionnaire consequently contained questions about the frequency with which students used different ICT applications outside of school. This usage included using the internet for social communication and using ICT for recreational activities.

The student questionnaire also included items designed to measure the extent to which students were confident in completing a range of ICT-related tasks. According to Bandura (1993), students’ confidence in their ability to carry out specific tasks in an area (self-efficacy) is strongly associated with their performance as well as their perseverance, emotions, and later study or career choices. Moos and Azevedo (2009) concluded from their review of research on computer self-efficacy that this variable plays an integral role in learning in computer-based learning environments.

The ICILS student questionnaire also collected information on students’ enjoyment of using computers to complete tasks and on their ICT self-concept, both of which reflect their perceptions of their ability to cope with a certain learning area (Branden, 1994; Marsh & Shavelson, 1985). Scholars have found associations between both factors and students’ effective use of ICT (see, for example, Dede, Ketelhut, Clarke, Nelson, and Bowman, 2005; OECD, 2005; Pekrun, Goetz, Titz, & Perry, 2002).

Data collection and ICILS instruments

The main survey data collection took place in the 21 participating countries between February and December 2013. Countries with a Northern Hemisphere school calendar completed the survey between February and June 2013; those with a Southern Hemisphere school calendar between October and December 2013. ICILS used six instruments to collect data: two for students, one for teachers, one for school ICT-coordinators, one for school principals, and one for staff in the study’s national research centers.

The student instruments were delivered using purpose-designed software administered primarily via USB drives attached to school computers. In some cases, sets of notebook computers were provided to schools for the assessment. The software could have been delivered via the internet, but the USB delivery ensured a uniform assessment environment for students regardless of the quality of internet connections in participating schools. After administration of the student instruments, data were either uploaded to a server or delivered on the USB drives to national research centers.

The two student instruments were:

• The international student test of computer and information literacy: This consisted of questions and tasks presented in four 30-minute modules. A module was a set of questions and tasks based on a real-life theme and following a linear narrative structure. Each module had a series of small discrete tasks (each of which typically took less than a minute to complete) followed by a large task that typically took 15 to 20 minutes to complete. Each student completed two modules randomly allocated from the set of four. In total, the modules comprised 62 tasks and questions corresponding to 81 score points.

• A 30-minute international student questionnaire: This included questions relating to students’ background characteristics, their experience of and use of computers and ICT to complete a range of different tasks in school and out of school, and their attitudes toward using computers and other forms of ICT.

The three instruments designed to gather information from and about teachers and schools could be completed on computer (over the internet) or on paper, depending on the availability of resources in schools and countries. These instruments were:

• A 30-minute teacher questionnaire: This asked some basic background questions followed by questions relating to teachers’ reported use of ICT in teaching, their attitudes about the use of ICT in teaching, and their participation in professional learning activities relating to using ICT in teaching.

• A 10-minute ICT-coordinator questionnaire: This asked ICT-coordinators about the resources available in the school to support the use of ICT in teaching and learning. The questionnaire addressed both technological (e.g., infrastructure, hardware, software) as well as pedagogical support (e.g., through professional development learning).

• A 10-minute principal questionnaire: Principals provided information about school characteristics and school approaches to providing CIL-related teaching as well as about incorporating ICT in teaching and learning.

ICILS national research coordinators (NRCs) coordinated information procured from national experts in response to an online national contexts survey. This information concerned the structure of the country’s education system, the presence and nature of CIL-related education in national curricula, and recent developments in CIL-related education.

The ICILS instruments were developed in three phases:

• Phase 1 encompassed writing the test and questionnaire items. This work was guided by the ICILS assessment framework. Before developing the tasks and items in detail, writers consulted with NRCs in order to reach agreement on module concepts. Instrument development also included extensive consultation with the study’s national project coordinators and expert consultants.

• Phase 2 saw the instruments field trialed in all participating countries. Subsequent analysis of the collected data informed judgments about the suitability of the contents of each instrument for inclusion in the ICILS main survey data collection.

• Phase 3 included a final revision of the instruments in light of the field trial results and further feedback from national centers and expert consultants.

Given the importance of ensuring comparability and appropriateness of the measures in this study across the diverse range of participating countries, the ICILS field trial test and questionnaire data underwent a thorough review of crossnational validity.²

Report context and scope

This report presents the outcomes of the analyses of data collected across the 21 countries participating in the ICILS main survey in 2013. All data are reported at the international level.

Our aim in this report is to provide overarching international perspectives on the ICILS data relative to the ICILS research questions. Another aim is to provide researchers with observations and questions that may provide the catalyst for further investigation into CIL education within and across countries.

In addition to this current chapter, the report has eight others.

• Chapter 2 describes the national contexts for CIL education in ICILS countries. Here we address common patterns as well as policies, curriculum, resources, and practices in specific countries and groups of countries.

• In Chapter 3, we report on the levels of CIL proficiency across countries. We describe how the ICILS student test was used to measure CIL and present the ICILS scale of CIL proficiency. We also document variance in student achievement scores on the CIL scale across the participating countries.

• Chapter 4 focuses on the associations between aspects of student background and CIL. Also included is the contribution of aspects of student background to variations in CIL achievement.

• In Chapter 5, we draw on student questionnaire data to explore students’ use of and engagement with ICT. Throughout the chapter, standardized scale indices are used to report students’ use of and attitudes toward using ICT for a range of purposes. Gender-based differences in this regard and in terms of CIL achievement are also reported, and associations between individual and home characteristics with CIL achievement are identified.

• Our focus in Chapter 6 is on the roles of schools in CIL education. The data pertinent to this chapter derive mainly from the teacher, ICT-coordinator, and principal questionnaires. The chapter also describes variation in approaches to providing CIL-related education in schools.

• In Chapter 7, we examine the roles of teachers with respect to CIL education. We also use data from the teacher questionnaire to detail teachers’ use of and attitudes toward the use of ICT in their teaching.

• Chapter 8 presents the outcomes of the multivariate and multilevel models that we used to explain variations in CIL within countries.

• Chapter 9 summarizes and discusses the results of ICILS. We also provide in this final chapter a summary of the main findings emerging from ICILS in relation to the research questions and discuss the possible implications of these for policy and practice.