1 Introduction

SITES2006 is part of a series of studies the IEA is conducting in this area. Following the first Computers in Education (CompEd) study, the IEA decided in the late 1990s to start the Second Information Technology in Education Studies (SITESFootnote 1) consisting of a number of phases or modules.

The central theme of the SITES studies is to foster our understanding of how Information Technologies are affecting the way students learn in schools. A first major issue that is being addressed concerns the question whether there is any evidence that the objectives of education and/or educational practices are changing. From statistical indicators as well as policy analyses that were conducted during SITES-M1 (Pelgrum & Anderson, 2001) it appeared that in many countries policy actions were undertaken to initiate ICT-supported pedagogical reforms in schools, but that the implementation of ICT in the teaching–learning (T/L) processes was not widespread. In SITES-M2 more than 170 qualitative case studies were conducted to study in-depth examples of ICT supported pedagogical practices in order to investigate what were the characteristics of these innovations (Kozma, 2003). From the SITES results as well as extensive additional policy document analyses (Plomp, Anderson, Law, & Quale 2003) one may conclude that in quite a number of educational systems throughout the world a need is felt for integrating ICT in education in order to implement pedagogical changes related to changes in our societies. This need may be induced by different reasons, such as:

  • ➢ The expectation that in the Information Society the shelf-life of knowledge is so short that all citizens should be life-long learners who feel responsible for keeping their knowledge and skills up to date and who are able to use IT to access the latest know-how in a particular domain

  • ➢ The experience of many teachers who are early innovators that the use of ICT is motivating for the students as well as for the teachers themselves

  • ➢ The argument that in business and industry teamwork (often on distance) and active, independent knowledge and skills acquisition is increasingly becoming important and that ICT is a facilitating tool for realizing these objectives.

This reasoning is very much consonant with research on constructivist approaches for teaching and learning.

Policy makers throughout the world have initiated in recent years actions to promote the use of ICT in their national education systems, for example by:

  • ➢ Changing the curriculum (Singapore, The Netherlands)

  • ➢ Equipping schools with computers and network connections

  • ➢ Creating national web sites or even international ones (e.g., European Schoolnet) where teachers can find source materials for web based learning and/or cooperation with other colleagues

  • ➢ Training teachers to use ICT and change their pedagogy

  • ➢ Creating data bases with exemplary best practice materials

A major overarching policy evaluation question after a roughly 10-year period of investments is ‘What has been the impact of all investments, and how can we measure this impact’? Major derived questions are ‘Are pedagogical practices in schools and classrooms indeed changing?’ and ‘To what extent and how is ICT used in educational practice and how does it support and enhance pedagogical practices?’

Obviously, these questions can be investigated within a national educational system, which is and has been done in many countries. Although national research can show whether the education system is developing in the desired direction, it is difficult to evaluate the rate of development as long as reference data (or benchmarks) are not available.

Major aims of SITES2006 are to provide international benchmarks of (1) how in the information society pedagogical practices are changing, (2) the extent to which ICT is used in education and (3) how the use of ICT is associated with (changing) pedagogical practices. Another aim of the study is, building upon the large number of case studies of innovative pedagogical practices supported by ICT, to investigate which factors are associated with the use of ICT and the nature of pedagogical practices found in schools and among teachers. In doing so, the outcomes will assist national policy-makers to make informed judgments about developments in their national education system as compared to other countries.

In the remaining of this paper we will present a conceptual analysis of the policy questions mentioned above resulting in a conceptual framework and the major research questions that will be addressed in SITES2006. This will be followed by a summary of the research design and planning.

2 Conceptual analysis–Conceptual framework

Many teachers feel the pressure to ‘do something with ICT’, because in many societies a common belief exists that students, wherever possible, need to be able to work in an environment where digital devices are a normal component of every situation in which one is learning and/or working. Statistics that were and are collected in (international) assessments regarding ICT in educationFootnote 2 have shown that albeit the use of ICT in education is increasing, for the majority of teachers this is still a tool that is used in the margins of the educational process.

Researchers have continuously re-iterated that the integration of the full functionalities of ICT in education is a complex task, which can only succeed if a number of conditions are fulfilled simultaneously. The use of ICT in teaching and learningFootnote 3 in schools is influenced by a number of factors external to the school, such as the visions and prescriptions that are reflected in (national) curriculum and national policies on (ICT in) education. But how ICT is used for teaching and learning depends within schools amongst others on factors such as (1) vision on education or the teaching/learning process (both on school and teacher level), (2) knowledge and skills of the teaching staff, (3) availability of content (educational software) and (4) the hardware infrastructure. These factors can be elaborated both at school and teacher level, and for each school and each teacher, the ‘balance’ between these factors determines not only the options and freedom of choice that are available within the context of the school but also their choice of pedagogical approach and the use of ICT in the teaching/learning processes for which they are responsible.

In order to assess the use of ICT in pedagogical practices it is necessary to elaborate the concepts ‘teaching and learning’ as well as ‘use of ICT’ and define how these concepts are related. Moreover we need to elaborate concepts such as vision on education, teachers’ knowledge and skills of ICT, the availability of content and the hardware infrastructure. Although the last two concepts have been operationalized in various studies, recently in e.g., SITES-M1 and various national studies (e.g. in the USA, England, Norway, The Netherlands), the operationalization of the first two has been undertaken in a limited number of large-scale surveys which SITES2006 can use as preliminary starting point, such as the ones organized by the Dutch Foundation for ICT at school.

The concepts discussed above are summarized in a preliminary conceptual framework (that will be elaborated later on by the participants of SITES2006) that will serve as the basis for concept and indicator development. From Fig. 1 one may infer that the main focus for SITES2006 is on the primary process of teaching and learning and that contextual factors are seen as conditional and explanatory with regard to variations that will be observed between schools and countries regarding these primary processes. Such conditions may be conceived as indicators of the readiness of schools and teachers to innovate pedagogical practices and to use ICT in them.

Fig. 1
figure 1

Initial conceptual framework for SITES2006

Full size image

The framework visualizes that teaching and learning at classroom level and the use of ICT in it is influenced by factors on school and teacher level like the ones mentioned above, which in their turn are influenced by school external factors (policies, community, parents, etc).

This prime attention for the primary process of teaching and learning can also be phrased as ‘what one may actually perceive if one walks into a school and would observe teachers and students activities’. Klein (1991) and recently Van den Akker (2003) have identified a number of concepts, which are core to describing teaching/learning activities: content, learner activities, teacher activities, grouping, time, evaluation, materials and resources. Also in the context of SITES-M1 and SITES-M2 other important components influencing the use of ICT in innovative pedagogical practices were identified (Kozma, 2003; Law, 2003; Mioduser, Nachmias, Tubin, & Forkosh-Baruch 2003). We may call all these the components of the primary process.

Very important also are concepts that reflect the secondary process (such as school regulations, infrastructure, curriculum, vision, teachers’ objectives, etc.). With regard to each of these components one may pose a number of questions: what is happening, who makes decisions about this component and how is ICT-use related to this? When considering the literature on ICT in education and how ICT use may differ in an industrialized society from that in an information society, it seems that for answering these questions the notion of degrees of freedom or degrees of variability may be useful. Increased freedom and variety of choices and possibilities in realizing teaching/learning situations is a continuous thread that runs through the current discussions and publications on educational reform in the information society (referring to for example, lifelong learning, more autonomy for students, the need for authenticity and flexibility in the curriculum, freedom of choice for schools, etc.)

3 Research questions

The following main research questions were derived from the conceptual framework that was presented above:

  1. 1.

    What are the pedagogical practices applied in school and how is ICT used in them?

  2. 2.

    How is ICT supporting and enhancing learning and teaching?

    From these main questions a number of sub-questions will be derived, such as:

    • What ICT is available in the schools and how is it used in teaching & learning?

    • What are the pedagogical visions of schools and teachers?

    • In what pedagogical practices is ICT used and in what way?

    While the first question is mainly descriptive and focuses on the primary process, the third question addresses the secondary process and its relationships with the primary process:

  3. 3.

    To what extent are factors associated with the use of ICT and the nature of pedagogical practices found in schools and among teachers?

    This question regards several concepts, which are mentioned in Fig. 1. Some of these concepts have been addressed in previous studies as well. For instance in the IEA CompEd study and SITES M1 a number of indicators were developed for measuring student: computer ratios, integration of computers in existing subjects, teacher skills and competencies, and student computer/ICT-literacy.

    Examples of derived research questions are:

    • To what extent are school’s and teachers’ pedagogical visions related?

    • What is situation with respect to factors such as staff competencies, access to content, student-centred pedagogy, etc?

    • What is the ICT-related vision of education in the school and how does this translate into policies and practices?

    • How are teachers communicating and sharing their knowledge and experiences?

    • To what extent are parents and other members of the community involved in education processes?

As some of these questions were also included in SITES-M1, this part of the SITES2006 will allow for examining trends regarding the availability and use of ICT in those countries that also participated in M1.

4 Research design

A number of a priori decisions about the design of SITES2006 has been taken in order to increase the timeliness and the relevance of the study. Before summarizing research design issues a number of these decisions and the underlying arguments are reviewed.

  • ➢ The study should present timely results

    Timeliness is an important constraint for SITES 2006, in particular because the area of ICT is perceived as quickly changing. Therefore study has been designed in a way that participating countries should be able to publish their results within half a year after the data collection. A priori decisions to make this happen were:

    • Limiting the size of the study by focusing on the lower secondary education level. The target grades will be grades 8 and 9, but in countries where grade 8 is the final grade of a school type, grades 8 and 7 will be the target grades.

    • Exploring the feasibility of online data collection (ODC) for school- and teacher-questionnaires (as an alternative next to the paper-and-pencil mode) to increase the efficiency of both data collection and subsequent file building. However, it has to be shown that online data collection instruments yields the same results as paper and pencil instruments if both are to be used in the same study interchangeably.

  • ➢ The study should minimize the burden for respondents: maximum response time 45–50 min.

    ➢ The study should result in information that also can be used at school level.

    The study should allow for incorporating regional options, i.e. addressing issues that are of specific interest to a particular region in the world, so as to increase the relevance of the study for policy makers. It is anticipated that some European countries will be interested in a European regional option in SITES 2006.

The a priori decisions as identified above have consequences for several aspects of the design of SITES 2006.

Given the objectives of SITES 2006, it is appropriate to distinguish two survey components; namely (a) a survey of schools, and (b) a survey of teachers.

4.1 Populations and sampling

4.1.1 Survey of schools

The desired target population is defined as all schools where students are enrolled in the target grade, which is the grade that represents 8 years of schooling, counting from the first year of ISCED Level 1. In each country a representative sample of maximally 500 schools will be drawn, proportional to the size of the school. This number will be lower for countries with a relative small number of schools. The respondent units are defined as school principals and school technical experts (ICT coordinators).

4.1.2 Survey of teachers

This population will consist of all math and science teachers at the target grades. In each sampled school, a minimal sample size of two math teachers (or all math teachers when there are less than two in the school) and two science teachers (or all science teachers) is required.

4.2 Instrument development

Data will be collected through survey questionnaires to school principals, ICT coordinators and teachers. The questionnaires will focus on the key concepts depicted in Fig. 1. The instruments will be developed following a number of steps, viz. creating an initial assessment framework (derived from Fig. 1), identifying indicators that have to be constructed, collecting and/or writing items to measure the indicators, which will result in preliminary drafts of the framework, indicators and questionnaires that will be reviewed by the National Research Coordinators (NRCs) resulting in pilot testing of the draft instruments, on the basis of which the final instruments will be constructed.

All the instruments will be pretested for various reasons, such as (1) trial of the draft instruments with the purpose of selection of the candidate questions for the final instruments, (2) test of procedures and logistics, (3) training of researchers in conducting this specific assessment, and (4) trial of online data collection methods.

4.3 Data collection and processing

Where possible online data collection will be applied, which will allow for more flexibility and freedom to administer questionnaires independent of time and place. But, in order to avoid unforeseeable complications, countries will be given the option of paper-and-pencil data collection.

Data cleaning and processing will follow that standard IEA procedures (see e.g. Schwippert, Itzlinger, & Bos, 1999) so as to assure consistent quality of the data sets that will be used for national and international analysis and reporting.

4.4 Quality monitoring

Quality monitoring in SITES2006 will consist among others of the following activities:

  • Judgment by the international coordination center and the IEA-secretariat of translation (and verification) procedures as reported by NRCs

  • Judgment of sampling plans and monitoring sampling procedures by the sampling coordinator

  • Checks on whether rather time-invariant indicators (e.g. average school size, gender composition) yield the same estimates as in TIMSS-2003

  • Checks on the quality of the data-entry procedures

4.5 Reporting

The following reports will result from the project:

  • Statistical overview on the Web: 0,5 year after data collection

  • Report to schools (generated via the online data collection tool): 2 months after data collection

  • International descriptive report and first analyses: 1 year after data collection

Moreover, data bases will be made available to the research community for secondary analyses.

5 From concepts to indicators to instruments

As stated above, instrument development will follow a stepwise procedure. Of crucial importance are the steps from concepts to indicators and from there to instruments. This process is illustrated here by summarizing how, starting from Fig. 1, the focus on the teaching/learning (T/L) processes as arranged by teachers within the context of the school is operationalized.

Although this process is not yet completed, the following illustrate the key concepts and indicators that will be addressed in the various areas:

  1. 1.

    Teaching/learning (T/L) practice with key concepts and indicators like:

    • Curriculum goals (reported for a particular practice)

    • Materials & resources (ICT, specific configuration)

    • Methods (specific activity organizations like experiments, project work, whole class teaching, use of ICT, ...)

    • Teacher activities (provide whole group instruction, give individual feedback, organize students activities, ...)

    • Student activities (define own learning goal, follow instruction, ...)

    • Organization (such as space–time configuration, involvement of external parties, ...)

    • Assessment practices

    In elaborating the T/L practices, distinction will be made between the preparation, execution and regulation of these processes.

  2. 2.

    Teacher key concepts and key indicators such as:

    • Academic/professional background/other demographic

    • ICT competence and use of ICT (both personally & professionally)

    • Pedagogical beliefs/orientation

    • Pedagogical competence in dealing with new pedagogical approaches (new assessment methods, group work, etc)

    • Understanding of the role of ICT in supporting learning and teaching

    • Attitude towards new pedagogical orientations and towards use of ICT

    • Aspiration of the teacher towards change and readiness for change

    • Perception of institutional support, the system support for change

    The Appendix illustrates how one of these indicators, viz. teachers’ ICT competence, is being elaborated in a draft instrument.

  3. 3.

    School key concepts and key indicators such as

    • School vision and beliefs

    • History of innovation

    • ICT infrastructure (access and availability (time, place), functionality)

    • Strategy for ICT integration (incentive scheme, recognition, ...)

    • Professional development strategy (availability, intensity, recency, adequacy)

    • Technical support

    • Pedagogical support

    • Organizational structure to support ICT integration

6 Planning and participation

The main data collection for the study will take place in northern hemisphere countries in the period February–May, 2006, and in the southern hemisphere countries half a year later. The following education systems are collecting data within the framework of this study: Canada-Alberta, Canada-Ontario, Chile, Chinese Taipei, Denmark, Estonia, Finland, France, Hong Kong SAR, Israel, Italy, Japan, Lithuania, Norway, Russian Federation, Slovak Republic, Slovenia, Singapore, South Africa, Spain-Catalonia.and Thailand.

Those countries (or education systems) will be able to report their national results half a year after data collection. The first international data will be released in December 2007.