Keywords

13.1 Introduction

As outlined in Chaps. 1 and 2, concerns regarding validity and reliability do not end with the successful collection of data. One of the primary ongoing concerns for IEA is that the data collected for each of the studies is used to inform educational reform and improvement. While data are initially provided for and made available to the participating countries’ analysts, these data represent a vast resource of information on the functioning of the participating educational systems and provide a form of public accountability, which can and (in IEA’s view) should be made accessible to the wider research community.

Users of IEA data have access to not only the learning outcomes of the target populations that participate in the study but also a vast array of background information about the nature of the education systems in each country, the participating schools and the teachers that provided instruction, the students’ home backgrounds, their demographics, and, in the case of some of the assessments, background information from students’ parents or guardians.

In order to achieve its broader mission of educational improvement and the development of the community of educational researchers, IEA has taken a number of actions to not only facilitate the use and accessibility of the data it produces but also ensure that data are used appropriately.

The following sections outline some of the key strategies that IEA employs to support a positive outcome in terms of consequential validity arguments. They include the development of analytical tools, facilitating access to data, supporting training on the use of the data, and scientific collaboration and exchange.

13.2 Data Access

As noted, IEA explicitly encourages researchers around the world to use the data collected from its studies for further in-depth analysis, making significant efforts to provide data access along with comprehensive documentation. Data gathered in IEA studies is available via the IEA website (www.iea.nl) and can be downloaded from IEA’s data repository (IEA 2020a). Comprehensive technical documentation, including data code books, accompany the data. This documentation enables researchers to navigate the contents of the international databases (IDBs), and understand the methods used to collect, process, and structure the data. The documentation also presents information on how achievement and background scales were calculated and how these should be used for analysis. The materials provide users with sufficient information to conduct secondary analysis that go beyond the mostly descriptive scope of the primary publications. By providing the data at individual and variable level, they also allow users to calculate new variables or scales. For example, researchers may want to recode variables by collapsing categories, or create new indexes. Expert users may reproduce or engage in rescaling of achievement items, modifying the conditioning model by entering new variables merged from other data sources, or optimizing the item response theory (IRT) model for specific (groups of) countries.

IEA data are made available in both SAS (SAS Institute Inc. 2013) and SPSS (IBM Corporation 2016) formats. These formats can be easily converted into file formats usable for various other contemporary statistical software packages handling international large-scale assessment (ILSA) data such as R (R Core Team 2014), STATA (StataCorp LLC 2019), Mplus (Muthén and Muthén 2012), or WesVAR (Westat Inc. 2008), to name the most popular. To keep the structure clear, data is made available in individual files for different countries and data sources. Algorithms for merging and appending data for specific analysis are explained in the accompanying user guides. Codebooks provide detailed information on each variable in the database. These display names, labels, and valid ranges, describe field values and missing schemes, and variable allocation to specific domains. Even though many of the actual achievement items cannot be published (items that are to be reused in future cycles for trend measurement purposes have to be maintained secure), item responses and descriptive information are available for all items, sufficient for additional item or scale level analysis. Released items (i.e., those that will not be used in future cycles) can also be obtained from the IEA by completing and submitting a permission request form. To protect the identity of those participating in the studies, some variables are omitted from the public use data files, as they bear a low, but non-negligible risk of disclosing information about single schools or even individuals participating in an IEA study. As IEA assures full anonymity to participants and commits to data security and confidentiality laws, such as the European Union’s General Data Protection Regulation (GDPR; see https://eugdpr.org/), suppressing information that would potentially identify participants is a priority. Researchers interested in using the variables not included in the public dataset for analysis can apply for access to the restricted use data. Applicants need to provide comprehensive information about how they intend to use the data and sign an agreement not to disclose details of any school or individual.

Additional support for analysts is provided through syntax in the format of ready-to-run programs that allow items to be scored or specific databases to be merged. Depending on the study and scope, additional information is accessible, such as national context survey data and encyclopedias, item almanacs presenting item parameters and descriptive statistics, or data on countries’ curricula and how they match with the tests.

Meticulous study documentation supplementing the data is also freely available. Each IEA study produces its own technical report, detailing information about the instruments, sampling, survey implementation, and analytic procedures used to derive scale scores and indexes. User guides complement this information, giving users a comprehensive overview of the structure and content of the international database of each study, the main features, analytical possibilities, and limitations. The user guides also contain the international versions of the contextual questionnaires, together with an overview of national adaptations, example items, and information on derived variables; the user guides are excellent resources to familiarize users with the data and studies.

Data files and technical documentation are also available from each study’s webpages. For example, users can find the IEA’s Trends in International Mathematics and Science Study (TIMSS) and Progress in International Reading Literacy Study (PIRLS) data at their dedicated website (https://timssandpirls.bc.edu), along with results and technical information for all cycles.

IEA also supports another important website, designed as a central point of entry to international large-scale assessments in education: the ILSA Gateway (https://ilsa-gateway.org). Here, users can find comprehensive information on all contemporary ILSAs. The information available is not limited to IEA studies, but also includes studies commissioned by other international organizations. The ILSA Gateway includes direct links to data and documents on the various study websites, a database of study-related research papers, and announcements of recent news and upcoming events. The Gateway was designed for a worldwide audience, first and foremost researchers, but is also a useful tool for policymakers, decision makers, and all others interested in education research. It is intended to facilitate access to ILSA-related resources, encourage knowledge exchange and discussions, and inspire future research.

Finally, IEA provides researchers with support related to the use of data from IEA studies. The IEA website contains contact information on support options, and interested researchers are encouraged to make contact when they need guidance.

13.3 Facilitating Analysis: The IEA IDB Analyzer

The ILSAs conducted by IEA are also known as group-score assessments because they are designed with the purpose of reporting results for different groups of interest within a population, and not at the individual level. This population is generally a specified group of students within a country, but in some cases, it could be those within a jurisdiction of interest, such as a municipal school system or a subnational system.

The qualification “large-scale” refers to the broadness of the domains covered, and the size of the population represented by those participating in the assessment, but not necessarily to the size of the sample or the length of the instrument administered to those taking part in the assessment. As such, ILSAs rely on the evaluation of subsets of individuals from the population with subsets of items from the domain, in order to make inferences to the entire population about the entire domain assessed. This is what is called multiple matrix sampling (Gonzalez and Rutkowski 2010; Lord 1962; Shoemaker 1973). Using multiple matrix sampling results in great efficiencies in the assessment process since only some of the people have to be assessed using some of the items. This significantly reduces the burden on the individual participants (due to reduction in the assessment time) and on the organizations implementing the assessment (due to reduction in the number of individuals that need to be assessed). But, as a consequence, these design features need to be accounted for, and sampling weights, complex variance estimation methods, and plausible values have to be used to analyze the resulting data with the corresponding adjustments.

The IEA International Database Analyzer (IDB Analyzer; IEA 2020b) is an application that can be used to combine and analyze data from IEA’s large-scale assessments, as well as data from most major large-scale assessment surveys.

The IDB Analyzer creates SPSS or SAS syntax that can be used to combine files from across different countries and levels (e.g., student, parent, teacher, or school), and perform analyses with these international databases that take into account the sample and assessment designs. It generates SPSS or SAS syntax that takes into account information from the sampling design in the computation of sampling variance and handles the plausible values. The code generated by the IDB Analyzer enables the user to conduct a variety of analyses including, at the most basic level, descriptive statistics, and also allows users to conduct statistical hypothesis testing among groups in the population without having to write any programming code.

The IDB Analyzer is licensed free of charge and available for use in accordance with the terms of the licensing agreement. The IDB Analyzer consists of two modules: the Merge Module and the Analysis Module (Table 13.1).

Table 13.1 Functions of the IDB analyzer
Full size table

When calculating these statistics, the IDB Analyzer is capable of using any continuous or categorical variable in the database, or can make use of achievement scores in the form of plausible values. When using plausible values, the IDB Analyzer generates code that takes into account the multiple imputation methodology in the calculation of the variance for statistics as it applies to the corresponding study.

All procedures offered within the Analysis Module of the IDB Analyzer make use of appropriate sampling weights and standard errors of the statistics that are computed according to the variance estimation procedure required by the design as it applies to the corresponding study. For more information about the workings of the IDB Analyzer, please refer to the Help Manual for the IDB Analyzer included with the software (IEA 2020b).

13.4 Capacity Building: Workshops

Conducting ILSAs in education with the highest quality standards is always a collaborative endeavor, involving many actors at international and national levels. It can only be successful if all contributors have acquired thorough knowledge about their specific tasks within the project. From the early years, IEA trained staff from national teams to achieve standardized implementation of the assessments in all participating countries, thereby building research capacities that were often used to develop and implement national educational monitoring systems. However, IEA realized that many countries lacked the capacity to thoroughly analyzing the data collected from the assessments, given its complex structure and statistical constraints. Often, the studies’ value to national policymakers and educational stakeholders was limited to translating the international reports into national languages, when the available data could instead be further exploited to address specific issues of national interest; in other words, IEA data was underused. This inspired IEA to offer a broad portfolio of learning opportunities by means of professional development workshops, conducted by experts in the field, and designed to cater all interested in learning about the concepts underpinning ILSA. The key objective of these workshops is to provide researchers with sufficient skills and knowledge to use the data collected optimally.

IEA workshops are arranged in thematic areas ranging from introductory courses for designing surveys, to the implementation of complex survey operations procedures. IEA also offers workshops on using various complex statistical methods, and report writing and publishing. Each theme consists of a series of both basic and advanced courses, often building on each other. In response to latest research developments in the field, workshop content is continually updated (see Table 13.2).

Table 13.2 Examples of IEA workshops
Full size table

While workshops build research capacity, they also provide a forum for fruitful exchange of ideas and foster the development of research networks. The methods applied in IEA workshops are distinctively selected to accommodate this goal. All workshops are conducted as in-person training (i.e., participants and trainers meet in one location, as opposed to remote training methods, such as webinars). Each workshop combines lectures and tutorials with opportunities for participants to not only apply and practice what they have learned but also discuss emerging issues with each other. Experienced instructors are on hand to provide guidance and advice to participants and explicitly encourage exchange and collaboration. Participants share their ideas, approaches, and findings with the group, resulting in an inspiring atmosphere. Workshops benefit from the fact that the trainees come usually from multicultural backgrounds, bringing multiple perspectives on the topics. Catering to this audience, workshops are mostly conducted in English, but some are also available in German, Spanish, Arabic, and French (a workshop brochure can be download from the IEA website; IEA 2020c).

Workshops can be tailored to specific occasions, topics, studies, and/or audiences, and vary in length, from a 90 min introductory course to a one-week long intensive training course. Workshops are conducted at international conferences, at focused academies, or on demand; some are anchored within specific capacity-building projects.

13.4.1 Promoting High-Quality Research Based on Large-Scale Assessment Data

In keeping with its stated aims and commitment to the development of the wider research community with a particular focus on ILSA, IEA has initiated a series of publications and actively supports other research related activities. IEA publishes its own journal, Large-scale Assessments in Education, in association with Educational Testing Service (ETS; www.ets.org) under the brand of the IEA-ETS Research Institute (IERI) (see Sect. 12.4), conducts its own focused biennial research conference (see Sect. 12.5), and encourages scholars in their work by distinguishing outstanding research with special awards (see Sect. 12.3). IEA also funds researchers to conduct comprehensive in-depth analysis of IEA data; the results are published in the Springer open access series, IEA Research for Education (see Sect. 12.3). Finally, with the IEA Compass: Briefs in Education series (see Sect. 12.6), IEA established a special publication format addressing policymakers and other stakeholders in education outside the scientific research community. Each publication in the series aims to connect research findings to recurrent and emerging questions in education policy debates at the international and national levels. Using appropriate text formats and illustrations, the series intends to translate these findings for an audience with less technical backgrounds, making sure the results reach broader target audiences.

As reported in Chap. 12, all publications undergo rigorous review procedures at several stages, involving international renowned experts in the respective fields. One important review stage is conducted by IEA’s Publication and Editorial Committee (PEC; see Sect. 12.7). In this way, IEA attempts to provide support and guidance to researchers in a way that helps to ensure the study data is being used appropriately. Good examples of methodologically sound or extraordinary innovative research are highlighted via IEA’s social media platforms, by inviting authors for special publication formats, and by awarding prizes.

13.4.2 The IEA-ETS Research Institute (IERI)

In 2007, the IEA partnered with ETS to form IERI. IERI is a collaborative effort between the Research and Development Division at ETS and the IEA that focuses on improving the science of large-scale assessments. IERI undertakes activities around three broad areas of work that include research studies related to the development and implementation of large-scale assessments, professional development and training, and dissemination of research findings and information gathered through large-scale assessments. As part of the activities of IERI, a monograph series was started in 2007, which later became the online open journal Large-scale Assessments in Education (see Sect. 12.4).

This journal focuses on articles that contribute to the science of large-scale assessments, help disseminate state-of-the-art information about empirical research using these databases and make the results available to policymakers and researchers around the world. Submissions to the journal have to undergo and receive favorable technical, substantive, and editorial review prior to acceptance and publication. The journal uses a double-blind peer-review system, where the reviewers do not know the names or affiliations of the authors, and the reviewer reports provided to the authors that assess the quality of their manuscript are also anonymous. Independent researchers in the relevant research area are chosen to assess submitted manuscripts for originality, validity, and significance.

13.4.3 IEA International Research Conference

At the turn of the millennium, even though IEA had been active in the field of educational research for several decades, IEA study related research publications and presentations at notable international or national conferences were relatively rare. Early efforts to change this led to symposia initiated by smaller research networks,Footnote 1 but there was no dedicated platform for presenting and sharing IEA study findings. To foster collaboration among scholars using IEA data for their research, with the initiative and leadership of Constantinos Papanastasiou,Footnote 2 IEA launched its first International Research Conference (IRC). This first conference took place in Lefkosia, Cyprus, in 2004 and has followed a generally biennial cycle since.Footnote 3 The purposes and goals of the event were set high from the very beginning. According to Papanastasiou and Papanastasiou (2004, p. 202), “the aim of the IRC-2004 Conference was to provide an international forum for the exchange of ideas and information on critical … issues in education evolving from secondary analyses of data from IEA studies. With its objective to foster creative dialogue among scholars and researchers, the conference aimed at providing greater understanding of the numerous roles that education plays in the development of nations and in shaping individuals. Because of its international scope, the conference also aimed to examine issues in both a comparative and global context with the ultimate aim to enhance pedagogical knowledge and implement positive change. A final goal of this conference was to create a global network of researchers.” These initial goals of the conference remain, however, the conference has significantly broadened its contents and scope over time.

Since 2008, the two days prior to the conference have been dedicated to capacity building. Experts in the field educate participants in various, often methodological topics related to ILSA. Statistical software and analysis approaches are introduced, and issues related to the dissemination of results to varying audiences are tackled.

Due to the increasing amount of high-quality proposals, a poster session format was introduced in 2017, together with a poster award, particularly designed to encourage early career researchers to present their work. Symposia were introduced in 2015, as well as panels on policy impact and on facilitating the use of IEA data for practitioners.

Another important development starting in 2015 was the inclusion of methodological focus sessions. New approaches related to the science of large-scale assessments are discussed in these sessions, fostering further development and implementation of new methods of analysis and stimulating critical exchange among international scholars about the applied methodologies used in ILSAs.

There are some significant factors distinguishing the IRC from other research conferences in the educational field. Most of these factors are related to its highly focused nature, conditioned by its restrictive content (proposals must make use of IEA data or be directly related to IEA study methodology). This focus strongly enhances the opportunities for networking: all attendees are to some extent familiar with the studies, which sets the basis for efficient exchanges. Participants can connect with peers who may have encountered similar problems and can share solutions or find new partners for future projects. Moreover, data users are connected with data producers; the experts involved in designing the methodologies for data collection and analysis for the IEA studies are present at the conference. They can give users invaluable hints and insights about the studies and, at the same time, benefit from the outsider perspectives of the scholars making use of their data.

Information on past and upcoming IEA Research Conferences can be obtained from the IEA website (IEA 2020d).

13.4.4 Academic Visitors/Scholars

On a regular basis, IEA invites scholars to work within IEA’s premises under its academic visitor programs. The programs are partly collaborative efforts with other internationally renowned institutions or organizations, and can be viewed as part of IEA’s efforts to contribute to the development of a worldwide network of researchers in educational assessment and evaluation. Interested academics and early research fellows can apply for the programs with a research proposal using IEA data. If selected, they develop their own research project while benefiting from the individual support of IEA experts working in different fields related to ILSAs. More information on the visitor program can be found on the IEA website (IEA 2020e).

13.4.5 IEA Awards

One of the earliest attempts to stimulate the proper use of IEA data was the establishment of a series of research awards (see also Sect. 12.3). These include the Bruce Choppin award (which recognizes a Masters or Doctoral level dissertation that employs empirical research methods with IEA data; the Richard M. Wolf award (which recognizes the author or authors of a paper published in a refereed journal, monograph or book based on analysis of IEA data), and, more recently, the Constantinos Papanastasiou poster prize (which is presented at IEA’s biennial research conference).

13.5 Conclusions

In order to address potential concerns related to issues of consequential validity, namely the correct use and interpretation of the data, IEA has developed a successful and elaborate strategy for addressing these concerns, which includes several key components. These relate to user-friendly data access, meticulous technical documentation, and a strong focus on training and capacity building. IEA also stimulates secondary analysis of the data through its awards, journal, policy briefs, and the research conference (see also Chap. 12) to ensure that the valuable information collected by IEA’s ILSAs contributes effectively to educational policy reform and improvement.