Keywords

12.1 Introduction

The International Association for the Evaluation of Educational Achievement (IEA) has a long history of publications and, over successive decades, has developed a dissemination strategy that is intended to maintain focus on the central objective of supporting educational progress around the world. Assessment studies run the risk of simplistic headlines. IEA’s constant endeavor has been to ensure that study findings are used primarily to deepen understanding of the policies and practices that foster educational reform and, where necessary, to challenge the simplistic use of assessment data. This runs alongside the objective of helping educational systems to develop their own research and analysis capacities.

From the early ground-breaking research reports of the 1960s to the extensive contemporary output, IEA publications have been at the forefront of developments in international large-scale assessments (ILSAs). Besides making available unique datasets on student achievement worldwide, which have wide resonance nationally and internationally, IEA’s ILSAs embody outstanding scholarship and have helped to shape and to advance the science of large-scale assessment. In addition, these datasets and publications underpin a vast number of journal articles and university dissertations by independent scholars. Most importantly, they have contributed, and continue to contribute, to educational reform and the enhancement of student learning around the world.

All IEA publication activity derives from the projects but, for purposes of this chapter, here we categorize the publication output as follows:

  • Core project publications

  • Project-related publications

  • Academic journal

  • International research conference

  • IEA Compass briefs.

Each of these publication strands is subject to rigorous quality assurance reviews to ensure that they meet the highest standards of research and scholarship.

We also draw attention to IEA’s dissemination activity, which serves both to take study findings to diverse audiences and to generate publications in its own right.

12.2 Core Project Publications

Each IEA study generates a set of open-access publications that are freely accessible for download (IEA 2020a). The starting point, naturally, is an assessment framework, a document that articulates the basic structure of each study and lays down the parameters for its design and execution. The framework sets out what is to be measured and how it will be measured. It details the background information that will be collected to enable analysis of the student achievement data. It also outlines an assessment design, including information about the population(s) to be sampled, measurement instruments, and test administration logistics. Assessment frameworks build on previous cycles of the studies but also evolve: they maintain continuity with previous studies so as to permit the measurement of trends over time, but they must also take account of changes in the educational environment so as to incorporate new thinking and remain educationally relevant.

Assessment frameworks start by specifying the content of the assessment. The four most recent assessment frameworks for IEA studies serve as good examples.

IEA’s Trends in International Mathematics and Science Study (TIMSS) 2019 presented two content assessment frameworks, one for mathematics and one for science (Mullis and Martin 2017). Each one was organized around two dimensions: (1) content, which specifies the subject matter to be assessed; and (2) cognitive, which specifies the thinking processes to be assessed. Thus, the content for grade 8 assessment in mathematics was grouped under number, algebra, geometry, and data and probability, and in science under biology, chemistry, physics, and earth science. For both, the cognitive processes assessed remain the same: knowing, applying, and reasoning.

The Progress in International Reading Literacy Study (PIRLS) 2016 assessment framework was described in terms of the major purposes of reading and the comprehension processes to be tested (Mullis and Martin 2015). The purposes of reading are: reading for literary experience, and reading to acquire information. Four processes were assessed: (1) retrieving specific information; (2) making inferences from text; (3) interpreting information in a text; and (4) evaluating content.

The International Computer and Information Literacy Study (ICILS) 2018 assessment framework (Fraillon et al. 2019) was developed in terms of two dimensions: (1) computer and information literacy; and (2) computational thinking. Meanwhile, the International Civic and Citizenship Education Study (ICCS) 2016 assessment framework (Schulz et al. 2016) was organized around four content domains: (1) civic society and systems, (2) civic principles, (3) civic participation, and (4) civic identities. In addition to this there were two cognitive domains: (1) knowing, and (2) reasoning and applying (seen as a single domain). In addition, and in recognition of the importance of students’ attitudes toward civic engagement, the ICCS 2016 framework included an affective-behavioral dimension comprising two further domains: (1) attitudes, and (2) engagement.

The second component of the assessment frameworks relates to background information. This encompasses, as appropriate for each study, information on national context, national curriculum, school and classroom factors, teacher background, student perceptions, and home background. The final component expands on the study design, specifying the populations to be sampled, the instruments (tests and questionnaires) and item types to be used, and coverage of framework domains, along with practical details on test administration and questionnaire management. Increasingly, studies are moving some or all of their activities from paper to computer modality, and details are presented as necessary, including any linking arrangements that may be required.

For PIRLS and TIMSS, and sometimes ICCS, the assessment framework is supplemented by an encyclopedia (see IEA 2020a for some examples). These publications draw on public data, responses to study-specific national questionnaires and structured reports from national experts. The encyclopedias present demographic and economic data for participating countries and outline the structure of their education system. They describe the relevant curricula for the grade(s) being assessed, along with instructional practices and school organization. Other topics covered include teacher education, and specifically teachers’ preparedness for the curriculum area in question, and national assessment and quality assurance procedures. These encyclopedias serve a two-fold purpose: they provide an additional lens for analyzing student achievement data within countries; and they constitute a rich source of information on school practices globally regarding reading, mathematics and science, and civics and citizenship education.

The key publication for each study is the international report. This presents overall study findings alongside background data that put student achievement results in context. Thus, for each participating country, the TIMSS 2015 mathematics report (Mullis et al. 2016) described students’ achievement in mathematics, on average and at benchmark points, for both content and cognitive domains. It also reported on factors such as home background, school resources, school climate, teacher preparedness, and classroom instruction. The ICCS 2016 report describes how well young people are prepared for their role as citizens: it presents data on their civic knowledge, civic engagement and attitudes toward key issues in society, alongside school and broader contextual factors.

Once studies have been completed and the international report released, technical reports, the datasets, and supporting documentation are made publicly available. Technical reports are detailed documents which provide, for each study, an overview of the entire study. They document the development of test and questionnaire items, along with the rigorous translation verification procedures used and any national adaptations to the instruments. They give details on sampling design and implementation, field operation and quality assurance procedures, data collection and management, database construction, weighting procedures, and the construction of scales. Full data for IEA studies are freely available from the IEA Data Repository (IEA 2020b). Data are available in SPSS and SAS formats.

There are a variety of free software tools and additional resources which help in analyzing IEA datasets, all of which are available from the IEA website (www.iea.nl). They include the International Database Analyzer (IDB Analyzer; see IEA 2020c), which is a tool for combining and analyzing data from all IEA datasets (and from some other large-scale assessment datasets); user guides, which contain information on the structure of datasets and the variables contained within them, and how to conduct analyses using the IDB Analyzer (for a full overview of available user guides see IEA 2020a); the IEA Data Visualizer (IEA 2020c), which assists in visualizing trends over time and regional variations in TIMSS and PIRLS data; and the ILSA Gateway (ilsa-gateway.org/), which provides easy access not only to IEA studies but also to other international large-scale assessments. The IEA website also highlights upcoming training opportunities and provides online video tutorials for the IDB Analyzer (IEA 2020c, d).

12.3 Project-Related Publications

International project reports are the starting point for more detailed scrutiny of student performance within individual countries. Many factors impinge on the latter, and in-depth analysis is necessary to undertand the patterns of student achievement within countries. Following the international release of study results, many national centers of participating countries publish national reports in the national language (see IEA 2020a). These present study findings within the context of the specific country and serve a valuable function in highlighting policy and practice implications at national level.

In 2016, IEA launched the open-access IEA Research for Education series (IEA 2020e) as an additional initiative to support analysis of the data and encourage dialogue focusing on policy matters and technical evaluation procedures. This series is dedicated to promoting in-depth analyses of IEA data and has a twin focus on significant policy issues and methodological innovation. IEA issues two calls to tender each year, and successful bidders are given financial support to carry out an agreed program of work. A wide range of topics has been covered to date, from gender differences in computer and information literacy and teaching tolerance in a globalized world, to the globalization of science curricula and the link between teacher quality and student achievement in mathematics.

IEA studies have led to a vast number of academic papers and dissertations. These papers can be found in many academic journals and testify to IEA’s significant impact on assessment scholarship. Likewise, student dissertations in many universities draw on IEA datasets and publications. As a measure of support for this scholarship, IEA provides two annual awards (IEA 2020f): the Richard M. Wolf award and the Bruce H. Choppin award (Wolf and Choppin were well-renowned psychometricians who made significant contributions to the development of IEA studies). The Wolf award recognizes the author or authors of “a paper published in a refereed journal, monograph, or book that includes analysis of data from one or more IEA studies,” and the Choppin Award recognizes outstanding “master’s theses or doctoral dissertations that employ empirical research methods and use IEA data.”

12.4 Academic Journal

IEA also publishes its own journal, Large-scale Assessments in Education (IEA 2020g), in association with Educational Testing Service (ETS; www.ets.org). This is dedicated to the science of large-scale assessment and publishes articles that utilize not only IEA data but also assessment data collected by any other similar large-scale studies of student achievement, such as the ILSAs undertaken by the OECD and the US National Assessment of Educational Progress (NAEP). In addition to research articles, the journal publishes reviews and articles presenting methodological and software innovations relevant to the analysis of large-scale assessment data.

Large-scale Assessments in Education is published on an open-access basis by Springer, and all articles accepted for publication are made permanently available online without subscription charges. It also provides the opportunity to publish or link to large datasets, support static graphics or moving images, and display data in a form that can be read directly by other software packages so as to allow readers to manipulate the data for themselves.

12.5 IEA International Research Conference (IRC)

IEA’s international research conference is a well-established biennial event that brings together researchers working with IEA data to present their findings to the wider research community. These can be cross-national studies or explorations of, for example, curriculum or teaching practice within particular countries, as well as an opportunity to explore different research methodologies. The conferences enable colleagues to share perspectives and enhance their professional development. In addition to the main conference activities, pre-conference workshops are held on specialized topics related to large-scale assessment.

These conferences have also generated a large number of publications, as IEA conference proceedings (IEA 2020h) and as subsequent articles in academic journals or policy papers. In 2017, IEA established the Constantinos Papanastasiou Poster Prize for the best poster submitted to the IEA IRC (IEA 2020f). The prize is awarded in recognition of Professor Papanastasiou’s enthusiastic, long-term contributions to building and supporting the educational research community, including founding and hosting the inaugural IRC in 2004 at the University of Cyprus.

12.6 IEA Compass Briefs

IEA studies are complex, and project reports along with the analyses deriving from them are, necessarily, detailed and require close reading. This indeed is one of the reasons why media headlines can so often mislead. Technical language and methodological qualifications do have a role, and when they are set aside without understanding miscommunication becomes likely.

There is still, however, a need to communicate study findings beyond the research community. The studies have implications and raise questions of relevance to policymakers, curriculum developers, teacher educators and teachers, as well as the general public. To address this need, IEA has developed the IEA Compass: Briefs in Education series (IEA 2020i). This is a set of brief documents that use IEA study data to address issues of interest to various educational stakeholders. Each publication in the series aims to connect study findings to recurrent and emerging questions in education policy and practice. Recent briefs have covered topics as diverse as: how safe primary-aged students feel at school and how perceptions of safety affect their learning; Latin American students’ support for dictatorships; and using TIMSS data to reform mathematics education.

12.7 Quality Assurance in Publications

Considerable effort is taken to ensure that all IEA studies are of high quality. It is important that they are methodologically rigorous and educationally relevant. The fundamental way of achieving this is to establish study teams which are highly competent and conduct their studies in a rigorous and transparent way. Each IEA study is supported by an array of expert committees, with regular reports to the IEA Standing Committee (this committee comprises six national representatives elected by the IEA General Assembly membership and serves as a board of directors for IEA) and to national research coordinators from all participating countries. When testing takes place within a country, quality monitors visit schools to help ensure that all procedures are followed correctly. In addition, the IEA convenes a Technical Executive Group (TEG), a small committee of expert psychometricians and methodologists, who advise on the technical aspects of all projects.

A further level of quality assurance is provided by the IEA Publications and Editorial Committee (PEC). This is a group of scholars from around the world who review documents prior to publication. PEC includes both IEA and non-IEA scholars to ensure an adequate measure of independent scrutiny. PEC provides expert, robust feedback to authorial teams and helps to ensure excellence in IEA publications.

12.8 Public Dissemination of IEA’s Work

IEA is highly respected among the research community but often not well known outside of academic circles. To address this gap, since 2015, IEA has placed a key strategic focus on communicating IEA research, data, and study findings to diverse audiences, including educators, policymakers, media, and the public while also retaining researchers as a main audience for IEA.

The IEA website (www.iea.nl) is the primary source of information about IEA studies, new publications, activities, and opportunities. The news and events sections are updated regularly and website users have the opportunity to subscribe to a quarterly email newsletter, IEA Updates, for further information.

IEA has an active social media presence across Twitter, LinkedIn, Facebook, and YouTube (see www.iea.nl for links), using these platforms to share information about IEA, to support the network of IEA study participants (for example, by sharing external publications based on IEA data), and to communicate IEA study results. In particular, IEA focuses on targeted social media posts to share findings that are related to current events. This helps to bring IEA data and resources to new audiences and to expand beyond the existing networks of people who are already aware of IEA’s activities.

A range of promotional materials provide support for IEA activities, including the annual IEA Insider publication, study brochures, promotional flyers, infographics and short videos (IEA 2020a). These materials are shared by IEA staff and collaborators alike as part of wider networking and outreach initiatives.

Promoting IEA publications (study-specific publications, the journal, IEA Research for Education series and IEA Compass: Briefs in Education series) is another key area of activity. New publications are shared on the website, social media, and newsletter channels in addition to being promoted at conferences and other events. New IEA publications provide a basis for producing articles aimed at more general audiences. These include blog posts and contributions to external platforms.

IEA hosts information stands at international conferences to promote the wealth of data, publications, and training opportunities that are available to the education research community. In some cases, researchers may be aware of some of IEA’s studies but unaware that the IEA is the organization behind those studies. Promoting all IEA activities together (studies, data, publications, and research services) helps to highlight the full range of what IEA can offer to the research community.

In addition to contributing to external events, IEA-specific events also promote the work. New international reports from IEA studies are released as part of large, media-focused events, usually held in partnership with external organizations such as UNESCO. IEA produces and disseminates press releases, infographics, and videos to promote the key findings of each study to diverse audiences. All materials are shared with participating countries so that they may be translated and adapted for national use. Any media outreach or interviews are always conducted in cooperation with the participating countries.

IEA’s communications and dissemination activities shine a light on the wealth of high-quality, international data and research expertise available as valuable resources for diverse audiences. With these activities, IEA aims to bridge the gap between academic research, and education policy and practice.

12.9 Conclusions

IEA studies generate a vast body of publications: study findings and documentation, academic and policy papers based on the studies, training materials, and dissemination documents. IEA’s own documents (IEA 2020a) are subject to rigorous review prior to publication, and papers in academic journals are subject to normal blind peer review. IEA’s portfolio of publication activity, which is constantly growing, constitutes an enormous contribution to comparative educational research and the science of educational measurement.