Keywords

5.1 The ICCS 2022 Instruments

The ICCS 2022 instruments collect outcome data, as well as contextual variables. Given the specific nature of a study on civic and citizenship education, outcome variables are assessed through cognitive test materials and a student questionnaire. Contextual data that explain variation in outcome variables are collected through student, teacher, and school questionnaires, as well as through the national contexts survey.

Table 5.1 shows the instruments used in ICCS 2022 together with their respective assessment mode (computer-based assessment = CBA; paper-based assessment = PBA), length and type of respondent. For student instruments, countries chose the standard assessment mode (computer or paper) for all respondents. For teacher and school questionnaires, countries decided for each instrument whether to administer them only on paper, only in online mode, or depending on the respondent’s preference, in either of the two modes. The national contexts survey was only offered in online format to national centers.

Table 5.1 ICCS 2022 instruments
Full size table

The ICCS 2022 test of civic knowledge also includes 55 items from five clusters that were used in ICCS 2016 in order to estimate changes over time for those countries participating in both surveys. These 55 items were integrated across the eleven ICCS 2022 test item clusters (comprising 121 items in total) that are common to both computer-based and paper-based assessments. This was done to ensure an appropriate content balance within each cluster given that, for this cycle, approximately one-half of the newly developed items related to two areas of increased focus: global citizenship and sustainable development. Larger numbers of items reflecting these two areas have also been included in the student, teacher, and school questionnaires.

The computer-based test instrument included three clusters of items in addition to the eleven clusters common to both the computer-base and paper-based tests. The computer-based test instrument consequently comprised 14 clusters and the paper-based instrument comprised 11 clusters. Each of the three clusters of items, unique to the computer-based instrument, comprised five items associated with a narrative theme. In each cluster, at least one item provided some form of dynamic feedback to students that could not be achieved in a paper-based testing environment.

Table 5.2 shows the numbers and percentages of items from ICCS 2016 and those newly developed for ICCS 2022. For the student test and the European student questionnaire more than half of the item material was newly developed for the third ICCS cycle. For the teacher and school questionnaires slightly less than half of the content was added in ICCS 2022. For the student questionnaire, about one third of the item material addresses the areas of global citizenship and sustainable development. Due to delays with the implementation of the field trial in all Latin American ICCS 2022 countries caused by the COVID-19 pandemic, no field trial data were available for an evaluation of newly developed item material for the ICCS 2022 Latin American student questionnaire. For this reason, this regional instrument consists almost entirely of material that was included in the previous cycle.

Table 5.2 Numbers and percentages of items from ICCS 2016 and those newly developed for ICCS 2022 contained within the main survey instruments
Full size table

5.2 Item Types

The ICCS 2022 instruments include a range of different item types in order to assess a diversity of cognitive, affective-behavioral or contextual aspects, which vary across instruments.

The cognitive test contains the following item types:

  • Multiple-choice items (MC): Each item has four response options, one of which is the correct response and the other three of which are distractors.

  • Open-ended response items (OR): Students are requested to write a short response to an open-ended question. The responses are scored by scorers working for the national centers according to international scoring guides.

  • Drag & Drop items (DD): Students are requested to drag elements within a computer-based environment and drop them in other places in response to a question.

  • Large-task items (LT): Students provide answers by selecting different options on a computer in response to more complex tasks (e.g., through putting together web-page information) and receive some form of dynamic feedback based on their selections.

Table 5.3 illustrates the distribution of item formats in the ICCS 2022 survey. For this cycle, the option for a computer-based delivery also includes a number of computer-enhanced items that are only delivered on a computer and will not be comparable with any paper-based items. For items in clusters that are common across both delivery modes (computer- and paper-based) about 10 percent of the items are constructed response items and 90 percent have a multiple-choice format, DD and LT formats are only found as part of the computer-enhanced items that also include items with an MC or CR format.

Table 5.3 Item formats in the ICCS 2022 student test
Full size table

As in ICCS 2009 and ICCS 2016, the student, teacher, and school questionnaires for ICCS 2022 include the following item types that were displayed in similar ways on both computer and on paper:

  • Likert-type items (LK): For each item, respondents are asked to rate a number of statements, typically on a four-point scale. For most of these items, the rating scale ranges from (1) strongly agree to (4) strongly disagree. The rating scales for other questions indicate frequencies (never, rarely, sometimes, often) or levels of interest, trust, or importance.

  • Multiple-response items (MR): Respondents are asked to indicate the three aspects they view as most important.

  • Categorical response items (CR): Respondents are required to choose one out of two or more response categories that they view as most appropriate. These questions are primarily used for collecting contextual information (for example, on gender, educational level of parents, books in the home, subjects taught at school, and public or private school management).

  • Open-ended response items (OR): Respondents are asked to write a short response that is coded by the national centers; these items are used only for collecting information on parental occupation as part of the international student questionnaire.

5.3 Coverage of Framework Domains

The ICCS 2022 item material was developed to provide adequate coverage of the different types of domains and areas specified in the assessment framework. Given that in ICCS 2022 there are distinct frameworks for cognitive and affective-behavioral measures, the coverage is reviewed separately for each of these two types of measures.

The coverage of cognitive and content domains shown in Table 5.4 illustrates how many of the items in both paper-based assessments (PBA) and computer-based assessments (CBA) relate to each cognitive and content domain. The highest number of items is related to the content domain civic principles, followed by civic institutions and systems and civic participation, while the lowest number of items is related to civic roles and identities. About two thirds of the common items relate to the cognitive domain reasoning and applying and one third to knowing. The 15 computer-enhanced items are all related to the content domain civic participation and the cognitive domain reasoning and applying.

Table 5.4 Coverage of cognitive and content domains in ICCS 2022 test
Full size table

Table 5.5 illustrates the coverage of affective-behavioral areas across the three student questionnaires (international, European, and Latin American). Within the area attitudes, for both the international and European student questionnaires, most items relate to the subarea attitudes toward civic issues and institutions, in the Latin American questionnaire most items pertain to the subarea attitudes toward civic principles. Generally, the subarea attitudes toward civic role and identities has somewhat less coverage across instruments. For the area engagement, most items reflect expected future engagement, followed by experiences with engagement, while the lowest number of items is related to dispositions toward engagement.

Table 5.5 Coverage of affective-behavioral areas and subareas in the ICCS 2022 student questionnaires
Full size table

5.4 The ICCS 2022 Test Design and the Described Achievement Scale

For the student test, ICCS 2022 uses rotated designs for test administration, making it possible to include more test material and thus ensure greater coverage of the assessment framework without increasing the testing time for each student. This procedure also enables enough score points to be generated to provide the basis for comprehensive descriptions of the scale. Rotating the clusters throughout the booklets (or modules when delivered on a computer) ensures that the different tests are linked.

In countries conducting paper-based delivery, eleven test clusters are administered in a rotated design across eleven booklets, with each cluster appearing in one of the three possible positions at the beginning, in the middle, or at the end of the booklet (Table 5.6). The design for computer-based delivery follows the same principles, however, only 14 clusters are administered (including three clusters consisting of computer-enhanced items, displayed on a shaded background in Table 5.7), each also appearing once in each of the three possible positions (Table 5.7).

Table 5.6 Main survey test booklets design (paper-based assessment)
Full size table
Table 5.7 Main survey test booklets design (computer-based assessment)
Full size table

As ICCS 2022 is the first cycle of ICCS to include computer-based administration, it is necessary to review and adjust for mode effects across paper-and computer-based delivery (both across ICCS 2022 participants and compared to the paper-based assessment in ICCS 2016). After a first review using data from a mode-effects study as part of the ICCS 2022 field trial, results showed some limited mode effects. Consequently, it will be necessary to confirm these results and provide a basis for estimating potential adjustments to make data across modes fully comparable. Therefore, the ICCS 2022 main survey includes a bridging study: In countries completing the computer-based assessment that also participated in the previous cycle, an additional separate sample of students complete the test on paper. The paper-based test instrument used in the bridging study comprises only the first eight clusters of the paper-based assessments, which are administered in a similarly rotated design (Table 5.8).

Table 5.8 Main survey test booklets design (bridging study)
Full size table

Test items will be scaled using IRT (item response theory) (Bond & Fox, 2007; Hambleton et al., 1991) specifically with the One-Parameter Rasch model (Rasch, 1960). The cognitive test items for ICCS 2022 will be scaled to obtain civic knowledge scores. This scale will cover student knowledge and understanding encompassing the four content domains (civic systems and society, civic principles, civic participation, and civic roles and identities) and the two cognitive domains (knowing and reasoning and applying). Items will be used to describe students’ knowledge and understanding at different levels of student proficiency. ICCS 2016 trend items will provide the basis for equating civic knowledge scores across cycles.

As in the previous survey cycles, ICCS 2022 test items were designed to provide the basis for deriving a described scale of civic knowledge, which has consisted, since ICCS2016, of four levels of proficiency. The proficiency-level descriptions are syntheses of the item descriptors within each level. They describe a hierarchy of civic knowledge in terms of increasing sophistication of content knowledge and cognitive process. Because the scale was derived empirically rather than from a specific model of cognition, increasing levels on the scale represent increasingly complex content and cognitive processes as demonstrated through performance. The scale does not, however, simply extend from simple content at the bottom to reasoning and analyzing at the top.

The cognitive processes of knowing and of reasoning and applying can be seen across all four levels of the scale, depending on the content issues to which they apply. The scale includes a synthesis of the common elements of civic and citizenship content at each level and the typical ways in which students use that content. Each level of the scale references the degree to which students appreciate the interconnectedness of civic systems, as well as the sense students have of the impact of civic participation on their communities. The scale broadly reflects development encompassing the concrete, familiar, and mechanistic elements of civics and citizenship through to the wider policy and institutional processes that determine the shape of our civic communities, with the following four levels (see Appendix C for a more detailed description):

  • Level D: Students with civic knowledge at this level are expected to demonstrate basic familiarity with concrete, explicit content and examples relating to the basic features of democracy, to identify intended outcomes of simple examples of rules and laws, and to recognize the explicit function of key civic institutions and the rights of others.

  • Level C: Students with civic knowledge at this level are able to understand fundamental principles and broad concepts underpinning civics and citizenship, are familiar with some of the “big ideas” of civics and citizenship, recognize social justice issues in familiar contexts, and demonstrate an understanding of the basic operations of civic and civil institutions.

  • Level B: Students with civic knowledge at this level are expected to demonstrate specific knowledge and understanding of the most pervasive civic and citizenship institutions, systems, and concepts, and to understand the interconnectedness between civic and civil institutions as well as the processes and systems through which they operate.

  • Level A: Students with civic knowledge at this level demonstrate integrated rather than segmented knowledge and understanding of civic and citizenship concepts. They have the ability to make evaluative judgments with respect to the merits of specific policies and behaviors in view of different perspectives, to provide justifications for positions or propositions, and to hypothesize expected outcomes based on their understanding of civic and citizenship systems and practices.

5.5 Questionnaire Scales

ICCS 2022 will report on outcomes of civic and citizenship education and contexts based on several scales derived from the international and regional student questionnaires and the teacher and school questionnaires. Typically, items will be scaled using the IRT Rasch partial credit model (Masters & Wright, 1997).

The international student questionnaire includes items that will be used to obtain the following expected constructs (new aspects are displayed in italics):

Attitudes

  • Students’ perceptions of the value of student participation at their schools (5 items)

  • Students’ attitudes toward the political system (9 items, two dimensions expected : cynicism and support for political system)

  • Students’ perceptions of threats to democracy (9 items)

  • Students’ perceptions of good citizenship (13 items, three dimensions expected: conventional behavior, social-movement-related behavior and global citizenship behavior)

  • Students’ attitudes toward restrictions in national emergencies (9 items)

  • Students’ attitudes toward equal rights for immigrants and non-immigrants (5 items)

  • Students’ attitudes toward gender equality (7 items)

  • Students’ attitudes toward environmental protection (5 items)

  • Students’ attitudes toward equal rights for all ethnic groups (international option, 5 items)

  • Students’ trust in institutions (14 items, six items for measuring construct: trust in civic institutions)

  • Students’ perceptions of threats to the world’s future (11 items)

  • Students’ attitudes toward the influence of religion in society (international option, 6 items)

Engagement

  • Students’ engagement with digital media (5 items)

  • Students’ (past or present) involvement in organizations and groups outside of school (5 items)

  • Students’ (past or present) involvement in school activities (7 items)

  • Students’ sense of citizenship self-efficacy (7 items)

  • Students’ expectations of future school participation (4 items)

  • Students’ expectations to participate in civic action to express opinions about important issues (13 items, three dimensions expected: Expected legal activities, expected pro-environmental activities, expected illegal activities)

  • Students’ expectations of participation as adults (10 items, two dimensions expected: expected electoral participation and expected active political participation)

Home and school contexts

  • Students’ reports on media consumption and discussions about political and social issues (7 items, 4 items to measures political discussions with parents and peers)

  • Students’ perceptions of open classroom climates for discussion of political and social issues (6 items)

  • Students’ reports on civic learning at school (9 items)

  • Students’ perceptions of their school climate (9 items, two dimensions expected: perceptions of teacher-student relationships at school and perceptions of social interaction between students at school)

The European regional student questionnaire includes items that will be used to obtain the following indices (new aspects are displayed in italics):

  • Students’ sense of European identity (4 items)

  • Students’ reports of learning opportunities about Europe at school (5 items)

  • Students’ attitudes toward freedom of movement for European citizens within Europe (6 items, two dimensions expected: endorsement of freedom of movement and endorsement of restrictions)

  • Students’ attitudes toward cooperation among European countries (7 items)

  • Students’ attitudes toward environmental cooperation in Europe (5 items)

  • Students’ perceptions of discrimination in Europe (10 items)

  • Students’ expectations regarding the future of Europe (13 items, two dimensions expected: positive and negative perceptions)

  • Students’ expectations regarding their own individual future (5 items)

  • Students’ perceptions of the importance of aspects for their future life (9 items)

  • Students’ reports of political and ethical consumerism behaviors (6 items)

  • Students’ reports of their sustainable behaviors (8 items)

  • Students’ attitudes toward the European Union (10 items)

The Latin American regional student questionnaire includes items that will be used to obtain the following indices (new aspects are displayed in italics):

  • Students’ perceptions of their own individual future (8 items)

  • Students’ attitudes toward authoritarian government practices (7 items)

  • Students’ attitudes toward corrupt practices (5 items)

  • Students’ attitudes toward disobedience to the law (10 items)

  • Students’ attitudes toward homosexuality (5 items)

  • Students’ perception of discrimination of minorities in Latin American societies (10 items)

The teacher questionnaire includes items to derive the following contextual indices (new aspects are displayed in italics):

  • Teachers’ participation in school governance (6 items)

  • Teachers’ perceptions of social problems at school (9 items)

  • Teachers’ perceptions of student activities in the community (10 items)

  • Teachers’ perceptions of classroom climate (4 items)

  • Teachers’ perceptions of student participation in decision-making processes at classroom level (6 items)

  • Teachers’ reports of activities to deal with diversity among students (6 items)

  • Teachers’ perceptions of the effects of cultural and ethnic diversity on teaching and class contexts (6 items)

  • Teachers’ perceptions of the effects of social and economic diversity on teaching and class contexts (6 items)

  • Teachers’ reports of activities related to environmental sustainability (6 items)

  • Teachers’ reports of activities related to the use of digital technologies (4 items)

  • Teachers’ perceptions of good citizenship (13 items, three dimensions expected: conventional, social-movement-related and global citizenship behavior)

  • Teachers’ reports of class activities related to civic and citizenship education (international option, 10 items)

  • Teachers’ reports of activities related to global issues (international option, 5 items)

  • Teachers’ preparation for teaching topics related to civic and citizenship education (international option, 13 items)

  • Teachers’ reports of their participation in training courses about topics related to civic and citizenship education (international option, 13 items)

  • Teachers’ perceptions of students’ opportunities to learn about topics related to civic and citizenship education (international option, 13 items)

  • Teachers’ reports of their training in teaching methods (international option, 6 items)

The school questionnaire includes items to derive the following contextual indices (new aspects are displayed in italics):

  • Principals’ perceptions of teacher participation in school governance (5 items)

  • Principals’ perceptions of school community participation (10 items, two dimensions expected: students’ and parents’ participation in decision-making at school)

  • Principals’ perceptions student contributions to decision-making processes at school (5 items)

  • Principals’ reports on communication between school and parents/guardians (4 items)

  • Principals’ reports of collaboration between the school and the local community (4 items)

  • Principals’ perceptions of student opportunities to participate in community activities (10 items)

  • Principals’ reports of activities related to diversity at school (6 items)

  • Principals’ reports of activities related to environmental sustainability (9 items)

  • Principals’ reports of the extent to which activities related to global citizenship education and education for sustainable development (6 items)

  • Principals’ reports on training activities undertaken at school on the use of digital technologies for civic and citizenship education (6 items)

  • Principals’ perceptions of the availability of resources in the local community (11 items)

  • Principals’ perceptions of social tension in the community (12 items)

  • Principals’ perceptions of school autonomy for the delivery of civic and citizenship education (7 items)