1 Introduction

International large-scale assessment studies play a major role in shaping the landscape of relevant international data in the education sector. They can influence both the public discourse in a country and the political agenda. A prominent example in Germany is the first cycle of the Programme for International Student Assessment (PISA). The phenomenon widely known as “PISA shock” refers to the below-average performance of German pupils in the first international PISA study in 2001 which was developed by the Organisation for Economic Co-operation and Development (OECD).Footnote 1 In-depth discussions on the results followed and reforms were subsequently initiated within the German educational landscape: in the aftermath of PISA and based on the recommendations of the national experts on education (Forum Bildung 2002), the German federal states developed educational curricula for early childhood. PISA therefore also indirectly exerted an influence on the field of early childhood education and care (ECEC) (Eibeck 2005; Schreiber 2009). This example illustrates the power of international studies and highlights the impact that the organisations behind them—in this case, the OECD—have. PISA has also helped to further increase the acceptance and reputation of international large-scale surveys.

Besides PISA, the OECD has established various comparative studies for different target populations at different educational levels. Strategies for gaining sound data on education include the use of a variety of different research approaches, but also the implementation of similar assessment instruments for different educational sectors. This can have its advantages but also present challenges. The alignment of different projects or surveys is typical of the OECD’s work, which systematically seeks opportunities for synergies as part of its strategic scientific programme (OECD 2015).

A recent example is TALIS Starting Strong, a survey initiated by the OECD to gain international comparable data on staff working in ECEC. Whereas already existing international indicator-based reports like the OECD’s Education at a glance series, the Eurydice-Report by the EU or other projects on ECEC staff (e.g. seepro‑r project, Oberhuemer and Schreyer 2018) mainly focus on structural aspects of ECEC systems, TALIS Starting Strong was supposed to fill this research gap in that it gives ECEC staff a voice and also includes information on staff’s practices, working conditions and job satisfaction. This quantitative survey is based on the Teaching and Learning International Survey (TALIS) of teachers and principals at schools in different OECD countries. TALIS has been conducted multiple times since 2008 to complement the PISA data on students’ competencies with information on teaching in different countries. In 2018, TALIS Starting Strong was implemented for the first time in nine OECD countries. Researchers at the International Centre of Early Childhood Education and Care (ICEC) at the German Youth Institute (DJI) acted as national project team for Germany and as authors of this article.

Hence, this article is written from a German perspective and with a focus on TALIS Starting Strong, since Germany has not (yet) participated in TALIS despite a lively debate on the subject.Footnote 2 When recapping the genesis of TALIS Starting Strong the authors build on the work of Bloem (2018), who described the development process before final implementation of the study. As the authors implemented the study in Germany and analysed the collected data, they thus seamlessly pick up on this initial work and are able to provide a comprehensive first-hand description of all study steps. This insider knowledge enables a better classification of processes and products within the study and avoidance of misinterpretations, which can especially occur with data of large-scale studies. Gorur (2020, p. 190) also points this out: “Data collected by anonymous others may be just at our fingertips, but they are also very distant at the same time—distant from the objects they seek to represent, and distant in the sense that we cannot know their antecedents, the assumptions that underpin them or the (mis)adventures that resulted in their creation.”

To understand how TALIS influenced the development of a new survey for the ECEC sector, the genesis of TALIS Starting Strong is reviewed and the effects of having an elder sibling highlighted.Footnote 3 OECD studies like TALIS Starting Strong are embedded in complex structures. Many different stakeholders with diverse interests are involved, for example, the OECD, participating countries or contractors commissioned with conducting the study. This can lead to compromises and influence the decisions made. Thus, this article first focuses on the role of different stakeholders while recapping the study’s development.

Second, we will examine the study design and provide insights into the target population and methodology. Here, also the advantages and challenges of large-scale international surveys in general are reviewed with relation to TALIS Starting Strong. TALIS greatly influenced the development of the TALIS Starting Strong study and its questionnaires. It is highlighted that the alignment of these similar surveys for different educational levels can be seen as advantageous as established questionnaire elements can be used. However, it can also give rise to particular challenges, as some elements of TALIS may not fit seamlessly to a different educational level.

Lastly, we will discuss the new structural developments for the second cycle of the survey: TALIS Starting Strong 2024 will be a module of TALIS and embedded in the same organisational structures. This will strengthen the links between the two and could have consequences in different areas. It will also be interesting to observe the extent to which the content is further developed. Like with other recurring surveys, it is a fine line between capturing developments in the sector and including new questions on pressing issues (e.g. digitalisation or inclusion).

2 Strategic choices and research gap: Reasons for the emergence of TALIS Starting Strong

Eight years after the first PISA study, the OECD decided to also survey various topics related to the work context of teachers and school leaders for the first time within the TALIS study. This seemed to be a logical step—after all, to determine which improvements are necessary and learn more about the relationships between teaching and students’ competencies, it is essential to consider both the learners (students at ISCED level 2Footnote 4 whose performance is measured in PISA) and the staff working within the educational level. Or, as the then OECD Secretary-General Angel Gurría called it, to gain “an insight into the world of education” “through the eyes of those who are on the front line” (Gurría 2009).

Building on this and following a coherent logic of expansion within the education system, the first ever international comparative survey of pedagogical staff and centre leaders in early childhood education and care followed in 2018: TALIS Starting Strong. This study is based on questioning ECEC leaders and staff in two target populations: those who work with children aged under three (ISCED 0.1) and those who work with children between the age of three and school entry (ISCED 0.2). For the ECEC sector, an approach very similar to the school sector was chosen in which PISA and TALIS examine learners and teachers. While TALIS Starting Strong should focus on the professionals’ perspectives, the International Early Learning and Child Well-being Study (IELS) was designed to investigate the learner side, respectively preschool children’s abilities.Footnote 5 However, it is worth mentioning, that to date, it has not been possible to link professionals’ perspectives and children’s abilities directly, mainly due to methodological concerns and national interests within the ECEC and school sector.Footnote 6 This shows that although the OECD envisions transferability and comparability of studies across different education sectors, in practice, this can compete with evolved structures within the respective sectors as well as with the individual countries’ resources, cultures and needs.

Nonetheless, there is great—and growing—interest in TALIS and TALIS Starting Strong: while 24 jurisdictions initially participated in TALIS 2008, 55 signed up for TALIS 2024 and while 9 ISCED 0.2 and 4 ISCED 0.1 ECEC systems were observed in TALIS Starting Strong 2018, 17 ISCED 0.2 and 7 ISCED 0.1 participants are already included in TALIS Starting Strong 2024.

The new survey name refers to the OECD Starting Strong series, which presents a comparative review of ECEC in different member states as well as policy recommendations.Footnote 7 The third edition, Starting Strong III, was published in 2012 with the participation of 31 countries. A quality toolbox was developed comprising five policy levers to further develop or ensure high quality in the ECEC sector. One goal involved “advancing data collection, research and monitoring” and ultimately provided the impetus for the development of TALIS Starting Strong (OECD 2012). The study was then conceptualised by the OECD Directorate of Education and Skills secretariat, mainly with the support of the OECD ECEC Network.Footnote 8

Another actor, the Education Policy Committee (EDPC), who functioned as the parent body, also helped to get the study off the ground. In 2014, the EDPC officially commissioned the OECD Secretariat to develop and implement TALIS Starting Strong (Bloem 2018). The following year, an informal subgroup, called the OECD Extended Network, which specifically addresses aspects of the survey itself, was established.

In addition to the different groups and internal structures of the OECD, the countries participating in the Starting Strong series also indirectly promoted the study through their participation. They—besides the beforehand mentioned existing data gap with respect to a self-report survey of staff working in ECEC—provided evidence that additional data was needed. The OECD recognised this need, interest among countries to participate and the possibility for their own profiling by establishing a new international study in a dynamic, high-growth education segment. Various actors were thus involved in the project before concrete development of the study even began.

3 Key stakeholders’ influence in the different study phases

Prior to internal commissioning of the study in 2014 and establishment of the OECD Extended ECEC Network in the following year, a discussion took place on what kind of study could fill the existing data gaps. Another possibility beside a self-report survey was observational studies. The advantage would be improved objectivity and thus less social desirability compared to self-reports. However, the latter was ultimately rejected due to high costs and challenges such as comparability between countries and settings (Bloem 2018). Another pragmatic reason for the decision for a self-report survey on ECEC staff was that the experiences of TALIS could be built upon, synergies used, and the policy relevance of an established survey used and transferred to the ECEC sector. This was considered a win-win situation by the interested countries and OECD alike:

From the OECD’s perspective a new “product” could be established with the help of an existing label while enhancing the visibility of TALIS and TALIS Starting Strong at the same time. In addition, new synergies and opportunities for comparability across different levels of education emerge. Ultimately, this also strengthens the position of the OECD as an active player for research and policy within the education sector. From the countries’ perspective, joining TALIS also allows them to compare multiple educational levels and, in the case of TALIS Starting Strong, to build on an established study. Potential risks such as project failure or immature instrument design are reduced, as these were already in the field, albeit in a different form. Overall costs for the study development can be reduced, too, and the status and visibility of ECEC increased within the education sector.

Bäumer (2014), who was commissioned by the OECD specifically to analyse whether TALIS contents and methodology could be transferred to the ECEC context, also confirmed this optimistic attitude. The expert concluded that the new study would cover many identical content areas. Following adaption to the ECEC context, a large number of questionnaire items from TALIS could be used in TALIS Starting Strong (Bäumer 2014). However, the ECEC network and participating countries emphasised from the outset that particularities and the autonomy of the ECEC sector needed to be kept in mind and that comparability with results from TALIS was of secondary importance.

While these aspects were discussed among different stakeholders, the OECD was in charge of steering and furthering this process. This is also a crucial point of what Bloem (2018) describes as “institutional dynamics” and how the OECD positions itself as an actor within the development process. An overarching mission statement and corresponding projects lead to individual studies being fitted into a larger structure, connected to overarching strategies (e.g. quantification and comparability of education) or established projects such as PISA or the “Future of Education and Skills 2030” project (OECD 2022). The OECD not only drives the emergence process at various points, but also in certain directions. By developing ideas for new projects, it gains policy-making power and the ability to decide or at least suggest which areas are investigated in different countries around the world. In doing so, however, the OECD depends on how potential participants perceive their project ideas. Only if enough countries can be convinced of a concept, this idea can be advanced. Otherwise, various resources (e.g. for the development of observational studies in ECEC) are wasted or at least postponed otherwise. If this “hurdle” can be overcome and a “critical mass” of interested actors (i.e. countries) achieved, the OECD is better able to drive the concept in a certain direction. Taking the respective countries’ interests into account remains a balancing act however. After all, the financing and thus also implementation of the projects depends entirely on the participating countries. This process is essentially a project pitch in the field of scientific studies with the involvement of different countries.

Sufficient interest could be aroused in TALIS Starting Strong for concrete implementation of the study to begin. From this point on, a variety of stakeholders responsible for different tasks were involved (see Fig. 1 for an overview). A broad exchange was initiated within the OECD Network on ECEC, coordinated by the OECD, to compile and discuss the latest issues and pressing questions while at the same time encouraging participation. A priority rating exercise was then conducted among interested countries to further systematise and narrow down the issues to address and determine the breadth and depth of the questionnaire. The majority of countries favoured greater breadth—not least due to their different areas of interest. In 2015, the final voting and definition of topics took place, with a particular focus on ensuring alignment with TALIS, but also taking national interests into account. Nine countries committed to participate in the ISCED 0.2 study (Chile, Denmark, Germany, Israel, Iceland, Japan, Korea, Norway and Turkey), whereby four of them (Denmark, Germany, Israel and Norway) additionally focused on ECEC for children aged under three. With the support of different international advisory expert groups and under the control of an international study contractorFootnote 9, preliminary survey instruments were developed based on the main topics and questions identified in the priority rating exercise. These were translated into the national languages and piloted in all nine participating countries in autumn 2016. With the aim of testing the questionnaires for relevance, comprehensibility and completeness, focus group discussions were conducted with ECEC staff in various countries as a pilot study. Feedback from ECEC professionals could thus systematically be gathered and fed back into the questionnaire development.

Fig. 1
figure 1

Overview of TALIS Starting Strong 2018 stakeholders and their interrelations. (Source: Authors’ illustration)

Full size image

Based on the revised questionnaires, a field trial was conducted in 2017 to test the main study on a smaller scale. In Germany, around 50 ECEC centres with almost 300 pedagogical staff and leaders participated. In addition to the further optimisation and testing of the survey instruments, organisational processes in the national study centres were tested, such as the recruitment of respondents or survey modes (online/offline). The questionnaires also had to be shortened so that it would not take longer than 45 to 60 min to complete in the main study 2018. To obtain feedback on as many questions as possible, different versions of the questionnaire were used as part of a rotation design, while the main study relied on one version only.

In 2018, the main survey was carried out in all nine participating countries by the respective “National Study Centres”. The International Centre for Early Childhood Education and Care (ICEC) was commissioned with management and implementation of the study in Germany. In addition, various expert advisory groups contributed their knowledge to different steps and phases of the study. In Germany, an advisory board consisting of experts from professional practice, science, politics and associations additionally accompanied the process (cf. Turani et al. 2022).

4 Adapting the TALIS methodology to TALIS Starting Strong

We now consider how the existing TALIS survey affected the development of TALIS Starting Strong and address different elements of the study design, such as participant selection and methodology.

4.1 Approaches to reduce social desirability and cultural bias

For data to be considered sound and be used by ECEC stakeholders to develop or adapt policies or measures, it is essential that the technical procedures and instruments used in the survey fulfil various scientific standards (OECD 2019a). In this respect, TALIS Starting Strong was able to benefit from its elder sibling TALIS, as the lead contractor (IEA/DPC) was also responsible for the implementation of TALIS and could draw on two cycles (2008, 2013) of experience regarding both operational aspects and questionnaire design. In an international self-report study, questionnaire development involves reducing two types of biases: social desirability bias, caused by individuals’ tendency to present themselves in a more favourable light (e.g. Holden and Passey 2009), and cultural bias, caused by the “tendency to interpret a word or action according to culturally derived meaning assigned to it” (e.g. Purtilo et al. 2018, p. 60). In some cultures, for example, physical contact (e.g. a hug) is considered intimate, whereas in others it is more of a casual gesture between acquaintances. So, if survey participants are asked about the frequency of physical contact with children in their ECEC setting, respondents from different cultures will have different notions of “physical contact”.

To avoid social desirability bias, for some questions TALIS Starting Strong adopted from TALIS the use of questions that asked about staff’s practices in the ECEC centre as a whole rather than about staff’s individual practices (e.g. TALIS Starting Strong 2018: To what extent do the following apply to staff at this ECEC centre?; TALIS 2018: Thinking about the teachers in this school, how strongly do you agree or disagree with the following statements?). To reduce possible cultural bias, two approaches were adopted from TALIS. Firstly, parallel to TALIS, TALIS Starting Strong used focus groups to pilot draft material with the aim to compare how the target population in different countries judge the relevance and comprehensibility of items. To ensure that the collected data was comparable across countries, the staff characteristics for the focus groups were clearly defined and standardised probing questions as well as standardised feedback sheets were used. Although these measures aim to ensure consistency across the different national research teams, group discussions can be dynamic and some questions might need to be dropped or altered in the flow of the discussion or are not answered by all members of the focus group. Regarding the documentation, aggregation and analysis of participants’ feedback, there can also be some variance, first at the level of the national research teams and then at the level of the international study consortium (e.g. IEA/DPC and Questionnaire Expert Group). Secondly, measurement invariance testingFootnote 10 was conducted after data collection to test whether the same constructs are being measured across all groups and participating countries. This enabled decisions on the extent to which certain scales (e.g. concerning pedagogical practices or well-being) could be used for international comparative analyses.Footnote 11

4.2 The focus on high participation rates—adequate for the ECEC context?

High minimum school participation rates (e.g. 75% after replacement) have been introduced since the first TALIS cycle in order to “minimise the negative impact of non-response biases” (OECD 2013, p. 131). Although this is an important goal and allows the results to be presented to the public and school or ECEC stakeholders as highly valid, the adequacy of a focus on high response rates is controversial. Groves et al. (2008, p. 1) point out, for example, that “nonresponse rates are poor indicators of nonresponse error of survey estimates. However, the field (and the public at large) has been taught to use response rates as quality indicators”. The OECD took the first-time success with TALIS 2008 as confirmation of their high standards: “experience and knowledge gained from TALIS 2008 showed that the targets set for TALIS 2008 participation were realistic and so could be reiterated for the 2013 cycle” (OECD 2013, p. 131). The OECD sought to apply equally high standards to TALIS Starting Strong.Footnote 12 However, in the first cycle of TALIS Starting Strong, two out of nine countries struggled to meet the high standard set for centre participation rates. How appropriate these high standards are for the ECEC sector should be reflected upon for future cycles. One aspect to consider in surveys in the ECEC sector is, for example, that centre leaders are more difficult to reach than school principals. While the latter are usually supported by a secretariat, in some countries (e.g. Germany) most ECEC centres are not. Centre leaders often also work with the children in the group and are therefore difficult to reach to discuss a survey (e.g. OECD 2020 for an international comparison of the time leaders spend with the children).

One further aspect that makes recruiting survey participants in the ECEC sector more challenging is that in several countries there is greater variety of ECEC providers than is the case for schools. The latter are organised more centrally, i.e. are run by larger public entities like the government or government-dependent private institutions (OECD 2021). A more decentralized system may also lead to less sophisticated official statistics on ECEC settings compared to statistics on the school sector. This may pose some challenges when listing the population of ECEC settings and selecting a representative sample. Survey participation can also depend on whether the survey is connected to other studies or is mandatory. For example, in a country where participation is linked to a census (e.g. Iceland) or other national studies, or is per se mandatory (e.g. Israel ISCED 0.2), the selected ECEC centres probably feel more obliged to participate. So, although high standards regarding minimum (centre) participation rates are an important goal to ensure reliable results, more obstacles can arise when reaching out to participants in countries with more decentral ECEC systems and high provider autonomy. Further quality indicators or approaches that ensure that the collected data is valid and sound could therefore be considered for future cycles.

4.3 International staff categories

To ensure comparability between staff groups across countries, TALIS Starting Strong adopted the staff categories of “Leader” (in TALIS: “Principal”) and “Teacher” from TALIS and introduced a new category “Assistant”. Each participating country could assign their staff groups working in ECEC to these (OECD 2019a). While it may seem clear who is meant with the “Principal” at a school and the “Leader” in an ECEC setting, the latter role is in fact somewhat trickier to define, because in ECEC contexts leadership tasks are sometimes shared (e.g. pedagogical vs. administrative tasks). Respondents therefore might be unable to accurately answer all leader-specific questions. Leadership teams comprising two or more leaders are also growing in popularity in some countries (e.g. Fonsén and Mäntyjärvi 2019 for Finland; Autorengruppe Fachkräftebarometer 2021 for Germany). Some settings are even specifically dividing up leadership tasks among staff and parents. In Germany, for example, there are “parent initiatives” (Elterninitiativen) in which typically only a pedagogical leader is defined and parents take care of the other tasks (e.g. finances, staff recruitment, admission criteria). Such parent initiatives make up about 7% of all ECEC settings in Germany (Autorengruppe Fachkräftebarometer 2021). In these cases, the pedagogical leader might not be as knowledgeable about some aspects and needs to confer with the parents to complete the survey. These various phenomena may need to be taken into account when developing TALIS Starting Strong further.

In contrast to the “Leader” and “Principal” categories, the definition for the “Teacher” staff category differed in TALIS and TALIS Staring Strong. In TALIS “Teacher” referred to someone, who actively and regularly teaches (OECD 2019c). In TALIS Starting Strong “Teacher”, referred to someone regularly working with children and was also the person with the most responsibility for a group of children. For TALIS Starting Strong, the main assumption was that ECEC professionals worldwide work in teams. This led to the introduction of “Assistants” as a second main staff category for TALIS Starting Strong, to refer to someone, who regularly works with children and supports the “Teacher” with a group of children.Footnote 13 This new category is very important to detect how the roles in ECEC teams differ and determine differences between the “Teacher” and “Assistant” staff groups, for example, regarding qualifications, professional development needs or subjective well-being. These comparisons can be interesting for countries where persons with the same qualification level can fulfil both roles (e.g. Germany) as well as for countries where persons in the “Teacher” role typically have a higher qualification than those in the “Assistant” role (e.g. Norway).Footnote 14

Within some ECEC teams, the official staff roles only play a minor role in everyday operations. In the German main survey in 2018, for example, some centre leaders mentioned that official staff roles were not particularly relevant in their daily work and that each team member felt equally responsible for the group. One question that cannot yet be answered using TALIS Starting Strong data, but would undoubtedly be interesting for further research is the role ECEC staff assign themselves if responsibility for the group is shared equally between team members. While we would argue that the use of international staff categories and particularly the introduction of the new “Assistant” category worked for the first cycle of TALIS Starting Strong, potential exists to take a closer look at the team dynamics and to what extent team members work according to their “official” roles.

4.4 Transferring items from TALIS to TALIS Starting Strong

The TALIS questionnaire was a good starting point for development of the content for TALIS Starting Strong 2018, particularly for constructs like staff well-being. However, it was hardly possible to transfer identical items for other areas, for example, with regard to the pedagogical practices. In the conceptual phase of questionnaire development, it was anticipated that approximately 70% of the content could be taken from TALIS with no or only minimal adaptations (OECD 2019a, pp. 54–56). Table 1 shows that this initial estimation was quite optimistic. Already in the pilot study only 54% of the TALIS leader questionnaire items and 35% of the staff questionnaire items were used for the equivalent TALIS Starting Strong questionnaires. This overlap proportion stayed roughly the same for the main survey (leader questionnaire: 49%, staff questionnaire: 30%). Significant is that the number of identical items were reduced by half comparing the pilot versions with the final versions in the main study.Footnote 15 The eliminated identical items often referred to staff’s pedagogical practices, for example, original TALIS items like I give students projects that require at least one week to complete (adapted in TALIS Starting Strong 2018 pilot version: I give children projects that require more than one day to complete) or I let children use ICT for projects or class work (adapted in TALIS Starting Strong 2018 pilot version: I let children use ICT for projects [e.g. computers, tablets, iPads]) were thrown out after the pilot due to their lower relevance for the work with preschool children. So, although TALIS was a good starting point for the TALIS Starting Strong questionnaires, the Questionnaire Expert Group for ECEC right from the start made necessary adaptations and at the same time the feedback from the pilot study (and field trial) had an extremely important function in ensuring that the transferred content was relevant for the work in ECEC centres.

Table 1 Overlap between TALIS Starting Strong 2018 and TALIS 2018 across different study phases
Full size table

4.5 The “target group” assessment approach

As already discussed, the TALIS Starting Strong questionnaire is strongly modelled on the TALIS questionnaire, which mainly consists of dichotomous yes/no questions or questions that mostly use four-point Likert scales. One key assessment approach adapted from TALIS is the “target group”. Whereas this approach has proven helpful in the school context, some challenges exist when it is transferred to the ECEC context. Respondents should not think about children in general, but rather about a particular group at a certain time. The aim of this approach is to enhance accuracy and comparability. In the “target group” section of TALIS Starting Strong, pedagogical staff were instructed to think about their first group of children on the last working day before they completed the questionnaire. In contrast, TALIS asked school teachers to think about a class “last Tuesday at 11 a.m.”. For a teacher, the group of pupils in a certain subject is generally fixed (if it is a regular class), the teacher should be familiar with the group and have some routine regarding teaching practices. For staff in ECEC settings with (mainly) fixed groups, the situation is similar and they mostly work with the same group of children. However, different group concepts are used in ECEC settings. So there might generally be fixed groups, but also periods during the day when children from different groups come together (garden or yoga time, etc.) or when children are allowed to move freely between rooms (e.g. switching from the “creative room” to the “sports room”). Some ECEC settings even implement a completely open group concept where staff potentially work with a different group of children every day. Open concept settings encourage children to choose from different activities which are offered by different staff each day.Footnote 16 Staff working with mainly fixed groups face different conditions compared to staff working with changing groups of children and this could potentially influence how accurately staff can describe children’s features and the extent to which certain practices are employed. Given the great variety in how ECEC centres can structure their day, one could consider assessing information on the group concept in future cycles. This would not only provide greater insights into the organisation of pedagogical practice, but also allow more accurate analyses of the team composition in relation to the children being cared for. When looking at target group data in TALIS Starting Strong, it is important to keep in mind that staff—in contrast to the school sector—can think of very different situations when answering these questions.

5 Conclusion and outlook for TALIS 2024

In this article, we aimed to analyse how an established survey for school teachers and principals—TALIS—impacts the design of a new staff survey for the ECEC context: TALIS Starting Strong.

Our review of the evolution of TALIS Starting Strong showed that “copying” from TALIS was worthwhile: the new study could benefit from the experience gained from TALIS in several ways. First, it could build on established structures and procedures as well as use a known name to raise awareness. Additionally, the possibility to investigate similarities and differences between different educational levels helped to raise the study’s appeal among potential participating countries. On the downside, it was discussed that the minimum participation rates, which were originally established for the school sector, should be reviewed taking into account that some ECEC systems are organised in a less central way than school systems. In addition, the affinity with TALIS led to concerns that ECEC could be seen merely as an extension of the school sector. Hence, the autonomy of ECEC was emphasised by different stakeholders during all stages of study development. The example of TALIS Starting Strong also demonstrated how the OECD shapes study development processes and establishes new studies that fit seamlessly into their research programme. The OECD can both influence the political agenda within the education sector of its member states and establish itself as an important international research actor in this way.

Second, the discussion on adaptation of the methodology also showed that TALIS Starting Strong could benefit from TALIS. When dealing with methodological challenges of international large-scale self-report studies such as social desirability or cultural bias, it can be helpful to have a role model with established standards to guide a new study. That being said, international large-scale assessments not only need to be valid in a technical sense, but also with regard to the content (e.g. Gorur 2017). The smaller than anticipated overlap between items in TALIS and TALIS Starting Strong, clearly points to the specificities of both sectors. While TALIS provided a good basis for constructs like staff well-being, it was important to develop new items on pedagogical practices, which take into account that interactions between staff and children in ECEC settings can differ from and be more dynamic than those in a school setting. When comparing the categories used to classify staff, it became apparent that it is often difficult to make such role assignments in ECEC contexts where staff and even leader roles sometimes change or merge. The attempt to transfer the concept of the “target group” also shows that there is often a greater diversity of team constellations and different forms of work in ECEC centres compared to school settings.

International large-scale staff surveys are an opportunity to inform policymakers about how ECEC professionals feel about their job and to learn whether staff are providing the pedagogical quality aimed for in ECEC curricula. Ideally, the results can be used to identify areas of improvement, which can be addressed with new or adapted policies. However, at the same time, policymakers are criticised for (mis)using the data for their own agenda by cherry-picking only those results that back up the policies they wish to support (e.g. Gorur 2017). While TALIS Starting Strong highlighted some of the advantages of such a study, the discussion also pointed to some specific challenges. The preparations for TALIS Starting Strong 2024 are currently underway and many of the issues addressed above are being discussed by the stakeholders involved. Although the impact of TALIS on TALIS Starting Strong was notable during the first cycle, the links between both studies will be strengthened further for the second cycle. In line with its long-term data strategy, the OECD has moved TALIS Starting Strong “formally under the TALIS umbrella” and integrated it “in the work programme of the TALIS Governing Board” (OECD 2019b, p. 4). This will lead to further alignments, for example in terms of operational processes including schedules or meetings, but also conceptual developments (ibid, pp. 7–8). It will be interesting to observe the consequences of this development.

It goes without saying that it is important to collect data that can be used as a starting point to initiate necessary improvements in ECEC. For this goal, it is essential that we continue discussing what international categories or assessment approaches are helpful, for example, to assess staff roles and group concepts in ECEC settings in an international comparative survey. Doing so can help to further the critical and constructive debate on the function and use of international large-scale data.