Talking About Education: How Topics Vary Between International Organizations

In this chapter, Raphael Heiberger and Michael Windzio examine which topics are important for major education international organization (IOs). IOs in the field of education follow different ideological paradigms in the global education discourse. Yet, it is an open question as to whether different types of IOs focus on different topics and thereby support different paradigms of education. Based on more than 1000 documents with over 40 million words published by the World Bank, UNESCO, the ILO, the OECD, ISESCO, and SEAMEO, they explore education issues addressed in this sample. Using standardized methods of quantitative text analysis and topic modeling, Heiberger and Windzio reveal that major topics found in these documents do indeed differ between the different types of organizations.


Introduction 1
Policymakers usually regard education as a domestic policy field, where responsibility rests either with the national government or, as in most federal states, is delegated to subnational units. That education policy is heavily influenced by transnational entities, and by international organizations (IOs) in particular, is a fairly new insight within the realm of academic research and also a new experience for some policymakers (Martens et al. 2010(Martens et al. , 2014. Tracing the influences of these IOs in detail is challenging for several reasons, one being the great variety of IOs-currently around 30-that are active in the policy field of education (Niemann and Martens 2021). While many industrialized countries have responded to the results of the Organisation for Economic Co-operation and Development's (OECD) Programme for International Student Assessment (PISA) study over the years, it remains an open question whether and how other organizations influence domestic education policies, and, if so, which countries are affected.
This study takes one step back from such a causal analysis and instead aims at analyzing the programs and recommendations communicated by IOs in their major documents in a descriptive manner. If we assume that IOs are more or less powerful actors in the global world system and that they try to influence domestic education policies (Martens et al. 2007), then we should first attempt to understand their missions and preferred goals in the field of education. In line with the studies by Niemann (Chap. 5) and Krogmann (Chaps. 7 and 8), we analyze the goals and ideas of IOs where education is concerned but apply a different methodological approach. We created a comprehensive text corpus (over 40 million words) from documents published by the six IOs that were selected to be analyzed in this book (Chap. 1) and applied structural topic models (STMs) to analyze the basic topics these organizations are dealing with. We selected the IOs according to a two-dimensional classification that distinguishes between derivative and dedicated education IOs, on the one hand, and IOs with open or restricted membership criteria, on the other hand.
According to our results, the six IOs do indeed focus on different issues. As we will see, this is basically due to the outstanding degree of homogeneity of topics communicated in the documents of those IOs that are dedicated to the issue of education, but at the same time restricted in their membership. These IOs are the Islamic Word Educational, Scientific and Cultural Organization (ICESCO) and the Southeast Asia Ministers of Education Organization (SEAMEO).

Theoretical Background
How can we explain why IOs communicate about education in different ways? This is an important question given that we know IOs can considerably influence domestic education policies. When the OECD published the results of its first PISA study in 2000, policymakers and researchers controversially discussed the comparisons of educational performance and their policy implications. Many countries took the opportunity to implement reforms after being disappointed with their poor performance. The OECD thus had a strong influence on their domestic education policies, even though the organization itself does not have any formal authority. Switzerland, for example, was receptive to the study because education reforms had been considered overdue for decades; however, they were difficult to implement in a federal system with many veto players (Bieber and Martens 2011). In Germany, education policy changed so dramatically after the PISA study that scholars even used metaphors such as "Turn of the Tide" (Niemann 2010) or "After the Big Bang" (Niemann 2014).
Not all countries in the world are members of the OECD, but its concepts and suggestions for economic development are rather general and might also be applicable for non-members. Similarly, the United Nations Educational, Scientific and Cultural Organization (UNESCO) is well known for its focus on the role of education for economic and social development in the Global South but is not at all limited to these issues. Other organizations focus on particular world regions or on countries where a particular culture or religion prevails.
Communication theories often regard communication as a system embedded in surrounding social systems (Luhmann 2000). Mass communication operates in specific ecologies where different media compete for niches to survive, maintain their communication, and get attention. Niche theory argues that the overall semantic space of communication at a given moment results from variation and selection within this competitive discourse environment. If a given ecology is unfavorable toward a particular communication, actors often seek to establish their own niche or try to fill existing, but alternative, niches (Riedlinger and Rea 2015).
For instance, a new niche was established when communication on the Internet started, and what we observe today is an increasing differentiation of niches that becomes apparent in various different online platforms that engage in tight competition for attention. Despite this fine-grained differentiation, the proliferation of certain niches such as YouTube is a challenge for traditional media. Not only the form but also the content of a newly emerged discourse niche might even displace previously dominant niches in the future (Ha and Fang 2012).
According to this niche competition argument, instead of there being just one discourse on education led by one dominant and universalistic IO, there might be different discourses in distinct niches. Such niche differentiation might be brought about by perception that the dominant and universalistic IO does not put sufficient weight on a particular issue, or the perception that the discourse of the dominant IO is somehow biased by a hegemonic ideology. This argument might be particularly applicable to the field of education because education is closely related to the intergenerational reproduction of culture. In the long run, the persistence of different "cultural spheres" (Chap. 1) also depends on how countries organize the form and content of their education systems.
We thus expect there to be considerable differences in how these IOs "talk" about education. To date, however, there is no standardized analysis of the most prevalent topics being discussed by these IOs. Education meets various functional requirements in a society such as generating literacy in the population, supporting ideas of legitimacy appropriate to the respective institutional order, laying the foundation of human capital investments, integrating migrants and their descendants into society, and preventing juveniles from becoming delinquent (Windzio 2013). It is likely that these and other functions of the education system are weighted differently in the discourses of these IOs. According to the "cultural spheres" argument ( Chap. 1), which highlights the importance of education for the intergenerational reproduction of particular cultures, we expect that the most prevalent topics differ between the IOs depending on whether they are derivative or dedicated, open or restrictive. We thus classify active IOs in the field of education along the two dimensions derivative-dedicated and open-restricted (Chap. 1). An IO is derivative with respect to education if it mainly deals with more general issues, such as economic development, while covering education as just one issue among many. A dedicated IO is one whose major topic is in fact education. Furthermore, the nature of IOs' membership rules can be open to any state or be restricted according to specific criteria such as geographical region or religion. Table 9.1 shows the categorization of the IOs and the number of documents considered in the analysis. The International Labour Organization (ILO), the World Bank, and the OECD are organizations that are not primarily focused on education, whereby the latter IO is also restricted in its membership rules and essentially only includes economically developed countries. In contrast, UNESCO, ICESCO, and SEAMEO all have education as their main focus, but only UNESCO membership is open to any state. ICESCO only includes countries with a high share of Muslims and thus restricts membership to a particular religion, while SEAMEO membership consists of 11 Southeast Asian countries and is thus restricted to a particular world region. Given that there already existed an IO that supposedly handles the educational concerns of all countries in the world (UNESCO), there must be a reason why these member-restrictive organizations were formed and continue to be active today. We assume that this is because these IOs focus their communication specifically on their region, as in the case of SEAMEO, or on their particular culture and religion, as is the case with ICESCO.
In the following sections, we apply structural topic models to a large text corpus of IO documents in order to trace the major issues these organizations are talking about.

Data Preprocessing
As a first step in the creation of the text corpus, we defined a search string in order to restrict the selected documents to the issue of education by using the keyword "education." After the exclusion of clearly inappropriate documents, these keywords yielded a set of N = 1603 documents (see Table 9.1). The clearing of the text corpus included the removal of numbers, punctuation, and special characters. Subsequently, we identified English stop words and extended the set of stop words by 21 corpusspecific words (e.g., "chapter," "table," "director"). Words with less than three letters were deleted.
We then lemmatized the corpus by using the function lemmatizestring from the R package textstem. Finally, we removed tokens that appeared in less than three documents.

Structural Topic Models
To reduce the semantic space of the documents published by the six IOs to meaningful dimensions, we applied a recent variety of topic models known as Structural Topic Modelling (Roberts et al. 2014(Roberts et al. , 2016. Topics are directly derived from "bag-of-words" (meaning that word order and grammar are ignored) by probabilistic algorithms. The main idea behind all topic models is that words co-occurring in and across documents describe meaningful themes (or topics). All topic models are generative models of word counts using a two-step method of clustering. A topic is defined as a mixture of words, where each word has a probability of belonging to a topic (1st cluster step), and a document is a mixture of topics, meaning that each document is composed of multiple topics (2nd clustering). As such, the sum of the topic proportions across all topics for a document is one (also known as θ, the topic load of a document). In addition, the sum of the topic probabilities for a word across all topics is also one (also known as β, a word's contribution to a topic). Thus, the more often words co-occur in documents, the higher the probability that they constitute a topic.
Compared to other topic models (Jordan and Mitchell 2015), STM allows for the improvement of the estimation of topics by using document meta-data as covariates. STM does not assume that the distribution of words is the same for all documents, but words in documents with the same covariates (e.g., year, source) have a higher likelihood of being clustered together and forming a topic. It has been shown that the inclusion of covariates improves the quality of topic selection substantially (Roberts et al. 2014(Roberts et al. , 2016, especially for documents covering longer time periods (Farrell 2016).

Validation
Although the proposed STM solves other technical issues, such as finding the optimal starting parameters and providing consistent results by a "spectral initialization" (Roberts et al. 2016), it does not solve the issue of selecting an appropriate number of topics, which is crucial for any further analysis. Comparable to efforts in cluster analysis to determine the optimal number of clusters, there is no "right" answer to the question of how many topics are appropriate for a given corpus (Grimmer and Stewart 2013); though there do exist better and worse choices. Given the importance of the question, a careful examination and consideration of different topic solutions is key to every scientific dimensional reduction technique.

Internal Validation
The idea of internal validation is to identify a model with topics that best reflect weighted bags of words that are used by interviewees. In other words, we test which topics (respectively the most defining words of a topic) best predict interviewees' responses (aka the text). For that general purpose, semantic coherence and exclusivity are widely used measures Roberts et al. 2016).
The coherence of a semantic space addresses whether a topic is internally consistent by calculating the frequency with which high-probability topic words tend to co-occur in documents. Coherence grows with the likelihood of a topic's most probable words co-occurring together. The authors who introduced this measure, Mimno and Blei (2011), validated it for academic writing. They showed that it had high correspondence with the judgments of National Institutes of Health (NIH) officials on a set of NIH grants .
However, semantic coherence alone can be misleading since high values can simply be a result of very common words of a topic that occur together in most documents. To account for the desired statistical discrimination between topics, we may consider a second metric proposed by Roberts et al. (2014) and measure the exclusivity of a topic. Exclusivity provides us with the extent to which the words of a topic are distinct from it. Both exclusivity and coherence complement each other and are examined in concert to give us a first impression of where topics represent word distributions in documents and at the same time provide distinct dimensions.
The developers of STM recommend that researchers look for the "semantic coherence-exclusivity frontier"-namely, the specification after which allowing for more topics fails to produce models that dominate others in terms of semantic coherence and exclusivity (Roberts et al. 2014, 7). Thus, what we are looking for is when both indicators build a plateau, that is, they do not improve (much) with higher Ks. This gives us an upper limit for a reasonable K. Figure 9.1 reveals a "jump" in coherence at K = 70. Before and after that jump it decreases with higher values of K. Looking at exclusivity, we see that it increases only marginally after values of K are greater than 50. Figure 9.1 displays the distribution of exclusivity and semantic coherence. In general, we see that exclusivity gets better with a rising number of topics, but coherence gets worse.
In addition, we look at the held-out likelihood. This is "a measure of predictive power to evaluate comparative performance," in this case among models that allow for different numbers of topics (Roberts et al. 2016, 22). To obtain the held-out likelihood of an STM, we first subset 10% of the documents in the corpus and hold out half of the words in them. We then evaluate the likelihood of the held-out words. Higher likelihoods indicate a more predictive model. In general, the loglikelihood of K is falling (more general topics, more predictive power). Thus, we are looking for some "breaks" in a straight line. Figure 9.2 shows a corresponding "plateau" to 1 around K = 70. That means the LL is not improving much between steps 60 and 70, where we would expect to see a linear falling curve.

Robustness
In addition to internal validation, we may want to include a second pillar by checking robustness across K. It is probably the most important way to show that our choice of K is not changing the whole semantic space and hence our (descriptive) analysis.
For this purpose, we use an indicator developed for comparing clustering solutions called "Fowlkes-Mallows Index" (FMI). It basically tests the relation of true and false positives/negatives across cluster assignments, or in our case, the assignment of a document to a topic. We used the max-approach to assign topics to documents, that is, max(theta) (maximum topic load) of a document defines its topic. The FMI may give us a straightforward way of measuring consistency by investigating the rate of change with regard to topic assignments across different Ks. K on the x-axis represents the similarity of topic assignments for all documents between two consecutive Ks, that is, an STM with k-topics is compared to the next smallest STM with k = 10 (e.g. k = 10 and k = 20 are compared at the first x-tick). Figure 9.3 shows the results for the FMI. We see consistent values above 0.7 for K = 70. This means that more than 70% of documents' maximum topic assignments at K = 60 are the same at the next highest K. Thus, K = 70 again marks a local peak, this time in terms of "robust" topic assignments. In general, we note that the finer the semantic grid with rising K, the lesser the changes necessary to classify documents sufficiently. Therefore, all validation results point to K = 70 as the most valid and consistent choice for the number of topics. However, we see in Fig. 9.3 that the choice of K is not changing the semantic dimensions (i.e., topics) in a drastic way.

Results
According to the diagnostics presented in the previous section, we opted for a solution with 70 topics. Each of these topics has a specific prevalence, that is, a specific share in the overall set of these topics. Taken together, all 70 topics make up 100% of the semantic space. Figure 9.4 shows the ten most important topics according to their topic prevalence and the five most important words that best represent the respective topic. For an understanding of what these topics mean, it is necessary to inspect the topic-word list in the Appendix. Topic 57 refers to the level of education of students and the OECD, whereas topic 64 refers to national policies related to countries' development. Topic 49 appears similar to topic 64 but focuses more on communities. Topic 13 focuses on the OECD, students, and policies in conjunction with schools. In contrast, topic 18 is more focused on teachers and their profession, while topic 53 is clearly centered on vocational training and the OECD. Topic 60 is also focused on teachers, but it additionally relates to recommendations and committees. Topic 26 is clearly related to labor training and the ILO, while topic 39 is centered on tertiary educational institutions (particularly universities) and topic 40 is again more focused on skills, training, and development. At first glance, topics 57 and 13 seem to be similar, but closer inspection reveals that topic 13 is limited to secondary education, whereas topic 57 also deals with tertiary education. As this example shows, it can sometimes be quite difficult to clearly discern the content of all 70 topics, but aside from such instances, the top ten topics of our analysis are actually quite easy to distinguish between since they are dealing with distinct topics such as performance levels of secondary education, teacher-related issues, tertiary education, vocational training, or labor-related issues.
More important for our research question is the relative importance of the 70 topics for the different IOs. Figure 9.5 shows pie plots with the three most important topics for each of our six IOs. The higher the proportion covered by the three topics, the more focused the documents of the respective IO are on fewer issues. Contrariwise, if an organization discusses a variety of different issues in equal proportion, then the topic heterogeneity and overall share of the three most important topics are relatively low. It is thus striking in Fig. 9.5 that for ICESCO and SEAMEO, at least three quarters of their communication are concentrated among the three most important topics. As argued in the theoretical section of this chapter, these dedicated organizations are focused on education. Even though UNESCO  is a dedicated IO whose membership is open to all countries in the world, ICESCO and SEAMEO create and maintain their own distinct thematic niches, which implies that, in their view, UNESCO does not appropriately address these niches (see Appendix for an overview of all 70 topics). A closer inspection of the content of these topics reveals that this does in fact seem to be the case. ICESCO devotes 54.7% of its communication to topic 63, which is about "Islam, culture, and the state." The remaining two most important topics make up only a small proportion of around 4-5%. Accordingly, the main focus of ICESCO is clearly on Islam, the social world, the state, and rights. The concentration of topics is even more pronounced for SEAMEO. Here, 78.6% of its communication is concentrated on topic 7, which is about "development, SE Asia" as well as on the countries Thailand and Malaysia. The remaining two most important topics make up around 3% each and focus on "early childhood and care" (topic 62), and "teacher and digital technology" (topic 33).
In contrast to ICESCO and SEAMEO, the three most important topics of the OECD make up only one-third of its overall communication combined. At the same time, the distribution of the topic proportions is more equal: The most important issue has a share of 12.1% and is focused on "earnings, tertiary and non-tertiary" (topic 57). In the overall text corpus, this is the most prevalent topic (Fig. 9.4). In addition, topic 53, which makes up 7.2% of the OECD's communication, ranks sixth place among the most important topics in the overall corpus and is clearly focused on "vocational training and apprenticeship," whereas topic 13 ("assessment, average, and Latvia") makes up 8.3% and relates to schools, secondary education, and national averages. Even though the OECD is an organization with restricted membership, it is nevertheless rather broad and heterogeneous in its communication about education. Much like the OECD, the three most important topics for UNESCO also make up around one-third of its communication. Topic 64 is its most important topic, accounting for 13.1%, and rather broadly addresses issues on "support national development" and training. In total, 10.3% of UNESCO's communication is again devoted to issues relevant to development, such as "problems of adult learning." Interestingly, UNESCO also communicates considerably (7.1%) on topic 63 ("Islam, culture, and the state"). As we have seen already, this is by far the most important topic for ICESCO (54.7%), and despite being remarkably less relevant for UNESCO, it nevertheless accounts for one of its three most important topics. This probably has to do with UNESCO's activities in fields such as the preservation of cultural heritage and culture in general.
The World Bank shows the highest heterogeneity in its topics compared to the other IOs, since only around 15% is concentrated on the three most important topics, which are topic 49 ("development projects and community"), topic 48 ("South Africa, development, and telecommunication"), and topic 40 ("vocational training and East Asia"). The ILO shows a concentration of topics: "labor, train, development, and worker" (topic 26, 15.1%), "selection of teachers" (topic 60, 14.2%), and "war, employment, and wages" (topic 55, 10.6%).
To sum up, what we can learn from this standardized analysis of the comprehensive text corpus is that our six IOs do indeed talk differently about education. These differences are best described by the combination of two characteristics, in the lower right cell of our 2 × 2 table (Table 9.1). If IOs dedicated to the issue of education are at the same time restricted in their membership (e.g., through their focus on either a particular cultural sphere or world region), then they will tend to focus their communication on specific issues much more than either derivative IOs or UNESCO, which is a dedicated IO but not restricted in membership. Interestingly, the case of the OECD underscores that restrictions on membership do not necessarily imply that the communication is limited to just a few topics (Niemann and Martens 2021). Despite its membership restrictions, the OECD seems to communicate on a huge variety of issues rather than focus on just a few specific interests.

Conclusion
We began our study by describing the classification of our six IOs into a 2 × 2 table, defined by the dimensions derivative-dedicated and openrestricted (Chap. 1). If comprehensive, universalistic IOs such as the World Bank or UNESCO claim to represent the global world system, the mere existence of dedicated IOs with restricted membership is a puzzling occurrence in and of itself. Why do regional or culturally specific IOs exist? Universalistic IOs such as UNESCO and the World Bank do not restrict membership. Nevertheless, it seems that some groups of countries do not feel represented by them. UNESCO is not regarded as an appropriate representation of certain countries when it comes to the cultural aspects of education, while the World Bank's inaccessibility stems from its emphasis on the role education plays for the economy.
Our analysis of a huge text corpus of IO documents showed that the major topics communicated by the respective organizations clearly differ. The most important distinction is between ICESCO and SEAMEO, on the one hand, that is, the two IOs that fall in both the restricted and dedicated categories, and UNESCO, the OECD, and the World Bank, on the other hand. The former IOs show a rather homogeneous, highly focused content when it comes to their communication; there is basically one "big issue" they are dealing with. ICESCO's "big issue" is clearly the topic "Islam, culture, and the state," regardless of the fact that this topic is also somewhat important to UNESCO. Nevertheless, UNESCO deals with a variety of different issues and pays only minor attention to this particular topic. Perhaps this is the reason why ICESCO exists: religion and culture might be so important in the cultural sphere of Muslim countries (Chap. 2) that they consider it necessary to have a specific IO for this purpose. As Krogmann (Chap. 7) has shown, preserving the Islamic religion and culture while at the same time strengthening the economic and political power of Muslim countries is the major goal of ICESCO. Unsurprisingly, as a universalistic IO, it is not part of UNESCO's major goals to "allow the Islamic world to regain its leading role in building human civilization" (ISESCO 2017, 7), even though "Islam, culture, and the state" do play a role for UNESCO as well. Our results are thus in line with the idea that cultural spheres are important in global education policies. However, it remains an open question how important and effective ICESCO actually is in shaping education policies as well as systems in its member countries.
The goals of SEAMEO are less clearly defined. It is focused on its member countries and thereby on the particular region of Southeast Asia. Education policy is considered important for the development of this region, but it is again an open question why countries in this region do not consider themselves to be well represented by the World Bank or UNESCO. In contrast to these two IOs, the other four show much more diversity in their major topics. Even though the ILO is focused on labor issues, its communication seems to be spread more evenly over different topics, and these topics address all countries in the world.
Education is a special policy field since it is related to the reproduction of national or regional cultures. At least with respect to ICESCO, it is quite reasonable to assume that the reason this IO exists is because of the importance of Islamic civilization in the eyes of its member countries. An interesting question is whether we would detect regional or culturally specific IOs in other policy areas as well. If so, we could ask similar questions: What motivates countries to create and maintain activities in dedicated IOs and to maintain and occupy specific thematic niches? Why do these countries not consider themselves well represented by the major universalistic IOs? Is education the only policy area in which, for example, culture is so important that specialized IOs become active? This might be an interesting research agenda for studies on IOs in other areas of social policy.

No. Topic
Coherence Exclusivity Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/ by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence and indicate if changes were made. The images or other third party material in this chapter are included in the chapter's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.