Keywords

The intention of this research project has been to examine how policymakers mobilize evidence to advance educational reforms. Specifically, we sought to inspect how policymakers in five countries—Denmark, Finland, Iceland, Norway, and Sweden—link different types of evidence to policy, and to explore differences and commonalities between these countries. This set of objectives required a careful look into the architecture of policy knowledge, which includes the visible links between policy documents and other knowledge artifacts, such as articles, chapters, books, reports, and statistical analyses. In what follows, we describe the method behind this research project and the procedures we employed in each of the chapters. We detail important decisions about our methodological approach for analyzing and presenting data from policy documents.

Theoretical and Methodological Inspirations

Two lines of research inspired the research design: (a) social network analysis and (b) sociology of knowledge. In this section, we discuss the premise of each literature and the main concepts. We posit that a synthesis of these two literatures could offer a new way to examine how policymakers mobilize evidence to advance educational reforms.

The first is social network analysis (SNA), which is a broad research paradigm that includes theory, substance, and methodology. The basic definition of a social network is “a finite set or sets of actors and the relation or relations defined on them” (Wasserman & Faust, 1994, p. 20). In other words, a network is a set of socially relevant nodes that are possibly connected by one or more relations (Marin & Wellman, 2011). While some actors are connected, other actors might be disconcerted or isolated from each other. The key premise of SNA is that relationships between actors influence outcomes (e.g., attitudes and behaviors) beyond the actor’s characteristics alone (Valente, 2010). Further, relationships between actors determine in part what happens to a group of actors as a whole. Therefore, social network scholars examine the structure that emerges from these social patterns with the objective of understanding the ways in which this structure contributes to specific outcomes. Because actors and relationships are fundamental components in SNA, we now turn to defining these concepts.

Actors (or nodes) are discrete units or groups. Examples of actors are students in a classroom, schools within an educational system, non-governmental organizations (NGOs) in a given field, or nation-states in the world system. Most social network studies focus on collections of actors that are all the same type (e.g., schools in the same education system). However, some studies include actors from different sets (e.g., schools and NGOs that support them). The former is often called a one-mode network, whereas the latter is often called a two-mode network. Actors have attributes—or characteristics—that describe and distinguish them. For example, students could be described by their sex/gender, age, grade, and socio-economic background; schools could be described by their location (rural/urban), selectivity, and extent of important resources.

Relationships (or edges) are the ties that connect actors. The defining feature of a relationship is that it establishes a linkage between two actors. Relationships among actors can be of different kinds, and each type facilitates a corresponding network. Borgatti and Ofem (2010) offer a useful typology of relations studied in SNA that divides relations into five types: (a) similarities include spatial and temporal proximities, co-membership in groups or events, and sharing an attribute; (b) social relations are ties such as kinship and friendship; (c) mental relations are perceptions of and attitudes toward others; (d) interactions are discrete events that can be tallied over a period; and (e) flows are interactions that are transmitted.

SNA posits that an actor’s position in each network shapes the opportunities and constraints that the actor will encounter. This perspective is distinctive from traditional social science, which focuses on the characteristics of actors as predictors of different outcomes. In traditional social science, we might explain differences in the performance of individuals or groups by certain qualities or characteristics. In contrast, SNA considers the web of relationships in which individuals or groups are embedded.

In this research project, actors/nodes are documents produced and used in a given policy space.Footnote 1 Specifically, we operationalize the architecture of policy knowledge as a two-mode network that consists of source documents and reference documents. Source documents include a set of white papers (WPs) and green papers (GPs); reference documents include other artifacts referenced in the source documents. We posit that policy documents lend themselves as a strategic site to examine how policymakers draw on evidence and justify their political decisions. We describe each document with the following attributes: year of publication, type of publication (e.g., articles, chapters, books, reports, and statistical analyses), author (either individual or institutional), publisher, and place of origin (e.g., domestic, regional, international). We focus on relationships/edges between policy documents and other knowledge artifacts. Specifically, we examine citations as particularly important connections between policy documents and evidence. We assume that authors of policy documents mobilize evidence—and thus cite knowledge artifacts—to persuade audiences of the legitimacy of a policy statement. In other words, references/citations are interactions through which authority flows from policy documents to evidence.

The second line of research that informs this research project is the sociology of knowledge, which explores the production of knowledge (any knowledge) as a social activity. For example, recent work in the sociology of knowledge demonstrates a direct link between scientific collaboration networks and the structure of ideas. Moody (2004) analyzes patterns of co-authorship in all English journal articles listed in sociological abstracts that were published between 1963 and 1999. The results show that research specialty and methodology shape participation in the sociology collaboration network. Pizmony-Levy (2016) analyzed joint membership patterns in the Comparative and International Education Society (CIES) Special Interest Groups (SIGs), arguing that membership in these groups indicates a commitment to the fields of interest. The research suggests that thematic SIGs (e.g., Globalization and Education and Higher Education) are more central than regional SIGs.

Specifically, we draw on bibliometric analysis, which identifies prominent authors, documents, and journals within a scientific community (Börner et al., 2003). Basic bibliometric analysis uses descriptive statistics to document trends in topics and research approaches used by scholars. More advanced bibliometric analysis uses network analysis to provide a deeper and more comprehensive view of relational and structural features of a given corpus of knowledge. For example, Menashy and Read (2016) examined the references in World Bank publications to identify the disciplinary foundation and the geographic representation of Bank knowledge on the theme of private sector engagement in education. Verger et al. (2019) looked at the references in the education privatization literature to explore the bibliographic coupling of academic and international agency’s body of knowledge on education privatization.

Citations—also known as bibliographic references—are the building blocks of any bibliometric analysis. Martyn (1975) argues that bibliographic references “expressly state a connection between two documents, one which cites and the other which is cited” (p. 290). Merton (1973) asserts that citations are designed to “prove the historical lineage of knowledge and to guide readers of new work to sources they may want to check or draw upon themselves” (p. VI). Indeed, scientific tradition requires that scientists, when reporting their own research, refer to earlier works that relate to their research (Nicolaisen, 2007). Each bibliographic reference is an inscription (Latour & Woolgar, 1986, pp. 45–53) describing a certain text by a standardized code that includes author name, title, journal name, publisher, year of publication, and page numbers. The impact of a publication is often gauged by the number of times it has been cited by other authors.

In this research project, we combine SNA and bibliometric analysis to examine the architecture of policy knowledge in five countries. We investigate the extent to which policy documents cite reference documents included in our database. Citation networks, such as the one we study, are more of a sociocultural network in that authors of policy documents may cite other authors they have never met or could not possibly have met (White, 2011). Whereas bibliometric analysis of scientific papers often uses existing databases of scientific and scholarly research (e.g., Web of Science), there are no similar databases for policy documents. Therefore, in the next section, we describe the data and methods we adopt for this research project.

Constructing Networks

Researchers have been creative in obtaining data on social networks from diverse sources. In addition to using surveys and questionnaires, scholars have used archival sources and other documents extensively (Marsden, 1990). Interested in international student mobility, for example, Shields (2013) assembled information from country reports to the UNESCO Institute of Statistics on incoming students and their country of origin. Addressing questions about the system of “reference societies” in the context of education reform, Kessler and Pizmony-Levy (2020) extracted information from news stories published following the release of OECD/PISA results in 23 countries.

To understand the architecture of policy knowledge, we examined a sample of official policy documents from each of the five countries participating in the study. Each national team has identified a set of key policy documents that reflected official/state policy knowledge that the government had used in preparation for the most recent school reform in each country. Table 3.1 presents the titles of the reforms in all five countries. We first selected WPs that outlined the policy proposal in each country. The purpose of WPs is to launch a debate with stakeholders, including the public, unions, civil society, parliament, and the government. We then selected GPs that were explicitly cited in the WPs. GPs are written by government-appointed expert commissions to stimulate discussion on given topics; they often reflect insights from multiple sources and relevant parties. In countries where WPs and GPs are not a compulsory part of the institutionalized education policy process, each national team carefully identified official policy documents that are functionally equivalent to the WPs and GPs, respectively, in their policymaking contexts (see Chap. 9 for comparative discussion on the reform and policymaking contexts of the five Nordic countries). In this book, we use the terms “WP” and “GP” to refer to white papers and green papers as well as their functional equivalents. Our final sample of policy documents includes 8 WPs and 30 GPs (see Table 3.2).

Table 3.1 List of reforms by country
Full size table
Table 3.2 Policy documents (source), by country
Full size table

There are strengths and weaknesses to this sampling strategy. On the one hand, this strategy ensures the comparability of concepts and results in the project. By drawing on policy documents published in the context of a recent education reform, we can also assess the prominence of similar international knowledge artifacts, such as OECD/PISA reports. On the other hand, our focus on official policy documents excludes texts produced by other stakeholders that participate in the policymaking process. Therefore, we have a limited perspective on the kind of evidence that is mobilized in the policymaking process. Future research could address this limitation by incorporating policy documents produced by stakeholders such as labor unions, think tanks, and civil society organizations.

Once we settled on the sample of policy documents for each country, we implemented a standardized procedure for coding individual references from each document. We trained and supervised national research teams from the five countries participating in the study. Each team included two to three members; all of them had sufficient fluency and familiarity with education politics and policy to read and code policy documents. The research team from each participating country was responsible for their country’s data entry. All research teams followed a detailed protocol and used an Excel spreadsheet to enter the data. Data entry began by extracting all the items in the bibliography or reference list. That is, we coded references and not in-text citations. Research teams coded every reference in each source document (WPs and GPs) as the unit of analysis; they noted the content of the reference (e.g., author, year, title, publisher, type, and location). References were categorized as one of the five document types (reports, books, journal articles, government-published documents, and others) as well as one of the three location groups (domestic, regional/Nordic, and international). As international collaborations and multinational co-authorship increase, it has become more challenging to classify the location of a publication. In this project, the location was coded based on the location of the publisher. Figure 3.1 shows the first page of the reference section for a GP in Norway: NOU 2015:8 Fremtidens skole (The School of the Future). The page includes 23 citations; each of them was entered as a unit/record in the database. In addition to coding the relationship between the GP and the references, we also coded the references’ attributes. For example, we coded background information for the following citations:

Original citation in GP

Identification number

Author

Year

Publisher

Type

Location

Beck, U. (1992) Risk Society. London: Sage Publications.

578–1096

Beck, U.

1992

Sage Publications

Academic book

International

Bjørkeng, B. (red). (2013). Ferdigheter i voksenbe-folkningen. Resultater fra den internasjonale undersøkelsen om lese- og tallforståelse (PIAAC). Rapport 2013/42. Oslo/Kongsvinger: Statistisk sentralbyrå.

578–1128

Bjørkeng, B.

2013

SSB

Report

Domestic (Norway)

Fig. 3.1
A page titled references lists the names starting from A and B in 2 columns.

Green paper from Norway, NOU 2015:8 Fremtidens skole. Fornyelse av fag og kompetanser [The School of the Future. Renewal of Subjects and Competences]

Full size image

The content of the reference was entered as it was listed in the source documents; however, when any errors were suspected or observed, national teams checked the cited document to address the issues. Throughout the data entry and coding process, we communicated closely with the national teams to ensure validity and reliability across the countries.

After the initial data entry, the national research teams prepared the data for network analysis. First, they assigned a unique identification number to each source document and reference document. The identification number included three digits country (ISO-UN code) and three-four digits for the specific document. Second, they combined variants of the same text under one identification number. For example, they merged under one identification number references to the original and translated versions of John Hattie’s book Visible Learning: A Synthesis of Over 800 Meta-analyses Relating to Achievement (2008). Furthermore, one identification number was assigned for different editions of the same document if other contents of the reference, such as authors and publishers, remained the same. For serial publications such as OECD’s Education at a Glance, only publications with the same title and author(s), published in the same year, were assigned the same identification number.

The national research teams sent the data to the technical team at Teachers College, Columbia University for further review and cleaning. The main purpose of this cleaning was to ensure that all information in the database was ready for SNA. After multiple rounds of data cleaning, we finalized the database and constructed the network matrices for the analysis.

Finally, we produced five sets (one for each country) of a social network file and an attributes file. The network file includes a two-mode matrix, with source documents (WPs or GPs) in columns and reference documents in rows. A cell in the matrix is coded one (1) if the source document cites the reference document and coded zero (0) otherwise. The attributes file includes background variables describing the documents (see above). Figure 3.2 illustrates this process.

In the figure, circles mark nodes/actors and lines mark reference relationships. White Paper #1 cites two Green Papers (#1 and #2) and one reference document. Green Paper #1 cites two reference documents (#1 and #2). Green Paper #2 cites three reference documents (#1, #3, and #4). Reference document #1 has an in-degree of three; that is, three source documents cite this reference document. All other reference documents have an in-degree of one. The following matrix represents the relationships in the figure as a two-mode network:

Fig. 3.2
An organizational chart of W P categorizes into G P 1 and 2. G P 1 categorizes into references 1 and 2, and G P 2 categorizes into references 1, 3, and 4.

Illustrative example of a policy knowledge network

Full size image
 

WP #1

GP #1

GP #2

Reference 1

1

1

1

Reference 2

0

1

0

Reference 3

0

0

1

Reference 4

0

0

1

Analyzing Networks: Exploring the Architecture of Policy Knowledge

In this book, the authors report data from policy knowledge networks in five Nordic countries: Denmark, Finland, Iceland, Norway, and Sweden. Our analytical strategy included two steps. First, we calculated measures of centrality (degree) and centralization (density). Second, we produced visuals or maps of the networks. We conducted all analyses with UCINET 6.708 (Borgatti et al., 2002). UCINET is a comprehensive package for the analysis of social network data; most importantly, it can handle large networks.

At the micro-level, we calculated the degree centrality of reference documents. Degree centrality is the simplest centrality measure to compute; it is simply a count of how many connections an actor has with other actors (Wasserman & Faust, 1994). In this research, the degree centrality is equal to the number of source documents that cite a given reference document. For example, if WP 1 and GP 2 mention the same reference document, then the degree centrality for that reference document is equal to two. Following past research, we assume that documents with a higher number of citations are prominent or important in the context of the policy process.

At the meso-level, we examined descriptive statistics of the degree centrality (i.e., minimum and maximum, average, and standard deviation). Using these simple indicators, we assessed variability across the reference documents (Wasserman & Faust, 1994). Low variability suggests that reference documents are homogeneous in their degree centrality and structural position in the network. High variability suggests that reference documents are more heterogeneous in their degree centrality and structural position in the network.

At the macro-level, we examined the density of the entire network. Density is the ratio of the number of links to the maximum possible number of links. Higher density means that policy documents (i.e., WPs and GPs) draw on similar sources for evidence.

Finally, we used NetDraw to visualize the relationships between source documents and reference documents in the dataset. All figures use a Multidimensional Scaling (MDS) layout with node repulsion and equal edge length bias (Borgatti et al., 2002). This approach puts two documents (nodes) closer together if they are more similar (in terms of their connections to other nodes). The distances between documents and the direction (or location) are interpretable.

Limitations

Our study has three limitations that readers should consider. First, we opted to code whether a source document cites a reference document (binary variable yes/no) and to ignore the number of times a source document cites a reference document. This means that our analysis does not distinguish between documents cited only once and documents cited in a variety of different ways and places over the course of the source document. Second, our coding protocol did not address the context in which source documents cited references. Like bibliographical analysis of books and articles, we expect that authors of policy documents can use references constructively or critically (Mayrl & Wilson, 2020). Third, our coding protocol focuses on explicit references. Thus, it overlooks implicit references to culture, norms, and values (Steiner-Khamsi & Waldow, 2018; Waldow, 2017) that might be included in the text. Lastly, we did not carry out any multivariate analyses or statistical network modeling at this stage of the project. Despite these limitations, the database and analytic approaches utilized in this research project provide a unique opportunity to examine and map how policymakers mobilize evidence to advance educational reforms.

Conclusion

In this methodological note, we presented the theoretical and methodological inspirations behind this research project. We defined key concepts—network, actors/nodes, relationships/edges, and bibliometric analysis—and described the process that led to the database. The following chapters draw on the database and analysis we described above. Some chapters also draw on additional methodological approaches such as semantic and content analysis of policy documents and interviews with policy actors to address some limitations discussed above; the authors of these chapters provide additional information about their methodology.

Our approach to the study of the architecture of policy knowledge could be applied to other cases and domains. We hope scholars will find this methodological note useful as they extend this research. The database we generated through this research project provides an opportunity to explore many more questions about policy knowledge. For example, scholars could explore the selection process of references into WPs (i.e., why are some references, but not others, included in WPs?) Scholars could contribute to the mapping of the policy knowledge domain by pointing out the relationships of co-cited authors (i.e., which authors are co-cited frequently?) Also, scholars could examine the titles/abstracts of documents in the database to examine topical patterns and frequent labels/words.