Keywords

In Nordic countries, systematic inquiries have played a crucial role in the nation-states’ efforts to reform public education. However, in recent decades, various stakeholders have raised serious concerns about the legitimacy of such inquiries. Both in media and in research, critiques have targeted the quality of professional knowledge, suggesting that education policy should draw on scientific evidence to reform and evaluate the education system. This chapter examines the institutional response to this critique by examining how national authorities have made policy into an evidence-based pursuit of ministries and their governmental bodies.

By inquiring about two white and eight green papers published by the national authorities in Norway, we ask the following questions: How do policymakers and experts provide evidence and expertise in issuing school reforms for basic education? How do they identify options for school reform by deploying knowledge through argumentative forms of reasoning? How do various types of evidence structure these options by connecting policy realms and systems? We have divided the chapter into three main parts.

First, the chapter begins by establishing the policy context, the theoretical background for our study, and a typology that classifies various knowledge sources we will use for analytical purposes. We present some contextual information about the school reform we examined and give an overview of the bibliographic meta-data that we collected and the research strategies we used to examine the mediation and use of policy-relevant knowledge. We also introduce perspectives from sociological system theory based on Luhmann (2018) and associated scholars (Andersen, 2019). This theory helps us examine how references to knowledge sources are semantically translated within policymaking processes and how evidence informs thematic areas in school reform policy, such as curriculum and assessment. In addition, the theory serves as a link between policy processes and policy systems, such as science, politics, and education.

Second, we present the results obtained from our bibliometric network analysis in the form of the frequency and distribution of prominent references in our dataset. By looking into the ways two white papers and eight green papers refer to various knowledge sources (e.g., research reports, reviews, and governmental documents), we demonstrate how policymakers use bibliographic references to strengthen their arguments for reforming basic education in Norway. In our analysis, we identify how some references acquire a prominent role by being referenced by several sources within and across policy realms. We also uncover their prominence by mapping whether these or other knowledge sources are frequently and explicitly in-text referenced and thereby influential within translations made by the authors.

Third, we present a semantical analysis of how state authorities translate knowledge sources in-text referenced within the two white papers. These documents prepared for political decision-courses within the Norwegian parliament (see Chap. 10 for further details about the process). We analyze how policymakers formulated options for school reform through postulations and aspirations about structures, processes, and outcomes. Thus, we agree with Colebatch and Hoppe’s argument that “policy may be seen as both ex ante intention, ex durante becoming, and ex post outcome” (Colebatch & Hoppe, 2018a, p. 6). Moreover, according to Luhmann (1990), a medium, such as a policy document and its references, can provide meaning through forms of reasoning by semantically linking structures, processes, and outcomes via input/output schemes. Through these schemes, policy formulation and translation processes become programmatic by character (Luhmann, 2018).

Finally, we question whether the most cited and prominent knowledge has influenced the semantic repertoire of themes and arguments and thereby the configuration of the decision programs built into the national school reform. Based on a comparison of how references are used within and across two main thematic areas covered by the white papers, we conclude that evidence has a structuring impact on the formation of policy realms and, as a result, options for school reforms. The empirical investigation shows that claims and recommendations within the white papers are shaped by the type of knowledge source policymakers use, which policy realms they reference, and how they translate knowledge through argumentative modes of reasoning. We also demonstrate how policy documents mediate evidence that structurally connects the political system with the education system. This demonstration leads to a discussion of whether and how professional knowledge is restrained or constricted by evidence-based reform within the education sector.

The Use of Evidence and Expertise Within Policymaking Processes

During the last decade, interest in evidence-informed policy and practice has increased in Europe and beyond. Such policy often favors scientific methodologies and empirical research as a frame of reference. Importantly, the term evidence does not bear much significance in itself since it is associated with various actors, different types of knowledge sources, and multiple forms of knowing (Boaz et al., 2019, p. 5). Moreover, any material base that carries knowledge or any source of information may provide evidence depending on context variables (Sedlačko, 2018). Therefore, what counts as evidence and how evidence is used in reform policies are highly contingent questions.

This chapter examines how evidence serves as a medium in policymaking among experts who were mandated to develop official policies aimed at reforming a national education system. We investigate in particular how policymakers and experts selected research-based knowledge alongside other types of documentation, and how the policy documents mediated this knowledge as evidence to inform argumentation and decision-making processes. As Colebatch and Hoppe (2018b) have noted, “Both the documentation and the widespread ‘consultation’ within administrations can be part of the signature of policy-evidence that ‘due process’ was followed” (p. 15). As such, evidence-based policy is not merely a question of what kind of knowledge and information are produced and selected; rather, it is a question of how policy processes facilitate the usage and translation of evidence that various actors call for.

Because the term evidence has become a buzzword with no clear definition, both policymakers and practitioners are juggling multiple forms of evidence within these processes and involving various groups of actors that provide support for defining the “best evidence.” Within this perspective, the process of evidence use is considered a highly pragmatic enterprise. Despite attempts to establish clear hierarchies of evidence, few policy realms reflect a master plan for how to make use of expertise in reform-making processes. For example, in the Oslo Institute for Research on the Impact of Science (OSIRIS) project, Thune (2019) examined the use of evidence within public administrations in Norway. Thune found that both policymakers and practitioners deployed a variety of methods to access information and knowledge, such as contacting colleagues and conducting web searches. Moreover, policymakers used different types of media to collect knowledge sources, such as newspapers, publications, presentations, and informal dialogues. Against this background, one might wonder if there are any patterns that actually structure evidence use in policymaking processes.

Despite the micro-politics of providing evidence, researchers have argued for the importance of observing patterns or forms of patterning of policy processes. From this point of view, policy researchers consider the character and strength of the ties of actors as one possible structure that characterizes the use and translation of information (Honig & Venkateswaran, 2012). Among several other researchers, Thune (2019) found that provision of evidence is dependent on the capabilities of actors. According to Thune’s survey results, persons who are in senior positions, are well educated, and have work experience from other sectors, especially research sectors, are more capable of collecting and using knowledge sources than persons in junior positions. Moreover, the influence of these groups of actors is dependent on the relational ties that connect these actors.

O’Day’s (2002) study of accountability reform in Chicago during the early 2000s serves as another excellent example of how such ties evolve. O’Day investigated how policymakers and professionals interacted with various forms of governance through the reception and translation of information. In an empirical mixed-methods study, she uncovered critical mechanisms that enforced and constricted the flow of information within the policy-praxis nexus. A key assumption underlying this and similar studies is the changing role of bureaucratic governance. When studying the emergent impact of accountability systems through ties between administrative levels, O’Day (2002) identified impacts of both an outcome-based bureaucratic mode of governance and a professional mode of governance. Different from traditional bureaucratic government, these new modes centered on practice-based knowledge, performance-based standards, and bureaucratic accountability.

In our chapter, we look at similar patterns, but primarily by focusing on documents considered powerful media for policymakers to create options for reform and change. We examine how policy documents mediate knowledge and information through their selection of references and deploy argumentative modes of reasoning that connect politics, policies, and practices. Similar to O’Day (2002), we refer in particular to two models for how policies bring about change by referencing and translating knowledge across policy realms and levels: a traditional bureaucratic mode for policymaking that aligns broad outcomes with general mandates and an outcome-based bureaucratic mode that pursues means-end reasoning along with performance standards. Both these modes operate within the same context and involve different degrees of normativity and prescriptive routes of actions, and they can also be combined into mixed forms, as demonstrated in recent research on policy borrowing and lending (Sivesind et al., 2016).

In our study, we draw on Luhmann’s (2018) distinction between two program forms to map these alternatives. According to this theory, a conditional program and a purposive program differ by representing two sets of conceptual schemes for observing policymaking processes. According to Luhmann (2018, p. 213), a key difference between the two program forms is that they reflect various distinctions for observing decisions. While conditional programs divide between conditions and consequences to build up an argument about change, purposive programs distinguish between means and ends to observe problem-solving processes. This difference leads in the next step to various modes of reasoning, based on expectations that can be more or less normative and future-oriented and that can be more or less comprehensive or narrow by ways of issuing a reform within the education sector.

Moreover, by looking at policy documents as media for observing decisions about reform, we can also assess how evolving semantical structures are loose or fixed based on how they are formed. By examining how documents are linked to other media (e.g., other documents) or to systems (e.g., science, education, and politics), we can assess to what degree policymaking is structured (Andersen, 2019, p. 81). This examination may lead to an interesting discussion about the structuring role of evidence in policymaking processes if, for example, references to scientific evidence condition a reform by fixating particular modes of reasoning along with a purpose that national and international stakeholders have actually called for (Burns & Schuller, 2007).

We analyze more or less fixed patterns of evidence use by comparing how documents are connected within and across two thematic areas or realms: curriculum reform and assessment practices. We demonstrate how policy documents for each realm and their references to knowledge sources connect reform and change both in bibliometric networks and through semantical patterns. We show how policymakers create decision programs for education that make up policy options that are more or less future-oriented and normative by their use of evidence and language (Luhmann, 2018, p. 2015). By looking into the way mediating links are configured between the source documents and referenced documents, by comparing the patterns across policy realms, and by assessing how they connect with science, politics, and education as surrounding systems, we are able to analyze how evidence structures policy options for school reform in both a fixed and a loose form. Thus, we identify how the inter-mediational roles of both knowledge sources and arguments become decisive for the configuration of policy processes in the field of school reform.

Reform, Data, and Methods

In this study, we have analyzed the distribution and networks of bibliometric reference in two white and eight green policy papers that were prepared for political processes within the Norwegian parliament on the most recent school reform in basic education (Years 1–13). They were all written under the auspices of the Norwegian government and the Ministry of Education and Research. The reference use and the arguments made for reforming and renewing the education system prepared for political decision-making processes within the parliament. Through the next steps, these documents resulted in the renewal of the Knowledge Promotion Reform (2016/2020) within the Norwegian education system (Baek et al., 2018; Steiner-Khamsi et al., 2020). The national government launched the reform in 2017, and it was formally put into effect in August 2020. This reform set out to renew the curricula for the core subjects in primary and secondary education and provided a new introductory part that aimed at establishing a coherent framework for organizing and assessing teaching and learning in schools (Sivesind & Karseth, 2019).

To begin our analysis, we examined the bibliometric references in the two most prominent governmental reports (white papers, referenced here as WP#1 and WP#2) that were prepared for decision processes within the parliament. Thereafter, we included references in all public inquiry reports that were referenced in these white papers. Altogether, this sample made explicit references to 2312 knowledge sources, which were listed within the documents’ bibliographies and footnotes. By conducting a bibliometric network analysis, we identified the most co-cited knowledge sources within this corpus of documents.

For the bibliographic network analysis, our data consist of 2312 references from the reference lists of two governmental reports (white papers) and eight official reports (green papers, known as Norges offentlige utredninger [NOUs], written by experts who initially evaluated existing education systems and who addressed how to renew education in the future) referenced in the two white papers (Table 7.1).Footnote 1

Table 7.1 Sampling for the bibliographic network analysisFootnote

For a full overview of source documents, see Chap. 2.

Full size table

This sampling reflects the logic of the reform-making process and allows us to analyze the “official knowledge” from which bureaucrats and politicians eventually draw. The references of our ten source documents were analyzed and edited with the software programs UCINET and Netdraw. These programs generated descriptive statistics and visualized relationships between the documents. In this way, we identified the most prominent references of our dataset: those that were most often cited in the reference lists and played crucial roles in the reform discourse. We established a cut-off point at five, resulting in a list of 12 documents that were cited five to eight times in the whole dataset. Since we consider these publications essential knowledge in the process of reform formulation, we decided to track them back to the text in which they were actually quoted. These documents are the main data for our semantical analysis.

In addition to the bibliometric study, we conducted a semantical analysis of how policymakers translated references through their argumentative modes of reasoning. In this part of the study, we examined how policymakers formulated arguments to make recommendations for how to reform education in and across policy realms, such as curriculum reform and assessment practices. The school reform was formally built on two white papers (WP#1 and WP#2) written on behalf of the Norwegian government. These two white papers referred to various expert commissions and other sources that included references to articles, book chapters, and research reports, among others. Report No. 28 to the Parliament, Subjects, In-Depth Learning—Understanding. A Renewal of the Knowledge Promotion Reform (WP#1; Ministry of Education and Research of Norway, 2015), was published in 2015–2016 and presented a clear framework for curriculum revision. The report suggested continuing with and further developing competence descriptions, setting prioritization by defining core elements in the subjects, including basic skills in curricula, and creating more connections between the subjects by defining three prioritized cross-disciplinary topics (i.e., democracy and citizenship, sustainable development, and health and mastery of life). In addition, the report suggested new goals and assignments for the forthcoming renewal process. Report No. 21 to the Parliament, Eager to Learn—Early Intervention and Quality in Schools (WP#2; Ministry of Education and Research of Norway, 2016), was published in 2016–2017 and addressed diversity problems regarding students needing extra support. WP#2 also presented a comprehensive framework to improve equity and quality with new models for monitoring education.

For analytical and heuristic purposes, we identified two main groups of arguments: (a) those that maintained traditional governance modes that guide actions by regulating conditions and (b) those that purposively emphasized the need for reforming education more strategically. For this part of the analysis, we used Luhmann’s (2018) program forms (i.e., conditional versus purposive programs). Conditional programs are past-oriented and emphasize formal and substantial rationales for reforming education, while purposive programs are future-oriented and emphasize value-based and standard-based tools for achieving certain outcomes (Sivesind et al., 2016). For identifying how prominent references were used to legitimize arguments for school reforms and investigate the underlying communication patterns, we consider the combination of semantic and bibliometric network analysis to be a fruitful research approach (Froehlich, 2020).

Finally, we conducted a structural comparison of reference use within and across the two white papers (Sivesind, 1999). This comparison aimed at developing insights into the third research question, addressing how various types of evidence structure policy options by linking thematic areas and by connecting systems such as policy with education, politics, and science (Andersen, 2019). First, we examined if and how the most cited references shaped argumentative modes of reasoning through their tendency to favor a conditional program, a purposive program, or both. Through this comparison, we investigated if and how certain types of evidence can actually structure policy options by representing a particular type of knowledge. Second, we synthesized our findings and interpretations about the argumentative translation of evidence within the white papers and asked how policymakers shape policy options through the expertise they deploy during the writing process. Moreover, we examined whether we found different patterns between the two policy realms in terms of how evidence was translated. Finally, we compared the ways in which the most prominent references in the two white papers connected policy realms by referencing the surrounding world of systems, such as education, science, and politics.

Typology

Along with Stephen Toulmin (1958), we consider a persuasive argument as consisting of both claims and assertions backed by evidence. These assertions refer to both facts in terms of data and information, and include reasoning that enhances a persuasive use of the argument. In addition, we categorize various types of knowledge sources from our dataset that were used to build an argument. Moreover, to compare the influence and usage of various types of knowledge as manifested within the policy papers, we have developed a typology that fits with our data that we can use to build arguments ourselves. This typology is based on existing classifications of research and information that we combined for our particular purpose.

Doyle (2003) referred to Noblit and Hare (1988) when presenting three approaches to conducting research reviews: meta-analysis, literature review, and meta-ethnography. These approaches differ in their degree of incorporating contextual knowledge. Meta-analysis synthesizes numbers by aggregating findings to explain a phenomenon, such as what might increase learning achievement. In this approach, researchers gather all available information from studies that have measured the same variables and explain key mechanisms via either the exhaustive collection of research articles or random sampling. On this basis, experts can present generalizations as a universal standard that can be applied (more or less) independent of contextual knowledge.

The second alternative is the literature review, where the overall purpose is to create a chain of reasoning that helps illuminate the phenomenon under study. In this case, both the theory and the results are considered relevant knowledge. Data collection must be exhaustive, and researchers are challenged to logically bridge summaries of results and interpretations for different studies.

Within the third group, where reviews are based on meta-ethnography, the validity issue is particularly challenging. Doyle (2003) argued that the aim of generalizability in meta-ethnography is based on case studies where the local context shapes the ways in which findings and results are interpreted. Still, she asserted that it is possible to draw conclusions across case studies through synthetization, which generalizes knowledge beyond what is valid for single cases in the study. Such a generalization requires a particular methodology, which includes alternative analytical steps and aims at developing an increased understanding of a particular phenomenon. In many ways, this procedure is consistent with what Gough et al. (2012) labeled configurational synthesis, or the compilation of descriptions, interpretations, and results that are reorganized based on analytical steps and concepts rather than on pre-determined concepts or aggregated results.

Beyond evidence-based reviews, knowledge sources can be scientific by being built on empirical data collected by researchers who follow known procedures and general standards to examine a particular research problem. Such studies can be academic without any purpose of intervention, they can be clinical and result in knowledge to be applied for particular purposes, or they can be part of research projects and evaluations that guide actions through recommendations. According to Rasmussen et al. (2007), knowledge sources of this kind can serve different functions. Therefore, they suggested dividing educational research developed for scientific purposes from educational knowledge sources and theories used for programmatic and practical purposes. Rasmussen et al. (2007) classified evidence-based studies belonging to the first category as applied research that can be useful for policymakers and practitioners. A core aim in such applied research is to serve clinical purposes and project best practices by the use of assessment standards.

In our study, we include four types of reference sources that provide knowledge and information to support arguments within policy documents: (a) formal documents that are not primarily a result of research but can indirectly mediate research and serve as one type of evidence, (b) meta-analysis, (c) configurational reviews, and (d) empirical research studies. These four types of knowledge sources do not cover all knowledge and information that could be used; for example, we have not included experience, practice-based innovations, or theoretical knowledge in our analysis. However, through our policy documentation, we can examine how these four types of knowledge sources are used and translated as well as how they serve as a structural condition for how policy options for school reform are expedited.

Bibliometric Network Analysis

An earlier examination of the 2020 reform revealed that the most cited knowledge sources within the bibliographies were predominantly of domestic origin and represented an interesting mixture of references, such as formal documents, research reports, and scientific publications (Baek et al., 2018). Moreover, the most cited references were research and policy reports that had not undergone a scientific peer-review procedure.

Figure 7.1 shows the network structure of all references in our database. Source documents are visualized as circles and ordinary references as squares. The sizes of the nodes indicate their in-degree centrality, which is an indicator of the impact of a text assessed by its number of citations in other works. The shading shows the geographic origins of the references with regional nodes colored gray, Nordic references white, and international references black.

Fig. 7.1
A network of interconnected nodes has several node clusters that represent the references in a database. The node clusters are made up of square nodes. The clusters are interconnected via circular nodes.

Complete network structure of all references in our database. (Note: The network structure reveals that the publications are highly specialized and issue-centered, with little overlap between the various reports)

Full size image

The bigger nodes located in the middle visualize references cited by more than one report and how they are connected to each other. These co-cited references represent the core knowledge shared by one or more reports and are therefore the focus of our attention. It becomes apparent that there are a relatively small number of co-cited references, which means that the cited publications are highly specialized and issue-centered, with little overlap between the various reports (Baek et al., 2018). For greater insight into the network of co-cited references, we made the following list of the 12 most cited references (i.e., cited more than five times) in the database (see Table 7.2).

Table 7.2 Co-cited references in the full database
Full size table

The document at the top of the list, the white paper Culture for Learning, is a governmental report to the Norwegian parliament from 2003 to 2004. This foundational paper initiated the Knowledge Promotion Reform that was launched in 2006 and created systemic change in Norwegian education policy by introducing a national test system. Since policymakers presented the new renewal of the Knowledge Promotion Reform (2016/2020) as incremental and not foundational, we find it reasonable that this document from 2003 is the most referenced paper.

The second listed publication is Hattie’s Visible Learning (2009), a meta-analysis of studies on the effect of different educational influences on student performance. This book is not only omnipresent in media and research about direct instruction in schools, but it also is on the top of what is considered to provide evidence according to academic classifications (Doyle, 2003). Number three on the list is a Danish report on teaching competence and student learning in kindergarten and school that synthesizes international research about the relationship between teacher competence and student learning in schools. Nordenbo et al. (2008) authored this report on behalf of the Norwegian Ministry of Education. Because the synthesis builds on existing research on student learning with a focus on “what works,” it represents another typical example of evidence-based knowledge.

The rest of the publications are predominantly governmental or official reports (NOUs), which we have categorized, together with the white paper Culture for Learning, as formal governmental reports. The two empirical research studies from Aasen et al. (2012) and Nordahl and Hausstätter (2009) contain results from evaluation projects on the Knowledge Promotion Reform from 2006. While Aasen et al. (2012) presented a comprehensive evaluation of reform and governance within the Knowledge Promotion Reform, Nordahl and Hausstätter (2009) analyzed a specific group of students, those in need of special support. The research article from Durlak et al. (2011) reviewed existing knowledge about the improvement of student performance by fostering social and emotional learning. The study by Durlak et al. (2011) was published in the journal Child Development, which has a comparatively high impact factor of 5.024, according to the publisher’s website. The article referred to student performance and provided evidence-based knowledge about factors that can improve this performance. Along with Hattie’s Visible Learning, we classified the Durlak et al. (2011) article as a scientific publication in a narrow sense.

All in all, the list consists of publications that provide an interesting overview of prominent knowledge referenced by the school reform. This list reflects a mixture of formats (i.e., governmental and official reports, academic publications, and policy reports) and origins (i.e., national reports, international research, and one regional/Nordic report). Moreover, the group of knowledge sources represented in Figure 7.1 can be analyzed and compared according to their levels of evidence. Drawing on Doyle (2003), Gough et al. (2012), and Rasmussen et al. (2007), we have differentiated between the following types of knowledge sources: meta-analyses, configurational reviews (i.e., theory-driven reviews of qualitative and quantitative studies), empirical research studies, and governmental papers. Table 7.2 shows the allocation of references to categories.

The Argumentative Translation of References

As a first step in our analysis of the argumentative translation of references, we identified knowledge sources that were most often referenced within the two white and eight green papers. In addition, we identified how often references appeared as in-text citations within the two white papers that are the basis of our semantical analysis. As shown in Table 7.2, the report on the evaluation of the Knowledge Promotion Reform (Aasen et al., 2012) had an overwhelming in-text citation quote of 20, while the white paper Culture for Learning from 2003–2004, which was the most co-cited document within the full database, was cited only six times. Moreover, John Hattie’s Visible Learning was co-cited seven times within the bibliographies of the ten texts and referenced eight times within the texts themselves. This pattern indicates that Hattie’s meta-analysis has been among the most influential references to project options for reforming schools in Norway. However, a comparison of the actual citations with those from the bibliometric network analysis makes it clear that the report by Aasen et al. (2012) was the most influential reference for the translation of evidence within the two white papers, as it was prominent on the overall lists of the ten documents and was the most cited source in the white papers themselves.

To put the most cited references in perspective of our two white papers, we decided to compare the prominent sources within the full database with the total number of quotations of all references listed in the two white papers. The frequency of Aasen et al.’s (2012) evaluation report in this analysis was topped only by NOU (2015, p. 8) The School of the Future. Renewal of Subjects and Competences, which was cited 35 times altogether. The rest of the references in the two white papers were cited eight times or fewer, which underlines the importance of the Aasen et al. (2012) report in the two governmental papers (see Table 7.3) alongside the green paper authored by the public inquiry commission about the school of the future (NOU, 2015, p. 8).

Table 7.3 Frequency of references (in-text) in the two white papers, more than eight citations
Full size table

In the list of the most referenced in-text citations (Table 7.3), we also found an expert report (Dahl et al., 2016) that was not a commissioned paper, but still of crucial importance in argumentation for a new reform. In addition, we discovered another research evaluation report by Dale et al. (2011) that analyzed the national curriculum and its implementation. The list also includes WP#1, which was referenced in WP#2.

To trace the translations of these references and how they were shaped by argumentative modes of reasoning, we analyzed the paragraphs where the knowledge sources were referenced according to Luhmann’s (2018) program forms, namely, the past-oriented conditional program versus the future-oriented purposive program. This analytical distinction helped us to assess if and how different types of reviews and reports were used to stabilize conditions that were already in place or to change education in a more strategic way. In this part of the analysis, we looked at the meaning units within the text, which consisted of one to three paragraphs following the reference. Moreover, we evaluated whether the argumentation manifested within the document referred to normative or descriptive statements. In so doing, we identified whether statements and arguments referred to past conditions and actions (e.g., decisions about what to teach in schools) or future activities and results (e.g., what to accomplish and achieve). This way of conducting semantic analysis helped to clarify how the documents symbolized certain program forms and a regulative or a strategic use of references. Based on this analysis, we classified how the references were translated within a matrix (see Tables 7.4 and 7.5 in the Appendix).

For example, when statements within the texts referred to governmental papers authorized by state authorities in the past, we classified them in the first column (regulative; structures). In this case, we considered the reference use to reflect a formal regulation for how to provide education as a service to the population. When the documents made use of arguments to project expectations about performance and/or to discuss standards for how to assess the competence of the future learner, we classified them in the fourth column (cognitive; outcomes). In the second and third columns we divided sources into arguments that appeared to be merely informed by knowledge in the present (the second column: substantive; content) and arguments that reflected normative-oriented statements in the present (third column). Regarding the first column, it is important to add that normative statements are considered regulative, while normative arguments in the third column are regarded as axiological (i.e., value-based) as they refer to the soft governing side of ongoing activities by expecting learning and change. These alternatives resulted in a four-field classification (Sivesind et al., 2016).

In addition to this categorization, we added the titles and subtitles of the chapters in the matrix (Tables 7.4 and 7.5), which helped to create an overview of the thematic realms covered by the documents. The table includes the titles of only those chapters in the two white papers that contained citations of prominent knowledge sources, and the titles are our own translations.

Finally, we conducted a comparative analysis of the translation of references in the two white papers. In the following section, we present the argumentative use of co-cited sources and most cited in-text references that belong to different categories or levels of evidence: meta-analysis, configurational reviews, empirical research studies, as well as official and governmental papers.

A Comparative Analysis of Reference Use

Meta-analyses

The following publications were allocated to this category:

In correspondence with their characteristics as meta-analyses, the two publications refer to studies based on student performance as a benchmark for successful learning and teaching. They present strategies, techniques, and approaches intended to increase student performance and, in their own terms, quality in education. Accordingly, they are mostly translated in a purposive way by giving normative directions for teaching and learning and projecting future goals.

While the quotations in WP#1 are more general, those in WP#2 are more specific and contain precise results and knowledge from the meta-analysis. With one exception, the quotations present suggestions or recommendations that are clearly future-oriented as they refer to increasing student performance as a criterion for success. The following example uses Hattie to legitimize a core feature of the reform (i.e., three specific, core curricular goals); as with all quoted examples, the translations are our own:

The department has defined the following three sector goals for basic education, which sum up core elements in both the objects clause within the education act and the national curriculum framework:

  • The students shall have a good and inclusive learning environment

  • The students shall master basic skills and have good subject-specific competence

  • More students and apprentices shall complete secondary education (Years 13–16)

The three goals are connected to each other and sum up the school’s task for society. A good and inclusive learning environment is both a goal in itself and a tool for increasing the students’ learning outcomes. The objects clause points out, among other things, that schools and apprenticeship companies should meet students and apprentices with confidence and respect while working against all forms of discrimination. All children and youth should feel comfortable and be included. Schools that focus on a good and inclusive learning environment also reach better learning outcomes. (Hattie, 2009, Bakken & Seippel, 2012) (WP#2, pp. 15)

The paragraph states that a good and inclusive learning environment leads to better learning outcomes, which indicates that the matter of student well-being is subordinate to learning outcomes. Since the paragraph explicitly refers to learning outcomes, we have categorized this citation as reflecting a cognitive and learning-outcome-oriented program.

The scientific article by Durlak et al. (2011) is cited once in each of the two white papers, in both cases for the same argument: that soft skills contribute to student performance. The first citation presents knowledge on student learning, while the second one argues that soft skills also need to be supported from early on. By referring to the increase of student performance while subordinating soft skills to it, the reference to this publication clearly promotes a purposive program that suggests specific measures that, in a normative way, puts student performance on top of the agenda, as shown by the following example:

Social and emotional skills like patience, mastering one’s own feelings, curiosity, and mastering resistance, play an important role in student learning. (Backer-Grøndahl & Nærde, 2015; Birch & Ladd, 1997; Heckman & Kautz, 2013; Durlak et al., 2011) (WP#2, p. 42)

In both cases, the journal article is referenced together with additional sources. The purpose of citing this scientific article is probably to counter an often-raised critique against testing frameworks.

In general, the two white papers draw on meta-analyses in the following subject areas: student learning and the pedagogical work at school (WP#1) and quality in education, collaboration, early intervention, and competence development in the municipalities (WP#2). By underlining knowledge in these thematic areas, the meta-analyses support core points of the two white papers, namely, the need to strengthen student learning, quality development, collaboration, and competence development on the municipality level, even though some of these areas are not covered by the main arguments in the publications by Hattie (2009) and Durlak et al. (2011). Also striking is that Hattie’s book is not used to argue for specific methods in teaching, such as direct instruction or assessment practices, even though a “fundamental change in teaching” is announced in WP#2. It is more or less used as a justification for strengthening a purposive orientation to school reform in general.

Configurational Reviews

The following reference was allocated to this category:

  • Nordenbo et al. (2008)

Nordenbo et al. (2008) are cited surprisingly few times compared to other sources in this study. The report is referenced seven times in the whole dataset, whereas it is cited only twice in WP#1 and once in WP#2. It serves as a core source for demanding more competence development to increase collaboration among teachers, to improve student performance, and to demand more teachers with subject-specific training, especially in subjects that do not (yet) require subject-specific education, such as art and handicraft education. These demands represent purposive programs by anticipating improved student performance, which will follow from a set of normative expectations expressed in this way:

It is thoroughly documented in research that the teachers’ subject matter knowledge has an impact on the students’ learning outcomes. Teachers who feel confident about their subject matter are less dependent on predefined teaching designs and methods, and they can make variations and develop their own teaching further (Nordenbo et al., 2008). For example, it is documented that it is important that teachers possess the practical skills that their students are expected to develop (Espeland, 2011). For this reason, the Ministry of Education will evaluate whether one should introduce new competence requirements for teaching in several subjects at the primary school level. (WP#1, p. 74)

An interesting aspect in the example above is how the paragraph introduces Nordenbo et al. (2008). The argument starts with “it is thoroughly documented in research,” indicating that Nordenbo et al.’s report is indeed considered a significant knowledge source that provides the best research evidence, even though the report draws on several references that are, in fact, not what is typically regarded as scientific. We categorize all three citations as reflecting a purposive policy program, two of them as a normative type of reference and one as merely informative. This means that Nordenbo et al. (2008) is used to project best practices by creating cognitive expectations in one case. For the other two cases, the reference is used for a normative purpose, indicating what should be done to develop competence among teachers in schools.

Interestingly, the white papers make in-text references to Nordenbo et al. (2008) under the following headings: professions in the school, competence development and capacity building, and competence requirements for hiring and teaching. In only one case does the reference appear together with another source, which is an expert report about the role of the teacher (Dahl et al., 2016). This official report also synthesizes education research and is used for partly purposive reasons (five times within WP#2 with a purposive aim and two times within the substantive category, reflecting a conditional orientation). Although this expert report describes the teacher role and elaborates on the need for professionalization of teachers, it covers several of the themes in WP#2, such as collaboration among teachers, the role of the teacher, and the department’s assessment of the quality of the evaluation system. Thus, this report, which includes some configurational reviews, covers a broader set of themes than Nordenbo et al.’s (2008) analysis. This finding shows that configurational reviews that focus on certain evidence might serve different functions and purposes than expert reports that are more comprehensive in terms of the knowledge sources they draw on. Yet, by comparing the use of various types of reviews, we find that configurational analyses are used less frequently to specify cognitive outcomes than the meta-analyses of Hattie (2009) and Durlak et al. (2011).

Empirical Research Studies

The following references were allocated to this category:

  • Aasen et al. (2012)

  • Nordahl and Hausstätter (2009)

  • Dale et al. (2011)

It is perhaps self-evident that the evaluation of the Knowledge Promotion Reform plays a crucial role in the two white papers. These research-based evaluations were funded by the state and can be classified as policy research (Christensen & Holst, 2017). Aasen et al.’s (2012) report on the evaluation of the Knowledge Promotion Reform accordingly has the highest number of citations in the two white papers but is co-cited only five times in the whole dataset. Apart from a number of citations that present background knowledge in the reports, Aasen et al. (2012) is mainly used to legitimize the renewal of the Knowledge Promotion Reform (2016/2020) by showing weaknesses and creating options for discussing new issues in the context of the current reform. In so doing, it delivers core arguments for the basic measures of the renewal of the Knowledge Promotion Reform (2016/2020), including that basic skills should be integrated in the subject matter curricula, that goals and content need to be more explicit, and that the connection between subject matter and the new general part of the curriculum should be strengthened. In addition, Aasen et al. (2012) is used in WP#1 to argue for strengthening the on-site work on curricula, activating the political level (school owners), clarifying the responsibilities and the relationship between different levels in the education system, improving schools’ ability to understand and make use of quality assessment results, and promoting a more decentralized competence development system in which municipalities should gain more scope of action.

This long list of demands shows the impact of the publication and reveals that the authors of WP#1 obviously counted on the evidence provided by Aasen et al.’s (2012) report. However, most of the quotations (12 out of 20) appear in WP#2. These quotations mainly deal with the question of how different levels in the municipalities and the schools share responsibilities through collaboration and process the information of student assessment tests across contexts. This information is used as background information and as additional support for the demand for a more enhanced system of competence development at schools, as the following example shows:

The evaluation of the Knowledge Promotion Reform showed that the reform contributed to increasing the quality of basic education, but that many small municipalities and schools experienced challenges in implementing the reform. From the perspective of the county governors, many municipalities have gained a stronger grip on the role of school principals and the responsibility for schools in their municipality. But the regional governor also perceives differences among the municipalities. These differences are first of all connected to the size of the municipalities, but also to their way of organizing. Small municipalities are in many cases vulnerable concerning their economy and competence. Some of them compensate for this with an established network for collaboration on the level of the municipalities and the schools. (Aasen et al., 2012) (WP#2, p. 31)

The citation is not normative as such but describes in a rather open way how municipalities have implemented the 2006 reform. It does not suggest a specific solution but mentions that some municipalities draw on networks of collaboration for coping with the challenges they are facing. We have categorized the citation as oriented to substantial issues and thereby belonging to a conditional type of program.

In WP#1, however, quotations are explicitly used to justify both specific demands and the renewal of the curriculum in general, as below:

There is research that indicates that the General Part and the Principles for Education are part of the local work with curricula only to a small extent under the Knowledge Promotion Reform (Aasen et al., 2012). […] In order to create better cohesion in the curriculum framework, the Department wants to renew the current General Part, Principles for Education, and the subject curricula. This is supposed to contribute to a more holistic curriculum framework, updated for today’s and the future’s society. (WP#1, p. 19)

All in all, we categorized citations from Aasen et al. (2012) either as substantially oriented, reflecting a conditional approach, or as normative, reflecting a purposive approach. WP#1 uses the research study in a normative way, while WP#2 uses it in a more substantive way. This finding is an interesting difference because we could have expected the opposite pattern. One might interpret this pattern as a dynamic use of evidence: in issues that are more conditionally oriented, such as the curriculum, the reference points to the purposive side; in thematic areas that are more purposive, such as quality development, Aasen et al. (2012) is used to strengthen the conditional side. Another explanation could be that the temporal order of the two white papers and the mandate framed how the policymakers utilized evidence in the two white papers.

The argument that draws on the reference to Nordahl and Hausstätter (2009) is quite inconspicuous. By citing the number of students who are in need of support for special needs in the course of their education, WP#2 offers insight into conditions for organizing special needs education. We categorized the citation as informative, associated with a conditional type of program, opposite of a normative or cognitive argument that promotes strategic actions. Both the Aasen et al. (2012) and Nordahl and Hausstätter (2009) publications provide national and local knowledge about the former school reform, the Knowledge Promotion Reform of 2006.

Dale et al. (2011) authored another research report, based on the same conditions for conducting research. This report is not among the most frequent co-cited publications, but it is one of the most cited in-text references in WP#1. It is cited eight times, four of which are in conjunction with Aasen et al. (2012). Only one of the eight citations uses the source merely to inform about reform conditions in a substantive way. Instead, WP#1 uses the Dale et al. (2011) source to strengthen the purposive orientation of the reform proposal. None of the reports in this category are, however, used to create narratives or specifications about outcomes or best practices, which is also an interesting observation.

Formal Documents: Governmental and Official Reports

The following publications were allocated to this category:

  • St. Meld. 30 (2003–2004). Kultur for læring [Culture for learning]. (White Paper No. 30 to the Storting). Ministry of Education and Research.

  • NOU 2003: 16 I første rekke: forsterket kvalitet i en grunnopplæring for alle [In the First Row. Increased Quality Within a Basic Education System for Everyone]. The Committee for Quality in Primary and Secondary Education in Norway.

  • St. Meld. 16 (2006–2007) … og ingen sto igjen. Tidlig innsats for livslang læring [And No One Is Left Behind. Early Intervention for Lifelong Learning]. Ministry of Education and Research.

  • St. Meld. 29 (1994–1995) Om prinsipper og retningslinjer for 10-årig grunnskole—ny læreplan [Principles and Guidelines for 10-Year Compulsory Schooling—New Curriculum]. Ministry of Education, Research and Church Affairs.

  • St. Meld. 20 (2012–2013) På rett vei [On the Right Path]. Ministry of Education and Research.

  • St. Meld. 11 (2008–2009) Læreren—Rollen og utdanningen [The Teacher’s Role and Education]. Ministry of Education and Research.

  • Ot.prp. nr. 46 (1997–1998) Om lov om grunnskolen og den vidaregåande opplæringa (opplæringslova) [Law Draft on Act Relating to Primary and Secondary Education and Training (The Education Act)].

  • NOU: 8 Fremtidens skole: fornyelse av fag og kompetanser [The School of the Future: Renewal of Subjects and Competences]. Ministry of Education and Research.

  • Dahl, T., Askling, B., Heggen, K., Kulbrandstad, L. I., Lauvdal, T., Qvortrup, L., Salvanes, K. G., Skagen, K., Skrøvset, S., Thue, F. W., & Mausethagen, S. (2016). Om lærerrollen. Et kunnskapsgrunnlag [On the Role of the Teacher. A Knowledge Basis]. Fagbokforlaget.

  • St. Meld. 28 (2015–2016). Fag– Fordypning—Forståelse. En fornyelse av Kunnskapsløftet [White Paper: Subjects—In-Depth Learning—Understanding. A Renewal of the Knowledge Promotion Reform]. Ministry of Education and Research.

Formal documents make up more than half of the co-cited references on the list of the most cited references. They play a crucial role in the reform because they serve as a link to previous reforms and arguments for reforming education in earlier periods. Mostly they serve as normative points of reference, from which specific further steps in the renewal of the Knowledge Promotion Reform (2016/2020) are deducted. In this way, former written and published official knowledge is credited with legitimized power that does not need further justification.

The following quotation is the starting point for arguing for more collaboration among teachers and principals to enhance quality development and student learning:

One of the ambitions of the Knowledge Promotion Reform was that schools should develop a culture for learning to a larger extent. The starting point for this goal was the St. Meld. 30 (2003–2004) Culture for Learning, which stated that schools have to be learning organizations. The goal was that the schools’ ability and willingness to learn and develop them further should be improved. (WP#2, p. 27)

The governmental and official reports establish continuity in the reform process by providing a history that connects all previous reforms with those that will be implemented. The use of formal documents in the two white papers appears to be evenly distributed, with a slight tendency for WP#2 to lean more heavily on formal documents. With only a few exceptions, we categorized the use of formal documents as regulative, reflecting a conditional program for reforming education and schooling; as such, they stabilize the reform rather than renewing and transforming the schools in radical terms.

Conclusions

In this chapter, we analyzed the distribution and argumentative use of co-cited and in-text references within and across policy realms of Norway’s most recent school reform. We aimed to examine how policymakers and experts responsible for writing two white papers and eight public inquiry reports responded to critiques and trends that have generated demands for a more evidence-based policy. We asked how state administrative bodies such as ministries and inquiry bodies made policy evidence-based by referencing different types of knowledge sources and how their modes of arguing for reform translated knowledge in terms of two decision programs that each covers two types of arguments. As a result, we assessed how references were embedded in regulative, informative, normative, or cognitive arguments.

Both our bibliometric network and semantical analyses revealed that the authors and bodies involved in this work have reacted to the critique that inquiries have been too practical and general in earlier reform periods. This reaction is documented in the extensive usage of references within the white and green papers. Throughout this chapter, we have demonstrated that Norwegian reform documents authorized by the Ministry of Education are thereby heavily evidence-based, as shown through their references to various types of knowledge (e.g., research reviews including meta-analyses and configurational analyses) and their use of phrases like “as thoroughly documented in research.” The core basis of the most cited documents within the full database contains not only typical evidence-based reviews, but also many governmental documents; in fact, more than half of the most co-cited references are governmental reports or documents. In addition, by taking in-text references into consideration, we showed that the report of Aasen et al. (2012) plays a crucial role by setting the scene for reform and renewal across a variety of topics within both policy realms related to the curriculum reform and to educational governance of the basic education system.

The argumentative use of the references in the two white papers corresponds, more or less, to the type of category they represent. In other words, reviews in the form of meta-analyses and configurational analyses tend to be used in a purposive manner, while governmental papers are mainly used to ensure that certain conditions are in place and formal procedures are followed (conditional manner). However, some observations show interesting variations and unexpected translations.

First, there is a commonly held view that reform options become standardized through evidence-based policy that makes use of measurements as quantifiable expressions of performance. In our study, relatively few citations specify standards for best practices (see “cognitive outcomes” on the right side of Tables 7.4 and 7.5). Few publications that were cited in the white papers provide measures and standards for how to improve student learning and outcomes by purpose. The absence of knowledge provided by international organizations such as the Organisation for Economic Co-operation and Development (OECD) among the most co-cited references is possibly a reason for this pattern. However, another important finding is that the scientific reviews are in no case used to regulate or inform about past policies or practices in a conditional form, but only in terms of arguments that call for purposive, future-oriented changes within the education system. Thus, the use of scientific reviews directs attention toward performance-oriented outcomes.

Our analysis shows that meta-analyses and configurational analyses serve as sources for normative recommendations for how to improve student learning instead of explicitly referring to results of performance studies or specifying cognitive outcomes. This finding holds particularly true for WP#1, whereas WP#2 references Hattie (2009) five times to specify cognitive outcomes. Interestingly, another important finding is that the scientific reviews are in no case used in a conditional way; instead, they are used as arguments for purposive, future-oriented changes within the education system.

Second, our analysis shows that there is significant use of governmental reports for regulative use, which means that arguments in the two white papers refer to formal procedures or decisions documented in those governmental reports. The use of formal documents seems to help with identifying the intention of past decision processes. In addition, they stabilize the reform by creating memories of the past. However, in WP#1, governmental reports are used not only in a regulative manner, but also in substantive, normative, and cognitive arguments. The use of NOU, 2015: 8 is the prime explanation for this pattern, as it is used frequently to legitimize both conditional and purposive program forms. This source is the only one among the most prominent references that is used to make all four types of arguments (i.e., regulative, informative, normative, and cognitive). A dominant pattern is that NOU, 2015: 8 serves a purposive role to legitimize the renewal of the national curriculum. The report was cited in chapters like “competence aims within the subjects,” “an increased cross-disciplinary orientation in teaching and learning,” “strengthening clearer priorities within the subjects,” “emphasizing improved coherence and progression,” “learning strategies and reflection on one’s own learning,” and “competence development and capacity building” (see Table 7.4). The pattern is in no way surprising, as the authors of WP#1 were mandated to legitimate the renewal of the reform by drawing on earlier governmental and official reports, particularly NOU, 2015: 8 The School of the Future. The document, however, is not included as a reference in WP#2, which addresses issues related to early intervention and quality in schools.

Third, we found that the program evaluation reports that contribute empirical research on education reforms are used for various purposes. Interestingly, Aasen et al. (2012) is distributed over two categories, reflecting both substantive and normative forms of use. While it tends to be translated in a normative-oriented way in WP#1 in tandem with the Dale et al. (2011) report, authors of the WP#2 apply Aasen et al. (2012) in a substantive, retrospective way. In WP#2, the use of Aasen et al. (2012) differs from references to Dahl et al. (2016), which was formally commissioned by the state as an expert report. Dahl et al. (2016) summarized knowledge about the teacher as a key agent within the education system, for which it is used in a normative way for several themes, such as “teachers’ collaborations in school.” For the sections that cover quality assessment and the evaluation system, neither the reference to Aasen et al. (2012) nor Dahl et al. (2016) sets the direction in a purposive way (see Table 7.5).

Finally, the reference patterns we have uncovered through our analysis help us explore the role of evidence in education policy. Evidence seems to be structured by the knowledge sources that are used while being deeply dependent on the arguments formulated by policymakers. Looking for examples in the content of the references, we see a clear pattern in which formal documents or governmental papers connect reform options with political ambitions and jurisdictions. These connections exist because the formal documents offer information about various conditions for organizing education and because they are mandated to do so. Moreover, empirical research studies and reviews that refer to the inner life of schools can be used to inform policymakers about educational concerns in renewing teaching and learning. Thus, these references are affiliated with topics that policymakers potentially address when they explore options for education reform. If they are also written under the auspices of state-funded programs to evaluate the reform, this condition is likewise an explanation for their prominence. We discovered that the two white papers frequently cite evaluation reports about the former curriculum reform, the Knowledge Promotion Reform 2006. This finding underlines how the evaluation and use of research reports enable a tight connection between the previous and the current reform. Apparently, results and interpretations from the evaluation of the Knowledge Promotion Reform inform policymakers; more importantly, they serve as a normative basis for formulating specific policy options that aim at curriculum renewal and improvement.

No matter their intended use, references serve the function of bringing evidence into policy, in terms of either bibliometric networks or explicit use to formulate arguments within the bodies of the texts. Through this function, references align systems of reasoning that are highly differentiated in practice. In this way, policy knowledge plays an inter-mediational role by shaping reform options connecting both policy realms and systems of reasoning.