Keywords

3.1 Introduction

This chapter focuses on the analysis of popularization discourse. Before we introduce our own analytical framework in Chap. 4, we take a step back to give an overview of current frameworks for popularization discourse and to critically discuss their usefulness and applicability. The chapter contains an overview of methodological considerations that are relevant when constructing frameworks, and an overview of the academic literature on current frameworks and rubrics to analyze and assess popularized texts.

3.2 Text Analysis: Goals and Formats

While the previous chapter discussed how scientific data and insights can be communicated to the general public through texts, we will now turn to those texts themselves. In other words, we will look at those popularized texts as research objects or as generators of academic data and insights. Often, analytical frameworks are used for this cause, also sometimes called coding schemes. Frameworks and coding schemes ideally provide a way to analyze texts objectively, reliably, and in collaboration with multiple researchers or analysts.

The overarching aim of text analysis is to reconceptualize text as data to be analyzed. In other words, by treating text as a research object, text analysis can provide insight, either qualitative or quantitative, about it. Multiple research traditions have their own forms of text analysis, such as linguistics, computer sciences, and social sciences. In this book, we will focus solely on the linguistic tradition, where the main aim of text analysis is to describe text structure (Roberts, 2000). Broadly seen, there are two types of linguistic text analysis: qualitative (Kuckartz, 2014, 2019) and quantitative (Roberts, 2000). Quantitative text analysis is either representational (researchers classify the intention of the writer) or instrumental (researchers apply a theory to interpret the text). Quantitative analysis always produces a data matrix, which can then be used in statistical analysis. Options for quantitative text analysis are thematic text analysis (occurrence of themes), semantic text analysis (relations among themes), and network text analysis (networks of interrelated themes) (Roberts, 2000). Qualitative text analysis, on the other hand, is defined through the use of categories. The aim is to reduce complexity through classification based on characteristics, which can be derived from theory (Kuckartz, 2014). Research that uses qualitative text analysis uses categories (or codes) to develop a category system or coding scheme. Categories can be developed in a construct-driven (deductive) or data-driven (inductive) way, or as a mix of these two options. These categories can be factual, thematic, evaluative, analytical, theoretical, natural, or formal (Kuckartz, 2019). The specific interpretation of what a category is, or how a category can be described or analyzed, often remains implicit—which is a problem that we time and time again encountered during our own research. This sentiment is reflected by Kuckartz:

The question of what exactly a category represents in empirical research is hardly addressed in literature on research methods, even in textbooks that focus on methods of qualitative data analysis, it is more or less assumed that people already know what a category is, based on common sense. Instead of a definition, you often find a collection of category attributes, particularly in textbooks about qualitative data analysis. There it can be read, for example, that categories should be ‘rich’, ‘meaningful’, ‘distinguishable’, or ‘disjunctive’. (Kuckartz, 2014, p. 39)

In the academic literature, presented strategies or categories often lack meaningful descriptions or explanations. Yet analytical frameworks are used because they allow for objective, reliable, and shared analysis. In this book, we have therefore added extensive notes on the strategy (that is, category) level to give a rich description of what each category in the analytical framework means, as such avoiding the pitfall that was discussed in the above quote by Kuckartz. We are proposing a framework that can be used both quantitatively and instrumentally (to score the occurrence of categories) or qualitatively and representationally (to explain the specific use of each category). But before we delve deeper into our framework, let us explore existing frameworks and coding schemes that are used to analyze popularization discourse.

3.3 Text Analysis of Popularization Discourse

In the current literature, popularization discourse is analyzed in multiple ways. Analysis can focus on the achievement of communicative goals (see Metcalfe, 2019), on content analysis (see Kessler, 2019; Shea, 2015), on a specific textual feature (see Rakedzon et al., 2017; Sharon & Baram-Tsabari, 2014 for the analysis of jargon; Riesch, 2015 for the analysis of humor), on componential analysis (August et al., 2020; Giannoni, 2008; Hyland, 2010; Luzón, 2013; Motta-Roth & Lovato, 2009; Nwogu, 1991), or on assessment of popularization discourse in educational settings (Moni et al., 2007; Poronnik & Moni, 2006; Rakedzon & Baram-Tsabari, 2017a, 2017b; Yuen & Sawatdeenarunat, 2020). In this chapter, we focus on two forms of analysis of popularization discourse, as they come closest to providing an overview of the textual features in the genre: componential analysis through frameworks and assessment in educational settings through rubrics. Frameworks can take many different forms, ranging from a list of components to an overview of scoring criteria. Rubrics, on the other hand, often follow a strict template. They are presented in the form of a table, with the assessment criteria (the skills that are graded through the rubric) on one axis and the grades on the other. At each criteria/grade intersection, the table provides an explanation of the assessment point, thus forming a guideline for teachers in their grading (Stevens et al., 2012). The following sections will first cover frameworks that describe popularization strategies, followed by an overview of rubrics that deal with the assessment of popularization discourse.

3.4 Frameworks for Popularization Strategies

Currently, structural or textual models of popularization discourse are scarce, and an overarching analytical framework—a framework that spans across text types and academic disciplines— is non-existent. Only a handful of researchers have analyzed the use of popularization discourse on the level of structural components, that is, textual strategies or structural categories (August et al., 2020; Giannoni, 2008; Hyland, 2010; Luzón, 2013; Motta-Roth & Lovato, 2009; Nwogu, 1991). We will briefly describe the set-up of each of these studies.

Nwogu (1991) analyzed 15 journalistic reported versions of medical texts to explore their discourse structure. Swales’ genre analysis model was used as a theoretical framework. The study resulted in an overview of eight moves and constituent elements. These moves are presented chronologically, that is to say, in a text they always appear in the same order. Giannoni (2008) studied popularization features in 40 journal editorials from medicine and applied linguistics and found seven popularization features. Motta-Roth and Lovato (2009) focused on the rhetorical organization of 30 popularization news articles. They used Nwogu’s (1991) framework as a basis and created a new list of six moves that occur in a specific order, combined with two types of discursive elements that can occur throughout the text. Hyland (2010) analyzed the use of proximity in 120 research articles versus popular science articles and found five thematic strategies. Luzón (2013) used an a priori coding scheme from the literature and then employed grounded theory to analyze 75 science blog posts. The research yielded a framework with three themes and 23 strategies. August et al. (2020) used manual coding of 337 sample articles and computational analysis to analyze the use of ten popularization strategies in a corpus of 128,000 documents.

The structural components, or strategies, that each of these studies delineated can be found in Table 3.1. Because the frameworks from Motta-Roth and Lovato (2009) and Nwogu (1991) work (partly) with a specific order of strategies, they are presented specifically as ‘moves.’ Two other studies worth mentioning are Nisbet et al. (2003) and Calsamiglia and Van Dijk (2004), but because their focus was on a specific topic (stem cells and the genome), the framing is too narrow to compare them with the other sources.

Table 3.1 Overview of popularization strategies mentioned in the literature
Full size table

3.5 Discussion of Current Frameworks

In this section we discuss similarities and differences between the frameworks, as well as critically analyzing their construction and the insights they generate. There are multiple similarities between the structural components of the frameworks. All frameworks mention the strategy of including main findings. Analogy/metaphor and describing the method are mentioned in four out of six frameworks; impact/implication and explanation of terms in three; and personalization, question, humor, reader engagement, opinion, and contextualization in two. Dissimilarities between frameworks also exist, with many textual components, such as the addition of a title or inclusive pronouns, being mentioned just once. Furthermore, mismatches are visible between the levels at which components are mentioned, for example as a main-level strategy versus a sub-strategy. Reference to the authors is mentioned as a sub-strategy of presenting new research and describing the data collection procedure in Nwogu’s (1991) framework, yet in Motta-Roth and Lovato’s (2009) it is part of presentation of the research and voice switching. Another aspect to keep in mind is the order of linguistic moves; only the frameworks by Nwogu (1991) and Motta-Roth and Lovato (2009) presume a specific order.

It thus becomes clear that although these frameworks are connected to the same research problem and show common delineators, there is no real consensus about which strategies or structural components appear in popularization discourse. The studies described are mostly isolated projects—the exception here is the study by Motta-Roth and Lovato (2009) that used the Nwogu (1991) study as a point of departure. This also points to a larger issue: the use of popularization strategies does not appear to be a standardized research topic and therefore there is no chronology in reasoning or projects building from one another.

Another important issue with the aforementioned studies has to do with their construction and validity. Datasets are generally (very) small, ranging from 15 to 120 texts, except for the research by August et al. (2020) in which a 128,000-document corpus was used. It should be noted though that this corpus was analyzed using computational analysis, with human coders hand-coding a sample of only 337 articles. This shows the boundaries of human coding in linguistics, with computational analysis offering a chance at coding corpora many times the size of what is usually achievable in hand-coded research. More generally, it is questionable how reliable the frameworks are when they are based on a small dataset, often consisting of texts from a specific subtype of popularization discourse or a single academic field/couple of academic fields. For a framework to be reliable and all-encompassing, it should be based on the analysis of multiple subtypes of popularization and draw sources from multiple disciplinary fields.

Half of the sources do not explain the methodological steps that were taken in the construction of their presented framework. Although Nwogu (1991) mentioned that a genre analysis model was used, no information was given on data analysis or construction of the framework. Likewise, Giannoni (2008) and Hyland (2010) provided little information on methodological steps taken. The other three sources do provide more detail. Luzón (2013) described how grounded theory was used in combination with an a priori framework based on literature. August et al. (2020) showed in detail how writing strategies were chosen, the computational analysis was conducted, and a subset of texts was hand-coded on a sentence level. Motta-Roth and Lovato (2009) explained how three rounds of analysis consisted of individual analysis, cross-analysis, and identification of linguistic components. Because not every source offers an equally clear explanation of the methodological steps that have been taken in the construction, this poses issues for the reproducibility of the research. It also means that it is difficult to estimate the reliability and overall applicability of the frameworks.

Most of these studies are single-author papers, meaning the analysis was also conducted by a single author, or, if multiple people worked on the analysis, this was never mentioned (Giannoni, 2008; Hyland, 2010; Luzón, 2013; Nwogu, 1991). Motta-Roth and Lovato (2009) did use multiple raters and checked for consensus in their analyses, although there is no report on inter-rater reliability. The exception is August et al. (2020) who used Krippendorff’s α to measure inter-rater agreement for the manual coding of annotations and reported findings on a strategy level. Ideally, a framework should be used consistently by different raters. By not using multiple raters to construct or evaluate the framework, no knowledge is available about inter-rater reliability, or in other words, the consistency of coding across raters (Holton, 2007; Hallgren, 2012; Kuckartz, 2014). A lack of data on inter-rater reliability hampers the usability of these frameworks, as there is no assurance that the findings that are produced through them are reliable across users.

The biggest issue that surfaces through the analysis of current frameworks is that none of these studies factually presents a framework for analysis, in the sense that they report solely on results of text analysis. These results are presented as a list of strategies, and in some papers examples are provided. Whether the strategies that are found in one subgenre of popularization or in sources from one discipline can be generalized to other texts or subgenres within popularization discourse remains unclear. In the same vein, these lists do not (or cannot) provide any information on how they should be used in other analytic studies; that is to say, no meta-text, coding information, size of coding, or coding manual is presented in any of these studies. Consequently, there is a lack of validation of strategies in all discussed studies. An exception here is August et al. (2020), who specified their coding to be on the sentence level and reported the accuracy of the computational analysis on a strategy level. Generally speaking, the lack of explicit insight into the methodological choices made in these studies makes it impossible for these lists of strategies to be used as analytical frameworks in follow up studies into popularization discourse.

3.6 Rubrics for Popularization Discourse

Apart from the frameworks discussed above, the structural components of popularization discourse can also be captured in rubrics, which are used in education for assessment purposes. The literature about the assessment of popularization discourse stems from the research field of science communication. Some studies focus on the learning goals that should be implemented in science communication courses (see Baram-Tsabari & Lewenstein, 2013, 2017a, 2017b; Bray et al., 2012; Mercer-Mapstone & Kuchel, 2015). Alternatively, studies focus on the assessment of popularization skills acquired in those courses, which is often conducted using rubrics (Moni et al., 2007; Poronnik & Moni, 2006; Rakedzon & Baram-Tsabari, 2017a, 2017b; Yuen & Sawatdeenarunat, 2020). The research field also includes rubrics that primarily consider speaking skills (see Alias & Osman, 2015; Sevian & Gonsalves, 2008; Murdock, 2017), but these were left out of the current overview, to retain the focus on writing skills.

We compared rubrics from five studies. Two studies focused on rubric construction (Rakedzon & Baram-Tsabari, 2017b; Yuen & Sawatdeenarunat, 2020), whereas the other three used a rubric as an assessment tool for learning outcomes of explicit instruction in an educational setting (Moni et al., 2007; Poronnik & Moni, 2006; Rakedzon & Baram-Tsabari, 2017a). Moni et al. (2007) used an opinion editorial rubric to assess learning outcomes for science communication skills in final-year physiology and pharmacology students. Poronnik and Moni (2006) used an opinion editorial rubric for peer review and to assess learning outcomes in undergraduate physiology students. Rakedzon and Baram-Tsabari (2017b) constructed and evaluated a rubric to assess L2 (non-native) STEM graduate students’ popularized writing, which was then used in a pretest-posttest intervention study by Rakedzon and Baram-Tsabari (2017a) to assess popularization skills in L2 science and engineering graduate students. Yuen and Sawatdeenarunat (2020) used popular news articles written by science undergraduate students in a rubric development cycle to construct a science communication rubric. Table 3.2 gives an overview of the assessment items mentioned in these rubrics.

Table 3.2 Overview of assessment items in popularization rubrics
Full size table

3.7 Discussion of Current Rubrics

Some similarities exist in the content of these rubrics. The rubrics by Moni et al. (2007) and Poronnik and Moni (2006) are very similar; these papers are in large part written by the same authors. The same goes for the two identical rubrics by Rakedzon and Baram-Tsabari (2017a, 2017b). Overall, more differences than similarities are visible between rubrics. Each can explain part of the picture of popularization discourse assessment, but the dissimilarities between the rubrics signify that none can give an overarching view (that is, spanning across text forms and academic disciplines). Rubric operationalization is largely the same in all studies; rubrics comprise an assessment grid with scoring options for a range of grades for each assessment criterion from 1 to 8% (Moni et al., 2007), 1–2% to 9–10% (Poronnik & Moni, 2006), 1 to 4 (Rakedzon & Baram-Tsabari, 2017a, 2017b), or 1 to 8 (Yuen & Sawatdeenarunat, 2020). Because of these differences in rating scale, the rubrics are not easily comparable. The rubrics by Moni et al. (2007) and Poronnik and Moni (2006) are specifically constructed for one text type, the opinion editorial. This makes them less broadly applicable and more difficult to compare to other rubrics. More generally, and this is true for all rubrics to some degree, the rating of a certain criterion is (partly) dependent upon a subjective assessment by the rater.

Rubric construction and validation are only discussed in three studies. Rakedzon and Baram-Tsabari (2017b) constructed their rubric in a five-stage model that included developing course goals, choosing assessment tasks, setting their standard, developing assessment criteria, and rating values for scoring. Its validation was conducted by checking scoring consistency with two raters. In Rakedzon and Baram-Tsabari (2017a), the rubric was constructed based on course materials and previous research. It was piloted in two rounds, after which empirically developed descriptors were added. Yuen and Sawatdeenarunat (2020) developed quality definitions in their rubric through the analysis of science-related newspaper articles. Student ability, rater severity, item difficulty, and quality of the rating scale were calibrated using a Many-facet Rasch Model. Raters were asked to also mark sample scripts and were interviewed about their marking. A survey was used to measure student perception of the rubric. In these studies, underlying methodologies differ greatly, which hampers the compatibility of rubrics—in other words, it is difficult to construct one overarching insight in assessment criteria. In Moni et al. (2007) and Poronnik and Moni (2006), the construction of the rubric is not discussed explicitly. These rubrics suffer from a methodological gap; it is unclear how sound their construction and how valid their use in practice is.

Just as is the case with the frameworks that were described in the previous two paragraphs, there seems to be no overarching line in academic advancement of rubrics, with researchers working independently and projects not being used for follow-up studies (the exceptions here are the studies conducted by mostly the same authors). This is a pity as it means that there is no clear academic advancement of insights, and each research team that is working on the topic seems to be reinventing a slightly different wheel.

What You Have Learned in This Chapter

  • Text analysis sees texts as research objects and turns them into data either through quantitative or qualitative analysis.

  • In the current literature, popularization discourse is either analyzed through frameworks or assessed through rubrics. The results from these methods show a whole range of structural components, strategies, or assessment criteria in popularization discourse.

  • It is impossible to produce one overarching overview of strategies because of differences in subtypes of popularization discourse and disciplines from which source materials are gathered.

  • There is a lack of explanation of methodological steps undertaken in some of these studies, which points to a bigger methodological issue in the research field.

  • There is a need for an analytical framework, or in other words, a coding scheme, for popularization discourse.