Keywords

Everyone is entitled to his own opinion, but not to his own facts. – Daniel Patrick Moynihan

A common expectation among engagement scholars is that participating in public engagements will lead to better-informed citizens (e.g., Gastil & Dillard, 1999; Selin et al., 2017). We want citizens to know about policies and technologies and then make decisions about them. We want citizens to have reactions that are informed by facts that experts can agree upon, rather than using “alternative facts.” Recent research on online rumors (e.g., Garrett, Weeks, & Neo, 2016) and boomerang effects (e.g., Hart & Nisbet, 2012) (when persuasion attempts result in people adopting the opposite of what the persuasion was pointing toward) reinforces the idea that simply being aware of a policy is not enough. This research suggests that rumors are prevalent in online contexts, and individuals will often cling to incorrect facts even after hearing that their information is incorrect. Knowledge is often at the forefront of discussions about the role that citizens should play in science policy discussions and was a crucial outcome examined in our own work. The research here deals with a few key questions: What situations will increase learning during public engagements? What features will help participants best understand the facts relevant to the policies and technologies under consideration? We address knowledge in our studies by assessing what the student engagement participants learned over time, and what helped or hindered that learning.

3.1 Why Does Knowledge Matter?

A disappointing but well-known fact is that Americans are especially ill-informed about a variety of political issues. Late-night comics make a joke of this in the common “man-on-the-street” interviews that highlight how little the average person knows as the audience laughs along. A 2014 video created by a group of Texas Tech students went viral when it illustrated fellow students’ lack of knowledge of the name of their vice president and who won the Civil War.Footnote 1 While we may laugh at our own expense, what people know does, at least theoretically, have important implications for their reactions to policy proposals. Democratic theorists assert that individuals need to be knowledgeable in order for them to participate effectively in the public sphere. Delli Carpini and Keeter (1996, pp. 5) state that “knowledge provides the raw material that allows citizens to use their virtues, skills, and passions in a way that is connected meaningfully to the empirical world.” Therefore, having knowledge is an important qualification in order to self-govern and to have meaningful debates about various issues.

Unfortunately, knowledge is not equally distributed among citizens, and how that can be ameliorated is the topic of decades of research in political science and communication. At the core of research on knowledge in political and policy-related contexts is the assumption that inequalities in knowledge will lead to inequalities in participation and ultimately inequalities in the benefits that can be obtained from participation (Dutwin, 2003). Thus, in addition to raising knowledge levels, it would be valuable to help people to reach more equal levels of knowledge through public engagement activities.

Deliberative democratic theorists assert that the best decisions will be made for the greatest number of people when citizens are informed about all sides of an issue. We can trace this line of thinking back to John Stewart Mill and the “Marketplace of Ideas.” Essentially, Mill believed that, if we let the information be known, the best ideas will rise to the top (Mill, 1860). However, being aware of something is not the same as having knowledge. The current debate about fake news and the speed with which rumors can spread online highlight that difference (e.g., Allcott & Gentzknow, 2017). Instead, the marketplace of ideas needs to inform us more substantially—we need to fully understand what is happening rather than simply being aware that a policy or debate over policy exists. The purpose of public engagements is to provide a variety of information and perspectives to individuals, with the hope that they leave more knowledgeable, and with clearer and more evidence- and fact-consistent beliefs (Delli Carpini, Cook, & Jacobs, 2004).

We need to note that knowledge is sometimes viewed as an inappropriate focus in studies of engagement , because it is often associated with so-called deficit models used in science communication (e.g., Sturgis & Allum, 2004). These models propose that citizens do not believe in or value various scientific findings or do not behave in a way that is desired, simply because they do not have adequate knowledge (or trust or empathy, etc.). In essence, deficit models assume an ignorant public and assert that everyone will agree once the problem of ignorance (or mistrust, etc.) is fixed. While there is some research to support this view (van der Linden, Leiserowitz, Feinberg, & Maibach, 2015), the vast majority of research paints a much more complex picture (Druckman & Bolsen, 2011; Sturgis & Allum, 2004). Some would argue that, even given the same set of facts, different individuals will come to different conclusions based on their own values and other considerations. Even those who critique the role of knowledge in producing other public engagement outcomes, however, do not question that knowledge is beneficial for citizen engagement. That is, while knowledge may not always lead to changes in behavior or attitudes, we would argue that knowing more would be an indication of a successful public engagement. In public engagement , deficit model thinking, or simply providing a set of facts that will remove ignorance and assure the “right” way of behaving, is not the focus. Discussion of pros and cons is often explicitly encouraged rather than asserting one way as preferred, and participants are encouraged to share their own perspectives.Footnote 2 However, engagement scholars believe that participants do need to be working from the same set of facts, which can be provided at several stages throughout the engagement activities (Goodin & Niemeyer, 2003; Muhlberger & Weber, 2006).

3.2 How Can Public Engagements Foster Increases in Knowledge?

Prior research on participation in public engagements have found that participants feel like they know more (Powell & Kleinman, 2008) and in many cases actually do know more facts (Fishkin, 1997; Luskin, O’Flynn, Fishkin, & Russell, 2014) about the topic at hand after participating. The research that has explicitly examined when individuals are most likely to learn suggest that most of the learning from participating in public engagements happens while individuals are reading the information provided prior to deliberation, and the discussion itself does not add much additional information (Goodin & Niemeyer, 2003; Muhlberger & Weber, 2006). This research is the exception however, and most look at the impact of the entire public engagement experience. Much of the research and theory about everyday discussion would suggest that the discussion is essential in helping individuals crystallize what they learn from other sources (Eveland & Hively, 2009; Mansbridge, 1999; Sanders, 1997); however, the research supporting that fact in the presence of readings and discussion is rather scant.

There are a host of theories that can help explain why participating in public engagements would lead to an increase in knowledge, but they can generally be tied to two different perspectives. One perspective is simply that providing information will lead to an increase in knowledge, with the caveat that the information is attended to in some fashion. A wrinkle to this perspective in the public engagement literature focuses on how the material is being presented. For example, the National Issues Forum recommends that information be presented in a balanced, nonpartisan fashion. The second perspective instead focuses on cognitive engagement with various aspects of the materials as the reason behind why individuals learn. Put simply, participating in an engagement will motivate you to think (or cognitively engage) with the concepts somehow, which in turn will foster learning. Along these lines, there is evidence that when participants know their input will matter, this increases motivation, and they are more likely to cognitively engage with the materials (Powell & Kleinman, 2008). However, the reasoning behind the “presentation of materials” arguments is directly tied to presenting the material in a way that will foster deeper or more effective cognitive engagement, so the two perspectives are not completely distinct. Over our series of studies, we examine both of these perspectives to try to better understand how best to foster increases in learning.

3.2.1 Informational Presentation

Public engagements will typically give participants some sort of information prior to coming together as a group. During this preliminary information presentation, which may be in the form of materials to be read in advance of the deliberative gathering or an initial information session during the gathering, the “facts” are stated in advance of the deliberative portion of the engagement. There is generally extensive care put into the creation of these documents to try to provide the nuance of various perspectives from a balanced, nonpartisan perspective. This is done so individuals will come to the deliberative engagement prepared to knowledgeably engage in the various sides of an issue. There are theoretical reasons to believe that how this information is presented can influence how citizens respond to that information. In general, practitioners argue that at a minimum, the information needs to be presented in a balanced manner (Burkhalter, Gastil, & Kelshaw, 2002; Lukensmeyer & Torres, 2006). However, even while meeting that criterion, there are various ways in which that information might be structured.

As noted in Chap. 2, we focused on two different informational presentation strategies that are typically used in public engagements. The format that we expected would have a positive impact on learning was a pro-con organization style. Such a style is typically used, for example, in the National Issues Forums. This strategy explicitly compares the various perspectives on the information that is being presented. The other presentation strategy was a topical organization that does not explicitly call out the different perspectives. Our expectation was the pro-con organization would facilitate learning based on prior educational research that has examined different note-taking styles. Scholars find that making comparisons while taking notes improves learning over simple linear methods (Robinson & Kiewra, 1995). Furthermore, texts structured in a compare-contrast form tend to be associated with improved recall, compared to texts organized linearly or descriptively (Bohn-Gettler & Kendeou, 2014).

The second experimental manipulation relevant to knowledge, which also focused on the presentation of the materials, was the inclusion of prompts encouraging participants to think critically. The justification for these prompts encouraging learning is tied to research in education that shows that students learn differently depending on their goals (Bohn-Gettler & Kendeou, 2014). Participants were simply asked either to provide feedback or to critically evaluate the material at several points throughout the reading. By encouraging students to use critical thinking skills , we expected that individuals would pay closer attention to the material and process the information more deeply, thereby facilitating learning. How individuals process information—or how they engage with the materials cognitively—is an important consideration in many literatures. This was also true for our own studies, and we assessed cognitive engagement several ways throughout our series of experiments.

3.2.2 Cognitive Engagement

The theoretical reasoning behind why a pro-con information organization or critical thinking prompts would facilitate learning all goes back to how individuals may process the material with which we present them. In essence, we were testing whether or not we could get participants to engage with our materials in a deeper and more effortful fashion. We also then explicitly measured the extent to which participants said they were engaged while participating. The need for individuals to process the information they are presented with in order to learn is central to several theories across a variety of fields. Communication scholars will often invoke the communication mediation model (Eveland, 2001; Shah et al., in press) which indicates that the extent to which individuals elaborate, which is engaging in additional thinking, or engage in perspective taking after being presented with information will influence how much they learn. In psychology, dual process models of attitude change—which have also been applied to learning—indicate that longer-lasting effects in attitudes occur when individuals systematically process information (Eagly & Chaiken, 1993; Petty & Cacioppo, 1986). Systematic processing implies individuals are both motivated and able to expend cognitive effort processing information . The theories that we relied on most heavily when examining cognitive engagement are those that reside in education psychology. In particular, that research suggests that the greatest learning will occur when individuals use deep rather than surface-level processing strategies, which is more likely when they are motivated.

3.2.3 Forms of Cognitive Engagement

Research in areas such as cognitive and educational psychology find that one of the most important predictors of whether, how much, and how robustly people learn is how they engage with information. Much of this research has been conducted in educational contexts and examines a variety of different types of study strategies, categorizing ways of studying new materials as involving surface-level versus deep cognitive processing, or as being strategic and conscientious. For our experiments here, we created eight scales designed to measure various facets of individual engagement (for an in-depth examination of the construction and validation of those scales, please see PytlikZillig et al., 2013). The eight types of engagement that we examined were active learning, conscientious, uninterested, creative, open-minded, close-minded, angry, and social (measures used to assess these are discussed in Chap. 2). Active learning and conscientious and uninterested engagement relate to participants’ motivations while engaging. Active learning engagement is assumed to occur when participants’ acknowledge that they are trying to deeply process the information with which they are presented. This deep processing is likely to be tied to increased knowledge. Conscientious engagement refers to an individual’s desire to be careful or thoughtful while examining information, which also would be associated with enhanced learning. Uninterested engagement is characterized by low motivation and boredom, which would likely impede learning. Creative, open-minded, and close-minded engagement focus more on how individuals are participating. Creative engagement focuses on whether individuals are attempting to “think outside the box” and potentially use multiple and unexpected perspectives when participating. While this could increase learning, it is also possible that creative engagement could have a negative impact on learning due to the possibility of distraction by irrelevant or incorrect information. Open-minded and close-minded engagement assess the extent to which individuals are willing to be open versus closed to other’s opinions. Angry engagement examines participants’ negative emotional engagement while participating, which may indicate that participants are engaging in defensive strategies and therefore are less likely to learn. Social engagement is designed to assess the extent to which a participant connected and interacted with others while participating .

3.2.4 Need for Cognition

The last element we examined was an individual difference variable that is frequently used when examining knowledge gain or learning—need for cognition (NFC ). NFC is the general tendency to enjoy and use effortful cognitive processing strategies (Cacioppo et al., 1996). Persons with high NFC should be most likely to learn when participating in public engagements. Such people have been identified as especially likely to participate in deliberations, be more resistant to the arguments of others, and have more influence (Delli Carpini et al., 2004). In our case, we were more interested in various features that could increase knowledge across the board once we accounted for differences in NFC. Essentially, we wondered if we could level the cognitive playing field with our various informational presentation strategies.

3.3 What Do We Mean by Knowledge?

Given our purpose here is to understand increases in knowledge, it is also relevant to discuss what we mean by knowledge, as knowledge can be variously defined. Throughout our studies, we examined both objective and subjective knowledge. It should be noted that integration (Neuman, 1981) or structural knowledge (Eveland & Hively, 2009) is another important facet of knowledge, but one that is typically ignored in research on public deliberations. Objective knowledge is probably what most would think about when hearing the term knowledge—being able to correctly identify explicit pieces of information—that is, facts. Factual knowledge is used extensively in communication and political science literatures when trying to assess what people know about various topics. Within this definition of knowledge, it is assumed that information can be either correct or false. Consequently, the more correct pieces of information people are able to access, the more knowledgeable they are. In public engagement contexts, this would be assessed by measuring how much participants know about the topic at hand prior to participating in the engagement, and again following participation, or in our case at multiple time points throughout the engagement. Beginning with Study 3, we asked knowledge questions at four different time points, at the beginning of the semester (A1), just prior to receiving the reading material (pre-A2), just after reading (post-A2), and in the final assessment at the end of semester (A4). Participants’ knowledge was not re-assessed immediately following the discussion activities (A3), so any additional learning from the discussion activities we would assume to see in the final assessment.

Objective knowledge is most often assessed by asking participants either true-false or multiple-choice questions. The literature on the extent to which individuals show large gains in knowledge during public engagements and via these measures is mixed at best. The majority does not show sharp increases in knowledge; however, there is some research that has found objective knowledge increases over time. The form of knowledge that is more consistently found to be improved via participation in public engagements is subjective knowledge—how much individuals feel they know.

3.4 What Did They Learn?

Before we go into what improved participants’ learning , it’s worth noting that we had to continually refine our objective knowledge measures . For us, measuring subjective knowledge was relatively straightforward, and we used similar items throughout all five studies (see Chap. 2 for details). To create the objective knowledge measures, in Study 3, we created items based on information that was provided in a document participants were asked to read and then examined pre-post engagement statistics for each knowledge question in order to identify the questions that were most sensitive to detecting pre-post changes. While refining our knowledge questions, we learned a lot about what participants did not learn. For instance, one question we had that performed poorly was a “select all that apply” question about what were current areas of nanogenomic research and development. Of the five options, two options showed high levels of correct responses that did not vary. The other three options had very low correct responses initially and got worse over time. True-false questions also posed problems, both in that some showed very high levels of knowledge at pre-exposure and also some questions that participants continued to do poorly on even after completing the reading and the discussions . We determined that these questions were either too intuitive in the case of questions where individuals knew the answer before the public engagement activities or were too difficult to ascertain from the readings in cases where we did not see any improvements over time.

Looking at the means in Table 3.1, we see that our measures did detect knowledge changes. Using both objective and subjective measures , we consistently see increased knowledge scores over the course of the public engagement . The increases are most pronounced when examined just after the readings, but the knowledge gains persist when comparing the end of the semester to the initial or pre-reading measures . These increases in knowledge over time are statistically significant. The knowledge means were tested with repeated measures ANOVA, and in all cases, the later knowledge measures were significantly greater than the initial, pre-reading measures. In a few cases, there was a decrease in knowledge from just after the reading and the end of the semester, but that should not be surprising. Some loss of memory for facts over several weeks is to be expected. But even with that decrease, participants knew significantly more at the end of the semester than they knew at the beginning of the semester.

Table 3.1 Knowledge means
Full size table

This suggests that there were some long-term objective effects of participating in this public engagement. That is, the public engagement as a whole did lead to citizens who were more knowledgeable about nanotechnology. This is also true for subjective knowledge. Our participants consistently reported that they felt they knew significantly more as the semester went on, with the most prominent effects observed after the reading. When we did have both post reading and end of the semester measures, we did not see significant increases between those time points. The group discussion occurred between the reading and the end of the semester, which suggests that reading the materials, rather than engaging in discussion with other participants, had the most profound effects on what they objectively knew and felt that they knew.

Knowing that the majority of the observed learning is occurring after the reading as opposed to after the discussion is critical from a theoretical perspective. When the learning happens in public engagements is frequently not explicitly addressed. Instead most scholars look at the effects of the engagement as a whole, regardless of what aspect most contirubuted to learning. Our findings here are consistent with others that show reading appears to have a stronger impact on learning (e.g., Muhlberger & Weber, 2006). Focusing on reading rather than discussion opens up a lot of possibilities when it comes to the ease of implementing engagement activities. It suggests that what is most important is the time to thoughtfully process new information; therefore remote deliberations, such as Becker and Slaton’s teledemocracy (2000), are possibilities.

Next, we examine the effects of information presentation on how much our participants learned or felt they learned. How we presented the information had minimal to no effect on how much the participants learned. Critical thinking prompts were used in all four studies, and in all cases, the prompts did not result in higher objective or subjective knowledge. In Studies 3 and 4, the information organization manipulation was used, and pro-con versus topical organizational strategies also did not have an impact on objective or subjective knowledge. The interaction between information presentation and critical thinking prompts was also not predictive. These findings could be interpreted as optimistic or encouraging to practitioners. It appears to suggest that if the information presented meets some minimal deliberative expectations (e.g., we strove to use balanced information, although less balanced information in Study 5 did not appear to decrease learning), we do not need to spend much time or effort on how to best structure the arguments or further encourage deep processing. It appears that participants learn at similar rates regardless of the information presentation, and they do still learn during the public engagement event. Whether these strategies matter for other outcomes will be addressed in subsequent chapters, but from a knowledge perspective, simply providing relatively clear and balanced information is enough within this context. Our subjects and context are unique, but this provides an optimistic starting point for future studies.

We now turn to examining whether or not the extent to which participants were engaged with the materials influenced how much they learned, or felt that they learned, by the end of the public engagement event. Table 3.2 shows all significant relationships obtained between our engagement scales and the end of the study measures of knowledge. As you can see, the relationship between the various engagement scales and knowledge was much more apparent when examining subjective knowledge. There were fewer and less consistent relationships between objective knowledge and the various forms of engagement, though a couple measures proved more consistent. In general, it appears that individuals do learn, and how they engage with materials effects the extent to which they felt they have learned, but less so how much they objectively learn.

Table 3.2 Knowledge and engagement
Full size table

Looking first at objective knowledge, conscientious and open-minded engagement were positive predictors of learning , both significantly predicting learning in three of the nine different measures of knowledge. Disinterested and angry engagement were negatively related to learning. Disinterested engagement was a negative predictor of four of the nine measures of knowledge, whereas angry engagement was a significant negative predictor in three of the nine measures of knowledge. Active learning, social engagement, creative engagement, and close-minded engagement were never significantly associated with objective measures. On the whole, there is some evidence of engagement influencing learning, but not consistently.

Consistently across the three studies, conscientious engagement, active learning engagement, and creative engagement had a positive relationship with subjective knowledge. That is, those who indicated they were paying closer, more conscientious attention, actively and metacognitively engaging with the materials and were thinking creatively about the issues were more likely to indicate that they learned more over the course of the deliberative event. Social and open-minded engagement also occasionally had positive effects on reported subjective learning outcomes . The negative forms of engagement were not consistently associated with subjective knowledge. Disinterested and angry engagement were associated with less subjective knowledge in Study 3, but only disinterested engagement was related in Study 4. None of the negative engagement items were related to subjective knowledge in Study 5.

In summary, we can say that participants do learn and feel that they have learned, and how they report engaging with materials has a fairly consistent effect on how much they feel they have learned, but how we structured the materials does not have an effect on how they learn. This opens up the question of whether or not how we structure the materials influences how individuals engage. This may mean that rather than our experimental manipulations having a direct effect on knowledge, engagement could serve to mediate the effect of our experimental manipulations on knowledge.

3.5 What Mediates Knowledge ?

Assessing the alternative information organizations used in Studies 3 and 4 indicates that the pro-con versus topical organization is not associated with any of the forms of engagement. However, there is a fairly consistent pattern for the critical thinking prompts in Studies 3 and 4. As noted in Chap. 2, beginning in Study 3 we used prompts that were less didactic and more gentle to “nudge” participants in the direction of critical thinking. Across Studies 3 and 4, we find consistently that conscientiousness, open-minded engagement, and active learning are positively associated with the critical thinking condition, while close-minded engagement is negatively associated with the critical thinking condition. This suggests that our critical thinking prompts resulted in more positive engagement with the pre-deliberation materials. Given the earlier presented findings indicating that positive engagement is associated with increases in knowledge, this suggests that the critical thinking condition has the potential to influence knowledge indirectly through increases in positive engagement and decreases in close-minded engagement. It should be noted, however, that the relationships between critical thinking and engagement observed in Studies 3 and 4 did not replicate with Study 5. There were no significant relationships between the critical thinking condition and the various forms of engagement in Study 5.

In order to further address the relationship between critical thinking and subjective knowledge , we employ a more stringent test of the relationships by using linear regression with controls and including all of the forms of engagement. The controls allow us to account for other variables that people might argue could account for the relationship between engagement and knowledge and include need for cognition, typical grades, gender, willingness to change their minds, and their prior familiarity with nanotechnology. Given that the relationship between engagement and subjective knowledge was the strongest, this is where we would expect to find significant relationships, which is indeed the case. These analyses indicate conscientious engagement positively predicts increases in subjective knowledge across Studies 3 and 4, even when including various controls and simultaneously examining the effects of the other forms of engagement. The regression also indicates that the critical thinking condition did not have a direct effect on subjective knowledge once the controls and varieties of engagements were considered. These relationships taken together provide evidence that critical thinking prompts might increase learning through influencing how individuals engage with the materials .

To examine if critical thinking had an indirect effect on knowledge, parallel mediation using Hayes (2013) PROCESS macro was utilized, and the conceptual figure is presented in Fig. 3.1. Parallel mediation via PROCESS allows all indirect relationships to be simultaneously tested in order to determine which variables are exerting the strongest influence on our outcome of interest. The results of the process model in study three indicate that two forms of engagement serve as significant mediators of the effect of being in the critical thinking condition on increases in subjective knowledge. That is, different types of engagement serve as the conduit of the influence of critical thinking prompts on subjective knowledge. Controlling for the same set of variables used in the regression analyses, and considering the potential mediating effects of all forms of engagement, there is a significant indirect effect of being in the critical thinking condition on perceived knowledge through increased levels of conscientious engagement and decreased levels of close-minded engagement. That is, being in the critical thinking condition increases conscientious engagement and decreases close-minded engagement, which are subsequently associated with perceived knowledge. In Study 4, conscientious engagement served as a significant mediator of the effect of being in the critical thinking condition and increasing perceived knowledge (utilizing the same analysis strategy as Study 3). Again, being in the critical thinking condition increased conscientious engagement, which in turn was associated with higher levels of perceived knowledge at the end of the semester . Again, the indirect effects were not observed for objective knowledge.

Fig. 3.1
figure 1

Conceptual figure for parallel mediation

Full size image

3.6 Summary and Conclusion

Returning to our motivating question posed in Chap. 1 (what works, for what purposes, under what conditions, and why?), we were able to shed light on this question when it comes to addressing knowledge. In regard to addressing what works when it comes to knowledge, the combined influence of the public engagement activities appeared to have a positive effect. In particular, individuals both felt that they learned, in addition to showing objective increases in what they learned, after completing the readings that were provided. The structure of the readings was not found to be important in and of itself. Simply reading, at least within the specific public engagement we had set up, was enough to lead to increases in knowledge of what nanotechnology is and what it can do. The impact of discussion, regardless of the various scenarios we put participants in (see Chap. 2 for details), was minimal to nonexistent.

Finding the readings to be more important in comparison to the discussion is consistent with other research, despite being counter to what would be expected from deliberative theory. However, this issue is perhaps addressed by our results showing that cognitively engaging with the materials was associated with knowledge gains. Individuals who are more deeply processing the reading materials are essentially engaging in what deliberative scholars would desire, but doing it internally rather than with others. The benefits of discussion in deliberation are presumed to occur due to individuals thoughtfully considering issues and perspectives that are provided by other participants in the deliberation. By measuring how individuals were engaging with the content, we were able to determine that thoughtfully considering issues does in fact increase what individuals feel that they know, and this increase can happen in isolation rather than with others. Further study is needed to determine if this can occur out of a classroom setting or if these observed relationships are only found with students in classrooms.

The element of effectiveness (the “for what purposes” component of our motivating question) that was examined in this chapter was learning, conceptualized and operationalized two different ways. Making distinctions between what participants felt they learned and objective measures of knowledge was shown to be important. Subjective learning, but not objective learning, showed some of the more interesting relationships. These two different measures of learning are clearly different from each other as they frequently did not correlate significantly, although the sign of the coefficients were always positive.

The context element (the “under what conditions” component of our motivating question) was crucial to understand what we actually discovered about learning in public engagements about nanotechnology. How we structured the reading did not directly contribute to whether participant learned or felt they learned. This suggests that how you present information does not matter at all when it comes to objective learning. However, the critical thinking prompts did prove to be important once we considered how participants engaged with the readings. We find that the critical thinking prompts in Studies 3 and 4 encourage deeper processing and discourage negative forms of engagement, which subsequently lead to an increase in subjective knowledge. Including mediators that speak to when different elements may be more successful allowed us to paint a much more nuanced picture of what is occurring during public engagement activities.

From a broader perspective, having three studies that contained the full knowledge battery also allowed us to see what consistently worked or what consistently did not work. Looking at Table 3.2 paints a very clear picture of the importance of conscientious engagement —which is engaging in more thoughtful and deep processing—across all measures of knowledge and across all studies. Given the so-called replication crisis plaguing the social sciences, ensuring that we understand and appreciate the robustness of our effects is going to be important. Multiple studies were able to confirm the importance of conscientiousness and also show areas where effects may not be as robust as expected. In Study 5, the indirect effects of the critical thinking condition on subjective learning was not found, contrary to Studies 3 and 4. Study 5 had a different set of experimental manipulations in comparison to Studies 3 and 4, even though the critical thinking prompts remained the same. While we did not find direct or interactive effects of these various manipulations on learning or engagement, it is possible that these changes in the design of Study 5 changed the observed relationships. However, at this point, we have more findings than we have adequate theory to cover them.