Significance of a Study: Revisiting the “So What” Question

Abstract

Every researcher wants their study to matter—to make a positive difference for their professional communities. To ensure your study matters, you can formulate clear hypotheses and choose methods that will test them well, as described in Chaps. 1, 2, 3 and 4. You can go further, however, by considering some of the terms commonly used to describe the importance of studies, terms like significance, contributions, and implications. As you clarify for yourself the meanings of these terms, you learn that whether your study matters depends on how convincingly you can argue for its importance. Perhaps most surprising is that convincing others of its importance rests with the case you make before the data are ever gathered. The importance of your hypotheses should be apparent before you test them. Are your predictions about things the profession cares about? Can you make them with a striking degree of precision? Are the rationales that support them compelling? You are answering the “So what?” question as you formulate hypotheses and design tests of them. This means you can control the answer. You do not need to cross your fingers and hope as you collect data.

Part I. Setting the Groundwork

One of the most common questions asked of researchers is “So what?” What difference does your study make? Why are the findings important? The “so what” question is one of the most basic questions, often perceived by novice researchers as the most difficult question to answer. Indeed, addressing the “so what” question continues to challenge even experienced researchers. It is not always easy to articulate a convincing argument for the importance of your work. It can be especially difficult to describe its importance without falling into the trap of making claims that reach beyond the data.

That this issue is a challenge for researchers is illustrated by our analysis of reviewer comments for JRME. About one-third of the reviews for manuscripts that were ultimately rejected included concerns about the importance of the study. Said another way, reviewers felt the “So what?” question had not been answered. To paraphrase one journal reviewer, “The manuscript left me unsure of what the contribution of this work to the field’s knowledge is, and therefore I doubt its significance.” We expect this is a frequent concern of reviewers for all research journals.

Our goal in this chapter is to help you navigate the pressing demands of journal reviewers, editors, and readers for demonstrating the importance of your work while staying within the bounds of acceptable claims based on your results. We will begin by reviewing what we have said about these issues in previous chapters. We will then clarify one of the confusing aspects of developing appropriate arguments—the absence of consensus definitions of key terms such as significance, contributions, and implications. Based on the definitions we propose, we will examine the critical role of alignment for realizing the potential significance of your study. Because the importance of your study is communicated through your evolving research paper, we will fold suggestions for writing your paper into the discussion of creating and executing your study.

We laid the groundwork in Chap. 1 for what we consider to be important research in education:

In our view, the ultimate goal of education is to offer all students the best possible learning opportunities. So, we believe the ultimate purpose of scientific inquiry in education is to support the improvement of learning opportunities for all students…. If there is no way to imagine a connection to improving learning opportunities for students, even a distant connection, we recommend you reconsider whether it is an important hypothesis within the education community.

Of course, you might prefer another “ultimate purpose” for research in education. That’s fine. The critical point is that the argument for the importance of the hypotheses you are testing should be connected to the value of a long-term goal you can describe. As long as most of the educational community agrees with this goal, and you can show how testing your hypotheses will move the field forward to achieving this goal, you will have developed a convincing argument for the importance of your work.

In Chap. 2, we argued the importance of your hypotheses can and should be established before you collect data. Your theoretical framework should carry the weight of your argument because it should describe how your hypotheses will extend what is already known. Your methods should then show that you will test your hypotheses in an appropriate way—in a way that will allow you to detect how the results did, and did not, confirm the hypotheses. This will, in turn, allow you to formulate revised hypotheses. We described establishing the importance of your study by saying, “The importance can come from the fact that, based on the results, you will be able to offer revised hypotheses that help the field better understand an issue relevant for improving all students’ learning opportunities.”

The ideas from Chaps. 1, 2, and 3 go a long way toward setting the parameters for what counts as an important study and how its importance can be determined. Chapter 4 focused on ensuring that the importance of a study can be realized. The next section fills in the details by proposing definitions for the most common terms used to claim importance: significance, contributions, and implications.

You might notice that we do not have a chapter dedicated to discussing the presentation of the findings—that is, a “results” chapter. We do not mean to imply that presenting results is trivial. However, we believe that if you follow our recommendations for writing your evolving research paper, presenting the results will be quite straightforward. The key is to present your results so they can be most easily compared with your predictions. This means, among other things, organizing your presentation of results according to your earlier presentation of hypotheses.

Part II. Clarifying Importance by Revisiting the Definitions of Key Terms

What does it mean to say your findings are significant? Statistical significance is clear. There are widely accepted standards for determining the statistical significance of findings. But what about educational significance? Is this the same as claiming that your study makes an important contribution? Or, that your study has important implications? Different researchers might answer these questions in different ways. When key terms like these are overused, their definitions gradually broaden or shift, and they can lose their meaning. That is unfortunate, because it creates confusion about how to develop claims for the importance of a study.

By clarifying the definitions, we hope to clarify what is required to claim that a study is significant, that it makes a contribution, and that it has important implications. Not everyone defines the terms as we do. Our definitions are probably a bit narrower or more targeted than those you may encounter elsewhere. Depending on where you want to publish your study, you may want to adapt your use of these terms to match more closely the expectations of a particular journal. But the way we define and address these terms is not antithetical to common uses. And we believe ridding the terms of unnecessary overlap allows us to discriminate among different key concepts with respect to claims for the importance of research studies. It is not necessary to define the terms exactly as we have, but it is critical that the ideas embedded in our definitions be distinguished and that all of them be taken into account when examining the importance of a study.

We will use the following definitions:

• Significance: The importance of the problem, questions, and/or hypotheses for improving the learning opportunities for all students (you can substitute a different long-term goal if its value is widely shared). Significance can be determined before data are gathered. Significance is an attribute of the research problem, not the research findings.

• Contributions: The value of the findings for revising the hypotheses, making clear what has been learned, what is now better understood.

• Implications: Deductions about what can be concluded from the findings that are not already included in “contributions.” The most common deductions in educational research are for improving educational practice. Deductions for research practice that are not already defined as contributions are often suggestions about research methods that are especially useful or methods to avoid.

Significance

The significance of a study is built by formulating research questions and hypotheses you connect through a careful argument to a long-term goal of widely shared value (e.g., improving learning opportunities for all students). Significance applies both to the domain in which your study is located and to your individual study. The significance of the domain is established by choosing a goal of widely shared value and then identifying a domain you can show is connected to achieving the goal. For example, if the goal to which your study contributes is improving the learning opportunities for all students, your study might aim to understand more fully how things work in a domain such as teaching for conceptual understanding, or preparing teachers to attend to all students, or designing curricula to support all learners, or connecting learning opportunities to particular learning outcomes.

The significance of your individual study is something you build; it is not predetermined or self-evident. Significance of a study is established by making a case for it, not by simply choosing hypotheses everyone already thinks are important. Although you might believe the significance of your study is obvious, readers will need to be convinced.

Significance is something you develop in your evolving research paper. The theoretical framework you present connects your study to what has been investigated previously. Your argument for significance of the domain comes from the significance of the line of research of which your study is a part. The significance of your study is developed by showing, through the presentation of your framework, how your study advances this line of research. This means the lion’s share of your answer to the “So what?” question will be developed as part of your theoretical framework.

Although defining significance as located in your paper prior to presenting results is not a definition universally shared among educational researchers, it is becoming an increasingly common view. In fact, there is movement toward evaluating the significance of a study based only on the first sections of a research paper—the sections prior to the results (Makel et al., 2021).

In addition to addressing the “So what?” question, your theoretical framework can address another common concern often voiced by readers: “What is so interesting? I could have predicted those results.” Predictions do not need to be surprising to be interesting and significant. The significance comes from the rationales that show how the predictions extend what is currently known. It is irrelevant how many researchers could have made the predictions. What makes a study significant is that the theoretical framework and the predictions make clear how the study will increase the field’s understanding toward achieving a goal of shared value.

An important consequence of interpreting significance as a carefully developed argument for the importance of your research study within a larger domain is that it reveals the advantage of conducting a series of connected studies rather than single, disconnected studies. Building the significance of a research study requires time and effort. Once you have established significance for a particular study, you can build on this same argument for related studies. This saves time, allows you to continue to refine your argument across studies, and increases the likelihood your studies will contribute to the field.

Contributions

As we have noted, in fields as complicated as education, it is unlikely that your predictions will be entirely accurate. If the problem you are investigating is significant, the hypotheses will be formulated in such a way that they extend a line of research to understand more deeply phenomena related to students’ learning opportunities or another goal of shared value. Often, this means investigating the conditions under which phenomena occur. This gets complicated very quickly, so the data you gather will likely differ from your predictions in a variety of ways. The contributions your study makes will depend on how you interpret these results in light of the original hypotheses.

Contributions Emerge from Revisions to your Hypotheses

We view interpreting results as a process of comparing the data with the predictions and then examining the way in which hypotheses should be revised to more fully account for the results. Revising will almost always be warranted because, as we noted, predictions are unlikely to be entirely accurate. For example, if researchers expect Outcome A to occur under specified conditions but find that it does not occur to the extent predicted or actually does occur but without all the conditions, they must ask what changes to the hypotheses are needed to predict more accurately the conditions under which Outcome A occurred. Are there, for example, essential conditions that were not anticipated and that should be included in the revised hypotheses?

Consider an example from a recently published study (Wang et al., 2021). A team of researchers investigated the following research question: “How are students’ perceptions of their parents’ expectations related to students’ mathematics-related beliefs and their perceived mathematics achievement?” The researchers predicted that students’ perceptions of their parents’ expectations would be highly related to students’ mathematics-related beliefs and their perceived mathematics achievement. The rationale was based largely on prior research that had consistently found parents’ general educational expectations to be highly correlated with students’ achievement.

The findings showed that Chinese high school students’ perceptions of their parents’ educational expectations were positively related to these students’ mathematics-related beliefs. In other words, students who believed their parents expected them to attain higher levels of education had more desirable mathematics-related beliefs.

However, students’ perceptions of their parents’ expectations about mathematics achievement were not related to students’ mathematics-related beliefs in the same way as the more general parental educational expectations. Students who reported that their parents had no specific expectations possessed more desirable mathematics-related beliefs than all other subgroups. In addition, these students tended to perceive their mathematics achievement rank in their class to be higher on average than students who reported that their parents expressed some level of expectation for mathematics achievement.

Because this finding was not predicted, the researchers revised the original hypothesis. Their new prediction was that students who believe their parents have no specific mathematics achievement expectations possess more positive mathematics-related beliefs and higher perceived mathematics achievement than students who believe their parents do have specific expectations. They developed a revised rationale that drew on research on parental pressure and mathematics anxiety, positing that parents’ specific mathematics achievement expectations might increase their children’s sense of pressure and anxiety, thus fostering less positive mathematics-related beliefs. The team then conducted a follow-up study. Their findings aligned more closely with the new predictions and affirmed the better explanatory power of the revised rationale. The contributions of the study are found in this increased explanatory power—in the new understandings of this phenomenon contained in the revisions to the rationale.

Interpreting findings in order to revise hypotheses is not a straightforward task. Usually, the rationales blend multiple constructs or variables and predict multiple outcomes, with different outcomes connected to different research questions and addressed by different sets of data. Nevertheless, the contributions of your study depend on specifying the differences between your original hypotheses and your revised hypotheses. What can you explain now that you could not explain before?

We believe that revising hypotheses is an optimal response to any question of contributions because a researcher’s initial hypotheses plus the revisions suggested by the data are the most productive way to tie a study into the larger chain of research of which it is a part. Revised hypotheses represent growth in knowledge. Building on other researchers’ revised hypotheses and revising them further by more explicitly and precisely describing the conditions that are expected to influence the outcomes in the next study accumulates knowledge in a form that can be recorded, shared, built upon, and improved.

The significance of your study is presented in the opening sections of your evolving research paper whereas the contributions are presented in the final section, after the results. In fact, the central focus in this “Discussion” section should be a specification of the contributions (note, though, that this guidance may not fully align with the requirements of some journals).

Contributions Answer the Question of Generalizability

A common and often contentious, confusing issue that can befuddle novice and experienced researchers alike is the generalizability of results. All researchers prefer to believe the results they report apply to more than the sample of participants in their study. How important would a study be if the results applied only to, say, two fourth-grade classrooms in one school, or to the exact same tasks used as measures? How do you decide to which larger population (of students or tasks) your results could generalize? How can you state your claims so they are precisely those justified by the data?

To illustrate the challenge faced by researchers in answering these questions, we return to the JRME reviewers. We found that 30% of the reviews expressed concerns about the match between the results and the claims. For manuscripts that ultimately received a decision of Reject, the majority of reviewers said the authors had overreached—the claims were not supported by the data. In other words, authors generalized their claims beyond those that could be justified.

The Connection Between Contributions and Generalizability

In our view, claims about contributions can be examined productively alongside considerations of generalizability. To make the case for this view, we need to back up a bit. Recall that the purpose of research is to understand a phenomenon. To understand a phenomenon, you need to determine the conditions under which it occurs. Consequently, productive hypotheses specify the conditions under which the predictions hold and explain why and how these conditions make a difference. And the conditions set the parameters on generalizability. They identify when, where, and for whom the effect or situation will occur. So, hypotheses describe the extent of expected generalizability, and revised hypotheses that contain the contributions recalibrate generalizability and offer new predictions within these parameters.

An Example That Illustrates the Connection

In Chap. 4, we introduced an example with a research question asking whether second graders improve their understanding of place value after a specially designed instructional intervention. We suggested asking a few second and third graders to complete your tasks to see if they generated the expected variation in performance. Suppose you completed this pilot study and now have satisfactory tasks. What conditions might influence the effect of the intervention? After careful study, you developed rationales that supported three conditions: the entry level of students’ understanding, the way in which the intervention is implemented, and the classroom norms that set expectations for students’ participation.

Suppose your original hypotheses predicted the desired effect of the intervention only if the students possessed an understanding of several concepts on which place value is built, only if the intervention was implemented with fidelity to the detailed instructional guidelines, and only if classroom norms encouraged students to participate in small-group work and whole-class discussions. Your claims of generalizability will apply to second-grade settings with these characteristics.

Now suppose you designed the study so the intervention occurred in five second-grade classrooms that agreed to participate. The pre-intervention assessment showed all students with the minimal level of entry understanding. The same well-trained teacher was employed to teach the intervention in all five classrooms, none of which included her own students. And you learned from prior observations and reports of the classroom teachers that three of the classrooms operated with the desired classroom norms, but two did not. Because of these conditions, your study is now designed to test one of your hypotheses—the desired effect will occur only if classroom norms encouraged students to participate in small-group work and whole-class discussions. This is the only condition that will vary; the other two (prior level of understanding and fidelity of implementation) are the same across classrooms so you will not learn how these affect the results.

Suppose the classrooms performed equally well on the post-intervention assessments. You expected lower performance in the two classrooms with less student participation, so you need to revise your hypotheses. The challenge is to explain the higher-than-expected performance of these students. Because you were interested in understanding the effects of this condition, you observed several lessons in all the classrooms during the intervention. You can now use this information to explain why the intervention worked equally well in classrooms with different norms.

Your revised hypothesis captures this part of your study’s contribution. You can now say more about the ways in which the intervention can help students improve their understanding of place value because you have different information about the role of classroom norms. This, in turn, allows you to specify more precisely the nature and extent of the generalizability of your findings. You now can generalize your findings to classrooms with different norms. However, because you did not learn more about the impact of students’ entry level understandings or of different kinds of implementation, the generalizability along these dimensions remains as limited as before.

This example is simplified. In many studies, the findings will be more complicated, and more conditions will likely be identified, some of which were anticipated and some of which emerged while conducting the study and analyzing the data. Nevertheless, the point is that generalizability should be tied to the conditions that are expected to affect the results. Further, unanticipated conditions almost always appear, so generalizations should be conservative and made with caution and humility. They are likely to change after testing the new predictions.

Contributions Are Assured When Hypotheses Are Significant and Methods Are Appropriate and Aligned

We have argued that the contributions of your study are produced by the revised hypotheses you can formulate based on your results. Will these revisions always represent contributions to the field? What if the revisions are minor? What if your results do not inform revisions to your hypotheses?

We will answer these questions briefly now and then develop them further in Part IV of this chapter. The answer to the primary question is “yes,” your revisions will always be a contribution to the field if (1) your hypotheses are significant and (2) you crafted appropriate methods to test the hypotheses. This is true even if your revisions are minor or if your data are not as informative as you expected. However, this is true only if you meet the two conditions in the earlier sentence. The first condition (significant hypotheses) can be satisfied by following the suggestions in the earlier section on significance. The second condition (appropriate methods) is addressed further in Part III in this chapter.

Implications

Before examining the role of methods in connecting significance with important contributions, we elaborate briefly our definition of “implications.” We reserve implications for the conclusions you can logically deduce from your findings that are not already presented as contributions. This means that, like contributions, implications are presented in the Discussion section of your research paper.

Many educational researchers present two types of implications: implications for future research and implications for practice. Although we are aware of this common usage, we believe our definition of “contributions” cover these implications. Clarifying why we call these “contributions” will explain why we largely reserve the word “implications” for recommendations regarding methods.

Implications for Future Research

Implications for future research often include (1) recommendations for empirical studies that would extend the findings of this study, (2) inferences about the usefulness of theoretical constructs, and (3) conclusions about the advisability of using particular kinds of methods. Given our earlier definitions, we prefer to label the first two types of implications as contributions.

Consider recommendations for empirical studies. After analyzing the data and presenting the results, we have suggested you compare the results with those predicted, revise the rationales for the original predictions to account for the results, and make new predictions based on the revised rationales. It is precisely these new predictions that can form the basis for recommending future research. Testing these new predictions is what would most productively extend this line of research. It can sometimes sound as if researchers are recommending future studies based on hunches about what research might yield useful findings. But researchers can do better than this. It would be more productive to base recommendations on a careful analysis of how the predictions of the original study could be sharpened and improved.

Now consider inferences about the usefulness of theoretical constructs. Our argument for labeling these inferences as contributions is similar. Rationales for predictions are where the relevant theoretical constructs are located. Revisions to these rationales based on the differences between the results and the predictions reveal the theoretical constructs that were affirmed to support accurate predictions and those that must be revised. In our view, usefulness is determined through this revision process.

Implications that do not come under our meaning of contributions are in the third type of implications, namely the appropriateness of methods for generating rich contributions. These kinds of implications are produced by your evaluation of your methods: research design, sampling procedures, tasks, data collection procedures, and data analyses. Although not always included in the discussion of findings, we believe it would be helpful for researchers to identify particular methods that were useful for conducting their study and those that should be modified or avoided. We believe these are appropriately called implications.

Implications for Practice

If the purpose of research is to better understand how to improve learning opportunities for all students, then it is appropriate to consider whether implications for improving educational practice can be drawn from the results of a study. How are these implications formulated? This is an important question because, in our view, these claims often come across as an afterthought, “Oh, I need to add some implications for practice.” But here is the sobering reality facing researchers: By any measure, the history of educational research shows that identifying these implications has had little positive effect on practice.

Perhaps the most challenging task for researchers who attempt to draw implications for practice is to interpret their findings for appropriate settings. A researcher who studied the instructional intervention for second graders on place value and found that average performance in the intervention classrooms improved more than in the textbook classrooms might be tempted to draw implications for practice. What should the researcher say? That second-grade teachers should adopt the intervention? Such an implication would be an overreach because, as we noted earlier, the findings cannot be generalized to all second-grade classrooms. Moreover, an improvement in average performance does not mean the intervention was better for all students.

The challenge is to identify the conditions under which the intervention would improve the learning opportunities for all students. Some of these conditions will be identified as the theoretical framework is built because the predictions need to account for these conditions. But some will be unforeseen, and some that are identified will not be informed by the findings. Recall that, in the study described earlier, a condition of entry level of understanding was hypothesized but the design of the study did not allow the researcher to draw any conclusions about its effect.

What can researchers say about implications for practice given the complexities involved in generalizing findings to other settings? We offer two recommendations. First, because it is difficult to specify all the conditions under which a phenomenon occurs, it is rarely appropriate to prescribe an educational practice. Researchers cannot anticipate the conditions under which individual teachers operate, conditions that often require adaptation of a suggested practice rather than implementation of a practice as prescribed.

Our second recommendation comes from returning to the purpose for educational research—to understand more fully how to improve learning opportunities for all students (or to achieve another goal of widely shared value). As we have described, understanding comes primarily from building and reevaluating rationales for your predictions. If you reach a new understanding related to improving learning opportunities, an understanding that could have practical implications, we recommend you share this understanding as an implication for practice.

For example, suppose the researcher who found better average performance of second graders after the intervention on place value had also studied several conditions under which performance improved. And suppose the researcher found that most students who did not improve their performance misunderstood a concept that appeared early in the intervention (e.g., the multiplicative relationship between positional values of a numeral). An implication for practice the researcher might share would be to describe the potential importance of understanding this concept early in the sequence of activities if teachers try out this intervention.

If you use our definitions, these implications for practice would be presented as contributions because they emerge directly from reevaluating and revising your rationales. We believe it is appropriate to use “Contributions” as the heading for this section in the Discussion section of your research paper. However, if editors prefer “Implications” we recommend following their suggestion.

We want to be clear that the terms you use for the different ways your study is important is not critical. We chose to define the terms significance, contributions, and implications in very specific and not universally shared ways to distinguish all the meanings of importance you should consider. Some of these can be established through your theoretical framework, some by the revisions of your hypotheses, and some by reflecting on the value of particular methods. The important thing, from our point of view, is that the ideas we defined for each of these terms are distinguished and recognized as specific ways of determining the importance of your study.

Part III. The Role of Methods in Determining Contributions

We have argued that every part of the study (and of the evolving research paper) should be aligned. All parts should be connected through a coherent chain of reasoning. In this chapter, we argue that the chain of reasoning is not complete until the methods are presented and the results are interpreted and discussed. The methods of the study create a bridge that connects the introductory material (research questions, theoretical framework, literature review, hypotheses) with the results and interpretations.

The role that methods play in scientific inquiry is to ensure that your hypotheses will be tested appropriately so the significance of your study will yield its potential contributions. To do this, the methods must do more than follow the standard guidelines and be technically correct (see Chap. 4). They must also fit with the surrounding parts of the study. We call this coherence.

Coherence Across the Phases of Scientific Inquiry

Coherence means the parts of a whole are fully aligned. When doing scientific inquiry, the early parts or phases should motivate the later phases. The methods you use should be motivated or explained by the earlier phases (e.g., research questions, theoretical framework, hypotheses). Your methods, in turn, should produce results that can be interpreted by comparing them with your predictions. Methods are aligned with earlier phases when you can use the rationales contained in your hypotheses to decide what kinds of data are needed to test your predictions, how best to gather these kinds of data, and what analyses should be performed (see Chap. 4 and Cai et al., 2019a).

For a visual representation of this coherence, see Fig. 5.1. Each box identifies an aspect of scientific inquiry. Hypotheses (shown in Box 1) include the rationales and predictions. Because the rationales encompass the theoretical framework and the literature review, Box 1 establishes the significance of the study. Box 2 represents the methods, which we defined in Chap. 4 as the entire set of procedures you will use, including the basic design, measures for collecting data, and analytic approaches. In Fig. 5.1, the hypothesis in Box 1 points you to the methods you will use. That is, you will choose methods that provide data for analyses that will generate results or findings (Box 3) that allow you to make comparisons against your predictions. Based on those comparisons, you will revise your hypotheses and derive the contributions and implications of your study (Box 4).

Fig. 5.1

The Chain of Coherence That Runs Through All Parts of a Research Study

We intend Fig. 5.1 to carry several messages. One is that coherence of a study and the associated research paper require all aspects of the study to flow from one into the other. Each set of prior entries must motivate and justify the next one. For example, the data and analyses you intend to gather and use in Box 2 (Methods) must be those that are motivated and explained by the research question and hypothesis (prediction and rationale) in Box 1.

A second message in the figure is that coherence includes Box 4, “Discussion.” Aligned with the first three boxes, the fourth box flows from these boxes but is also constrained by them. The contributions and implications authors describe in the Discussion section of the paper cannot go beyond what is allowed by the original hypotheses and the revisions to these hypotheses indicated by the findings.

For each hypothesis (and thus each research question ) in your study, you should be able to trace an entire chain of coherence . In a complex study with multiple hypotheses (and thus multiple research questions), it can be extremely helpful to diagram these connections (or make a table of them) so that you can explicitly link each research question to the data collected for that question, to the analyses that will be conducted on those data to address that question, to the results obtained for that question, and finally to the contributions related to that question. A diagram or table of these links can help you to maintain coherence both while conducting your study and while writing your research paper.

Methods Enable Significance to Yield Contributions

We begin this section by identifying a third message conveyed in Fig. 5.1. The methods of the study, represented by Box 2, provide a bridge that connects the significance of the study (Box 1) with the contributions of the study (Box 4). The results (Box 3) indicate the nature of the contributions by determining the revisions to the original hypotheses.

In our view, the connecting role played by the methods is often underappreciated. Crafting appropriate methods aligned with the significance of the study, on one hand, and the interpretations, on the other, can determine whether a study is judged to make a contribution.

If the hypotheses are established as significant, and if appropriate methods are used to test the predictions, the study will make important contributions even if the data are not statistically significant. We can say this another way. When researchers establish the significance of the hypotheses (i.e., convince readers they are of interest to the field) and use methods that provide a sound test of these hypotheses, the data they present will be of interest regardless of how they turn out. This is why Makel et al. (2021) endorse a review process for publication that emphasizes the significance of the study as presented in the first sections of a research paper.

Treating the methods as connecting the introductory arguments to the interpretations of data prevent researchers from making a common mistake: When writing the research paper, some researchers lose track of the research questions and/or the predictions. In other words, results are presented but are not interpreted as answers to the research questions or compared with the predictions. It is as if the introductory material of the paper begins one story, and the interpretations of results ends a different story. Lack of alignment makes it impossible to tell one coherent story.

A final point is that the alignment of a study cannot be evaluated and appreciated if the methods are not fully described. Methods must be described clearly and completely in the research paper so readers can see how they flow from the earlier phases of the study and how they yield the data presented. We suggested in Chap. 4 a rule of thumb for deciding whether the methods have been fully described: “Readers should be able to replicate the study if they wish.”

Part IV. Special Considerations that Affect a Study’s Contributions

We conclude Chap. 5 by addressing two additional issues that can affect how researchers interpret the results and make claims about the contributions of a study. Usually, researchers deal with these issues in the Discussion section of their research paper, but we believe it is useful to consider them as you plan and conduct your study. The issues can be posed as questions: How should I treat the limitations of my study? How should I deal with findings that are completely unexpected?

Limitations of a Study

We can identify two kinds of limitations: (1) limitations that constrain your ability to interpret your results because of unfortunate choices you made, and (2) limitations that constrain your ability to generalize your results because of missing variables you could not fit into the scope of your study or did not anticipate. We recommend different ways of dealing with these.

Limitations Due to Unfortunate Choices

All researchers make unfortunate choices. These are mistakes that could have been prevented. Often, they are choices in how a study was designed and/or executed. Maybe the sample did not have the characteristics assumed, or a task did not assess what was expected, or the intervention was not implemented as planned. Although many unfortunate choices can be prevented by thinking through the consequences of every decision or by conducting a well-designed pilot study or two, some will occur anyway. How should you deal with them?

The consequence of unfortunate choices is that the data do not test the hypotheses as precisely or completely as hoped. When this happens, the data must be interpreted with these constraints in mind. Almost always, this limits the researcher to making fewer or narrower claims than desired about differences and similarities between the results and the predictions. Usually this means conclusions about the ways in which the rationales must be revised require extra qualifications. In other words, claims about contributions of the study must be made with extra caution.

Research papers frequently include a subsection in the Discussion called “Limitations of the Study.” Researchers often use this subsection to identify the study’s limitations by describing the unfortunate choices, but they do not always spell out how these limitations should affect the contributions of the paper. Sometimes, it appears that researchers are simply checking off a requirement to identify the limitations by saying something like “The results should be interpreted with caution.” But this does not help readers understand exactly what cautions should be applied and it does not hold researchers accountable for the limitations.

We recommend something different. We suggest you do the hard work of figuring out how the data should be interpreted in light of the limitations and share these details with the readers. You might do this when the results are presented or when you interpret them. Rather than presenting your claims about the contributions of the study and then saying readers should interpret these with “caution” because of the study’s limitations, we suggest presenting only those interpretations and claims of contributions that can be made with the limitations in mind.

One way to think about the constraints you will likely need to impose on your interpretations is in terms of generalizability. Recall that earlier in this chapter, we described the close relationship between contributions and generalizability. When generalizability is restricted, so are contributions.

Limitations Due to Missing Variables

Because of the complexity of problems, questions, and hypotheses explored in educational research, researchers are unlikely to anticipate in their studies all the variables that affect the data and results. In addition, tradeoffs often must be made. Researchers cannot study everything at once, so decisions must be made about which variables to study carefully and which to either control or ignore.

In the earlier example of studying whether second graders improve their understanding of place value after a specially designed instructional intervention, the researcher identified three variables that were expected to influence the effect of the intervention: students’ entry level of understanding, implementation of the intervention, and norms of the classrooms in which the intervention was implemented. The researcher decided to control the implementation variable by hiring one experienced teacher to implement the intervention in all the classrooms. This meant the variable of individual teacher differences was not included in the study and the researcher could not generalize to classrooms with these differences.

Some researchers might see controlling the implementation of the intervention as a limitation. We do not. As a factor that is not allowed to vary, it constrains the generalizations a researcher can make, but we believe these kinds of controlled variables are better treated as opportunities for future research. Perhaps the researcher’s observations in the classroom provided information that could be used to make some predictions about which elements of the intervention are essential and which are optional—about which aspects of the intervention must be implemented as written and which can vary with different teachers. When revising the rationales to show what was learned in this study, the researcher could include rationales for new, tentative predictions about the effects of the intervention in classrooms where implementation differed in specified ways. These predictions create a genuine contribution of the study. If you use our definitions, these new predictions, often presented under “implications for future research,” would be presented as “contributions.”

Notice that if you follow our advice, you would not need to include a separate section in the Discussion of your paper labeled “Limitations.” We acknowledge, however, that some journal editors recommend such a subsection. In this case, we suggest you include this subsection along with treating the two different kinds of limitations as we recommend. You can do both.

Dealing with Unexpected Findings

Researchers are often faced with unexpected and perhaps surprising results, even when they have developed a convincing theoretical framework, posed research questions tightly connected to this framework, presented predictions about expected outcomes, and selected methods that appropriately test these predictions. Indeed, the unexpected findings can be the most interesting and valuable products of the study. They can range from mildly surprising to “Wow. I didn’t expect that.” How should researchers treat such findings? Our answer is based on two principles.

The first principle is that the value of research does not lie in whether the predictions are completely accurate but in helping the field learn more about the explanatory power of theoretical frameworks. That is, the value lies in the increased understanding of phenomena generated by examining the ability of theoretical frameworks (or rationales) to predict outcomes and explain results. The second principle, a corollary to the first, is to treat unexpected findings in a way that is most educative for the reader.

Based on our arguments to this point, you could guess we will say there will always be unexpected findings. Predicted answers to significant research questions in education will rarely, if ever, be entirely accurate. So, you can count on dealing with unexpected findings.

Consistent with the two principles above, your goal should be to use unexpected findings to understand more fully the phenomenon under investigation. We recommend one of three different paths. The choice of which path to take depends on what you decide after reflecting again on the decisions you made at each phase of the study.

The first path is appropriate when researchers reexamine their theoretical framework in light of the unexpected findings and decide that it is still a compelling framework based on previous work. They reason that readers are likely to have been convinced by this framework and would likely have made similar predictions. In this case, we believe that it is educative for researchers to (a) summarize their initial framework, (b) present the findings and distinguish those that were aligned with the predictions from those that were not, and (c) explain why the theoretical framework was inadequate and propose changes to the framework that would have created more alignment with the unexpected findings.

Revisions to initial hypotheses are especially useful if they include explanations for why a researcher might have been wrong (and researchers who ask significant questions in domains as complex as education are almost always wrong in some way). Depending on the ways in which the revised framework differs from the original, the authors have two options. If the revised framework is an expansion of the original, it would be appropriate for the authors to propose directions for future research that would extend this study. Alternatively, if the revised framework is still largely within the scope of the original study and consists of revisions to the original hypotheses, the revisions could guide a second study to check the adequacy of the revisions. This second study could be conducted by the same researchers (perhaps before the final manuscript is written and presented as two parts of the same report) or it could be proposed in the Discussion as a specific study that could be conducted by other researchers.

The second path is appropriate when researchers reexamine their theoretical framework in light of the unexpected findings and recognize serious flaws in the framework. The flaws could result from a number of factors, including defining elements of the framework in too general a way to formulate well-grounded hypotheses, failing to include a variable, or not accounting carefully enough for the previous work in this domain, both theoretical and empirical. In many of these cases, readers would not be well served by reading a poorly developed framework and then learning that the framework, which had not been convincing, did not accurately predict the results. Before scrapping the study and starting over, we suggest stepping back and reexamining the framework. Is it possible to develop a more coherent, complete, and convincing framework? Would this framework predict the results more accurately? If the findings remain unexpected based on the predictions generated by this revised, more compelling framework, then the first path applies.

It is likely that the new framework will better predict the findings. After all, the researchers now know the findings they will report. However, it is unlikely that the framework will accurately predict all the findings. This is because the framework is not built around the findings of this study of which authors are now aware (but have not yet been presented). Frameworks are built on research and theory already published. This means the redesigned framework is built from exactly the same empirical findings and theoretical arguments available before the study was conducted. The redesigned framework also is constrained by needing to justify exactly those methods used in the study. The redesigned framework cannot justify different methods or even slightly altered methods. The task for researchers is to show how the new theoretical framework necessarily generates, using the same methods, the predictions they present in the research paper. Just as before, it is unlikely this framework can account for all the findings. Just as before, after presenting the results the researchers should explain why they believe particular hypotheses were confirmed and why others should be revised, even in small ways, based on the findings reported. Researchers can now use these findings to revise the hypotheses presented in the paper. The point we are making is that we believe it is acceptable to reconstruct frameworks before writing research reports if doing so would be more educative for the reader.

Finally, the third path becomes appropriate when researchers, in reexamining their theoretical framework, trace the problem to a misalignment between the methods they used and the theoretical framework or the research questions. Perhaps the researchers recognize that the tasks they used did not yield data that could test the predictions and address the research questions. Or perhaps the researchers realize that the sample they selected would likely have been heavily influenced by a factor they failed to take into account. In other words, the researchers decide that the unexpected findings were due to a problem with the methods they used, not with the framework or the accompanying predictions. In this case, we recommend that the researchers correct the methodological problems and conduct the study again.

Part V. A Few Suggestions for Structuring Your Discussion Section

Writing the Discussion section of your research paper can be overwhelming given all our suggestions about what to include in this section. Here are a few tips that might help you create a simple template for this section.

We recommend the Discussion begin with a brief summary of the main results, especially those you will interpret in this section. This summary should not contain new data or results not previously presented in the paper.

The Discussion could then move to presenting the contributions in the ways we have described. To do this you could point out the ways in which the results differed from the predictions and suggest revisions to your rationales that would have better predicted the results. Doing this will show how the contributions of your study extend what is known beyond the research you drew on to build your original rationale. You can then propose how to extend your contributions to research by proposing future research studies that would test your new predictions. If you believe the revisions you make to your rationales produce new insights or understandings that could be helpful for educational practitioners, you can identify these contributions to practice as well. This comprises the bulk of the Discussion section.

If you have embedded the limitations in earlier sections of the paper, you will have presented your results and interpreted your findings constrained by these limitations. If you choose (or are asked) to describe limitations in the Discussion, you could identify the limitations and then point to the ways they affected your interpretations of the findings. Finally, the Discussion could conclude with the implications of the study for methodological choices that could improve research in the domain in which your study is located or how future studies could overcome the limitations you identified.

Because we are providing guidance on writing your research paper for publication, we will reiterate here that you should investigate the expectations and conventions of the journal to which you will submit your paper. Usually, it will be acceptable to use the terms “significance,” “contributions,” and “implications” as we have defined them. However, if the editors expect you to use the terms differently, follow the editors’ expectations. Our definitions in this chapter are meant to help you think clearly about the different ways you can make a case for the importance of your research. What matters is that you have carefully built and described a coherent chain of scientific inquiry that allows your study to translate the significance of your research problem into contributions to the field.

We began the chapter with the “So what?” question. The question looks simple and straightforward but is challenging and complicated. Its simple appearance can lead researchers to believe it should have a simple answer. But it almost never does. In this chapter, we tried to address the many complications that arise when answering the question. We hope you now have some new insights and new tools for answering the question in your next study.