Should researchers make causal inferences and recommendations for practice on the basis of nonexperimental studies?

Educators, teachers, and policy makers take actions and make decisions that significantly impact the life of pupils and students. Thus, these actions and decisions should be in line with the practical implications of research findings (e.g., Meehl, 1997). To foster the knowledge transfer from research to practice, researchers can make recommendations for practice in their publications. Recommendations for practice have become increasingly common in educational psychology articles in recent decades according to a review by Brady et al. (2023). At the same time, the proportion of experimental studies has decreased in recent decades. This led Brady et al. to warn against under-supported recommendations for practice. I agree with most of what is said by Brady et al. For example, I agree with them that recommendations for practice should ideally be made on the basis of experimental studies rather than studies with other research designs. However, researchers who read their article might get the impression that evidence from experimental studies is the only acceptable basis for practice recommendations. Thus, researchers might take away that they should refrain from informing educators, teachers, and policy makers about potential practical implications of nonexperimental findings. Without recommendations from researchers, it might be challenging for practitioners to appropriately incorporate nonexperimental findings into their actions and decisions, potentially leading to ill-founded actions and decisions. In the current commentary, I attempt to complement the review by Brady et al. by facilitating an understanding of when and how educational research can inform educators, teachers, and policy makers.

Are only experimental studies causally informative?

Brady et al. (2023) state “Typically, RFP [recommendations for practice] require rigorous randomized controlled trials in most scientific fields” (p. 4). I agree that randomized control trials can be very informative about cause-effect relationships and recommendations for practice. Nonetheless, I think it is important to clarify that cause-effect relations can not only be investigated with experimental designs but also with other designs (e.g., Diener et al., 2022; Grosz et al., 2020). Observational studies that exploit natural experiments are particularly suitable for investigating cause and effect relationships (e.g., Dunning, 2012; Grosz et al., 2023). For example, a regression discontinuity design might be used to unbiasedly estimate the causal effect of class size on student achievement (e.g., Angrist & Lavy, 1999). And this finding can inform recommendations for the optimal class size in schools. Even observational studies that do not exploit natural experiments can under certain assumptions be used to investigate cause-effect relationships, for example, when the observed covariances are compared to the covariances that are implied by a structural equation model (e.g., Bollen & Pearl, 2013; Dumas & Edelsbrunner, 2023). A misfit between observed and model-implied covariances would inform the researcher that not all cause-effect relationships are correctly specified in the model. A fit between observed and model-implied covariances would, in and by itself, not confirm the causal effects proposed by the structural equation model because there are usually several models with identical fit and these models can propose strongly discrepant causal effects (e.g., Tomarken & Waller, 2003). Similarly, observed correlations are often in line with many potential causal models (e.g., Meehl, 1990). Yet, there are strategies to minimize the plausibility of alternative causal explanations in nonexperimental studies (e.g., Brick & Bailey, 2020; Shadish et al., 2002). Thus, observational studies can provide valuable insights about causal effects, even if the degree of informativeness of their findings depends on the details of the study design and analysis.

Relatedly, Brady et al. (2023) argued that “misinterpretation and misuse of research is due to unsupported claims researchers make when they go beyond their data in making recommendations for practice” (p. 2). Here, I think it is important to clarify that we cannot directly observe causal effects and thus causal inference always goes beyond the data, even in a randomized experiment (e.g., Kant, 1781/2002; Waldmann et al., 2006). In a randomized experiment, the experimental manipulation might, for example, not only affect the treatment variable of interest but also other variables and these other variables (not the treatment variable of interest) might have caused the effect (e.g., Fiedler et al., 2021; Shadish et al., 2002). Imagine, for example, a study examining the effect of a new reading program on students' reading comprehension. The experimental group received the new program, while the control group continued with the traditional program. The teachers of the experimental group might have more accurately followed the instructions of the program than the teachers of the traditional program because the teachers in the experimental group were trained more recently. Thus, how accurate the instructions were adhered to rather than differences in the contents and methods of the two programs might have caused differences in reading comprehension across experimental and control groups.

Still, experimental studies will usually be more causally informative than nonexperimental studies. So should researchers make causal inference and recommendations for practice only on the basis of experimental evidence? I think educational researchers would duck the responsibility if they would not talk about causal effects and recommendations for practice in nonexperimental studies. Practitioners can usually not postpone decisions and actions and wait for studies that are more informative. And for some research questions, studies that provide robust information about causal effects are unethical or unfeasible. Thus, if nonexperimental researchers would not suggest and discuss causal interpretations and practical implications, practitioners would need to interpret the reported nonexperimental findings themselves or, worse, base their decisions and actions on the recollection of anecdotal evidence and personal experiences, which are less trustworthy than nonexperimental evidence (e.g., Hilbert, 2012; Meehl, 1997). Thus, I think it is desirable that researchers (who know the most about the strengths and limitations of their research) explicitly discuss the potential causal effects underlying their findings even if their design does not allow them to draw strong causal claims. Of course, the researchers should be transparent about relevant limitations and assumptions, discuss them critically, and take them into account when contemplating causal effects and implications for practice. Nevertheless, I think it should be the researcher’s responsibility (not the practitioners) to discuss causal effects and practical implications because the authors of a study are usually better equipped to properly interpret their findings than practitioners.

Can only causally informative research inform recommendations for practice?

The article by Brady et al. (2023) might elicit not only the impression that experimental evidence is necessary to draw causal inference but also that only causally informative studies can inform recommendations for practice. I agree that information about causal effects is very helpful for educators, teachers, and policy makers because knowledge about cause-effect relations (but not descriptive and predictive findings) can be used to purposefully intervene and effectively change things for the better. However, descriptive and predictive findings can inform recommendations for practice as well. For example, a study might find that bullying is more common in classes with pupils from racially and ethnically diverse backgrounds than in other classes. Based on this descriptive finding, researchers might recommend that classes with pupils from racially and ethnically diverse backgrounds should receive anti-bullying interventions while other classes might not necessarily need anti-bullying interventions. Although information about cause-effect relationships is required to find or develop effective anti-bullying interventions, the descriptive finding that bullying is more common in some classes than others has implications for practice in the sense that it identifies targets for the intervention. Thus, researchers can draw practical implications not only from causal informative findings but also from descriptive and predictive findings.

Conclusion

Taken together, I agree with Brady et al. (2023) that randomized experiments are more informative about cause-effect relationships and recommendations for practice than nonexperimental studies. Nonetheless, I believe it is important to recognize that both experimental and nonexperimental designs can to some degree inform us about cause-effect relationships and that even studies that hardly inform us about causal effects can have practical implications. Thus, in order to enhance the transfer from research to practice, I would encourage educational researchers to talk about implications and recommendations for practice in their articles regardless of the design and analysis they used. At the same time, researchers should clearly and transparently communicate the limitations and assumptions of their research and how they affect the practical implications. This transparency needs to be rewarded by editors and reviewers or else researchers are incentivized to overstate implications and hide away weaknesses. Finally, future reviews of trends in educational psychology might want to code not only whether studies provide recommendations for practice or not but also the type and strength of the recommendations for practice and the degree to which limitations, assumptions, and caveats were appropriately voiced. This would enable us to properly evaluate whether the recommendations for practice in publications align with the informativeness of the evidence and ultimately to ensure that educators, teachers, and policy makers are provided with adequate information to make actions and decisions in line with scientific evidence.