Ethical Considerations of Conducting Systematic Reviews in Educational Research
Ethical considerations of conducting systematic reviews in educational research are not typically discussed explicitly. However, systematic reviews are frequently read and cited in documents that influence educational policy and practice. Hence, ethical issues associated with what and how systematic reviews are produced and used have serious implications. It becomes imperative for systematic reviewers to reflexively engage with a variety of ethical issues associated with potential conflicts of interest and issues of voice and representation. This chapter discusses how systematic reviewers can draw upon the philosophical traditions of consequentialism, deontology or virtue ethics to situate their ethical decision-making.
Ethical considerations of conducting systematic reviews in educational research are not typically discussed explicitly. As an illustration, ‘ethics’ is not listed as a term in the index of the second edition of ‘An Introduction to Systematic Reviews’ (Gough et al. 2017). This chapter draws from my earlier in-depth discussion of this topic in the Qualitative Research Journal (Suri 2008) along with more recent publications by colleagues in the field of research ethics and methods of research synthesis.
Unlike primary researchers, systematic reviewers do not collect deeply personal, sensitive or confidential information from participants. Systematic reviewers use publicly accessible documents as evidence and are seldom required to seek an institutional ethics approval before commencing a systematic review. Institutional Review Boards for ethical conduct of research do not typically include guidelines for systematic reviews. Nonetheless, in the past four decades systematic reviews have evolved to become more methodologically inclusive and play a powerful role in influencing policy, practice, further research and public perception. Hence, ethical considerations of how interests of different stakeholders are represented in a research review have become critical (Franklin 1999; Hammersley 2003; Harlen and Crick 2004; Popkewitz 1999).
Educational researchers often draw upon the philosophical traditions of consequentialism, deontology or virtue ethics to situate their ethical decision-making. Consequentialism or utilitarianism focuses on maximising benefit and minimising harm by undertaking a cost-benefit analysis of potential positive and negative impacts of research on all stakeholders. Deontology or universalism stems from Immanuel Kant’s logic that certain actions are inherently right or wrong and hence ends cannot justify the means. A deontological viewpoint is underpinned by rights-based theories that emphasise universal adherence to the principles of beneficence (do good), non-maleficence (prevent harm), justice, honesty and gratitude. While both consequentialism and deontology focus on actions and behaviour, virtue ethics focuses on being virtuous, especially in relationships with various stakeholders. There are several overlaps, as well as tensions, between and across these philosophical traditions (Brooks et al. 2014; Cohen et al. 2018).
Recognising the inherently situated nature of ethical decision-making, I am selectively eclectic in drawing from each of these traditions. I discuss a variety of ethical considerations of conducting systematic reviews informed by rights-based theories, ethics of care and Foucauldian ethics. Rights-based theories underpin deontology and consequentialism. Most regulatory research ethics guidelines, such as those offered by British Educational Research Association (BERA 2018) and American Educational Research Association are premised on rights-based theories that emphasises basic human rights, such as liberty, equality and dignity. Ethics of care prioritises attentiveness, responsibility, competence and responsiveness (Tronto 2005). Foucauldian ethics highlights the relationship of power and knowledge (Ball 2013).
Informed subjectivity and reflexivity
Purposefully informed selective inclusivity
identifying an appropriate epistemological orientation
identifying an appropriate purpose
searching for relevant literature
evaluating, interpreting and distilling evidence from selected reports
constructing connected understandings
communicating with an audience
To promote ethical production and use of systematic reviews through this chapter, I have used questioning as a strategic tool with the purpose of raising awareness about a variety of ethical considerations among systematic reviewers and their audience
1 Identifying an Appropriate Epistemological Orientation
What philosophical traditions are amenable for guiding ethical decision-making in systematic reviews positioned along distinct epistemologies?
Practising informed subjectivity and reflexivity, all systematic reviewers must identify an appropriate epistemological orientation, such as post-positivist, interpretive, participatory and/or critical, that is aligned with their review purpose and research competence (Suri 2013, 2018).
Deontological ethics is more relevant to post-positivist reviewers who focus on explaining, predicting or describing educational phenomena as generalisable laws expressed through relationships between measurable constructs and variables. The ethical focus of post-positivist systematic reviews tends to be on minimising threats to internal validity, external validity, internal reliability and external reliability of review findings. This is typically achieve by using a priori synthesis protocols, defining all key constructs conceptually and operationally in behavioural terms, employing exhaustive sampling strategies and employing variable oriented statistical analyses (Matt and Cook 2009; Petticrew and Roberts 2006).
Teleological ethics is more relevant to interpretive systematic reviews aiming to construct a holistic understanding of the educational phenomena that takes into account subjective experiences of diverse groups in varied contexts. Ethical decision making in interpretive systematic reviews lays an emphasis on authentically representing experiences and perceptions of diverse groups, especially those whose viewpoints tend to be less represented in the literature, to the extent that is permissible from the published literature. Maintaining a questioning gaze and a genuine engagement with diverse viewpoints, interpretive systematic reviewers focus on how individual accounts of a phenomenon reinforce, refute or augment each other (Eisenhart 1998; Noblit and Hare 1988).
Ethics of care is amenable to participatory systematic reviews that are designed to improve participant reviewers’ local world experientially through critical engagement with the relevant research. Ethical decision making in participatory systematic reviews promotes building teams of practitioners with the purpose of co-reviewing research that can transform their own practices and representations of their lived experiences. Participant co-reviewers exercise greater control throughout the review process to ensure that the review remains relevant to generating actionable knowledge for transforming their practice (Bassett and McGibbon 2013).
Foucauldian ethics is aligned with critical systematic reviews that contest dominant discourse by problematizing the prevalent metanarratives. Ethical decision making in critical systematic reviews focuses on problematizing ‘what we might take for granted’ (Schwandt 1998, p. 410) in a field of research by raising ‘important questions about how narratives get constructed, what they mean, how they regulate particular forms of moral and social experiences, and how they presuppose and embody particular epistemological and political views of the world’ (Aronowitz and Giroux 1991, pp. 80–81).
2 Identifying an Appropriate Purpose
What are key ethical considerations associated with identifying an appropriate purpose for a systematic review?
In this age of information explosion, systematic reviews require substantial resources. Guided by teleological ethics, systematic reviewers must conduct a cost-benefit analysis with a critical consideration of the purpose and scope of the review and its potential benefits to various groups of stakeholders.
If we consider the number of views or downloads as a proxy measure of impact, then we can gain useful insights by examining the teleological underpinnings of some of the highly read systematic reviews. Review of Educational Research (RER) tends to be regarded as the premiere educational research review journal internationally. Let us examine the scope and purpose of the three ‘most read’ articles in RER, as listed on 26 September 2018. Given the finite amount of resources available, an important question for educators is ‘what interventions are likely to be most effective, and under what circumstances?’. The power of feedback (Hattie and Timperley 2007), with 11463 views and downloads, is a conceptual analysis primarily drawing from the findings of published systematic reviews (largely meta-analyses) conducted to address this important question. In addition to effectively teaching what is deemed important, educators also have an important role of critiquing what is deemed important and why. The theory and practice of culturally relevant education: A synthesis of research across content areas (Aronson and Laughter 2016), with 8958 views and downloads, is an example of such a systematic review. After highlighting the positive outcomes of culturally relevant education, the authors problematise the validity of standardised testing as an unbiased form of a desirable educational outcome for all. As education is essentially a social phenomenon, understanding how different stakeholders perceive various configurations of an educational intervention is critical. Making sense of assessment feedback in higher education (Evans 2013), with 5372 views and downloads, is an example of a systematic review that follows such a pursuit. Even though each of these reviews required significant resources and expertise, the cost is justified by the benefits evident from the high number of views and downloads of these articles. Each of these three reviews makes clear recommendations for practitioners and researchers by providing an overview, as well as interrogating, current practices.
How does the agenda of the funding source intersect with the purpose of the review?
How might this potentially influence the review process and findings? How will this be managed ethically to ensure integrity of the systematic review findings?
In case of sponsored systematic reviews, it is important to consider at the outset how potential ethical issues will be managed if the interest of the funding agency conflicts with the interests of relatively less influential or less represented groups. Systematic reviews funded by a single agency with a vested interest in the findings are particularly vulnerable to ethical dilemmas arising from a conflict of interest (The Methods Coordinating Group of the Campbell Collaboration 2017). One approach could be to seek funding from a combination of agencies representing interests of different stakeholder groups. Exploring the option of crowdfunding is another option that systematic reviewers could use to represent the interests of marginalised groups whose interests are typically overlooked in the agenda of powerful funding agencies. In participatory synthesis, it is critical that the purpose of the systematic review evolves organically in response to the emerging needs of the practitioner participant reviewers.
3 Searching for Relevant Literature
What are key ethical considerations associated with developing an appropriate strategy for sampling and searching relevant primary research reports to include in a systematic review?
A number of researchers in education and health sciences have found that studies with certain methodological orientations or types of findings are more likely to be funded, published, cited and retrieved through common search channels (Petticrew and Roberts 2006). Serious ethical implications arise when systematic reviews of biased research are drawn upon to make policy decisions with an assumption that review findings are representative of the larger population. In designing an appropriate sampling and search strategy, systematic reviewers should carefully consider the impact of potential publication biases and search biases.
Funding bias, methodological bias, outcome bias and confirmatory bias are common forms of publication bias in educational research. For instance, studies with large sample-sizes are more likely to attract research funding, being submitted for publishing and getting published in reputable journals (Finfgeld-Connett and Johnson 2012). Research that reports significantly positive effects of an innovative intervention is more likely to be submitted for publishing by primary researchers and being accepted for publishing by journal editors (Dixon-Woods 2011; Rothstein et al. 2004). Rather than reporting on all the comparisons made in a study, often authors report on only those comparisons that are significant (Sutton 2009). As a result, the effectiveness of innovative educational interventions gets spuriously inflated in published literature. Often, when an educational intervention is piloted, additional resources are allocated for staff capacity building. However, in real life when the same intervention is rolled out at scale, the same degree of support is not provided to teachers whose practice is impacted by the intervention (Schoenfeld 2006).
Even after getting published, certain types of studies are more likely to be cited and retrieved through common search channels, such as key databases and professional networks (Petticrew and Roberts 2006). Systematic reviewers must carefully consider common forms of search biases, such as database bias, citation bias, availability bias, language bias, country bias, familiarity bias and multiple publication bias. The term ‘grey literature’ is sometimes used to refer to published and unpublished reports, such as government reports, that are not typically included in common research indexes and databases (Rothstein and Hopewell 2009). Several scholars recommend inclusion of grey literature to minimise potential impact of publication bias and search bias (Glass 2000) and to be inclusive of key policy documents and government reports (Godin et al. 2015). On the other hand, several other scholars argue that systematic reviewers should include only published research that has undergone the peer-review process of academic community to include only high-quality research and to minimise the potential impact of multiple publications based on the same dataset (La Paro and Pianta 2000).
With the ease of internet publishing and searching, the distinction between published and unpublished research has become blurred and the term grey literature has varied connotations. While most systematic reviews employ exhaustive sampling, in recent years there has been an increasing uptake of purposeful sampling in systematic reviews as evident from more than 1055 Google Scholar citations of a publication on this topic: Purposeful sampling in qualitative research synthesis (Suri 2011).
Aligned with the review’s epistemological and teleological positioning, all systematic reviewers must prudently design a sampling strategy and search plan, with complementary sources, that will give them access to most relevant primary research from a variety of high-quality sources that is inclusive of diverse viewpoints. They must ethically consider positioning of the research studies included in their sample in relation to the diverse contextual configurations and viewpoints commonly observed in practical settings.
4 Evaluating, Interpreting and Distilling Evidence from the Selected Research Reports
What are key ethical considerations associated with evaluating, interpreting and distilling evidence from the selected research reports in a systematic review?
Systematic reviewers typically do not have direct access to participants of primary research studies included in their review. The information they analyse is inevitably refracted through the subjective lens of authors of individual studies. It is important for systematic reviewers to critically reflect upon contextual position of the authors of primary research studies included in the review, their methodological and pedagogical orientations, assumptions they are making, and how they might have influenced the findings of the original studies. This becomes particularly important with global access to information where critical contextual information, that is common practice in a particular context but not necessarily in other contexts, may be taken-for-granted by the authors of the primary research report and hence may not get explicitly mentioned.
Systematic reviewers must ethically consider the quality and relevance of evidence reported in primary research reports with respect to the review purpose (Major and Savin-Baden 2010). In evaluating quality of evidence in individual reports, it is important to use the evaluation criteria that are commensurate with the epistemological positioning of the author of the study. Cook and Campbell’s (1979) constructs of internal validity, construct validity, external validity and statistical conclusion are amenable for evaluating postpositivist research. Valentine (2009) provides a comprehensive discussion of criteria suitable for evaluating research employing a wide range of postpositivist methods. Lincoln and Guba’s (1985) constructs of credibility, transferability, dependability and confirmability are suitable for evaluating interpretive research. The Centre for Reviews and Dissemination (CRD 2009) provides a useful comparison of common qualitative research appraisal tools in Chap. 6 of its open access guidelines for systematic reviews. Herons and Reason’s (1997) constructs of critical subjectivity, epistemic participation and political participation emphasising a congruence of experiential, presentational, propositional, and practical knowings are appropriate for evaluating participatory research studies. Validity of transgression, rather than correspondence, is suitable for evaluating critically oriented research reports using Lather’s constructs of ironic validity, paralogical validity, rhizomatic validity and voluptuous validity (Lather 1993). Rather than seeking perfect studies, systematic reviewers must ethically evaluate the extent to which findings reported in individual studies are grounded in the reported evidence.
While interpreting evidence from individual research reports, systematic reviewers should be cognisant of the quality criteria that are commensurate with the epistemological positioning of the original study. It is important to ethically reflect on plausible reasons for critical information that may be missing from individual reports and how might that influence the report findings (Dunkin 1996). Through purposefully informed selective inclusivity, systematic reviewers must distil information that is most relevant for addressing the synthesis purpose.
Often a two-stage approach is appropriate for evaluating, interpreting and distilling evidence from individual studies. For example, in their review that won the American Educational Research Association’s Review of the Year Award, Wideen et al. (1998) first evaluated individual studies using the criteria aligned with the methodological orientation of individual studies. Then, they distilled information that was most relevant for addressing their review purpose. In this phase, systematic reviewers must ethically pay particular attention to the quality criteria that are aligned with the overarching methodological orientation of their review, including some of the following criteria: reducing any potential biases, honouring representations of the participants of primary research studies, enriching praxis of participant reviewers or constructing a critically reflexive account of how certain discourses of an educational phenomenon have become more powerful than others. The overarching orientation and purpose of the systematic review should influence the extent to which evidence from individual primary research studies is drawn upon in a systematic review to shape the review findings (Major and Savin-Baden 2010; Suri 2018).
5 Constructing Connected Understandings
What are key ethical considerations associated with constructing connected understandings in a systematic review?
Through informed subjectivity and reflexivity, systematic reviewers must ethically consider how their own contextual positioning is influencing the connected understandings they are constructing from the distilled evidence. A variety of systematic techniques can be used to minimise unacknowledged biases, such as content analysis, statistical techniques, historical methods, visual displays, narrative methods, critical sensibilities and computer-based techniques. Common strategies for enhancing quality of all systematic reviews include ‘reflexivity; collaborative sense-making; eliciting feedback from key stakeholders; identifying disconfirming cases and exploring rival connections; sensitivity analyses and using multiple lenses’ (Suri 2014, p. 144).
In addition, systematic reviewers must pay specific attention to ethical considerations particularly relevant to their review’s epistemological orientation. For instance, all post-positivist systematic reviewers should be wary of the following types of common errors: unexplained selectivity, not discriminating between evidence of varying quality, inaccurate coding of contextual factors, overstating claims made in the review beyond what can be justified by the evidence reported in primary studies and not paying adequate attention to the findings that are at odds with the generalisations made in the review (Dunkin 1996). Interpretive systematic reviews should focus on ensuring authentic representation of the viewpoints of the participants of the original studies as expressed through the interpretive lens of the authors of those studies. Rather than aiming for generalisability of the findings, they should aim at transferability by focusing on how the findings of individual studies intersect with their methodological and contextual configurations. Ethical considerations in participatory systematic reviews should pay attention to the extent to which practitioner co-reviewers feel empowered to drive the agenda of the review to address their own questions, change their own practices through the learning afforded by participating in the experience of the synthesis and have practitioner voices heard through the review (Suri 2014). Critically oriented systematic reviews should highlight how certain representations silence or privilege some discourses over the others and how they intersect with the interests of various stakeholder groups (Baker 1999; Lather 1999; Livingston 1999).
6 Communicating with an Audience
What are key ethical considerations associated with communicating findings of a systematic review to diverse audiences?
All educational researchers are expected to adhere to the highest standards of quality and rigour (AERA 2011; BERA 2018). The PRISMA-P group have identified a list of ‘Preferred reporting items for systematic review and meta-analysis protocols’ (Moher et al. 2015) which are useful guidelines to improve the transparency of the process in systematic reviews. Like all educational researchers, systematic reviewers also have an obligation to disclose any sources of funding and potential conflicts of interest that could have influenced their findings.
All researchers should reflexively engage with issues that may impact on individuals participating in the research as well as the wider groups whose interests are intended to be addressed through their research (Greenwood 2016; Pullman and Wang 2001; Tolich and Fitzgerald 2006). Systematic reviewers should also critically consider the potential impact of the review findings on the participants of original studies and the wider groups whose practices or experiences are likely to be impacted by the review findings. They should carefully articulate the domain of applicability of a review to deter the extrapolation of the review findings beyond their intended use. Contextual configurations of typical primary research studies included in the review must be comprehensively and succinctly described in a way that contextual configurations missing from their sample of studies become visible.
Like primary researchers, systematic reviewers should reflexively engage with a variety of ethical issues associated that potential conflicts of interest and issues of voice and representation. Systematic reviews are frequently read and cited in documents that influence educational policy and practice. Hence, ethical issues associated with what and how systematic reviews are produced and used have serious implications. Systematic reviewers must pay careful attention to how perspectives of authors and research participants of original studies are represented in a way that makes the missing perspectives visible. Domain of applicability of systematic reviews should be scrutinised to deter unintended extrapolation of review findings to contexts where they are not applicable. This necessitates that they systematically reflect upon how various publication biases and search biases may influence the synthesis findings. Throughout the review process, they must remain reflexive about how their own subjective positioning is influencing, and being influenced, by the review findings. Purposefully informed selective inclusivity should guide critical decisions in the review process. In communicating the insights gained through the review, they must ensure audience-appropriate transparency to maximise an ethical impact of the review findings.
- Aronowitz, S., & Giroux, H. A. (1991). Postmodern education: Politics, culture, and social criticism. Minneapolis: University of Minnesota Press.Google Scholar
- BERA. (2018). British Educational Research Associations’ ethical guidelines for educational research. Retrieved 4 April 2018 from https://www.bera.ac.uk/wp-content/uploads/2018/06/BERA-Ethical-Guidelines-for-Educational-Research_4thEdn_2018.pdf?noredirect=1.
- Brooks, R., Kitty, t. R., & Maguire, M. (2014). Ethics and Education Research. London: Sage.Google Scholar
- Cohen, L., Manion, L., & Morrison, K. (2018). Research Methods in Education (8th ed.). Abingdon: Routledge.Google Scholar
- Cook, T. D., & Campbell, D. T. (1979). Quasi-experimentation: Design & analysis issues for field settings. Chicago: Rand McNally.Google Scholar
- CRD. (2009). Systematic reviews: CRD’s guidance for undertaking reviews in health care. Retrieved April 10, 2019 from https://www.york.ac.uk/media/crd/Systematic_Reviews.pdf.
- Franklin, B. M. (1999). Discourse, rationality and educational research: A historical perspective of RER. Review of Educational Research, 69(4), 347–363.Google Scholar
- Glass, G. V. (2000, January). Meta-analysis at 25. Retrieved April 10, 2019, from http://www.gvglass.info/papers/meta25.html.
- Godin, K., Stapleton, J., Kirkpatrick, S. I., Hanning, R. M., & Leatherdale, S. T. (2015). Applying systematic review search methods to the grey literature: A case study examining guidelines for school-based breakfast programs in Canada. Systematic Reviews, 4(1), 138. https://doi.org/10.1186/s13643-015-0125.CrossRefGoogle Scholar
- Gough, D., Oliver, S., & Thomas, J. (Eds.). (2017). An introduction to systematic reviews (2nd ed.). London: Sage.Google Scholar
- Hammersley, M. (2003). Systematic or unsystematic, is that the question? Some reflections on the science, art, and politics of reviewing research. Paper presented at the Department of Epidemiology and Public Health, University of Leicester.Google Scholar
- Lather, P. (1993). Fertile obsession: Validity after poststructuralism. The Sociological Quaterly, 34(4), 673–693.Google Scholar
- Major, C. H., & Savin-Baden, M. (2010). An introduction to qualitative research synthesis: Managing the information explosion in social science research. London: Routledge.Google Scholar
- Matt, G. E., & Cook, T. D. (2009). Threats to the validity of generalized inferences. In H. M. Cooper, L. V. Hedges & J. C. Valentine (Eds.), The handbook of research synthesis and meta-analysis (2nd ed., pp. 537–560). New York: Sage.Google Scholar
- Moher, D., Shamseer, L., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., . . . Stewart, L. (2015). Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Systematic Reviews, 4(1–9).Google Scholar
- Popkewitz, T. S. (1999). Reviewing reviews: RER, research and the politics of educational knowledge. Review of Educational Research, 69(4), 397–404.Google Scholar
- Rothstein, H. R., & Hopewell, S. (2009). Grey literature. In H. M. Cooper, L. V. Hedges & J. C. Valentine (Eds.), The handbook of research synthesis and meta-analysis (2nd ed., pp. 103–125). New York: Sage.Google Scholar
- Rothstein, H. R., College, B., Turner, H. M., & Lavenberg, J. G. (2004). The Campbell Collaboration information retrieval policy brief. Retrieved 2006, June 05, from http://www.campbellcollaboration.org/MG/IRMGPolicyBriefRevised.pdf.
- Suri, H. (2014). Towards methodologically inclusive research synthesis. UK: Routledge.Google Scholar
- Suri, H. (2018). ‘Meta-analysis, systematic reviews and research syntheses’ In L. Cohen, L. Manion & K. R. B. Morrison Research Methods in Education (8th ed., pp. 427–439). Abingdon: Routledge.Google Scholar
- Sutton, A. J. (2009). Publication bias. In H. M. Cooper, L. V. Hedges & J. C. Valentine (Eds.), The handbook of research synthesis and meta-analysis (2nd ed., pp. 435–452). New York: Sage.Google Scholar
- The Methods Coordinating Group of the Campbell Collaboration. (2017). Methodological expectations of Campbell Collaboration intervention reviews: Reporting standards. Retrieved April 21, 2019, from https://www.campbellcollaboration.org/library/campbell-methods-reporting-standards.html.
- Tolich, M., & Fitzgerald, M. (2006). If ethics committees were designed for ethnography. Journal of Empirical Research on Human Research Ethics. Journal of Empirical Research on Human Research Ethics, 12(2), 71–78.Google Scholar
- Tronto, J. C. (2005). An ethic of care. In A. E. Cudd & R. O. Andreasen (Eds.), Feminist theory: A philosophical anthology (pp. 251–263). Oxford, UK Malden, Massachusetts: Blackwell.Google Scholar
- Valentine, J. C. (2009). Judging the quality of primary research. In H. M. Cooper, L. V. Hedges & J. C. Valentine (Eds.), The handbook of research synthesis and meta-analysis (2nd ed., pp. 129–146). New York: Sage.Google Scholar
Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.
The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.