Development of a critical appraisal tool for assessing the reporting quality of qualitative studies: a worked example

Abstract

Conducting a critical appraisal is considered as one of the essential steps in any type of qualitative synthesis. While various appraisal tools are available in the literature, the reliability and validity of these tools tend to be limited due to inconsistencies in their conceptualizations and operationalizations. In addition, limited practical guidelines are available for researchers to develop and use these appraisal form for their own synthesis studies. Our goal for this study is to use and modify components of existing tools and work to address the concerns that have been laid out in existing research. Specifically, we set out to (a) keep detailed notes of our development of the appraisal tool over time, (b) attend to the inter-rater reliability and content validity of the appraisal tool, (c) provide a guidance on how to use the tool, and (d) produce an appraisal tool that could be used by all levels of researchers. In this paper, we explain the process we undertook to establish a reliable and valid critical appraisal tool for qualitative syntheses, share the developed 22-question critical appraisal form, and provide an example of how to use the results of a critical appraisal in a sample qualitative synthesis. The developed tool will help focus qualitative synthesis researchers’ attention on the transparency and completeness of the information reported in primary studies and thereby increase the credibility of a qualitative synthesis.

Appendix

Critical appraisal tool and a coded example of the report by Pinsk et al. (2014).

Criteria	Question(s)	Code*	Evidence from the paper (Direct quote) and Reviewer’s notes
Purpose of study Description of the main purpose(s)/ aim(s)/objective(s) of the study and /or research question(s) guided research	Did author(s) provide research questions? If not, is there a purpose statement to guide the investigation?	Yes	“To what degree did the use of student-generated video discussion posts create social presence for those students?” (p. 268)
Research design Description of what and how qualitative research deign was used and justified	Is qualitative research used for addressing the research purpose? (This may include considering the qualitative component of a mixed methods study)	Yes	“A case study methodology was adopted for this project. “(p. 268)
	Did author(s) specify their selected qualitative method? (e.g., case study, phenomenology, ethnography, grounded theory, action research, narrative inquiry, and other)	Yes	“A case study methodology was adopted for this project. “(p. 268) References supported for the methodological selection was provided
	Did author(s) provide an explanation of why the selected qualitative method was chosen?	Yes	“when the researcher has clearly identifiable cases within established boundaries and seeks to provide a greater in-depth understanding of a phenomenon, and allows for the emergence of sometimes subtle themes that are consistent throughout the experience” (p. 268)
	Does the provided research question(s) include qualitative language (e.g., explore, describe, understand…) and avoid quantitative language (e.g., relate, influence, effect, cause…) ?	No	Reviewers agreed the questions were not written for the purpose of the study, i.e., “The study attempted to discover the influence that this type of action [lf-created video discussion posts for a class] had on establishing social presence in online courses” (p. 267). Reviewers agreed that “to what degree” does not suit for the case study
Sampling Description of setting and what, how and why participants/ documents/events are selected	Did author(s) describe the setting/context for data collection?	Yes	Online course (pp. 267–268) Described about online course learning environment, Blackboard and Kaltura-brand video tools and course settings. But the authors did not describe about the contexts beyond online course
	Did author(s) provide description about the participants? (e.g., Administrators; Secondary science teachers; 4-6th grade math students; History textbooks before 2010; etc.)	Yes	Section “Participants” (pp. 269- 270) Students in class. The authors gave detailed description (age, background, level of computer literacy) for each participant
	Did author(s) explain why the selected participants/documents/events were chosen?	Yes	"After these sessions, seven agreed to participate. Of the seven, five actually produced at least 3 video discussion responses and were interviewed for this study" (p. 269)
	Did author(s) describe the recruitment process? (If so, was there any discussion on criteria for exclusion and inclusion?)	Yes	“The five students that participated in this study were selected based on a convenience sample.” (p. 269)
	Did the author specify the sample size?	Yes
	5 (p. 269)
Data collection Description of how data collection carried out and justification of the procedure	Did author(s) specify their data collection method(s)? (e.g., interviews, observation)	Yes	60–90 min. semi-structured telephone interviews with cases (Data collection section, p. 270) “data collected by the university on ‘student engagement’ was also reviewed” (Data collection section, p. 270)
	Did author(s) provide an explanation of why the selected data collection methods were chosen?	No	No description is available to indicate why the interview method was chosen. However, it indicated why telephone interviews were conducted For student engagement data, it says “Case studies often also include the use of archival data or physical artifacts in addition to personal interviews.” (Data collection section p. 270)
	Did author(s) provide any description of the data collection procedure? (e.g., If interviews were used, are there details on how they were conducted)	Yes	“Interviews ranged from 60 to 90 min and were conducted via telephone, since the participants were part of an online class and were geographically dispersed.” (Data collection section p. 270)
Data analysis Description of analytic procedure to reach findings	Did author(s) specify the data analysis method(s) they used? (e.g., constant comparison, thematic analysis, cross-case analysis…)	Yes	Within-case and cross-case analysis “After the interviews were transcribed, they were initially reviewed. This within-case analysis provided initial familiarity with the data. The data was then coded for themes.” (Data analysis section. p. 270) “This necessitated the use of the second coder. Then, cross-case analysis was used to facilitate insights beyond preliminary impressions. Themes began to emerge as the data was coded and analyzed. Enumeration and categorization of the data was also performed at this stage.” (Data analysis section. p. 270)
	Did author(s) provide justification or explanation of why they chose the data analysis method(s)?	No	No description is available
	Did author(s) provide description of data analysis procedure? (e.g., If thematic analysis is used, is it clear how the categories/themes were derived from the data?)	No	Themes, categorization and enumeration were listed. However, the authors did not provide the detailed process of identification of themes, and they only provided the general description
Findings Description of “what” author(s) found in the study. These findings will serve as data in qualitative synthesis	Did the authors' reported results address the research questions?	Yes	“Three critical components emerged as important elements among the participants regarding the establishment of social presence within the online experience in this class: course engagement, projection of self, and connection to other members of the class and the instructor.” (p. 270) The authors also identified three sub-factors: time considerations, technical issues, and interest in being innovative, and these sub-factors are the same for all components
	Did author(s) provide verbatim evidence representing the participant’s voices? (e.g., direct quotations or transcript excerpts)	Yes	Included keywords, quotes (pp. 270–272)
Value of the research Description of scholarly contribution to the relevant filed and an implication for both practice and research	Did the author(s) describe implications of the study findings?	Yes	"This study demonstrates that there may be some significant potential in using video for online discussions as a means to establishing social presence for students.” (Future Research section, p. 273) “Further research in this area can establish appropriate parameters for the use of this technology.” (Future research section. p. 273)
	Did author(s) state the contribution to the field? Or did author(s) state how the study fits into the existing literature?	No	In the literature review, it says “most research on this topic has been with traditional college students” (p. 268). However, the authors do not clarify whether a goal of the study is to fill the gap in the literature Last sentence in the limitations of the study simply says, “the study does contribute to the overall body of knowledge in the field of online education.”
Trustworthiness and reliability Descriptions for checking trustworthiness/reliability may include but are not limited to inter-rater reliability, member checking, researcher bias, role of researchers, piloting, data triangulation, etc	Did the author(s) discuss methods used to enhance the quality of data collection instruments? (e.g., piloting interview protocols, getting expert feedback on survey questions…)	Yes	"The protocol was reviewed by two subject matter experts in order to establish the validity of the questions used in the process" (Data collection section, p. 270)
	Did author(s) describe methods used to enhance the reliability of their data analysis? (e.g., inter-rater reliability, member checking, clarifying researcher bias, data triangulation, etc	Yes	"In order to assure maximum reliability in the data, each interview was transcribed by two different researchers and then the transcripts were compared to assure accuracy of the data." (Data collection section, p. 270)

1. *Indicate the information is provided (Yes, No, Cannot tell).