Advertisement

Systematic Reviews

, 8:38 | Cite as

Measured and perceived effects of audit and feedback on nursing performance: a mixed methods systematic review protocol

  • Émilie DufourEmail author
  • Arnaud Duhoux
  • Jolianne Bolduc
Open Access
Protocol

Abstract

Background

The use of audit and feedback (A&F) interventions in health care has been demonstrated to generally be effective on medical teams. However, literature suggests that the response of nurses to this type of intervention may differ from that of other types of health professionals, in relation to their roles, power, and to the configuration of nursing care activities. To our knowledge, no review has been conducted on A&F interventions with nurses. The objective of this systematic review is to examine the evidence of measured and perceived effects of A&F interventions on nurses’ performance.

Methods

A mixed methods systematic review design with thematic and narrative synthesis is used. Studies reporting quantitative and qualitative data on the effects of A&F interventions specific to nursing care are considered for inclusion. Studies will be appraised for quality using the Mixed Methods Appraisal Tool. Quantitative and qualitative data will be summarized in narrative and tabular form and will be synthetized using a segregated methodologies approach.

Discussion

Results will describe the characteristics of A&F with nurses, as well as the measured and perceived effects specific to nursing care. The associations between the characteristics and the effects as well as the concordance between measured and perceived effects will be presented. We anticipate that combining the evidence from qualitative and quantitative studies will allow us to provide relevant insight that can inform the design of better suited A&F interventions for nurses. Audit and feedback interventions demonstrate potential for improving the performance of nursing care. As their effectiveness varies greatly depending on the context and the professionals involved, a better understanding of the associations between its characteristics and the measured and perceived effects is valuable for improving the effectiveness of A&F.

Systematic review registration

PROSPERO CRD42018104973

Keywords

Nursing audit Systematic review Quality improvement 

Abbreviations

A&F

Audit and feedback

GRADE

Grading of Recommendations Assessment, Development and Evaluation

MMAT

Mixed methods appraisal tool

PRISMA

Preferred Reporting Items for Systematic Reviews and Meta-Analyses

Background

Scientific literature on quality improvement in healthcare highlights the effectiveness of audit and feedback (A&F) [1, 2, 3, 4, 5]. A&F is an intervention comprising of two components: (1) evaluation of the performance (audit) and (2) feedback to the professionals. Most of the literature pertaining to A&F involves physicians [1, 5, 6, 7]. However, some authors suggest that the response of nurses to this type of intervention may differ from that of other types of health professionals [8] in relation to their roles, power, and the configuration of nursing care activities [9, 10, 11]. Since nurses often work in teams, quality improvement involves changes both at individual and collective levels. Collective improvements, requiring communication, cohesion, and coordination within the nursing team [11, 12], make interventions with this group of professionals particularly challenging.

A&F should focus on elements of practice over which professionals perceive to have control and to be accountable for [1, 4], which has been suggested to be a current limitation in A&F interventions involving nurses [9, 10, 11]. Although many studies report the development and use of nursing-sensitive indicators, the access to measurable process indicators is currently lacking [13, 14, 15, 16]. The use of process indicators, as they are more actionable and proximal to nursing care activities, could result in higher improvement rates of nursing performance [16]. The nature of the indicators reported to nurses could therefore influence the overall effectiveness of A&F. In order to design A&F interventions that are suitable for nursing activities, it is relevant to gather the current state of evidence on A&F targeting nurses.

To our knowledge, no systematic review has been conducted on A&F interventions with nurses. Mixed methods systematic reviews are a method of synthesizing knowledge which aims to combine several types of data concerning the same subject in order to present an integrated synthesis of the results produced through different methodologies [17]. A better understanding of how A&F unfolds in nursing practice settings could allow us to implement design elements according to the care settings and professionals involved. Given the paucity of literature on the effectiveness of A&F interventions with nurses [5] and the difficulty of quantitatively measuring nursing-specific effects [14, 18], it is relevant to study both the measured and perceived effects of this type of intervention. Performance, in the current paper, is addressed in a comprehensive way [19] and refers to dimensions of quality, effectiveness, and security.

Characteristics of A&F interventions

Although the efficacy of A&F interventions has been well demonstrated, it is well acknowledged that it has large variations depending on certain characteristics [1, 20, 21]. Some authors have identified A&F characteristics which are more likely to modify its delivery and would therefore explain those variations [1, 2]. These characteristics relate to the form, content, and frequency of interventions [2]. Until now, no consensus has been established to provide best practices for developing A&F interventions. The effectiveness of A&F may also be influenced by, among other things, the type of professional, the context of the intervention, and levels of baseline performance [5].

Measured effects

The effects of A&F interventions are usually assessed using quantitative benchmarks in the form of indicators, which monitor the action around the dimensions of performance [5]. As such, the structure-process-outcome triad developed by Donabedian [22] is a quality of care assessment framework designed to include all dimensions of performance of a health organization. To Donabedian [22], the structure component refers to all the elements relating to the framework in which care is provided, including both elements related to environment and those related to staff characteristics. The process component encompasses all the acts performed by the professional in their care delivery activities as well as the way these activities are performed. The outcome component refers not only to the effects of care on the health status of populations [22], but also to organizational outcomes, including the volume of care provided by a health facility [23]. In this review, this triad will be used to extract and analyze data from quantitative studies.

Perceived effects

Quantitative evaluation of these three categories of indicators requires access to data [24]. In nursing, access to data that are both valid and reliable that allows nursing indicators’ measurement is an issue [18, 25, 26, 27]. At present, the measurable effects of A&F interventions on nursing performance have some limitations, such as variable access to reliable nursing-specific indicators [14, 15, 16]. Hence, it seems highly relevant to review the perceived effects associated with this type of intervention, which are generally evaluated through qualitative studies. Given the small amount of literature related to nurse-specific effects of AF [5], the inclusion of perceived effects is intended to provide a better understanding of how this type of intervention unfolds with nurses influences the perception of their performance. Murray et al. [28] proposed a framework for describing and evaluating the effects of implementing a complex intervention, such as A&F, within a health organization from the perspective of the professionals involved. We believe this framework provides a global and relevant perspective to better understand the actions of the professionals involved in the implementation of a complex intervention. According to this approach, the effects perceived by professionals when implementing an intervention in their practice relate to four areas. The first area, that of coherence, relates to the meaning and value that participants place on intervention [28]. The second area, that of cognitive participation, refers to the commitment and motivation that the implementation of the intervention entails for participants. The third area, that of collective action, refers to the actions that intervention requires in participants’ practice changes, for example, in terms of time and training. The last area, the reflexive domain, refers to the perceived effects of changes in practice, for example, in improving the care provided [28]. The assessment of these domains makes it possible to consider the context of implementation of the intervention and its effects on the response of the participants [28]. This framework will be used in this review to ensure the systematic nature of the data extraction and analysis from qualitative studies.

Objective and research questions

The main objective of this review is to identify and appraise the existing evidence on the effects of A&F on nursing performance. It aims to answer the following questions:
  1. 1-

    What are the main features of A&F with nurses?

     
  2. 2-

    What are the measured effects of A&F on nurses’ performance?

     
  3. 3-

    What are the perceived effects of A&F on nurses’ performance?

     
  4. 4-

    What are the associations between the features of A&F and the measured and perceived effects?

     
  5. 5-

    What is the concordance between the measured and perceived effects of A&F with nurses?

     

Methods

Considering mixed methods reviews’ approaches are still emerging, we will conduct the data extraction, analysis, and synthesis using a various range of guidelines [17, 29, 30, 31, 32, 33]. The specific contributions and applications of these guidelines are presented in each section of the method. The Preferred Reporting Items for Systematic Reviews and Meta-Analyzes (PRISMA) grid [34] will be used as a reference for the preparation and writing of this review. This mixed methods systematic review is registered with the International Prospective Register of Systematic Reviews (PROSPERO) CRD42018104973.

Outcomes

The primary outcome will include any measured or perceived effect of the AF intervention on nursing-related dimensions of performance, such as quality and security of care and effectiveness. A measured effect is defined as any variable that is reported objectively and assessed through quantitative methods. A perceived effect is defined as any variable that is subjectively reported and assessed through qualitative methods.

Eligibility criteria

Criteria have been developed to specifically address the research questions. Thus, studies that meet the following criteria will been included: (1) A&F intervention corresponding to a feedback of the performance to nurses over a given period of time [1], (2) focusing on an A&F intervention with nurses or health professionals including nurses and with specific effects on nurses, (3) describing characteristics of the A&F intervention, and (4) presenting an A&F intervention related to at least one effect related to a dimension of performance. The literature that will meet the following criteria will be excluded: (1) providing feedback to nursing students and (2) focusing on an A&F intervention targeting patients.

Information sources

The strategy will be carried out in three stages. First, an initial search will be performed from CINAHL and MEDLINE to test the identified keywords and descriptors [17]. Then, a second search will be carried out based on the keywords and descriptors identified in all the selected databases. Then, the list of references of identified studies will be analyzed to identify additional articles. Authors recognized as experts in the field will be contacted by email to obtain any additional relevant study related to the research topic [35]. Online periodicals relevant to the area of interest will be identified and searched for additional and non-indexed articles in the databases [35]. Publications in English and French will be considered with no time restriction. The following databases will be used: CINAHL, Cochrane Controlled Register of Trials, Embase, MEDLINE, PubMed, Scopus, Web of Science, PsycINFO, and Global Health. The ProQuest database will be considered for tracking unpublished studies. Quantitative studies will include randomized controlled trials, nonrandomized studies, and descriptive studies [36]. Qualitative studies will include phenomenological studies, grounded theorization, ethnography, action research, case studies, and descriptive qualitative research [36]. Mixed methods studies, if any, will be considered.

Search strategy

The following concepts will be sought: “feed back” OR “action planning” OR feedback OR audit OR audits AND nurs* AND performance OR quality OR effectiveness OR security OR efficacy. The following descriptors will be used: (MH “Feedback”) OR (MH “Audit”) OR (MH “Nursing Audit”) OR (MH “Clinical Audit”) AND (MH “Nursing Role”) OR (MH “Nurses +”) OR (MH “Nursing Care +”) AND (MH “Quality of Health Care”) OR (MH “Quality Improvement”) OR (MH “Clinical Effectiveness”) (see Additional file 1). Search equivalents terms in French will be sought.

Data screening and extraction process

The lead review author (ED) will upload the search results and remove duplicates using EndNoteX8 (Clarivate Analytics). Search results will then be imported into Covidence (https://www.covidence.org), an online systematic review software, to conduct screening, data extraction and quality assessment. Two independent reviewers (ED and JB) will then screen titles and abstracts and classify studies (1) possibly meeting the eligibility criteria, (2) not meeting the eligibility criteria. Studies possibly meeting the inclusion criteria will then be obtained in full-text format. Two review authors will independently complete the full-text screening process (ED and JB). Any disagreement will be discussed and resolved with a third reviewer (AD). A PRISMA flow-chart diagram [34] will be used to report study selection process. Data extraction will be performed by two independent reviewers (ED and JB). For all studies, a grid will be used to extract data on the design, the geographic location, the context, the participants, and the objective of the included studies.

For all studies, a table inspired by the questioning formula what, why, who, when, and how that Colquhoun et al. [2] used to classify the characteristics of AF interventions with medical teams will serve the same purpose in our review. Since the object of interest constitutes a heterogeneous field of research [5], we chose to proceed with the steps of extraction, analysis, and presentation of results using reference frameworks previously selected by the research team to ensure a systematic process. Evidence from quantitative studies will be classified in a systematic way following the Donabedian’s quality of care triad [22]. Evidence from qualitative studies will be classified in a table inspired by Murray et al.’s framework [28]. This framework will be used to identify themes and classify perceived effects. Data on measured and perceived effects will be extracted from the results and discussion sections of the studies. Data will include statistical analyses, verbatim quotations, and narrative descriptive summaries of results. The extraction will draw on the method proposed by Sandelowski et al. [31] in order to preserve the methodological context of the findings.

Quality assessment of included studies

The quality assessment of included studies will be carried out using the tool Mixed Methods Appraisal Tool (MMAT) [37]. This tool was developed specifically to assess the quality of items with a quantitative, qualitative, or mixed methods design [37]. The quantitative studies will be first categorized according to their characteristics and then assessed according to their respective characteristics, including the sampling strategy, the measuring instruments, and the response rate. For qualitative studies, the assessment criteria will include context, data sources, and data analysis. The evaluation criteria for the mixed methods studies will include the integration of the methods and the limitations presented. Two reviewers (ED and JB), for each of the criteria, will independently assign a score based on “yes,” “no,” “unspecified,” or “not applicable” responses. Any disagreement between the reviewers will be resolved by discussion or by requesting the assessment of a third reviewer (AD).

Data analysis

Given the expected heterogeneity of quantitative data, the evidence on measured effects will be reported in a narrative form following a thematic analysis [38] based on Donabedian’s quality of care triad [22]. Qualitative data will be analyzed through the areas described in Murray et al. [28] framework. Data analysis of evidence from quantitative and qualitative studies will be conducted following the three-step narrative approach proposed by Popay et al. [33]. This approach is suited for inclusion of a wide range of research designs that generate both quantitative and qualitative findings on the effects of an intervention [33]. This approach encompasses (1) developing a preliminary synthesis, (2) exploring relationships in the data, and (3) assessing the robustness of the synthesis product [33]. A primary synthesis describing the patterns of direction and size of the effects will be undertaken from the extraction stage and considering the context and characteristics of the included studies [33]. This first synthesis will allow for an in-depth exploration of relationships between contexts, processes, and outcomes [33]. Patterns across studies will be identified by the reviewers in order to synthesis how characteristics and contexts influence the effects of A&F. Following this in-depth synthesis, its robustness will be assessed by considering the methodological quality of the included studies based on the MMAT [36].

Data synthesis

Based on the segregated methodologies proposed by The Joanna Briggs Institute [17, 30] and based on Sandelowski et al. [29], an individual synthesis of quantitative and qualitative data will be performed. In this approach, a clear distinction between quantitative and qualitative evidence is required prior to the mixed method synthesis [17]. This first individual synthesis will be conducted following the approach proposed by Popay et al. [33] as previously described.

A mixed method synthesis including all the generated findings will be then be conducted in order to study (1) the association between the A&F features and the measured or perceived effects and (2) the concordance between the measured effects and the perceived effects of A&F interventions with nurses. The combined findings will either support, contradict, or add to the quantitative and qualitative evidence [17]. This mixed method synthesis will follow a configuration approach [32, 39]. Using a configuration will allow to make connections between the evidence from both the quantitative and qualitative individual syntheses without merging them [17]. In this review, quantitative data are used to inform measured effects while qualitative data are used to inform perceived effects. Considering both types of effects have different implications for nursing performance, the findings will be used to complement rather than confirm each other [32]. Figure 1 presents the data analysis plan for this mixed method systematic review.
Fig. 1

Data analysis plan. Inspired by the segregated analysis approach of The Joanna Briggs Institute [17]

Assessing confidence in the evidence

An assessment of the confidence in the evidence will be conducted from the recommendations made by The Joanna Briggs Institute [40, 41, 42]. These recommendations have been formulated in a complementary approach to that of the Grading of Recommendations Assessment, Development and Evaluation (GRADE) tool [42] and are based on the assessment of the coherence of the evidence according to the study specifications [42]. Given the heterogeneity of approaches to evaluating the effects of A&F interventions [5] as well as the inclusion of qualitative studies, this approach is favored in order to make appropriate recommendations. The levels of evidence proposed by The Joanna Briggs Institute [42], in a similar approach to that of GRADE, include the feasibility, relevance, effectiveness, and meaningfulness of the intervention under study. The latter relates more particularly to the evidence assessment for qualitative studies [42].

Discussion

The primary objective of this mixed methods systematic review is to gather evidence on the measured and perceived effects of A&F interventions on nursing performance. Although systematic reviews on measured effects of A&F have been conducted on professionals’ practice and health outcomes [5], no review has focused on gathering effects specific to nursing practice. Furthermore, no review has aimed to assess evidence from qualitative as well as quantitative studies. We anticipate that combining these findings to qualitative studies will be highly relevant in providing insight on nurses’ perceptions on A&F and therefore to provide recommendations that can improve its design. This review will be valuable to health professionals interested in designing better suited A&F interventions for nurses. The strengths and the limits of the review, as well as recommendations for the implementation of A&F in nursing care will be discussed. One of the main limitations of this review is the heterogeneity in the literature regarding A&F in nursing. We anticipate A&F studies will take place in many different health settings, as well as in various forms and on different topics. We address this potential limitation by using frameworks specific to each type of data. The heterogeneity of the evidence will also be considered for the recommendations made following the evidence assessment.

Notes

Acknowledgements

We would like to thank Assia Mourid, librarian at the paramedical library of the University of Montreal, for the support in developing the search strategy.

Funding

This review did not receive any specific funding from financing of the public, commercial or non-profit sector.

Availability of data and materials

Not applicable.

Authors’ contributions

ED designed the review and the article under the supervision of AD. ED wrote the article. ED, AD, and JB have revised the content of the article. All authors have read and approved the manuscript.

Authors’ information

Not applicable.

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary material

13643_2019_956_MOESM1_ESM.docx (24 kb)
Additional file 1: Search strategy for CINAHL. (DOCX 24 kb)

References

  1. 1.
    Brehaut JC, Colquhoun HL, Eva KW, Carroll K, Sales A, Michie S, Ivers N, Grimshaw JM. Practice feedback interventions: 15 suggestions for optimizing effectiveness. Ann Intern Med. 2016;164(6):435–41.CrossRefGoogle Scholar
  2. 2.
    Colquhoun H, Michie S, Sales A, Ivers N, Grimshaw JM, Carroll K, Chalifoux M, Eva K, Brehaut J. Reporting and design elements of audit and feedback interventions: a secondary review. BMJ Quality & Safety. 2016;26(1):54–60.Google Scholar
  3. 3.
    Ivers N, Grimshaw J. Reducing research waste with implementation laboratories. Lancet. 2016;388(10044):547–8.CrossRefGoogle Scholar
  4. 4.
    Ivers NM, Sales A, Colquhoun H, Michie S, Foy R, Francis JJ, Grimshaw JM. No more ‘business as usual’ with audit and feedback interventions: towards an agenda for a reinvigorated intervention. Implement Sci. 2014;9(1):14.CrossRefGoogle Scholar
  5. 5.
    Ivers N, Jamtvedt G, Flottorp S, Young JM, Odgaard-Jensen J, French SD, O'Brien MA, Johansen M, Grimshaw J, Oxman AD. Audit and feedback: effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev. 2012;6:CD000259.Google Scholar
  6. 6.
    Larson EL, Patel SJ, Evans D, Saiman L. Feedback as a strategy to change behaviour: the devil is in the details. J Eval Clin Pract. 2013;19(2):230–4.CrossRefGoogle Scholar
  7. 7.
    Lu CY, Vinci LM, Quinn MT, Wilkes AE, Chin MH, Peek ME. Using feedback to change primary care physician behavior. J Ambul Care Manage. 2015;38(2):118–24.CrossRefGoogle Scholar
  8. 8.
    Giesbers APM, Schouteten RLJ, Poutsma E, van der Heijden BIJM, et van Achterberg T. Feedback provision, nurses’ well-being and quality improvement: towards a conceptual framework. J Nurs Manag. 2015;23(5):682–691.  https://doi.org/10.1111/jonm.12196.
  9. 9.
    Cheater FM, Baker R, Reddish S, Spiers N, Wailoo A, Gillies C, Cawood C. Cluster randomized controlled trial of the effectiveness of audit and feedback and educational outreach on improving nursing practice and patient outcomes. Med Care. 2006;44(6):542–51. Google Scholar
  10. 10.
    Leeman J, Baernholdt M, Sandelowski M. Developing a theory-based taxonomy of methods for implementing change in practice. J Adv Nurs. 2007;58(2):191–200.CrossRefGoogle Scholar
  11. 11.
    Holleman G, Poot E, Mintjes-de Groot J, van Achterberg T. The relevance of team characteristics and team directed strategies in the implementation of nursing innovations: a literature review. Int J Nurs Stud. 2009;46(9):1256–64.CrossRefGoogle Scholar
  12. 12.
    Rahn DJ. Transformational teamwork: exploring the impact of nursing teamwork on nurse-sensitive quality indicators. J Nurs Care Qual. 2016;31(3):262–8.CrossRefGoogle Scholar
  13. 13.
    Dubois CA, D'Amour D, Brault I, Dallaire C, Dery J, Duhoux, A, Zufferey A. Which priority indicators to use to evaluate nursing care performance? A discussion paper. J Adv Nurs. 2017;73(12):3154–67.  https://doi.org/10.1111/jan.13373.
  14. 14.
    Heslop L, Lu S. Nursing-sensitive indicators: a concept analysis. J Adv Nurs. 2014;70(11):2469–82.CrossRefGoogle Scholar
  15. 15.
    VanFosson CA, Jones TL, Yoder LH. Unfinished nursing care: an important performance measure for nursing care systems. Nurs Outlook. 2016;64(2):124–36.CrossRefGoogle Scholar
  16. 16.
    Xiao S, Widger K, Tourangeau A, Berta W. Nursing process health care indicators: a scoping review of development methods. J Nurs Care Qual. 2017;32(1):32–9.CrossRefGoogle Scholar
  17. 17.
    Joanna Briggs Institute. The Joanna Briggs Institute Reviewers’ Manual 2014. In: Methodology for JBI Mixed Methods Systematic Reviews. Adelaide: The Joanna Briggs Institute; 2014.Google Scholar
  18. 18.
    Dubois C-A, D’Amour D, Brault I, Dallaire C, Déry J, Duhoux A, Lavoie-Tremblay M, Mathieu L, Karemere H, Zufferey A. Indicateurs prioritaires pour évaluer la contribution infirmière à la qualité des soins : revue systématique des écrits. Montréal: Secrétariat international des infirmières et infirmiers de l’espace francophone; 2015.Google Scholar
  19. 19.
    Sicotte C, Champagne F, Contandriopoulos AP, Barnsley J, Beland F, Leggat SG, Denis JL, Bilodeau H, Langley A, Bremond M, et al. A conceptual framework for the analysis of health care organizations’ performance. Health Serv Manag Res. 1998;11(1):24–41.CrossRefGoogle Scholar
  20. 20.
    Colquhoun HL, Carroll K, Eva KW, Grimshaw JM, Ivers N, Michie S, Sales A, Brehaut JC. Advancing the literature on designing audit and feedback interventions: identifying theory-informed hypotheses. Implement Sci. 2017;12(1):117.CrossRefGoogle Scholar
  21. 21.
    Foy R, Eccles MP, Jamtvedt G, Young J, Grimshaw JM, Baker R. What do we know about how to do audit and feedback? Pitfalls in applying evidence from a systematic review. BMC Health Serv Res. 2005;5:50.CrossRefGoogle Scholar
  22. 22.
    Donabedian A. The quality of care: how can it be assessed? J Am Med Assoc. 1988;260(12):1743–8.CrossRefGoogle Scholar
  23. 23.
    Mant J. Process versus outcome indicators in the assessment of quality of health care. Int J Qual Health Care. 2001;13(6):475–80.CrossRefGoogle Scholar
  24. 24.
    Mainz J. Defining and classifying clinical indicators for quality improvement. Int J Qual Health Care. 2003;15(6):523–30.CrossRefGoogle Scholar
  25. 25.
    Kurtzman ET, Dawson EM, Johnson JE. The current state of nursing performance measurement, public reporting, and value-based purchasing. Policy, Politics, & Nursing Practice. 2008;9(3):181–91.CrossRefGoogle Scholar
  26. 26.
    Needleman J, Kurtzman ET, Kizer KW. Performance measurement of nursing care. Med Care Res Rev. 2007;64(2_suppl):10S–43S.CrossRefGoogle Scholar
  27. 27.
    Dufour É, Duhoux A, Contandriopoulos D. Reliability of a Canadian database for primary care nursing services’ clinical and administrative data. Int J Med Inform. 2018;117:1–5.CrossRefGoogle Scholar
  28. 28.
    Murray E, Treweek S, Pope C, MacFarlane A, Ballini L, Dowrick C, Finch T, Kennedy A, Mair F, O’Donnell C, et al. Normalisation process theory: a framework for developing, evaluating and implementing complex interventions. BMC Med. 2010;8:63.CrossRefGoogle Scholar
  29. 29.
    Sandelowski M, Voils CI, Barroso J. Defining and designing mixed research synthesis studies. Res Sch. 2006;13(1):29–40.PubMedPubMedCentralGoogle Scholar
  30. 30.
    Pearson A, White H, Bath-Hextall F, Salmond S, Apostolo J, Kirkpatrick P. A mixed-methods approach to systematic reviews. Int J Evid-Based Healthc. 2015;13(3):121–31.CrossRefGoogle Scholar
  31. 31.
    Sandelowski M, Leeman J, Knafl K, Crandell JL. Text-in-context: a method for extracting findings in mixed-methods mixed research synthesis studies. J Adv Nurs. 2013;69(6):1428–37.CrossRefGoogle Scholar
  32. 32.
    Voils CI, Sandelowski M, Barroso J, Hasselblad V. Making sense of qualitative and quantitative findings in mixed research synthesis studies. Field methods. 2008;20(1):3–25.CrossRefGoogle Scholar
  33. 33.
    Popay J, Roberts H, Sowden A, Petticrew M, Arai L, Rodgers M, Britten N, Roen K, Duffy S. Guidance on the conduct of narrative synthesis in systematic reviews. A product from the ESRC methods program. Lancaster University: Lancaster Institute of Health Research; 2006.Google Scholar
  34. 34.
    Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. 2009;6(7):e1000097.CrossRefGoogle Scholar
  35. 35.
    Cooper H. Research synthesis and meta-analysis. A step-by-step approach. 5th ed. Thousand Oaks: Sage Publications; 2017.Google Scholar
  36. 36.
    Pluye P, Hong QN. Combining the power of stories and the power of numbers: mixed methods research and mixed studies reviews. Annu Rev Public Health. 2014;35:29–45.CrossRefGoogle Scholar
  37. 37.
    Pluye P, Robert E, Cargo M, Bartlett G, O’Cathain A, Griffiths F, Boardman F, Gagnon MP, Rousseau MC. Proposal: a mixed methods appraisal tool for systematic mixed studies reviews. Montreal: Department of Family Medicine, McGill University; 2011.Google Scholar
  38. 38.
    Joanna Briggs Institute. Joanna Briggs Institute Reviewers’ Manual: 2014 edition. Adelaide: The Joanna Briggs Institute; 2014.Google Scholar
  39. 39.
    Dixon-Woods M, Agarwal S, Jones D, Young B, Sutton A. Synthesising qualitative and quantitative evidence: a review of possible methods. J Health Serv Res Policy. 2005;10(1):45–53.CrossRefGoogle Scholar
  40. 40.
    Joanna Briggs Institute. New JBI grades of recommendation. Adelaide: The Joanna Briggs Institute; 2014.Google Scholar
  41. 41.
    Joanna Briggs Institute. New JBI levels of evidence. Adelaide: The Joanna Briggs Institute; 2014.Google Scholar
  42. 42.
    Joanna Briggs Institute. Supporting document for the Joanna Briggs Institute levels of evidence and grades of recommendation. Adelaide: The Joanna Briggs Institute; 2014.Google Scholar

Copyright information

© The Author(s). 2019

Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Authors and Affiliations

  1. 1.Faculty of NursingUniversité de MontréalMontréalCanada
  2. 2.CR-CSIS (Centre de recherche Charles-Le Moyne - Saguenay-Lac-Saint-Jean sur les innovations en santé) Université de SherbrookeLongueuilCanada
  3. 3.ESPUM (École de santé publique de l’Université de Montréal)Université de MontréalMontréalCanada

Personalised recommendations