What is known

  • Many tools exist for assessing the quality of studies in systematic reviews of either quantitative or qualitative work.

  • There is a paucity of tools that assess the quality of studies within systematic reviews that include a diverse group of study designs, and mixed or multi-methods studies in particular.

  • The Quality Assessment Tool for Studies with Diverse Designs (QATSDD) published in 2012 was developed to assess the quality of studies with heterogenous designs primarily for use in the discipline of Psychology.

What this study adds

  • The Quality Assessment for Diverse Studies (QuADS) tool is a refined version of the QATSDD tool. The aim was to use survey and literature review data to enhance the applicability of the tool to health services research, and more specifically, to multi or mixed-methods research.

  • The QuADS tool demonstrates substantial inter-rater reliability and content and face validity.

Background

The inclusion of diverse types of evidence, such as qualitative and mixed or multi-methods research, is well-established in systematic reviews of health services research [1,2,3]. This is important because these methods can address the complexities within healthcare that cannot often be readily measured through a single method. Qualitative methods, when used alone, offer explanatory power to enhance understanding of multi-faceted and complex phenomena such as experiences of healthcare and systems [3]. When partnered with quantitative methods, qualitative data can support and add depth of understanding [4, 5].

The appraisal of the methodological quality, evidence quality and quality of reporting of individual studies and of studies included in a review collectively is firmly established for reviews of quantitative studies. There are more than 60 tools currently available to assess the quality of randomised controlled trials alone [6]. Appraisal of the quality of evidence is often used to assess bias, particularly in randomised controlled trials. More recently, quality appraisal tools have extended to tools for appraising qualitative research, with the emergence of multiple tools in this space [7] creating a topic of extended debate [7,8,9,10]. As a result, reviews that include both qualitative and quantitative research often utilise separate quality appraisal tools for the quantitative and qualitative studies within the review, often citing the lack of a standard, empirically grounded tools suitable to assess methodological quality, evidence quality and/or quality of reporting with a variety of study designs [11]. The use of a parallel approach to all aspects of quality appraisal offers strength in the ability to acknowledge the unique nature of qualitative research and its epistemological distinction from quantitative approaches. Yet, a dual approach does not facilitate the appraisal of methodological, evidence or reporting quality for mixed-methods research, and creates challenges in appraising these aspects of multi-methods work.

Thus, acknowledging that the underlying assumptions of quantitative and qualitative research are substantially different, a tool to appraise methodological quality, evidence quality and/or quality of reporting mixed- or multi-methods research is valuable in enabling researchers to consider the transparency and reporting of key elements of these approaches [12]. Moreover, a tool that is relevant to mixed- and multi-method approaches is significant in the context of growing recognition of the value of these methodologies in health systems and services research [4]. A single tool that can be used to evaluate methodological quality, evidence quality, and quality of reporting across a body of diverse evidence facilitates reviewers to reflect on the extent to which there is apparent transparency and congruency in the research purpose and its reporting and the implications for evidence quality. This is currently not available for mixed- and multi-methods work, with study heterogeneity as a key obstacle to evidence appraisal. Given the complexities of multiple individuals evaluating a diverse set of studies, a supporting tool may also provide an underpinning method to develop a shared understanding of what constitutes quality in research methods, evidence and reporting.

The authors published in 2012, a pragmatic approach to facilitate reviewers to appraise the methodological quality, evidence quality, and quality of reporting in reviews that included qualitative, quantitative, mixed- and multi-methods research using a single tool (QATSDD) [13]. The QATSDD has been cited more than 270 times to date and has been used in more than 80 reviews. The tool provides a framework for exploring the congruency, transparency and organised reporting of the research process for research grounded in post-positivistic or positivist methodology that informs multiple-methods or mixed-methods designs. The tool was not proposed as a basis for determining studies to be excluded from a review given that any cut-off points to indicate high or low quality would be arbitrary.

The QATSDD tool was originally developed for application in Psychology but has demonstrated wider relevance through its application in a broad range of health services research. Its wide use suggests that researchers value the ability to appraise quality of evidence from studies that employ or combine a range of methods. Yet the QATSDD tool has some limitations in its ease of use beyond the discipline of Psychology. We therefore aimed to revise, enhance and adapt the current QATSDD tool into an updated version; Quality Assessment for Diverse Studies (QuADS), for greater applicability to health services researchers appraising quality of methods, evidence and reporting in multi- and/or mixed-methods research.

Methods

Data sources and procedures

Studies citing the QATSDD tool were identified using Google Scholar, citations imported to the reference-management software (Endnote X9.2) and duplicates removed. Full-text screening of the identified studies and discussion between two authors (BJ and RH) was used to identify studies that included qualitative evaluative data or commentary regarding the QATSDD tool to inform its enhancement. The following data were extracted: first author, year of publication, country, research discipline, study synopsis, QATSDD reliability and validity data and qualitative evaluative comments about the use of the tool. Alongside the review of citing studies, all authors who had used the QATSDD in a published, publicly accessible paper (101 authors) were contacted to provide an opportunity for them to provide any additional feedback through a qualitative brief online survey form. Ethical approval to administer the survey form was granted from the UNSW Human Research Ethics Committee (HC190645). The survey form contained two open-ended, free-text response items: 1) ‘When applying the QATSDD in your research, what were the strengths of the tool and what did this enable you to achieve?’ and 2) ‘When applying the QATSDD in your research, what were the limitations or challenges you experienced and how could these be addressed in a revised version of the tool?’ The survey was administered by one author (BJ) to the email addresses of the study authors via the Qualtrics online survey software, with one follow-up reminder. Consent was implied through completion and submission of the survey form.

Data analysis and synthesis

A narrative synthesis [14] was then undertaken with the heterogeneous data emerging from the literature review in addition to the qualitative comments provided by the survey respondents. In the development of the primary synthesis, two authors (BJ, RH) independently undertook a line-by-line review of each study and survey content. The evaluative comments were labelled and merged into a table of the items arising. The authors then met to discuss the commonly occurring items and created initial themes. In a further stage, an exploration of the relationships in the data and an assessment of the robustness of the synthesis product was explored. The initial themes were discussed and refined with two further authors (RL, PH) into final themes, which were tabulated. The research team then collectively discussed areas for clarification and areas requiring changes to be made. An iterative process of making refinements to the tool drawing upon the synthesised data was undertaken through collaboration, review of the tool and discussion between the author team.

Preliminary internal assessment and external evaluation

Face and content validity were also explored through providing the revised QuADS tool to 10 researchers who had expertise in reviewing studies with diverse designs within systematic reviews. The researchers worked across different disciplines (psychology, sociology, health services research, pharmacy) and methodologies (quantitative, qualitative and mixed-methods) relevant to health in the UK or Australia. Each researcher was provided the tool via email and asked to 1) provide their feedback on the perceived suitability of the items within the tool to their own field and methods of research and 2) report any items that require clarification for ease of use or readability. Their feedback was discussed between the authors and used to revise the tool iteratively through a series of minor amendments to wording and ordering or the tool items. The resulting QuADS tool was also subject to inter-rater reliability analysis between a psychologist, public health and health services researcher through application to 40 studies a recent systematic review with a kappa of 0.65 published by a colleague within our department who was external to this study [15].

Results

Results of the review

One hundred and ninety-seven citations were attributed to Sirriyeh et al’s (2012) [13] Reviewing studies with diverse designs: the development and evaluation of a new tool article and 31 of these studies met the inclusion criteria by including evaluative data or comments (Table 1). The study selection process is shown in Fig. 1. Of the 101 authors who had cited the QATSDD paper and reported using the tool in their publication; 13 did not receive emails, 10 had moved institutions or were on leave,, 74 did not provide any additional feedback and 1 researcher replied stating they had not been the individual that had used the tool. Three respondents provided survey feedback which were synthesised with and aligned the findings from the reviewed articles.

Table 1 Summary of Included Studies
Full size table
Fig. 1
figure 1

Flow chart of study search and selection process

Full size image

Excluded studies

Reasons for exclusion of studies were that 97 had cited the paper but made no further comments, 38 had cited the tool and provided rationale for its selection as the preferred tool but did not produce reliability and validity data, and 21 produced reliability and validity data that confirmed consistently the tool was reliable and valid across multiple contexts but made no qualitative comments. A further two papers were incorrectly attributed to the article on Google Scholar and eight were non-English papers.

Findings regarding the QATSDD tool

The synthesis revealed a number of perceived areas of strength of the tool including its strong reliability and validity [33, 55, 66]. All of the reviews within the 39 included articles that used the tool confirmed its reliability and validity. Further strength were the ability of QuADS to be applied when appraising diverse study designs [66, 95, 96], and its comprehensive list of indicators [97]. The breadth of disciplines in which this tool had been applied was notable: psychology, medicine, health sciences, allied health, and health services. The final group of included studies reflected the range of disciplines in which the tool had been applied. Authors who had employed the tool commented that it was valued for its inclusion of a wider range of important issues relating to research quality such as the involvement of end users in the research design and process, facilitating a comprehensive analysis [31, 95] Further, the synthesis also revealed opportunities to clarify and improve the tool, with one study [12] that conducted a substantial commentary piece on the QATSDD tool and its applications. Five key areas in which there were opportunities for enhancements or further clarification emerged. A number of revisions were made as part of the present study to the tool in order to address the findings from this study and described in relation to each of the findings below.

Scope and purpose of the tool

Further clarification on the scope of the use of the tool appeared to be necessary to distinguish its focus predominantly on reviews of mixed- or multi-method studies but also its purpose in providing an approach to assess the transparency and quality of study reporting. Instances in which the tool had been applied to exclude studies from a review were noted and this appeared to be due to the lack of detail regarding the purpose of the tool available to reviewers. Lack of clarity regarding the method for scoring using the QATSDD was apparent, with queries including whether weighting was required for particular criteria and the need for a cut-off to delineate high and low quality studies [12, 55, 91, 98]. Such queries indicate that the purpose of the tool to stimulate discussion regarding the quality and transparency of reporting in relation to each study may not be clear. There is no evidence to suggest that any criterion is more important than another or that a particular score is indicative of high or low quality; therefore, any cut-off would be arbitrary. The tool enables researchers to consider and discuss each element of the study in the context of its research aims and to explore the extent to which each quality criterion is met. This may then stimulate discussion of its relative importance in the context of their own review. A summary of the purpose of the tool and its scope is included in a new ‘User guide’ (supplementary file 1) that accompanies the tool.

Examples for each criterion

The desire for more examples to be used as part of the tool’s criteria was highlighted by Fenton et al. [12] and Lamore et al. [66]. These papers provided commentary that the use of more explicit examples from both a quantitative and qualitative perspective would assist users when scoring. These authors found the tool challenging when examples did not match the methods used in the papers they were reviewing and highlighted an opportunity to be more inclusive of a wider range of possible research methodologies when providing examples. Furthermore, the inclusion of additional examples may address challenges of distinguishing between scores. Limiting the responses to a dichotomous scale or 3-point scoring system was suggested in one commentary but a dichotomous scale does not provide sufficient response options for many items that are more complex than a yes or no, and three-point scales are recognised as leading to the overuse of neutral responses [12].

Theoretical and conceptual framework

A common challenge identified was in applying the notion of a ‘theoretical framework,’ particularly outside the discipline of Psychology [12, 91]. Fenton et al. [12] highlighted the need for additional guidance regarding the a definition of a theoretical framework and specifically, whether the inclusion of reference to theoretical concepts or assumptions was relevant to this criterion. It was notable from the included reviews that few studies scored highly on this criterion, providing a further indicator that this may require review. To resolve this, the criterion ‘Theoretical framework’ was revised to ‘Explicit consideration of theories or concepts that frame the study in the introduction,’ with relevant exemplars.

Quantitative bias, appropriate sampling and analytic methods

Fenton et al. indicated that the tool held a quantitative bias [12], suggesting that the wording and selection of examples may favour quantitative studies. Clausen et al. [33] also suggested that qualitative studies performed poorly using the tool. Criteria related to appropriate sampling and analytic methods appeared to be challenging to assess and it was decided to update these in the light of current perspectives on qualitative methodology, particularly regarding matters such as the need (or lack of) for data saturation. Explicit examples and language were added to each descriptor to balance recognition of both qualitative and quantitative research. Criteria concerning sample sizes was revised and reduced to ‘Appropriate sampling to address the research aim/s.’

Discussion

Quality appraisal is both a widely-debated and dynamic area with emerging opportunities but also increasing demands [98]. The findings of this research show that the QATSDD tool was utilised in a wide variety of health fields including psychology, allied health, medicine, public health, nursing, health services and social sciences, and that the tool demonstrated high reliability. Nevertheless, a range of minor limitations regarding the scope of use of the tool, balance between qualitative and quantitative ontologies and ease of use through examples also came to light. In the context of increasing mixed and multi-methods research in health services, this paper has described the development of the QuADS tool which is an augmentation of the QATSDD, and aims to be one of few pragmatic tools that will enable quality assessment across a diverse range of study designs [99]. QuADS provides a basis for research teams to reflect on methodological and evidence quality, in addition to establish limitations in the quality of reporting of studies. There is complementary scope for application of QuADS with other tools that focused on appraising the methodological quality to provide an expanded analysis where needed [100].

Increasing recognition of the value of employing mixed methods approaches in health services research to address complex healthcare questions is reflected in more than 10 quality assessment methods for mixed-methods work [101]. Such approaches have focused to the justification for and application of mixed methods in the study, considering approaches to study design and data synthesis. Current methods to explore quality in mixed-methods studies may not readily apply in the context of multi-methods work or a collection of heterogenous studies in a systematic review [101]. Given the multitude of quantitative or qualitative quality appraisal tools, a segregated approach is often taken to explore quality when reviews include heterogeneous studies which limits researcher ability to comment on the body of evidence collectively.

Four tools, including the QATSDD, have been developed to date to enable an integrated quality assessment [13, 102,103,104]. Two of the available tools provide a segregated analysis of the qualitative and quantitative elements of research studies rather than a single set of items applicable to explore both [102, 104]. The remaining tool provides a method to explore completeness of reporting of studies with mixed or multiple methods [103]. In the context of existing tools, the QuADS enables a brief, integrated assessment to be undertaken across a body of evidence within a review.

Limitations

This manuscript reports the first stage in revising a pragmatic tool that can be used to help guide reporting of research and to make assessments of the quality of non-trial based mixed- and multi-methods studies. Methodological, evidence and reporting quality are three important areas and each complex in their own right. Addressing all of these elements with a single tool is valuable for stimulating discussion and reflection between reviewers but provides a high-level analysis of these different quality domains. Ultimately the tool does not therefore provide a conclusive outcome regarding the quality of the research that can be used to make decisions regarding the inclusion or exclusion of studies from a review. Despite the inclusion of a wide range of literature utilising the QATSDD tool, the response rate of authors in the survey component of this work was very low which may have shaped the information provided. This project benefited from drawing upon the insights of those who had utilised the tool to shape the design of the revised tool, yet it is possible that those experienced difficulty in using the QATSDD tool ultimately did not include the tool in their outputs and were not readily identifiable for inclusion in this project. As a result, we may not have identified all of the areas for refinement required. Whilst the study panel were all experienced in reporting studies with diverse designs in multiple locations internationally, the panel process did not constitute a formal Delphi approach required to register QuADS tool in the Equator Network as a reporting guideline. This further process is an important further subsequent step that we seek to complete to improve the rigour and evidence base for the new tool.

Conclusion

Quality appraisal continues to be a critical component of systematic review. Increasing recognition of the value of multi- and mixed methods research to address complex health services research questions requires a tool such as QuADS, demonstrating good reliability and which allows researchers to appraise heterogenous studies in systematic review.