Abstract
Risk of bias tools is important in identifying inherent methodical flaws and for generating evidence in studies involving systematic reviews (SRs) and meta-analyses (MAs), hence the need for sensitive and study-specific tools. This study aimed to review quality assessment (QA) tools used in SRs and MAs involving real-world data. Electronic databases involving PubMed, Allied and Complementary Medicine Database, Cumulated Index to Nursing and Allied Health Literature, and MEDLINE were searched for SRs and MAs involving real-world data. Search was delimited to articles published in English, and between inception to 20 of November 2022 following the SRs and MAs extension for scoping checklist. Sixteen articles on real-world data published between 2016 and 2021 that reported their methodological quality met the inclusion criteria. Seven of these articles were observational studies, while the others were of interventional type. Overall, 16 QA tools were identified. Except one, all the QA tools employed in SRs and MAs involving real-world data are generic, and only three of these were validated. Generic QA tools are mostly used for real-world data SRs and MAs, while no validated and reliable specific tool currently exist. Thus, there is need for a standardized and specific QA tool of SRs and MAs for real-world data.
Similar content being viewed by others
Avoid common mistakes on your manuscript.
Introduction
Systematic Reviews (SRs), evidence-based medicine, and clinical guidelines bring together trustworthy information by systematically acquiring, analysing, and transferring research findings into clinical, management, and policy arenas [1]. As such, findings of different work in medical literature on related topics are evaluated using SRs and meta-analyses (MAs), through the application of scientific strategies that limit bias and errors that occur by chance [2]. Availability of the best evidence obtained though SRs and MAs is necessary to help clinicians, policy makers and patients reach the best health care decisions [3]. However, SRs and MAs require resources, take time, and are labour-intensive, as well, they may not always be warranted or possible. For example, a study estimated the expense of SRs for academic institutions and pharmaceutical companies to cost approximately $141,194.80, and on average, the total cost of all SRs per year to academic institutions and pharmaceutical companies amounts to $18,660,304.77 and $16,761,234.71 [4]. Therefore, unnecessary duplication of SRs should be avoided for cost, as well as given the large unmet need for SRs of a wide range of questions and the need to keep reviews up-to-date [5].
To use the results of SRs and MAs, it is important to assess the methodological quality of the primary studies [6]. Methodological quality assessment (QA) is the process of assessing the design and conduct of the included studies, and it is useful to establish transparency of evidence synthesis and to guarantee the certainty of the body of evidence of the review objective [7, 8]. The main reason for assessing methodological quality of primary studies is to identify risks of bias [9] which may be due to poor reporting and several design features that are dependent on the research question. Poor reporting may prevent assessment of key features of design, making it difficult to evaluate whether the study methodology has been adequate [10]. According to National Health and Medical Research Council [11], “risks of bias refer to the likelihood that features of the study design or conduct of the study will give misleading results”, and thus bring about misused resources, un-thriftiness for effective interventions or harm to consumers [11].
A systematic review of methodological assessment tools for preclinical and clinical studies, and clinical practice guidelines show that there are a variety of methodological assessment tools for different types of study design [12]. Thus, it is critical to identify the study type before choosing the corresponding QA tool. In accordance, Zeng and colleagues [12] submit that further efforts in the development of critical appraisal tools are warranted for areas that currently lack such tools. However, there is an apparent dearth of specific QA tool for real-world evidence (RWE) studies. According to Food and Drugs Administrations [13], “RWE is the clinical evidence about the usage and potential benefits, or risks of a medical product derived from analysis of real-world data (RWD)”. Whereas RWD are routinely collected data pertaining to health status and/or health care delivery of the patient which are collected from a range of sources” [14] including claims, clinical studies, clinical setting, pharmaceuticals, and patient-powered platforms [15, 16].
The increasing use of electronic health records, and health information systems has led to repositories of large volumes of complex longitudinal RWD [17]. Thus, RWD are mostly diversified, but generally are medical records, prescription data and lifestyle-related information from health care providers, hospitals, and pharmacies [18]. For primary studies based on RWD, the quality of their data should be defined in context, clearly represented, and accessible [15, 19]. For example, Hyrich [20] concludes that RWD plays significant role in rheumatology because it helps to better understand disease progression and treatment outcomes beyond the conclusions of a clinical trial, as it provides a platform to "test" outcomes in an uncontrolled, real-life environment. Furthermore, the author posits that there is need to generate trustworthy conclusions from RWD by ensuring appropriate methodological and ethical considerations for handling RWD. Given the importance of RWD in research, population health, quality improvement, clinical decision support, and personalised medicine [21], it is necessary to explore the existing QA tools that have been used for SRs and MAs that involved RWD. Hence, this scoping review of QA tools used for SRs and MAs that involved RWD.
Methods
Scoping review
We conducted a scoping review, a type of literature review that is used when it is difficult to identify a narrow review question; no prior synthesis has been undertaken on the topic; studies in the review sources are likely to have employed a range of data collection and analysis techniques; and a quality assessment of reviewed sources is not going to be conducted [22].
Search strategy
An electronic database search was carried out by the reviewers through November 2022 using the following databases: PubMed, Allied and Complementary Medicine Database (AMED), Cumulated Index to Nursing and Allied Health Literature (CINAHL), and MEDLINE. The keywords used in the search included a combination of RWE, RWD, routinely collected data, electronic health records, claims and billing activities, registries, meta‐analysis, and systematic review (Appendix 2). Further, a manual search of reference sections of the included studies was also checked for additional studies. The search was delimited to articles published in English language.
Study selection and data extraction
One reviewer screened the abstracts of all publications obtained by the search strategies. Studies meeting the following inclusion criteria were selected for further review: interventional or observational studies, using real-world data, employed methodological QA tools. SRs or MAs not based on RWD and not methodological quality assessed were excluded. The potential eligible papers were retrieved, and the full articles were obtained and assessed for their relevance by two reviewers (TG & CEM) based on the preplanned criteria for inclusion. Any disagreement in study selection was resolved through discussion and consultation with a third reviewer (FF) where necessary.
A summary table was used to display the extracted data. The following data were extracted: authors and date, type of study, type of QA tool, number of items, domains, whether the tool is generic or specific, time to complete the tool, psychometric properties (validity and reliability), population/studies used to validate the tool, and name of the unit that developed the tool. The reviewers resolved differences through discussion to achieve consensus.
Data synthesis
Study data were extracted by three reviewers into a template. Findings for each study focusing on the QA tools used in SRs and MAs of RWD were then summarized by one reviewer, and the summaries discussed and modified by the research team as necessary, to generate an overall conclusion about the quality assessment (QA) tools used in SRs and MAs involving real-world data.
Results
The search strategy retrieved 4,954 (PubMed = 4369; AMED = 5; CINHAL = 182; Medline = 398) articles from four databases (Fig. 1). After duplicates removal, the tittles, and abstracts of 4,153 publications were screened. From this, only 75 studies were included for full-text screening and 16 articles met the inclusion criteria.
Characteristics of included studies
The characteristics of the included studies are presented in Table 1. The included studies were published between 2016 and December 2021. Seven of the included studies were observational type and the remaining were interventional and observational type of studies. The included studies applied various QA tools. The number of items used for QA within the included studies ranged from 4 to 22. Seven of the included studies comprised core domains that contains different questions employed for quality assessment. Only one [23] of the included studies utilised very specific tools for methodological quality assessment. Three [24,25,26] of the included studies employed validated QA tools. In order to validate the tools used in the included studies, they employed 39 non-randomised studies [24], 131 cohort studies [25] and 30 cost-effectiveness studies [26]. On the other hand, the QA tools utilised to the remaining thirteen of the included studies were not validated.
Non-summative four-point system
Non-summative four-point system is one of the included studies used a QA tool specific to real-world data [23]. The tool was developed by Wylde and colleagues, it is non-summative four-point system [19]. The tool consisted of four items used to assess selection bias (inclusion of consecutive patients and representativeness), bias due to missing data (follow-up rates) and bias due to inadequate consideration of confounding (multivariable or univariable analysis). Each item was rated as adequate, not adequate or not reported.
Discussion
In this paper, we reviewed the methodological QA tools for SRs and MAs used in RWE studies. The included studies in our review were published between 2016 and 2021, this finding aligns with the period of recent surge of use of methodological QA tools in real-world data studies. However, there is inadequate use of QA tool in RWD compared to other SRs and MA using randomised clinical trial [39]. The use of appropriate QA tools in SRs and MAs involving RWD is needed to generate trustworthy conclusions and acceptable evidence and recommendations to be used in health care [40]. The key point that is considered in the process of utilising evidence from SRs and MAs is whether critical appraisal is carried out or not [41]. For example, the findings of a study [42] that assessed the methodological, reporting and evidence quality of SRs and MAs of total glucosides of paeony for rheumatoid arthritis indicated that although included studies summarised that glucoside of paeony was effective and safe in the treatment of rheumatoid arthritis, the methodological and reporting quality and the quality of evidence was poor. As a result, the study recommended that decision-makers should be prudent when using glucosides of paeony in treating rheumatoid arthritis. Hyrich [20] in highlighting the key role of RWD in rheumatology, noted that methodological challenges in analysing RWD is a significant challenge to generating reliable scientific output using RWD.
Variation was observed within the QA tools used in the SRs and MAs with regard to content of domains, checklist, and scales. For example, some of the QA criteria such as inclusion of consecutive patients, representativeness, and follow-up were frequently reported in QA tools. Thus, the absence of a specific QA tool can restrict the process of consistent and reliable appraisal for SRs and MAs studies that have used RWD. In the current review, the authors observed that some of the QA tools were adapted or modified [23, 32, 34, 36], whereas others used generic QA tools. Overall, little consensus was observed around the QA tools of the SRs and MAs for RWE studies.
The absence of a standard and specific QA tool for SRs and MAs involving RWE studies have resulted in the use of different types of QA tools that have been developed for other studies with a different methodology such as randomised controlled studies, cross-sectional studies. Except one [23], all the included studies for the current review have used different sets of QA tools that are generic. The tool developed by Evans and colleagues [23] was specific and consists of four items including inclusion of consecutive patients, representativeness, percentage of follow-up and minimisation of potential confounding. However, this QA tool was not validated, as its psychometric properties are lacking. Psychometric properties of a test are tests that identify and define critical aspects of an instrument that include its adequacy, relevance, and usefulness (or its validity) [43]. Other authors argued that there should be a QA tool which is specific to SRs and MAs for RWE that have been psychometrically tested for their feasibility, reliability, and validity [44].
The criteria to be used for QA in each type of tools are different and no specific tool covers all the methodological aspects. It is due to these methodological differences that relevant evaluation tools are developed based on the characteristics of different types of study. Some evaluation tools are, for example, used without recommendations for critical appraisal of evidence [45]. There are also many types of research methods such as before-after study (time series) and nested case–control study that do not have QA tools [46]. It is important that efforts should be made on developing QA tools for SRs and MAs of RWD.
This scoping review has certain strength and limitations. In this review, we used a systematic approach such as the screening of numerous data bases, and the involvement of multiple reviewers. Only studies conducted in English language were included, therefore, there is the possibility that some other relevant studies in other languages could have been excluded. Nevertheless, this review serves as a foundation for further work on QA tools in SRs and MA using RWD. Identification of appropriate QA tool for a specific type of study should be the priority for those utilising evidence from them. This is because it will be useful to increase the transparency and reproducibility of scientific work in real-world evidence. This study could be a foundation by way of summarising the QA tools while pointing out potential improvements to be adopted in the future.
Conclusions
The findings of the present scoping review indicated that many different types of QA tools are currently used for RWD of SRs and MAs studies, while no validated and reliable specific tool currently exist. Thus, there is a need for a standardized and specific QA tool of SRs and MAs for RWD.
Data availability
All results from our analyses are published in the Supplementary Material, available at Rheumatology International online. Items/domains employed to the included studies and extracted by our investigators are available upon reasonable request.
References
Manchikanti L (2008) Evidence-based medicine, systematic reviews, and guidelines in interventional pain management, part I: introduction and general considerations. Pain Physician 11(2):161
Oxman AD, Schünemann HJ, Fretheim A (2006) Improving the use of research evidence in guideline development: 8. Synthesis and presentation of evidence. Health Res Policy Syst 4(1):1–10
Moosapour H, Saeidifard F, Aalaa M, Soltani A, Larijani B (2021) The rationale behind systematic reviews in clinical medicine: a conceptual framework. J Diabetes Metab Disord 20:919–929
Michelson M, Reuter K (2019) The significant cost of systematic reviews and meta-analyses: a call for greater involvement of machine learning to assess the promise of clinical trials. Contemp Clin Trials Commun 16:100443
Chinnock P, Siegfried N, Clarke M (2005) Is evidence-based medicine relevant to the developing world? PLoS Med 2(5):e107
Ma LL, Wang YY, Yang ZH, Huang D, Weng H, Zeng XT (2020) Methodological quality (risk of bias) assessment tools for primary and secondary medical studies: what are they and which is better? Mil Med Res 7:1–11
Reitsma JB, Rutjes AW, Whiting P, Vlassov VV, Leeflang MM, Deeks JJ (2009) Assessing methodological quality. Cochrane Handb Syst Rev Diagn Test Accuracy Version 1:1–28
Drucker AM, Fleming P, Chan AW (2016) Research techniques made simple: assessing risk of bias in systematic reviews. J Investig Dermatol 136(11):e109–e114
Shea BJ, Grimshaw JM, Wells GA, Boers M, Andersson N, Hamel C, Bouter LM (2007) Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol 7(1):1–7
Smidt N, Rutjes AW, Van Der Windt DA, Ostelo RW, Reitsma JB, Bossuyt PM, De Vet HC (2005) Quality of reporting of diagnostic accuracy studies. Radiology 235(2):347–353
NHMRC. Guidelines for Guidelines: Assessing risk of bias. https://nhmrc.gov.au/guidelinesforguidelines/develop/assessing-risk-bias. Last published 29 August 2019
Zeng X, Zhang Y, Kwong JS, Zhang C, Li S, Sun F, Du L (2015) The methodological quality assessment tools for preclinical and clinical studies, systematic review and meta-analysis, and clinical practice guideline: a systematic review. J Evid Based Med 8(1):2–10
US Food and Drug Administration (2018) Framework for FDA’s real-world evidence program. US Food and Drug Administration, Silver Spring
Chodankar D (2021) Introduction to real-world evidence studies. Perspect Clin Res 12(3):171
Makady A, de Boer A, Hillege H, Klungel O, Goettsch W (2017) What is real-world data? A review of definitions based on literature and stakeholder interviews. Value Health 20(7):858–865
US Food and Drug's Administration Real-World Evidence [Last accessed on 2023 May 22]. Available from: https://www.fda.gov/science-research/science-and-research-special-topics/real-world-evidence
Liaw ST, Guo JGN, Ansari S, Jonnagaddala J, Godinho MA, Borelli AJ Jr, Kahn MG (2021) Quality assessment of real-world data repositories across the data life cycle: a literature review. J Am Med Inform Assoc 28(7):1591–1599
European Medicines Agency (2017) Observational Data (Real World Data). European Medicines Agency.
Wylde V, Beswick AD, Dennis J, Gooberman-Hill R (2017) Post-operative patient-related risk factors for chronic pain after total knee replacement: a systematic review. BMJ Open 7(11):e018105
Hyrich KL (2019) Real world data in rheumatology. In: Seminars in arthritis and rheumatism. WB Saunders, 49(3), S22-S24.
Liyanage H, Liaw ST, Jonnagaddala J, Schreiber R, Kuziemsky C, Terry AL, de Lusignan S (2019) Artificial intelligence in primary health care: perceptions, issues, and challenges. Yearb Med Inform 28(01):041–046
Arksey H, O’Malley L (2005) Scoping studies: towards a methodological framework. Int J Soc Res Methodol 8(1):19–32
Evans JT, Evans JP, Walker RW, Blom AW, Whitehouse MR, Sayers A (2019) How long does a hip replacement last? A systematic review and meta-analysis of case series and national registry reports with more than 15 years of follow-up. The Lancet 393(10172):647–654
Coratti G, Cutrona C, Pera MC, Bovis F, Ponzano M, Chieppa F, Mercuri E (2021) Motor function in type 2 and 3 SMA patients treated with Nusinersen: a critical review and meta-analysis. Orphanet J Rare Dis 16(1):1–12
Alipour O, Gualti A, Shao L, Zhang B (2021) Systematic review and meta-analysis: real-world data rates of deep remission with anti-TNFα in inflammatory bowel disease. BMC Gastroenterol 21(1):1–11
Lu ZK, Xiong X, Lee T, Wu J, Yuan J, Jiang B (2021) Big data and real-world data based cost-effectiveness studies and decision-making models: a systematic review and analysis. Front Pharmacol 12:2
Halling AS, Loft N, Silverberg JI, Guttman-Yassky E, Thyssen JP (2021) Real-world evidence of dupilumab efficacy and risk of adverse events: a systematic review and meta-analysis. J Am Acad Dermatol 84(1):139–147
Hidayat K, Du X, Shi BM (2019) Risk of fracture with dipeptidyl peptidase-4 inhibitors, glucagon-like peptide-1 receptor agonists, or sodium-glucose cotransporter-2 inhibitors in real-world use: systematic review and meta-analysis of observational studies. Osteoporos Int 30(10):1923–1940
Kolmos M, Christoffersen L, Kruuse C (2021) Recurrent ischemic stroke–a systematic review and meta-analysis. J Stroke Cerebrovasc Dis 30(8):105935
van der List JP, Chawla H, Zuiderbaan HA, Pearle AD (2016) The role of preoperative patient characteristics on outcomes of unicompartmental knee arthroplasty: a meta-analysis critique. J Arthroplasty 31(11):2617–2627
Rahhal A, Kasem M, Orabi B, Hamou F, Abuyousef S, Mahfouz A, Ahmed E (2022) Effectiveness of sacubitril/valsartan in heart failure with reduced ejection fraction using real-world data: an updated systematic review and meta-analysis. Curr Problems Cardiol 2:101412
Nicholas JA, Edwards NC, Edwards RA, Dellarole A, Grosso M, Phillips AL (2020) Real-world adherence to, and persistence with, once-and twice-daily oral disease-modifying drugs in patients with multiple sclerosis: a systematic review and meta-analysis. BMC Neurol 20(1):1–15
Omarini C, Piacentini F, Sperduti I, Cerma K, Barbolini M, Canino F, Moscetti L (2022) T-DM1 efficacy in trastuzumab-pertuzumab pre-treated HER2 positive metastatic breast cancer patients: a meta-analysis. BMC Cancer 22(1):1–7
Tahra A, Bayrak O, Dmochowski R (2022) The Epidemiology and population-based studies of women with lower urinary tract symptoms: a systematic review. Turk J Urol 48(2):155–165
Fatoye F, Gebrye T, Odeyemi I (2019) Real-world incidence and prevalence of low back pain using routinely collected data. Rheumatol Int 39(4):619–626
Lin SQ, Vo NP, Yen YC, Tam KW (2022) Outcomes of sentinel node biopsy for women with breast cancer after neoadjuvant therapy: systematic review and meta-analysis of real-world data. Ann Surg Oncol 2:1–12
Alsadhan N, Almaiman A, Pujades-Rodriguez M, Brennan C, Shuweihdi F, Alhurishi SA, West RM (2022) A systematic review of methods to estimate colorectal cancer incidence using population-based cancer registries. BMC Med Res Methodol 22(1):1–15
Erdos J, Wild C (2022) Mid-and long-term (at least 12 months) follow-up of patients with spinal muscular atrophy (SMA) treated with nusinersen, onasemnogene abeparvovec, risdiplam or combination therapies: a systematic review of real-world study data. Eur J Paediatr Neurol. 2:2
Jørgensen L, Paludan-Müller AS, Laursen DR, Savović J, Boutron I, Sterne JA, Hróbjartsson A (2016) Evaluation of the Cochrane tool for assessing risk of bias in randomized clinical trials: overview of published comments and analysis of user practice in Cochrane and non-Cochrane reviews. Syst Rev 5:1–13
Shea B, Hamel C, Wells GA, Bouter LM, Kristjansson E, Grimsha J, Boers M (2009) AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. J Clin Epidemiol 62(10):1013–1020
Tunguy-Desmarais GP, Muckart DJ (2013) Evidence-based medicine should be based on science. SAMJ South Afr Med J 103(10):700–701
Zhu X, Shen X, Hou X, Luo Y, Fu X, Cao M, Feng Z (2020) Total glucosides of paeony for the treatment of rheumatoid arthritis: a methodological and reporting quality evaluation of systematic reviews and meta-analyses. Int Immunopharmacol 88:106920
Licona-Chávez AL, Velázquez-Liaño LR (2020) Quality assessment of a multiple choice test through psychometric properties. MedEdPublish 9(91):91
Kim SY, Park JE, Lee YJ, Seo HJ, Sheen SS, Hahn S, Son HJ (2013) Testing a tool for assessing the risk of bias for nonrandomized studies showed moderate reliability and promising validity. J Clin Epidemiol 66(4):408–414
Sanderson S, Tatt ID, Higgins JP (2007) Tools for assessing quality and susceptibility to bias in observational studies in epidemiology: a systematic review and annotated bibliography. Int J Epidemiol 36(3):666–676
Grimes DA, Schulz KF (2002) Cohort studies: marching towards outcomes. Lancet 359(9303):341–345
Funding
This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.
Author information
Authors and Affiliations
Contributions
TG participated in the design of the study, carried out the literature search and selection process, charted and modelled the data and drafted the paper. FF, CEM and ZH also participated in the design of the study, the literature selection process and the modelling of the data and helped to draft the paper. All the authors participated in modelling the data, drafting the paper and reading and approving the final version of this manuscript.
Corresponding author
Ethics declarations
Conflict of Interest
The authors have no conflict of interests to declare.
Ethical Approval
For this study ethical approval is not required.
Informed Consent
The patient’s written informed consent was not made, as this was a systematic review study.
Disclaimer
No part of this review is copied or published elsewhere in whole or in part in any languages. The information in Appendix 1 are Items/domains employed to the included studies. They are specific criteria developed to be used for quality assessment.
Data Sharing
All data related to this work are available in this research article.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
Below is the link to the electronic supplementary material.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Gebrye, T., Fatoye, F., Mbada, C. et al. A scoping review on quality assessment tools used in systematic reviews and meta-analysis of real-world studies. Rheumatol Int 43, 1573–1581 (2023). https://doi.org/10.1007/s00296-023-05354-x
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00296-023-05354-x