Skip to main content

Advertisement

SpringerLink
Log in

The Composite Quality Score (CQS) as a trial appraisal tool: inter-rater reliability and rating time

  • Discussion paper
  • Published:
Clinical Oral Investigations Aims and scope Submit manuscript

Abstract

Objective

To establish the CQS inter-rater reliability and rating time and to compare both against that of the Jadad scale and Cochrane’s Risk of Bias Tool (ROBT).

Material and methods

Four independent raters rated 45 trial reports. The inter-rater reliability was established by use of the Brennan-Prediger coefficient (BPC). The coefficients were compared using the two-sample z-test. Secondary analysis included comparison of the inter-rater reliability of the randomization component of all tools, as well as of the allocation concealment component of the CQS to that of the ROBT. The mean rating time with standard deviation (SD) for each tool was determined using one-way repeated measures analysis of variance. Post hoc comparisons were made using the Tukey–Kramer adjustment for three pair-wise multiple comparisons.

Results

The inter-rater reliability was significantly higher for the CQS (BPC, 95% CI: 0.95, 0.87–1.00) compared to Jadad (0.70, 0.58–0.82) (adjusted p = 0.0005) and most components of ROBT. The mean (SD) time to complete the CQS (4.0 (1.0) min) did not differ significantly from that of the Jadad scale 4.8 (1.1) min (adjusted p = 0.11), but was significantly shorter compared to that of the ROBT 15.3 (5.9) min (adjusted p < 0.0001).

Conclusions

The results suggest the CQS to be a very reliable and fast trial appraisal tool.

Clinical relevance

The higher the inter-rater reliability, the higher the probability that trial results reflect therapeutic truth. The CQS will need to take further bias sources into consideration, in order to increase its utility.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Akobeng AK (2008) Assessing the validity of clinical trials. J Pediatr Gastroenterol Nutr 47:277–282. https://doi.org/10.1097/MPG.0b013e31816c749f

    Article  PubMed  Google Scholar 

  2. Mickenautsch S, Berger VW (2019) The role of the randomised controlled trial in restorative dentistry and the correct purpose of observational data. Br Dent J. https://doi.org/10.1038/sj.bdj.2019.43

    Article  PubMed  Google Scholar 

  3. Mickenautsch S, Yengopal V (2013) Direct contra naïve-indirect comparison of clinical failure rates between high-viscosity GIC and conventional amalgam restorations: an empirical study. PLoS ONE 8:e78397. https://doi.org/10.1371/journal.pone.0078397

    Article  PubMed  PubMed Central  Google Scholar 

  4. Odgaard-Jensen J, Vist GE, Timmer A, Kunz R, Akl EA, Schünemann H, Briel M, Nordmann AJ, Pregno S, Oxman AD (2011) Randomisation to protect against selection bias in healthcare trials. Cochrane Database Syst Rev 4:MR000012. https://doi.org/10.1002/14651858.MR000012.pub3

    Article  Google Scholar 

  5. Mickenautsch S (2020) Are most of the published clinical trial results in restorative dentistry invalid? - An Empirical Investigation. Rev Recent Clin Trials 15:122–130. https://doi.org/10.2174/1574887115666200421110732

    Article  PubMed  Google Scholar 

  6. Jadad AR, Moore RA, Carroll D et al (1996) Assessing the quality of reports of randomized clinical trials: is blinding necessary? Controlled Clin Trials 17:1–12. https://doi.org/10.1016/0197-2456(95)00134-4

    Article  PubMed  Google Scholar 

  7. Higgins JP, Altman DG, Gøtzsche PC, Jüni P, Moher D, Oxman AD, Savovic J, Schulz KF, Weeks L, Sterne JA; Cochrane Bias Methods Group; Cochrane Statistical Methods Group (2011) The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. BMJ 343:d5928. https://doi.org/10.1136/bmj.d5928

    Article  Google Scholar 

  8. Berger VW (2006) Is the Jadad score the proper evaluation of trials? J Rheumatol 33:1710–1712

    PubMed  Google Scholar 

  9. Clark HD, Wells GA, Huët C, McAlister FA, Salmi LR, Fergusson D, Laupacis A (1999) Assessing the quality of randomized trials: reliability of the Jadad scale. Control Clin Trials 20:448–452. https://doi.org/10.1016/s0197-2456(99)00026-4

    Article  PubMed  Google Scholar 

  10. Hartling L, Bond K, Vandermeer B, Seida J, Dryden DM, Rowe BH (2011) Applying the risk of bias tool in a systematic review of combination long-acting beta-agonists and inhaled corticosteroids for persistent asthma. PLoS ONE 6:e17242. https://doi.org/10.1371/journal.pone.0017242

    Article  PubMed  PubMed Central  Google Scholar 

  11. Hartling L, Hamm MP, Milne A et al (2013) Testing the risk of bias tool showed low reliability between individual reviewers and across consensus assessments of reviewer pairs. J Clin Epidemiol 66:973–981. https://doi.org/10.1016/j.jclinepi.2012.07.005

    Article  PubMed  Google Scholar 

  12. Gwet KL (2010) Handbook of inter-rater reliability. (2nd ed.) Advanced Analytics, LLC: Gainsburg, MD, USA pp. 54, 159-160

  13. Mickenautsch S (2020) Reports of controlled clinical trials for directly placed restorations in vital teeth published between 1990 – 2015: References of trial reports per calendar year. Dataset: 10.6084/m9.figshare.12310841.v1

  14. Website. Random Number Generator. Available from http://www.randomnumbergenerator.com (accessed: May 15, 2020)

  15. Website. Random Number Generator. Available from http://www.randomdraws.com/ (accessed: May 17, 2020)

  16. Pocock SJ (1988) Clinical trials. A practical approach. John Wiley & Sons Ltd, Chichester, pp 126–134

    Google Scholar 

  17. Geigy (1970) Scientific tables, 7th ed. Geigy: Basle p. 28

  18. SAS Institute Inc., SAS Software, version 9.4 for Windows, Cary, NC, USA: SAS Institute Inc. (2002-2012)

  19. Landis JR, Koch GG (1977) The measurement of observer agreement for categorical data. Biometrics 33:159–174

    Article  Google Scholar 

  20. Berger VW, Alperson SY (2009) A general framework for the evaluation of clinical trial quality. Rev Recent Clin Trials 4:79–88. https://doi.org/10.2174/157488709788186021

    Article  PubMed  PubMed Central  Google Scholar 

  21. Bérard A, Andreu N, Tétrault J, Niyonsenga T, Myhal D (2000) Reliability of Chalmers’ scale to assess quality in meta-analyses on pharmacological treatments for osteoporosis. Ann Epidemiol 10:498–503. https://doi.org/10.1016/s1047-2797(00)00069-7

    Article  PubMed  Google Scholar 

  22. Glenny AM, Altman DG, Song F, Sakarovitch C, Deeks JJ, D’Amico R, Bradburn M, Eastwood AJ; International Stroke Trial Collaborative Group (2005) Indirect comparisons of competing interventions. Health Technol Assess 9:1–134

    Google Scholar 

  23. Alperson SY, Berger VW (2011) Opposing systematic reviews: the effects of two quality rating instruments on evidence regarding t’ai chi and bone mineral density in postmenopausal women. J Altern Complement Med 17:389–395. https://doi.org/10.3310/hta9260

    Article  PubMed  PubMed Central  Google Scholar 

  24. Sterne JAC, Savović J, Page MJ, Elbers RG, Blencowe NS, Boutron I, Cates CJ, Cheng HY, Corbett MS, Eldridge SM, Emberson JR, Hernán MA, Hopewell S, Hróbjartsson A, Junqueira DR, Jüni P, Kirkham JJ, Lasserson T, Li T, McAleenan A, Reeves BC, Shepperd S, Shrier I, Stewart LA, Tilling K, White IR, Whiting PF, Higgins JPT (2019) RoB 2: a revised tool for assessing risk of bias in randomised trials. BMJ 366:l4898. https://doi.org/10.1136/bmj.l4898

    Article  Google Scholar 

  25. Thomas J, Kneale D, McKenzie JE, Brennan SE, Bhaumik S (2020) Chapter 2: Determining the scope of the review and the questions it will address. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.1 (updated September 2020). Cochrane. Available from www.training.cochrane.org/handbook

  26. Deeks JJ, Higgins JPT, Altman DG (editors). Chapter 10: Analysing data and undertaking meta-analyses. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors) (2019) Cochrane Handbook for Systematic Reviews of Interventions version 6.0 (updated July 2019). Cochrane. Available from www.training.cochrane.org/handbook

Download references

Acknowledgements

The authors thank Dr Petra Gaylard from DMSA for her valuable advice concerning data statistics and for conducting the data analysis.

Author information

Authors and Affiliations

  1. Review Center For Health Science Research, 84 Concorde Road East, Bedfordview/Johannesburg, 2008, South Africa

    Steffen Mickenautsch

  2. Department of Community Dentistry, School of Oral Health Sciences, Faculty of Health Sciences, University of the Witwatersrand, 7 York Rd, Parktown/Johannesburg, 2193, South Africa

    Steffen Mickenautsch

  3. Department of Endodontics and Restorative Dentistry, School of Dental Medicine, University of Zagreb, Gunduliceva 5, 10 000, Zagreb, Croatia

    Ivana Miletić

  4. Clinic of Operative Dentistry, Periodontology & Preventive Dentistry, Saarland University Hospital, Building 73, 66421, Homburg, Germany

    Stefan Rupf

  5. Open University of Catalonia - University of Barcelona, Barcelona, Spain

    Jone Renteria

  6. Department of Human Development and Quantitative Methodology, University of Maryland, College Park, MD, 20742, USA

    Jone Renteria

  7. Department of Operative and Preventive Dentistry, Charité - Universitätsmedizin Berlin, Assmannshauser Str. 4 - 6, Berlin, 14197, Germany

    Gerd Göstemeyer

Authors
  1. Steffen Mickenautsch
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ivana Miletić
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Stefan Rupf
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jone Renteria
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Gerd Göstemeyer
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

SM: conceptualization, methodology, data curation, writing—original draft, supervision. IM: investigation, writing—review & editing. SR: investigation, writing—review & editing. JR: investigation, writing—review & editing. GG: investigation, writing—review & editing.

Corresponding author

Correspondence to Steffen Mickenautsch.

Ethics declarations

Ethical approval

This article does not contain any studies with human participants or animals performed by any of the authors.

Informed consent

For this type of study, formal consent is not required.

Conflict of interest

The authors declare no competing interests.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (XLS 1235 KB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mickenautsch, S., Miletić, I., Rupf, S. et al. The Composite Quality Score (CQS) as a trial appraisal tool: inter-rater reliability and rating time. Clin Oral Invest 25, 6015–6023 (2021). https://doi.org/10.1007/s00784-021-04099-w

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00784-021-04099-w

Keywords

Search

Navigation