Skip to main content
SpringerLink
Log in

Case study in major quotation errors: a critical commentary on the Newcastle–Ottawa scale

  • REVIEW
  • Published:
European Journal of Epidemiology Aims and scope Submit manuscript

Abstract

The Newcastle-Ottawa scale (NOS) is one of many scales used to judge the quality of observational studies in systematic reviews. It was criticized for its arbitrary definitions of quality items in a commentary in 2010 in this journal. That commentary was cited 1,250 times through December 2016. We examined the citation history of this commentary in a random sample of 100 full papers citing it, according to the Web of Science. Of these, 96 were systematic reviews, none of which quoted the commentary directly. All but 2 of the 96 indirect quotations (98%) portrayed the commentary as supporting use of the NOS in systematic reviews when, in fact, the opposite was the case. It appears that the vast majority of systematic review authors who cited this commentary did not read it. Journal reviewers and editors did not recognize and correct these major quotation errors. Authors should read each source they cite to make sure their direct and indirect quotations are accurate. Reviewers and editors should do a better job of checking citations and quotations for accuracy. It might help somewhat for commentaries to include abstracts, so that the basic content can be conveyed by PubMed and other bibliographic resources.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

References

  1. Wells GA, Shea B, O’Connell D, Peterson J, Welch V, Losos M, et al. The Newcastle–Ottawa Scale (NOS) for assessing the quality if nonrandomized studies in meta-analyses. http://www.ohrica/programs/clinical_epidemiology/oxfordasp. 2009.

  2. Reeves BC, Deeks JJ, Higgins JPT, Wells GA. Chapter 13: Including non-randomized studies. In: Higgins JBT, Green S, editors. Cochrane handbook of systematic reviews of interventions, version 510 (updated March 2011). www.handbook.cochrane.org: The cochrane collaboration. 2011.

  3. Deeks JJ, Dinnes J, D’Amico R, Sowden AJ, Sakarovitch C, Song F, et al. Evaluating non-randomised intervention studies. Health Technol Assess. 2003;7:iii–173.

    Article  CAS  Google Scholar 

  4. Stang A. Critical evaluation of the Newcastle–Ottawa scale for the assessment of the quality of nonrandomized studies in meta-analyses. Eur J Epidemiol. 2010;25:603–5.

    Article  Google Scholar 

  5. Lee SY, Lee JS. A survey of quotation accuracy in two Korean dermatological journals. Ann Dermatol. 1995;7:236–9.

    Article  Google Scholar 

  6. de Lacey G, Record C, Wade J. How accurate are quotations and references in medical journals? Br Med J (Clin Res Ed). 1985;291:884–6.

    Article  Google Scholar 

  7. Eichorn P, Yankauer A. Do authors check their references? A survey of accuracy of references in three public health journals. Am J Public Health. 1987;77:1011–2.

    Article  CAS  Google Scholar 

  8. Evans JT, Nadjari HI, Burchell SA. Quotational and reference accuracy in surgical journals. A continuing peer review problem. JAMA. 1990;263:1353–4.

    Article  CAS  Google Scholar 

  9. Tfelt-Hansen P. The qualitative problem of major quotation errors, as illustrated by 10 different examples in the headache literature. Headache. 2015;55:419–26.

    Article  Google Scholar 

  10. Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gøtzsche PC, Ioannidis JP, Clarke M, Devereaux PJ, Kleijnen J, Moher D. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. J Clin Epidemiol. 2009;62(10):e1–34. https://doi.org/10.1016/j.jclinepi.2009.06.006.

    Article  PubMed  Google Scholar 

  11. DerSimonian R, Laird N. Meta-analysis in clinical trials. Control Clin Trials. 1986;7(3):177–88.

    Article  CAS  Google Scholar 

  12. Deeks JJ, Higgins JPT, Altman DG. Chapter 9: Analysing data and undertaking meta-analyses. In: Higgins JPT, Green S, editors. Cochrane handbook of systematic reviews of interventions, version 510 (updated March 2011). www.handbook.cochrane.org: The cochrane collaboration. 2011.

  13. Garvin DA. What does product quality really mean? Sloan Manag Rev. 1984;26:25–43.

    Google Scholar 

  14. Dickie G. Aesthetics: an introduction. New York: The Bobbs-Merrill Company, Inc.; 1971.

    Google Scholar 

  15. On Gerwitz P. I know it when I see it. Yale Law J. 1996;105:1023–47.

    Article  Google Scholar 

  16. Crosby PB. Quality is free. New York: McGraw-Hill; 1979.

    Google Scholar 

  17. Broh RA. Managing quality for higher profits. New York: McGraw-Hill; 1982.

    Google Scholar 

  18. Kuehn AA, Day RL. Strategy of product quality. Harv Bus Rev. 1962;40:100–10.

    Google Scholar 

  19. Rodgers A, MacMahon S. Systematic underestimation of treatment effects as a result of diagnostic test inaccuracy: implications for the interpretation and design of thromboprophylaxis trials. Thromb Haemost. 1995;73:167–71.

    CAS  PubMed  Google Scholar 

  20. Higgins JPT, Altman DG, Sterne JAC. Chapter 8: Assessing risk of bias in included studies. Cochrane handbook for systematic reviews of interventions, version 510 (updated March 2011). www.handbook.cochrane.de: The cochrane collaboration. 2011.

  21. Greenland S. Quality scores are useless and potentially misleading—Reply to Re—a critical-look at some popular analytic methods. Am J Epidemiol. 1994;140:300–1.

    Article  Google Scholar 

  22. Greenland S, O’Rourke K. On the bias produced by quality scores in meta-analysis, and a hierarchical view of proposed solutions. Biostatistics. 2001;2:463–71.

    Article  CAS  Google Scholar 

  23. Donabedian A. Evaluating the quality of medical care. Milbank Q. 1966;44:166–203.

    Article  Google Scholar 

  24. Juni P, Witschi A, Bloch R, Egger M. The hazards of scoring the quality of clinical trials for meta-analysis. JAMA. 1999;282:1054–60.

    Article  CAS  Google Scholar 

  25. Rubin DR. Meta-analysis: literature synthesis or effect-size surface estimation? J Educ Stat. 1992;17:363–74.

    Article  Google Scholar 

  26. Lash TL, Fox MP, Fink AK. Applying quantitative bias analysis to epidemiologic data. Dordrecht: Springer; 2009.

    Book  Google Scholar 

  27. Ebert CS Jr, Drake AF. The impact of sleep-disordered breathing on cognition and behavior in children: a review and meta-synthesis of the literature. Otolaryngol Head Neck Surg. 2004;131:814–26.

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported by the German Federal Ministry of Education and Science (BMBF) [Grant no. 01ER1704]. The funding source had no role in the study design, in the collection, analysis and interpretation of data, in the writing of the report, and in the decision to submit the paper for publication.

Author information

Authors and Affiliations

  1. Director of the Center of Clinical Epidemiology, Institute of Medical Informatics, Biometry and Epidemiology, University Hospital of Essen, Hufelandstr. 55, 45147, Essen, Germany

    Andreas Stang

  2. Department of Epidemiology, Boston University School of Public Health, 715 Albany St, Boston, MA, 02118, USA

    Andreas Stang

  3. Department of Medical Informatics, RWTH Aachen University, Pauwelsstr. 30, 52057, Aachen, Germany

    Stephan Jonas

  4. Department of Epidemiology, Gillings School of Global Public Health, University of North Carolina, CB #7435, 135 Dauer Drive, Chapel Hill, NC, 27599-7435, USA

    Charles Poole

Authors
  1. Andreas Stang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Stephan Jonas
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Charles Poole
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Andreas Stang.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Stang, A., Jonas, S. & Poole, C. Case study in major quotation errors: a critical commentary on the Newcastle–Ottawa scale. Eur J Epidemiol 33, 1025–1031 (2018). https://doi.org/10.1007/s10654-018-0443-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10654-018-0443-3

Keywords

Search

Navigation