The Newcastle-Ottawa scale (NOS) is one of many scales used to judge the quality of observational studies in systematic reviews. It was criticized for its arbitrary definitions of quality items in a commentary in 2010 in this journal. That commentary was cited 1,250 times through December 2016. We examined the citation history of this commentary in a random sample of 100 full papers citing it, according to the Web of Science. Of these, 96 were systematic reviews, none of which quoted the commentary directly. All but 2 of the 96 indirect quotations (98%) portrayed the commentary as supporting use of the NOS in systematic reviews when, in fact, the opposite was the case. It appears that the vast majority of systematic review authors who cited this commentary did not read it. Journal reviewers and editors did not recognize and correct these major quotation errors. Authors should read each source they cite to make sure their direct and indirect quotations are accurate. Reviewers and editors should do a better job of checking citations and quotations for accuracy. It might help somewhat for commentaries to include abstracts, so that the basic content can be conveyed by PubMed and other bibliographic resources.
This is a preview of subscription content, log in to check access.
Buy single article
Instant access to the full article PDF.
Price includes VAT for USA
Subscribe to journal
Immediate online access to all issues from 2019. Subscription will auto renew annually.
This is the net price. Taxes to be calculated in checkout.
Wells GA, Shea B, O’Connell D, Peterson J, Welch V, Losos M, et al. The Newcastle–Ottawa Scale (NOS) for assessing the quality if nonrandomized studies in meta-analyses. http://www.ohrica/programs/clinical_epidemiology/oxfordasp. 2009.
Reeves BC, Deeks JJ, Higgins JPT, Wells GA. Chapter 13: Including non-randomized studies. In: Higgins JBT, Green S, editors. Cochrane handbook of systematic reviews of interventions, version 510 (updated March 2011). www.handbook.cochrane.org: The cochrane collaboration. 2011.
Deeks JJ, Dinnes J, D’Amico R, Sowden AJ, Sakarovitch C, Song F, et al. Evaluating non-randomised intervention studies. Health Technol Assess. 2003;7:iii–173.
Stang A. Critical evaluation of the Newcastle–Ottawa scale for the assessment of the quality of nonrandomized studies in meta-analyses. Eur J Epidemiol. 2010;25:603–5.
Lee SY, Lee JS. A survey of quotation accuracy in two Korean dermatological journals. Ann Dermatol. 1995;7:236–9.
de Lacey G, Record C, Wade J. How accurate are quotations and references in medical journals? Br Med J (Clin Res Ed). 1985;291:884–6.
Eichorn P, Yankauer A. Do authors check their references? A survey of accuracy of references in three public health journals. Am J Public Health. 1987;77:1011–2.
Evans JT, Nadjari HI, Burchell SA. Quotational and reference accuracy in surgical journals. A continuing peer review problem. JAMA. 1990;263:1353–4.
Tfelt-Hansen P. The qualitative problem of major quotation errors, as illustrated by 10 different examples in the headache literature. Headache. 2015;55:419–26.
Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gøtzsche PC, Ioannidis JP, Clarke M, Devereaux PJ, Kleijnen J, Moher D. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. J Clin Epidemiol. 2009;62(10):e1–34. https://doi.org/10.1016/j.jclinepi.2009.06.006.
DerSimonian R, Laird N. Meta-analysis in clinical trials. Control Clin Trials. 1986;7(3):177–88.
Deeks JJ, Higgins JPT, Altman DG. Chapter 9: Analysing data and undertaking meta-analyses. In: Higgins JPT, Green S, editors. Cochrane handbook of systematic reviews of interventions, version 510 (updated March 2011). www.handbook.cochrane.org: The cochrane collaboration. 2011.
Garvin DA. What does product quality really mean? Sloan Manag Rev. 1984;26:25–43.
Dickie G. Aesthetics: an introduction. New York: The Bobbs-Merrill Company, Inc.; 1971.
On Gerwitz P. I know it when I see it. Yale Law J. 1996;105:1023–47.
Crosby PB. Quality is free. New York: McGraw-Hill; 1979.
Broh RA. Managing quality for higher profits. New York: McGraw-Hill; 1982.
Kuehn AA, Day RL. Strategy of product quality. Harv Bus Rev. 1962;40:100–10.
Rodgers A, MacMahon S. Systematic underestimation of treatment effects as a result of diagnostic test inaccuracy: implications for the interpretation and design of thromboprophylaxis trials. Thromb Haemost. 1995;73:167–71.
Higgins JPT, Altman DG, Sterne JAC. Chapter 8: Assessing risk of bias in included studies. Cochrane handbook for systematic reviews of interventions, version 510 (updated March 2011). www.handbook.cochrane.de: The cochrane collaboration. 2011.
Greenland S. Quality scores are useless and potentially misleading—Reply to Re—a critical-look at some popular analytic methods. Am J Epidemiol. 1994;140:300–1.
Greenland S, O’Rourke K. On the bias produced by quality scores in meta-analysis, and a hierarchical view of proposed solutions. Biostatistics. 2001;2:463–71.
Donabedian A. Evaluating the quality of medical care. Milbank Q. 1966;44:166–203.
Juni P, Witschi A, Bloch R, Egger M. The hazards of scoring the quality of clinical trials for meta-analysis. JAMA. 1999;282:1054–60.
Rubin DR. Meta-analysis: literature synthesis or effect-size surface estimation? J Educ Stat. 1992;17:363–74.
Lash TL, Fox MP, Fink AK. Applying quantitative bias analysis to epidemiologic data. Dordrecht: Springer; 2009.
Ebert CS Jr, Drake AF. The impact of sleep-disordered breathing on cognition and behavior in children: a review and meta-synthesis of the literature. Otolaryngol Head Neck Surg. 2004;131:814–26.
This work was supported by the German Federal Ministry of Education and Science (BMBF) [Grant no. 01ER1704]. The funding source had no role in the study design, in the collection, analysis and interpretation of data, in the writing of the report, and in the decision to submit the paper for publication.
Conflict of interest
The authors declare that they have no conflict of interest.
About this article
Cite this article
Stang, A., Jonas, S. & Poole, C. Case study in major quotation errors: a critical commentary on the Newcastle–Ottawa scale. Eur J Epidemiol 33, 1025–1031 (2018). https://doi.org/10.1007/s10654-018-0443-3
- Newcastle-Ottawa Scale (NOS)
- Quotation Errors
- Indirect Quotation
- Citation History
- Systematic Review Authors