Clinical Oral Investigations

, Volume 16, Issue 3, pp 781–788 | Cite as

A new instrument for assessing the quality of studies on prevalence

  • Nikolaos Nikitas GiannakopoulosEmail author
  • Peter Rammelsberg
  • Lydia Eberhard
  • Marc Schmitter
Original Article


There are numerous scientific articles of studies on the prevalence of disorders with non-standardised examination and diagnostic protocols. Because their quality is heterogeneous, a new instrument has been developed for the assessment of such studies. The new instrument is based mainly on statistical criteria. The points assigned for each of the main criteria according to the information gained from each paper are summed up to form a Total Quality Score (TQS). The interrater reliability of the instrument was tested by employing Kappa and Interrater Correlation Coefficient (ICC) statistics. The latter was assessed on the results of three independent investigators. The new quality instrument appeared to be easy to use, and the instructions were comprehensible. The ICC(2,1) for the TQS ranged between 0.94 and 1.00 indicating almost perfect agreement between the investigators. The reliability of the new instrument enables its use for scientific review purposes. In this way, its validity will also be tested. The instrument could be adopted for assessment of scientific articles of studies on the prevalence of disorders in many, similar, scientific areas.


Prevalence studies Quality assessment Reliability Epidemiology 



The authors of this article would like to acknowledge the dentists J. Mahabadi, A. Hassel, R. Shahin, and W. Bömicke for their help with assessment of the new instrument's reliability.

Conflict of interest

The authors declare that they have no conflict of interest.


  1. 1.
    Gross AJ, Rivera-Morales WC, Gale EN (1988) A prevalence study of symptoms associated with TM disorders. J Craniomandib Disord 2:191–195PubMedGoogle Scholar
  2. 2.
    Cherkin DC, Deyo RA, Wheeler K, Ciol MA (1994) Physician variation in diagnostic testing for low back pain. Who you see is what you get. Arthritis Rheum 37:15–22PubMedCrossRefGoogle Scholar
  3. 3.
    Koes BW, van Tulder MW, Ostelo R, Kim Burton A, Waddell G (2001) Clinical guidelines for the management of low back pain in primary care: an international comparison. Spine (Phila Pa 1976) 26:2504–2513, Discussion, 2513–2504CrossRefGoogle Scholar
  4. 4.
    Bamji AN, Erhardt CC, Price TR, Williams PL (1996) The painful shoulder: can consultants agree? Br J Rheumatol 35:1172–1174PubMedCrossRefGoogle Scholar
  5. 5.
    Luime JJ, Koes BW, Hendriksen IJ, Burdorf A, Verhagen AP, Miedema HS, Verhaar JA (2004) Prevalence and incidence of shoulder pain in the general population; a systematic review. Scand J Rheumatol 33:73–81PubMedCrossRefGoogle Scholar
  6. 6.
    Egger M, Smith G, Altman D (2009) Systematic reviews in health care: meta-analysis in context, 2nd edn. BMJ Books, LondonGoogle Scholar
  7. 7.
    von Elm E, Altman DG, Egger M, Pocock SJ, Gotzsche PC, Vandenbroucke JP (2007) The strengthening the reporting of observational studies in epidemiology (STROBE) statement: guidelines for reporting observational studies. Lancet 370:1453–1457CrossRefGoogle Scholar
  8. 8.
    Shamliyan T, Kane RL, Dickinson S (2010) A systematic review of tools used to assess the quality of observational studies that examine incidence or prevalence and risk factors for diseases. J Clin Epidemiol 63:1061–1070PubMedCrossRefGoogle Scholar
  9. 9.
    Boyle MH (1998) Guidelines for evaluating prevalence studies. Evid Based Mental Health 1:37–39CrossRefGoogle Scholar
  10. 10.
    Beller EM, Gebski V, Keech AC (2002) Randomisation in clinical trials. Med J Aust 177:565–567PubMedGoogle Scholar
  11. 11.
    Krogh-Poulsen WG (1969) Management of the occlusion of the teeth. Examination, diagnosis, treatment. In: Chayes LSCM (ed) Facial pain and mandibular dysfunction. Saunders, Philadelphia, pp 251–258Google Scholar
  12. 12.
    Helkimo M (1974) Studies on function and dysfunction of the masticatory system. Ii. Index for anamnestic and clinical dysfunction and occlusal state. Sven Tandläk Tidskr 67:101–121PubMedGoogle Scholar
  13. 13.
    Levitt SR, Lundeen TF, McKinney MW (1988) Initial studies of a new assessment method for temporomandibular joint disorders. J Prosthet Dent 59:490–495PubMedCrossRefGoogle Scholar
  14. 14.
    Levitt SR, McKinney MW, Lundeen TF (1988) The TMJ scale: cross-validation and reliability studies. Cranio 6:17–25PubMedGoogle Scholar
  15. 15.
    Lundeen TF, Levitt SR, McKinney MW (1986) Discriminative ability of the TMJ scale: age and gender differences. J Prosthet Dent 56:84–92PubMedCrossRefGoogle Scholar
  16. 16.
    Fricton JR, Schiffman EL (1986) Reliability of a craniomandibular index. J Dent Res 65:1359–1364PubMedCrossRefGoogle Scholar
  17. 17.
    Fricton JR, Schiffman EL (1987) The craniomandibular index: validity. J Prosthet Dent 58:222–228PubMedCrossRefGoogle Scholar
  18. 18.
    McNeill C (1993) Temporomandibular disorders-guidelines for classification, assessment and management, 2nd edn. Quintessence Publishing, ChicagoGoogle Scholar
  19. 19.
    Okeson J (1996) Orofacial pain: guidelines for assessment, diagnosis, and management. Quintessence Publishing, ChicagoGoogle Scholar
  20. 20.
    Dworkin SF, LeResche L (1992) Research diagnostic criteria for temporomandibular disorders: review, criteria, examinations and specifications, critique. J Craniomandib Disord 6:301–355PubMedGoogle Scholar
  21. 21.
    Wahlund K, List T, Dworkin SF (1998) Temporomandibular disorders in children and adolescents: reliability of a questionnaire, clinical examination, and diagnosis. J Orofac Pain 12:42–51PubMedGoogle Scholar
  22. 22.
    Schmitter M, Kress B, Leckel M, Henschel V, Ohlmann B, Rammelsberg P (2008) Validity of temporomandibular disorder examination procedures for assessment of temporomandibular joint status. Am J Orthod Dentofacial Orthop 133:796–803PubMedCrossRefGoogle Scholar
  23. 23.
    Dworkin SF, Sherman J, Mancl L, Ohrbach R, LeResche L, Truelove E (2002) Reliability, validity, and clinical utility of the research diagnostic criteria for Temporomandibular Disorders Axis II Scales: depression, non-specific physical symptoms, and graded chronic pain. J Orofac Pain 16:207–220PubMedGoogle Scholar
  24. 24.
    John MT, Dworkin SF, Mancl LA (2005) Reliability of clinical temporomandibular disorder diagnoses. Pain 118:61–69PubMedCrossRefGoogle Scholar
  25. 25.
    Schmitter M, Ohlmann B, John MT, Hirsch C, Rammelsberg P (2005) Research diagnostic criteria for temporomandibular disorders: a calibration and reliability study. Cranio 23:212–218PubMedGoogle Scholar
  26. 26.
    List T, John MT, Dworkin SF, Svensson P (2006) Recalibration improves inter-examiner reliability of tmd examination. Acta Odontol Scand 64:146–152PubMedCrossRefGoogle Scholar
  27. 27.
    Cohen J (1960) A coeffiecient of agreement for nominal scales. Educ Psychol Meas 20:37–46CrossRefGoogle Scholar
  28. 28.
    Shrout PE, Fleiss JL (1979) Intraclass correlations: uses in assessing rater reliability. Psychol Bull 86:420–428PubMedCrossRefGoogle Scholar
  29. 29.
    Aggarwal VR, Macfarlane TV, Macfarlane GJ (2003) Why is pain more common amongst people living in areas of low socio-economic status? A population-based cross-sectional study. Br Dent J 194:383–387, Discussion, 380PubMedCrossRefGoogle Scholar
  30. 30.
    Dworkin SF, Huggins KH, LeResche L, Von Korff M, Howard J, Truelove E, Sommers E (1990) Epidemiology of signs and symptoms in temporomandibular disorders: clinical signs in cases and controls. J Am Dent Assoc 120:273–281PubMedGoogle Scholar
  31. 31.
    Heikinheimo K, Salmi K, Myllarniemi S, Kirveskari P (1989) Symptoms of craniomandibular disorder in a sample of Finnish adolescents at the ages of 12 and 15 years. Eur J Orthod 11:325–331PubMedGoogle Scholar
  32. 32.
    Locker D, Grushka M (1987) The impact of dental and facial pain. J Dent Res 66:1414–1417PubMedCrossRefGoogle Scholar
  33. 33.
    Nilsson IM, List T, Drangsholt M (2005) Prevalence of temporomandibular pain and subsequent dental treatment in Swedish adolescents. J Orofac Pain 19:144–150PubMedGoogle Scholar
  34. 34.
    Pedroni CR, De Oliveira AS, Guaratini MI (2003) Prevalence study of signs and symptoms of temporomandibular disorders in university students. J Oral Rehabil 30:283–289PubMedCrossRefGoogle Scholar
  35. 35.
    Rantala MA, Ahlberg J, Suvinen TI, Savolainen A, Kononen M (2004) Chronic myofascial pain, disk displacement with reduction and psychosocial factors in finnish non-patients. Acta Odontol Scand 62:293–297PubMedCrossRefGoogle Scholar
  36. 36.
    Raustia AM, Peltola M, Salonen MA (1997) Influence of complete denture renewal on craniomandibular disorders: a 1-year follow-up study. J Oral Rehabil 24:30–36PubMedCrossRefGoogle Scholar
  37. 37.
    Sadowsky C, Muhl ZF, Sakols EI, Sommerville JM (1985) Temporomandibular joint sounds related to orthodontic therapy. J Dent Res 64:1392–1395PubMedCrossRefGoogle Scholar
  38. 38.
    Tallents RH, Hatala M, Katzberg RW, Westesson PL (1993) Temporomandibular joint sounds in asymptomatic volunteers. J Prosthet Dent 69:298–304PubMedCrossRefGoogle Scholar
  39. 39.
    Landis JR, Koch GG (1977) The measurement of observer agreement for categorical data. Biometrics 33:159–174PubMedCrossRefGoogle Scholar
  40. 40.
    Schouten H (1986) Nominal scale agreement among observers. Psychometrika 51:453–466CrossRefGoogle Scholar
  41. 41.
    DuRant R (1994) Checklist for the evaluation of research articles. J Adolesc Health 15:4–8PubMedCrossRefGoogle Scholar
  42. 42.
    Downs SH, Black N (1998) The feasibility of creating a checklist for the assessment of the methodological quality both of randomised and non-randomised studies of health care interventions. J Epidemiol Community Health 52:377–384PubMedCrossRefGoogle Scholar
  43. 43.
    Macfarlane TV, Glenny AM, Worthington HV (2001) Systematic review of population-based epidemiological studies of oro-facial pain. J Dent 29:451–467PubMedCrossRefGoogle Scholar
  44. 44.
    Genaidy A, Lemasters GK, Lockey J, Succop P, Deddens J, Sobeih T, Dunning K (2007) An epidemiological appraisal instrument - a tool for evaluation of epidemiological studies. Ergonomics 50:920–960PubMedCrossRefGoogle Scholar
  45. 45.
    Juni P, Altman D, Egger M (2001) Assessing the quality of controlled trials. BMJ 323:42–46PubMedCrossRefGoogle Scholar
  46. 46.
    Berlin J, Rennie D (1999) Measuring the quality of trials: the quality of quality scales. JAMA 282:1083–1085PubMedCrossRefGoogle Scholar
  47. 47.
    Greenland S (1994) Quality scores are useless and potentially misleading. Am J Epidemiol 140:300–302Google Scholar
  48. 48.
    Moher D, Pham B, Jones A, Cook DJ, Jadad AR, Moher M, Tugwell P, Klassen TP (1998) Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses? Lancet 352:609–613PubMedCrossRefGoogle Scholar
  49. 49.
    Whiting P, Harbord R, Kleijnen J (2005) No role for quality scores in systematic reviews of diagnostic accuracy studies. BMC Med Res Methodol 5:19–27PubMedCrossRefGoogle Scholar
  50. 50.
    Tooth L, Ware R, Bain C, Purdie DM, Dobson A (2005) Quality of reporting of observational longitudinal research. Am J Epidemiol 161:280–288PubMedCrossRefGoogle Scholar
  51. 51.
    Soares HP, Daniels S, Kumar A, Clarke M, Scott C, Swann S, Djulbegovic B (2004) Bad reporting does not mean bad methods for randomised trials: observational study of randomised controlled trials performed by the radiation therapy oncology group. BMJ 328:22–24PubMedCrossRefGoogle Scholar
  52. 52.
    Nguyen Q, Bezemer P, Habets L, Prahl-Andersen B (1999) A systematic review of the relationship between overjet size and traumatic dental injuries. Eur J Orthod 21:503–515PubMedCrossRefGoogle Scholar
  53. 53.
    Ariëns G, van Mechelen W, Bongers P, Bouter L, van der Wal G (2000) Physical risk factors for neck pain. Scand J Work Environ Health 26:7–19PubMedCrossRefGoogle Scholar
  54. 54.
    Loney P, Chambers LW, Benett KJ, Roberts JG, Stratford PW (1998) Critical appraisal of the health research literature: prevalence or incidence of a health problem. Chronic Dis Can 19:170–176PubMedGoogle Scholar
  55. 55.
    Woodbury M, Houghton PE (2004) Prevalence of pressure ulcers in Canadian healthcare settings. Ostomy/Wound Manage 50:22–38Google Scholar
  56. 56.
    Silva LC, Ordunez P, Paz Rodriguez M, Robles S (2001) A tool for assessing the usefulness of prevalence studies done for surveillance purposes: the example of hypertension. Rev Panam Salud Pública 10:152–160PubMedCrossRefGoogle Scholar
  57. 57.
    Barclay P, Hollender LG, Maravilla KR, Truelove EL (1999) Comparison of clinical and magnetic resonance imaging diagnosis in patients with disk displacement in the temporomandibular joint. Oral Surg Oral Med Oral Pathol Oral Radiol Endod 88:37–43PubMedCrossRefGoogle Scholar
  58. 58.
    Schmitter M, Kress B, Rammelsberg P (2004) Temporomandibular joint pathosis in patients with myofascial pain: a comparative analysis of magnetic resonance imaging and a clinical examination based on a specific set of criteria. Oral Surg Oral Med Oral Pathol Oral Radiol Endod 97:318–324PubMedCrossRefGoogle Scholar
  59. 59.
    Emshoff R, Rudisch A (2001) Validity of clinical diagnostic criteria for temporomandibular disorders: clinical versus magnetic resonance imaging diagnosis of temporomandibular joint internal derangement and osteoarthrosis. Oral Surg Oral Med Oral Pathol Oral Radiol Endod 91:50–55PubMedCrossRefGoogle Scholar

Copyright information

© Springer-Verlag 2011

Authors and Affiliations

  • Nikolaos Nikitas Giannakopoulos
    • 1
    Email author
  • Peter Rammelsberg
    • 1
  • Lydia Eberhard
    • 1
  • Marc Schmitter
    • 1
  1. 1.Department of ProsthodonticsUniversity of HeidelbergHeidelbergGermany

Personalised recommendations