Prevention Science

, Volume 14, Issue 5, pp 497–502 | Cite as

How Reliable Are County and Regional Health Rankings?

  • Stephan Arndt
  • Laura Acion
  • Kristin Caspers
  • Phyllis Blood


Surveillance system indicators of morbidity, mortality, or behaviors are used to provide regional information for ever-smaller areas, most recently for ranking counties. These rankings are thought to provide information about the relative standing of regions and provide information about problem areas and the success of programs. We investigate the ability of such rankings to reliably assess health. We assess the reliability of several ranked health indices used at the county level and the consistency of an index's quality across different states. Reliability is assessed using an index of reliability, simulations, and the ability of an index to consistently identify the top 10 % (worst) counties within a state. There is marked variability across measures to provide consistent ranks, across states, and, many times, across states for a particular measure. A few health measures do consistently well, e.g., Teen Birth, Chlamydia, and Years of Potential Life Lost rates. While a few health rankings appear worthy of use for policy and in the identification of local problems, in general, they are not consistent across regions.


Health Services Research Quality Indicator Health Status Indicators Health Policy 



This project was funded, in part, by the Iowa Consortium for Substance Abuse Research and Evaluation.

Conflict of Interest

None of the authors have any financial disclosures.


  1. Arndt, S., Acion, L., Caspers, K., & Diallo, O. (2011). Assessing community variation and randomness in public health indicators. Population Health Metrics, 9, 3. doi: 10.1186/1478-7954-9-3.PubMedCrossRefGoogle Scholar
  2. Buehler, J. W., & Holtgrave, D. R. (2007). Challenges in defining an optimal approach to formula-based allocations of public health funds in the United States. BMC Public Health, 7, 44. doi: 10.1186/1471-2458-7-44.PubMedCrossRefGoogle Scholar
  3. Cochran, W. G. (1954). The combination of estimates from different experiments. Biometrics, 10, 101–129.CrossRefGoogle Scholar
  4. Cochran, W. G. (1983). Planning and analysis of observational studies. New York: John Wiley & Sons.CrossRefGoogle Scholar
  5. DerSimonian, R., & Laird, N. (1986). Meta-analysis in clinical trials. Controlled Clinical Trials, 7, 177–188.PubMedCrossRefGoogle Scholar
  6. Evans, M., Hastings, N., & Peacock, B. (1993). Statistical distributions (2nd ed.). New York: John Wiley & Sons.Google Scholar
  7. Flowers, J., Hall, P., & Pencheon, D. (2005). Mini-symposium—public health observatories: Public health indicators. Public Health, 119, 239–245.PubMedCrossRefGoogle Scholar
  8. Guilford, J. P. (1954). Psychometric methods. New York: McGraw-Hill.Google Scholar
  9. Hall, P., & Miller, H. (2010). Modeling the variability of ranks. The Annals of Statistics, 38, 2652–2677.CrossRefGoogle Scholar
  10. Higgins, J. P., & Thompson, S. G. (2002). Quantifying heterogeneity in a meta-analysis. Statistics in Medicine, 21, 1539–1558. doi: 10.1002/sim.1186.PubMedCrossRefGoogle Scholar
  11. Higgins, J. P. T., Thompson, S. G., Deeks, J. J., & Altman, D. G. (2003). Measuring inconsistency in meta-analyses. British Medical Journal, 327, 557–560. doi: 10.1136/bmj.327.7414.557.PubMedCrossRefGoogle Scholar
  12. Hofer, T. P., Hayward, R. A., Greenfield, S., Wagner, E. H., Kaplan, S. H., & Manning, W. G. (1999). The unreliability of individual physician “report cards” for assessing the costs and quality of care of a chronic disease. JAMA: The Journal of the American Medical Association, 281, 2098–2105. doi: 10.1001/jama.281.22.2098.CrossRefGoogle Scholar
  13. Hofer, T. P., & Haywood, R. A. (1996). Identifying poor-quality hospitals: Can hospital mortality rates detect quality problems for medical diagnoses? Medical Care, 34, 737–753.PubMedCrossRefGoogle Scholar
  14. Huedo-Medina, T. B., Sánchez-Meca, J., Marín-Martínez, F., & Botella, J. (2006). Assessing heterogeneity in meta-analysis: Q statistic or I² index? Psychological Methods, 11, 193–206. doi: 10.1037/1082-989x.11.2.193.PubMedCrossRefGoogle Scholar
  15. Jacobs, R., Goddard, M., & Smith, P. C. (2005). How robust are hospital ranks based on composite performance measures? Medical Care, 43, 1177–1184.PubMedCrossRefGoogle Scholar
  16. Kephart, G., & Asada, Y. (2009). Need-based resource allocation: Different need indicators, different results? BMC Health Services Research, 9, 122. doi: 10.1186/1472-6963-9-122.PubMedCrossRefGoogle Scholar
  17. Krieger, N., Chen, J. T., Waterman, P. D., Soobader, M.-J., Subramanian, S. V., & Carson, R. (2002). Geocoding and monitoring of us socioeconomic inequalities in mortality and cancer incidence: Does the choice of area-based measure and geographic level matter? American Journal of Epidemiology, 156, 471–482. doi: 10.1093/aje/kwf068.PubMedCrossRefGoogle Scholar
  18. Krieger, N., Chen, J. T., Waterman, P. D., Soobader, M.-J., Subramanian, S. V., & Carson, R. (2003). Choosing area based socioeconomic measures to monitor social inequalities in low birth weight and childhood lead poisoning: The public health disparities geocoding project (US). Journal of Epidemiology and Community Health, 57, 186–199. doi: 10.1093/aje/kwf068.PubMedCrossRefGoogle Scholar
  19. Louis, T. A., Jabine, T. B., & Gerstein, M. A. (2003). Statistical issues in allocating funds by formula. Washington, D.C.: National Academies Press.Google Scholar
  20. Marshall, E. C., Sanderson, C., Spiegelhalter, D. J., & McKee, M. (1998). Reliability of league tables of in vitro fertilisation clinics: retrospective analysis of live birth rates Commentary: How robust are rankings? The implications of confidence intervals. British Medical Journal, 316, 1701–1705. doi: 10.1136/bmj.316.7146.1701.PubMedCrossRefGoogle Scholar
  21. McGinnis, J.M. (2010). Observations on incentives to improve population health. Preventing Chronic Disease, 9, A92. Retreived from
  22. Mokdad, A. H., & Remington, P. L. (2010). Measuring health behaviors in populations. Preventing Chronic Disease, 7, A75.PubMedGoogle Scholar
  23. Muhuri, P.K., & Ducrest, J.L. (2007). Block grants and formula grants: A guide for allotment calculations. Rockville, MD: SAMHSA, Office of Applied Studies Retrieved from
  24. Nunnally, J. C. (1967). Psychometric theory. New York: McGraw-Hill.Google Scholar
  25. O'Brien, S. M., & Peterson, E. D. (2007). Identifying high-quality hospitals: Consult the ratings or flip a coin? Archives of Internal Medicine, 167, 1342–1344. doi: 10.1001/archinte.167.13.1342.PubMedCrossRefGoogle Scholar
  26. Oliver, T.R. (2010). Population health rankings as policy indicators and performance measures. Preventing Chronic Disease, 5, A101. Retrieved from
  27. Page, S., & Cramer, K. (2001). Maclean’s rankings of health care indices in canadian communities, 2000: Comparisons and statistical contrivance. Canadian Journal of Public Health, 92, 295–298.PubMedGoogle Scholar
  28. Peppard, P. E., Kindig, D. A., Dranger, E., Jovaag, A., & Remington, P. L. (2008). Ranking community health status to stimulate discussion of local public health issues: The Wisconsin county health rankings. American Journal of Public Health, 98, 209–212. doi: 10.2105/AJPH.2006.092981.PubMedCrossRefGoogle Scholar
  29. Remington, P. L., & Booske, B. C. (2011). Measuring the health of communities—how and why? Journal of Public Health Management and Practice, 17, 397–400. doi: 10.1097/PHH.0b013e318222b897.PubMedCrossRefGoogle Scholar
  30. Rohan, A. M. K., Booske, B. C., & Remington, P. L. (2009). Using the Wisconsin county health rankings to catalyze community health improvement. Journal of Public Health Management and Practice, 15, 24–32. doi: 10.1097/PHH.1090b1013e3181903bf3181908.PubMedCrossRefGoogle Scholar
  31. Vila, P., Booske, B., & Remington, P. (2006). Measuring mortality in the Wisconsin county health. Population Health Institute Technical Report., (Department of Population Health Sciences). Retrieved from

Copyright information

© Society for Prevention Research 2013

Authors and Affiliations

  • Stephan Arndt
    • 1
    • 2
    • 3
  • Laura Acion
    • 3
  • Kristin Caspers
    • 3
    • 4
  • Phyllis Blood
    • 5
  1. 1.Department of Psychiatry, Carver College of MedicineUniversity of IowaIowa CityUSA
  2. 2.Department of Biostatistics, College of Public HealthUniversity of IowaIowa CityUSA
  3. 3.Iowa Consortium for Substance Abuse Research and EvaluationUniversity of IowaIowa CityUSA
  4. 4.Department of Epidemiology, College of Public HealthUniversity of IowaIowa CityUSA
  5. 5.Division of Criminal and Juvenile Justice PlanningIowa Department of Human RightsDes MoinesUSA

Personalised recommendations