Health Care Management Science

, Volume 18, Issue 4, pp 431–443 | Cite as

Benchmarking clinical practice in surgery: looking beyond traditional mortality rates

  • Ricardo A. S. Castro
  • Pedro N. Oliveira
  • Conceição Silva Portela
  • Ana S. Camanho
  • João Queiroz e Melo


This paper proposes two new measures to assess performance of surgical practice based on observed mortality: reliability, measured as the area under the ROC curve and a living score, the sum of individual risk among surviving patients, divided by the total number of patients. A Monte Carlo simulation of surgeons’ practice was used for conceptual validation and an analysis of a real-world hospital department was used for managerial validation. We modelled surgical practice as a bivariate distribution function of risk and final state. We sampled 250 distributions, varying the maximum risk each surgeon faced, the distribution of risk among dead patients, the mortality rate and the number of surgeries performed yearly. We applied the measures developed to a Portuguese cardiothoracic department. We found that the joint use of the reliability and living score measures overcomes the limitations of risk adjustedmortality rates, as it enables a different valuation of deaths, according to their risk levels. Reliability favours surgeons with casualties, predominantly, in high values of risk and penalizes surgeons with deaths in relatively low levels of risk. The living score is positively influenced by the maximum risk for which a surgeon yields surviving patients. These measures enable a deeper understanding of surgical practice and, as risk adjusted mortality rates, they rely only on mortality and risk scores data. The case study revealed that the performance of the department analysed could be improved with enhanced policies of risk management, involving the assignment of surgeries based on surgeon’s reliability and living score.


Quality measurement Patient outcomes Simulation and modelling ROC curves 



We thank Professor José António Sarsfield Pereira Cabral for his advice and useful comments on the models used in this work. This work was supported by the scholarship SFRH/BD/76634/2011 and by the HOBE project, PTDC/EGE-GES/112232/2009, both funded by the Portuguese Foundation for Science and Technology (FCT).

Conflict of Interest

The authors declare that they have no conflict of interest.


  1. 1.
    Donabedian A (2005) Evaluating the quality of medical care. Milbank Q 83(4):691–729PubMedCentralCrossRefPubMedGoogle Scholar
  2. 2.
    Hollingsworth B (2008) The measurement of efficiency and productivity of health care delivery. Health Econ 17(10):1107–1128CrossRefPubMedGoogle Scholar
  3. 3.
    Chilingerian JA (2010) Evaluating clinical performance in healthcare services with data envelopment analysis. In: Jones R, Jenkins F (eds) Managing money, measurement and marketing in the allied health professions, 1st edn., chap 14. Radcliffe Publishing LtdGoogle Scholar
  4. 4.
    Chilingerian JA (1995) Evaluating physician efficiency in hospitals: a multivariate analysis of best practices. Eur J Oper Res 80(3):548–574CrossRefMATHGoogle Scholar
  5. 5.
    Chilingerian JA, Sherman HD (1996) Benchmarking physician practice patterns with DEA: a multi-stage approach for cost containment. Ann Oper Res 67:83–116CrossRefGoogle Scholar
  6. 6.
    Chilingerian JA, Sherman HD (2004) Health-care applications: from hospitals to physicians, from productive efficiency to quality frontiers. In: Cooper WW, Seiford LM, Zhu J (eds) Handbook on Data Envelopment Analysis. Kluwer Academic Publisher, Boston/Dordrecht/LondonGoogle Scholar
  7. 7.
    Iezzoni LI, Ash AS, Shwartz M, Daley J, Hughes JS, Mackiernan YD (1996) Judging hospitals by severity adjusted mortality rates: the influence of the severity adjustment method. Am J Public Health 86(10):1379–87PubMedCentralCrossRefPubMedGoogle Scholar
  8. 8.
    Iezzoni LI, Ash AS, Shwartz M, Daley J, Hughes JS, Mackiernan YD (1995) Predicting who dies depends on how severity is measured: implications for evaluating patient outcomes. Ann Intern Med 123(10):763–70CrossRefPubMedGoogle Scholar
  9. 9.
    Thomas JW, Hofer TP (1999) Accuracy of risk adjusted mortality rate as a measure of hospital quality of care. Med Care 37(1)Google Scholar
  10. 10.
    Lilford R, Mohammed AM, Spiegelhalter D, Thomson R (2004) Use and misuse of process and outcome data in managing performance of acute medical care: avoiding institutional stigma. The Lancet 363(9415):1147–1154CrossRefGoogle Scholar
  11. 11.
    Pitches DW, Mohammed AM, Lilford RJ (2007) What is the empirical evidence that hospitals with higher-risk adjusted mortality rates provide poorer quality care? A systematic review of the literature. BMC Health Serv Res 7:91PubMedCentralCrossRefPubMedGoogle Scholar
  12. 12.
    Shojania KG, Forster AJ (2008) Hospital mortality: when failure is not a good measure of success. Can Med Assoc J 179(2):153–157CrossRefGoogle Scholar
  13. 13.
    Rogers CA, Reeves BC, Caputo M, Ganesh JS, Bonser RS, Angelini GD (2004) Control chart methods for monitoring cardiac surgical performance and their interpretation. J Thorac Cardiovasc Surg 128(6):811–819CrossRefPubMedGoogle Scholar
  14. 14.
    Brunelli A, Berrisford RG, Rocco G, Varela G (2009) The European thoracic database project: composite performance score to measure quality of care after major lung resection. Eur J Cardiothorac Surg 35(5):769–74CrossRefPubMedGoogle Scholar
  15. 15.
    Brunelli A, Refai M, Salati M, Pompili C, Sabbatini A (2011) Standardized combined outcome index as an instrument for monitoring performance after pulmonary resection. Ann Thorac Surg 92(1):272–7CrossRefPubMedGoogle Scholar
  16. 16.
    Braga AC, Oliveira P (2003) Diagnostic analysis based on ROC curves: theory and applications in medicine. Int J Health Care Qual Assur 16(4):191–198CrossRefGoogle Scholar
  17. 17.
    Roques F, Nashef SAM, Michel P, Gauducheau E, de Vincentiis C, Baudet E, Cortina J, David M, Faichney A, Gabrielle F, Gams E, Harjula A, Jones MT, Pintor PP, Salamon R, Thulin L (1999) Risk factors and outcome in European cardiac surgery: analysis of the EuroSCORE multinational database of 19030 patients. Eur J Cardiothorac Surg 15(6): 816–22; discussion 822–3CrossRefPubMedGoogle Scholar
  18. 18.
    Nashef SAM, Roques F, Michel P, Gauducheau E, Lemeshow S, Salamon R (1999) European system for cardiac operative risk evaluation (EuroSCORE). Eur J Cardiothorac Surg 16(1):9–13CrossRefPubMedGoogle Scholar
  19. 19.
    Geissler HJ, Hölzl P, Marohl S, Kuhn-Régnier F, Mehlhorn U, Südkamp M, De Vivie ER (2000) Risk stratification in heart surgery: comparison of six score systems. Eur J Cardiothorac Surg 17(4):400–406CrossRefPubMedGoogle Scholar
  20. 20.
    Nilsson J, Algotsson L, Höglund P, Lührs C, Brandt J (2006) Comparison of 19 pre-operative risk stratification models in open-heart surgery. Eur Heart J 27(7):867–874CrossRefPubMedGoogle Scholar
  21. 21.
    Metz CE (1978) Basic principles of ROC analysis. Semin Nucl Med 8(4):283–298CrossRefPubMedGoogle Scholar
  22. 22.
    Hanley JA, McNeil BJ (1982) The meaning and use of the area under a Receiver Operating Characteristic (ROC) curve. Radiology 143:29–36CrossRefPubMedGoogle Scholar
  23. 23.
    Green DM, Swets JA (1966) Signal detection theory and psychophysics. Wiley, New YorkGoogle Scholar
  24. 24.
    Bamber D (1975) The area above the ordinal dominance graph and the area below the receiver operating characteristic graph. J Math Psychol 12(4):387–415MathSciNetCrossRefMATHGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2014

Authors and Affiliations

  • Ricardo A. S. Castro
    • 1
  • Pedro N. Oliveira
    • 2
  • Conceição Silva Portela
    • 3
  • Ana S. Camanho
    • 1
  • João Queiroz e Melo
    • 4
  1. 1.Faculdade de Engenharia da Universidade do Porto: Rua Dr. Roberto Frias, s/nPortoPortugal
  2. 2.Instituto de Ciências Biomédicas Abel Salazar da Universidade do Porto: Rua de Jorge Viterbo Ferreira, no. 228PortoPortugal
  3. 3.CEGE - Centros de Estudos em Gestão e Economia da UniversidadeCatólica Portuguesa - Centro Regional do Porto: Rua Diogo Botelho, no. 1327PortoPortugal
  4. 4.ICS - Instituto de Ciências da Saúde da Universidade Católica Portuguesa - Centro Regional do Porto: Rua Dr. Ant´onio Bernardino de Almeida, s/nPortoPortugal

Personalised recommendations