# Benchmarking clinical practice in surgery: looking beyond traditional mortality rates

## Abstract

This paper proposes two new measures to assess performance of surgical practice based on observed mortality: reliability, measured as the area under the ROC curve and a living score, the sum of individual risk among surviving patients, divided by the total number of patients. A Monte Carlo simulation of surgeons’ practice was used for conceptual validation and an analysis of a real-world hospital department was used for managerial validation. We modelled surgical practice as a bivariate distribution function of risk and final state. We sampled 250 distributions, varying the maximum risk each surgeon faced, the distribution of risk among dead patients, the mortality rate and the number of surgeries performed yearly. We applied the measures developed to a Portuguese cardiothoracic department. We found that the joint use of the reliability and living score measures overcomes the limitations of risk adjustedmortality rates, as it enables a different valuation of deaths, according to their risk levels. Reliability favours surgeons with casualties, predominantly, in high values of risk and penalizes surgeons with deaths in relatively low levels of risk. The living score is positively influenced by the maximum risk for which a surgeon yields surviving patients. These measures enable a deeper understanding of surgical practice and, as risk adjusted mortality rates, they rely only on mortality and risk scores data. The case study revealed that the performance of the department analysed could be improved with enhanced policies of risk management, involving the assignment of surgeries based on surgeon’s reliability and living score.

### Keywords

Quality measurement Patient outcomes Simulation and modelling ROC curves## Notes

### Acknowledgments

We thank Professor José António Sarsfield Pereira Cabral for his advice and useful comments on the models used in this work. This work was supported by the scholarship SFRH/BD/76634/2011 and by the HOBE project, PTDC/EGE-GES/112232/2009, both funded by the Portuguese Foundation for Science and Technology (FCT).

**Conflict of Interest**

The authors declare that they have no conflict of interest.

### References

- 1.Donabedian A (2005) Evaluating the quality of medical care. Milbank Q 83(4):691–729PubMedCentralCrossRefPubMedGoogle Scholar
- 2.Hollingsworth B (2008) The measurement of efficiency and productivity of health care delivery. Health Econ 17(10):1107–1128CrossRefPubMedGoogle Scholar
- 3.Chilingerian JA (2010) Evaluating clinical performance in healthcare services with data envelopment analysis. In: Jones R, Jenkins F (eds) Managing money, measurement and marketing in the allied health professions, 1
^{st}edn., chap 14. Radcliffe Publishing LtdGoogle Scholar - 4.Chilingerian JA (1995) Evaluating physician efficiency in hospitals: a multivariate analysis of best practices. Eur J Oper Res 80(3):548–574CrossRefMATHGoogle Scholar
- 5.Chilingerian JA, Sherman HD (1996) Benchmarking physician practice patterns with DEA: a multi-stage approach for cost containment. Ann Oper Res 67:83–116CrossRefGoogle Scholar
- 6.Chilingerian JA, Sherman HD (2004) Health-care applications: from hospitals to physicians, from productive efficiency to quality frontiers. In: Cooper WW, Seiford LM, Zhu J (eds) Handbook on Data Envelopment Analysis. Kluwer Academic Publisher, Boston/Dordrecht/LondonGoogle Scholar
- 7.Iezzoni LI, Ash AS, Shwartz M, Daley J, Hughes JS, Mackiernan YD (1996) Judging hospitals by severity adjusted mortality rates: the influence of the severity adjustment method. Am J Public Health 86(10):1379–87PubMedCentralCrossRefPubMedGoogle Scholar
- 8.Iezzoni LI, Ash AS, Shwartz M, Daley J, Hughes JS, Mackiernan YD (1995) Predicting who dies depends on how severity is measured: implications for evaluating patient outcomes. Ann Intern Med 123(10):763–70CrossRefPubMedGoogle Scholar
- 9.Thomas JW, Hofer TP (1999) Accuracy of risk adjusted mortality rate as a measure of hospital quality of care. Med Care 37(1)Google Scholar
- 10.Lilford R, Mohammed AM, Spiegelhalter D, Thomson R (2004) Use and misuse of process and outcome data in managing performance of acute medical care: avoiding institutional stigma. The Lancet 363(9415):1147–1154CrossRefGoogle Scholar
- 11.Pitches DW, Mohammed AM, Lilford RJ (2007) What is the empirical evidence that hospitals with higher-risk adjusted mortality rates provide poorer quality care? A systematic review of the literature. BMC Health Serv Res 7:91PubMedCentralCrossRefPubMedGoogle Scholar
- 12.Shojania KG, Forster AJ (2008) Hospital mortality: when failure is not a good measure of success. Can Med Assoc J 179(2):153–157CrossRefGoogle Scholar
- 13.Rogers CA, Reeves BC, Caputo M, Ganesh JS, Bonser RS, Angelini GD (2004) Control chart methods for monitoring cardiac surgical performance and their interpretation. J Thorac Cardiovasc Surg 128(6):811–819CrossRefPubMedGoogle Scholar
- 14.Brunelli A, Berrisford RG, Rocco G, Varela G (2009) The European thoracic database project: composite performance score to measure quality of care after major lung resection. Eur J Cardiothorac Surg 35(5):769–74CrossRefPubMedGoogle Scholar
- 15.Brunelli A, Refai M, Salati M, Pompili C, Sabbatini A (2011) Standardized combined outcome index as an instrument for monitoring performance after pulmonary resection. Ann Thorac Surg 92(1):272–7CrossRefPubMedGoogle Scholar
- 16.Braga AC, Oliveira P (2003) Diagnostic analysis based on ROC curves: theory and applications in medicine. Int J Health Care Qual Assur 16(4):191–198CrossRefGoogle Scholar
- 17.Roques F, Nashef SAM, Michel P, Gauducheau E, de Vincentiis C, Baudet E, Cortina J, David M, Faichney A, Gabrielle F, Gams E, Harjula A, Jones MT, Pintor PP, Salamon R, Thulin L (1999) Risk factors and outcome in European cardiac surgery: analysis of the EuroSCORE multinational database of 19030 patients. Eur J Cardiothorac Surg 15(6): 816–22; discussion 822–3CrossRefPubMedGoogle Scholar
- 18.Nashef SAM, Roques F, Michel P, Gauducheau E, Lemeshow S, Salamon R (1999) European system for cardiac operative risk evaluation (EuroSCORE). Eur J Cardiothorac Surg 16(1):9–13CrossRefPubMedGoogle Scholar
- 19.Geissler HJ, Hölzl P, Marohl S, Kuhn-Régnier F, Mehlhorn U, Südkamp M, De Vivie ER (2000) Risk stratification in heart surgery: comparison of six score systems. Eur J Cardiothorac Surg 17(4):400–406CrossRefPubMedGoogle Scholar
- 20.Nilsson J, Algotsson L, Höglund P, Lührs C, Brandt J (2006) Comparison of 19 pre-operative risk stratification models in open-heart surgery. Eur Heart J 27(7):867–874CrossRefPubMedGoogle Scholar
- 21.Metz CE (1978) Basic principles of ROC analysis. Semin Nucl Med 8(4):283–298CrossRefPubMedGoogle Scholar
- 22.Hanley JA, McNeil BJ (1982) The meaning and use of the area under a Receiver Operating Characteristic (ROC) curve. Radiology 143:29–36CrossRefPubMedGoogle Scholar
- 23.Green DM, Swets JA (1966) Signal detection theory and psychophysics. Wiley, New YorkGoogle Scholar
- 24.Bamber D (1975) The area above the ordinal dominance graph and the area below the receiver operating characteristic graph. J Math Psychol 12(4):387–415MathSciNetCrossRefMATHGoogle Scholar