This paper proposes two new measures to assess performance of surgical practice based on observed mortality: reliability, measured as the area under the ROC curve and a living score, the sum of individual risk among surviving patients, divided by the total number of patients. A Monte Carlo simulation of surgeons’ practice was used for conceptual validation and an analysis of a real-world hospital department was used for managerial validation. We modelled surgical practice as a bivariate distribution function of risk and final state. We sampled 250 distributions, varying the maximum risk each surgeon faced, the distribution of risk among dead patients, the mortality rate and the number of surgeries performed yearly. We applied the measures developed to a Portuguese cardiothoracic department. We found that the joint use of the reliability and living score measures overcomes the limitations of risk adjustedmortality rates, as it enables a different valuation of deaths, according to their risk levels. Reliability favours surgeons with casualties, predominantly, in high values of risk and penalizes surgeons with deaths in relatively low levels of risk. The living score is positively influenced by the maximum risk for which a surgeon yields surviving patients. These measures enable a deeper understanding of surgical practice and, as risk adjusted mortality rates, they rely only on mortality and risk scores data. The case study revealed that the performance of the department analysed could be improved with enhanced policies of risk management, involving the assignment of surgeries based on surgeon’s reliability and living score.
Quality measurement Patient outcomes Simulation and modelling ROC curves
We thank Professor José António Sarsfield Pereira Cabral for his advice and useful comments on the models used in this work. This work was supported by the scholarship SFRH/BD/76634/2011 and by the HOBE project, PTDC/EGE-GES/112232/2009, both funded by the Portuguese Foundation for Science and Technology (FCT).
Conflict of Interest
The authors declare that they have no conflict of interest.
Hollingsworth B (2008) The measurement of efficiency and productivity of health care delivery. Health Econ 17(10):1107–1128CrossRefPubMedGoogle Scholar
Chilingerian JA (2010) Evaluating clinical performance in healthcare services with data envelopment analysis. In: Jones R, Jenkins F (eds) Managing money, measurement and marketing in the allied health professions, 1st edn., chap 14. Radcliffe Publishing LtdGoogle Scholar
Chilingerian JA (1995) Evaluating physician efficiency in hospitals: a multivariate analysis of best practices. Eur J Oper Res 80(3):548–574CrossRefMATHGoogle Scholar
Chilingerian JA, Sherman HD (1996) Benchmarking physician practice patterns with DEA: a multi-stage approach for cost containment. Ann Oper Res 67:83–116CrossRefGoogle Scholar
Chilingerian JA, Sherman HD (2004) Health-care applications: from hospitals to physicians, from productive efficiency to quality frontiers. In: Cooper WW, Seiford LM, Zhu J (eds) Handbook on Data Envelopment Analysis. Kluwer Academic Publisher, Boston/Dordrecht/LondonGoogle Scholar
Iezzoni LI, Ash AS, Shwartz M, Daley J, Hughes JS, Mackiernan YD (1996) Judging hospitals by severity adjusted mortality rates: the influence of the severity adjustment method. Am J Public Health 86(10):1379–87PubMedCentralCrossRefPubMedGoogle Scholar
Iezzoni LI, Ash AS, Shwartz M, Daley J, Hughes JS, Mackiernan YD (1995) Predicting who dies depends on how severity is measured: implications for evaluating patient outcomes. Ann Intern Med 123(10):763–70CrossRefPubMedGoogle Scholar
Thomas JW, Hofer TP (1999) Accuracy of risk adjusted mortality rate as a measure of hospital quality of care. Med Care 37(1)Google Scholar
Lilford R, Mohammed AM, Spiegelhalter D, Thomson R (2004) Use and misuse of process and outcome data in managing performance of acute medical care: avoiding institutional stigma. The Lancet 363(9415):1147–1154CrossRefGoogle Scholar
Pitches DW, Mohammed AM, Lilford RJ (2007) What is the empirical evidence that hospitals with higher-risk adjusted mortality rates provide poorer quality care? A systematic review of the literature. BMC Health Serv Res 7:91PubMedCentralCrossRefPubMedGoogle Scholar
Shojania KG, Forster AJ (2008) Hospital mortality: when failure is not a good measure of success. Can Med Assoc J 179(2):153–157CrossRefGoogle Scholar
Rogers CA, Reeves BC, Caputo M, Ganesh JS, Bonser RS, Angelini GD (2004) Control chart methods for monitoring cardiac surgical performance and their interpretation. J Thorac Cardiovasc Surg 128(6):811–819CrossRefPubMedGoogle Scholar
Brunelli A, Berrisford RG, Rocco G, Varela G (2009) The European thoracic database project: composite performance score to measure quality of care after major lung resection. Eur J Cardiothorac Surg 35(5):769–74CrossRefPubMedGoogle Scholar
Brunelli A, Refai M, Salati M, Pompili C, Sabbatini A (2011) Standardized combined outcome index as an instrument for monitoring performance after pulmonary resection. Ann Thorac Surg 92(1):272–7CrossRefPubMedGoogle Scholar
Braga AC, Oliveira P (2003) Diagnostic analysis based on ROC curves: theory and applications in medicine. Int J Health Care Qual Assur 16(4):191–198CrossRefGoogle Scholar
Roques F, Nashef SAM, Michel P, Gauducheau E, de Vincentiis C, Baudet E, Cortina J, David M, Faichney A, Gabrielle F, Gams E, Harjula A, Jones MT, Pintor PP, Salamon R, Thulin L (1999) Risk factors and outcome in European cardiac surgery: analysis of the EuroSCORE multinational database of 19030 patients. Eur J Cardiothorac Surg 15(6): 816–22; discussion 822–3CrossRefPubMedGoogle Scholar
Nashef SAM, Roques F, Michel P, Gauducheau E, Lemeshow S, Salamon R (1999) European system for cardiac operative risk evaluation (EuroSCORE). Eur J Cardiothorac Surg 16(1):9–13CrossRefPubMedGoogle Scholar
Geissler HJ, Hölzl P, Marohl S, Kuhn-Régnier F, Mehlhorn U, Südkamp M, De Vivie ER (2000) Risk stratification in heart surgery: comparison of six score systems. Eur J Cardiothorac Surg 17(4):400–406CrossRefPubMedGoogle Scholar