Abstract
Purpose
More recent severity scores should be more reliable than older ones because they account for the improvement in medical care over time. To provide more insight into this issue, we compared the predictive ability of the Simplified Acute Physiology Score (SAPS) II and SAPS 3 (originally developed from data collected in 1991–1992 and 2002, respectively) on a sample of critically ill patients.
Methods
This was a prospective observational study on 3,661 patients from 103 Italian intensive care units. Standardized mortality ratios (SMRs) were calculated. Assessment of calibration across risk classes was performed using the GiViTI calibration belt. Discrimination was evaluated by means of the area under the receiver operating characteristic analysis.
Results
Both scores were shown to discriminate fairly. SAPS 3 largely overpredicted mortality, more than SAPS II (SMR 0.63, 95 % CI 0.60–0.66 vs. 0.87, 95 % CI 0.83–0.91). This result was consistent and statistically significant across all risk classes for SAPS 3. SAPS II did not show relevant deviations from ideal calibration in the first two deciles of risk, whereas in higher-risk classes it overpredicted mortality.
Conclusions
Both scores provided unreliable predictions, but unexpectedly the newer SAPS 3 turned out to overpredict mortality more than the older SAPS II.
Similar content being viewed by others
References
Knaus WA, Wagner DP, Draper EA, Zimmerman JE, Bergner M, Bastos PG, Sirio CA, Murphy DJ, Lotring T, Damiano A et al (1991) The APACHE III prognostic system. Risk prediction of hospital mortality for critically ill hospitalized adults. Chest 100:1619–1636
Zimmerman JE, Kramer AA, McNair DS, Malila FM (2006) Acute Physiology and Chronic Health Evaluation (APACHE) IV: hospital mortality assessment for today’s critically ill patients. Crit Care Med 34:1297–1310
Sirio CA, Shepardson LB, Rotondi AJ, Cooper GS, Angus DC, Harper DL, Rosenthal GE (1999) Community-wide assessment of intensive care outcomes using a physiologically based prognostic measure: implications for critical care delivery from Cleveland health quality choice. Chest 115:793–801
Minne L, Eslami S, de Keizer N, de Jonge E, de Rooij SE, Abu-Hanna A (2012) Effect of changes over time in the performance of a customized SAPS-II model on the quality of care assessment. Intensive Care Med 38:40–46
Le Gall JR, Lemeshow S, Saulnier F (1993) A new Simplified Acute Physiology Score (SAPS II) based on a European/North American multicenter study. JAMA 270:2957–2963
Metnitz PG, Moreno RP, Almeida E, Jordan B, Bauer P, Campos RA, Iapichino G, Edbrooke D, Capuzzo M, Le Gall JR (2005) SAPS 3–From evaluation of the patient to evaluation of the intensive care unit. Part 1: objectives, methods and cohort description. Intensive Care Med 31:1336–1344
Capuzzo M, Moreno RP, Le Gall JR (2008) Outcome prediction in critical care: the simplified acute physiology score models. Curr Opin Crit Care 14:485–490
Boffelli S, Rossi C, Anghileri A, Giardino M, Carnevale L, Messina M, Neri M, Langer M, Bertolini G; Italian Group for the Evaluation of Interventions in Intensive Care Medicine (2006) Continuous quality improvement in intensive care medicine. The GiViTI Margherita Project—Report 2005. Minerva Anestesiol 72:419–432
Poole D, Rossi C, Anghileri A, Giardino M, Latronico N, Radrizzani D, Langer M, Bertolini G (2009) External validation of the Simplified Acute Physiology Score (SAPS) 3 in a cohort of 28,357 patients from 147 Italian intensive care units. Intensive Care Med 35:1916–1924
Moreno RP, Metnitz PG, Almeida E, Jordan B, Bauer P, Campos RA, Iapichino G, Edbrooke D, Capuzzo M, Le Gall JR (2005) SAPS 3–From evaluation of the patient to evaluation of the intensive care unit. Part 2: Development of a prognostic model for hospital mortality at ICU admission. Intensive Care Med 31:1345–1355
Campbell G (1994) Advances in statistical methodology for the evaluation of diagnostic and laboratory tests. Stat Med 13:499–508
Lemeshow S, Hosmer DW Jr (1982) A review of goodness of fit statistics for use in the development of logistic regression models. Am J Epidemiol 115:92–106
Bertolini G, D’Amico R, Nardi D, Tinazzi A, Apolone G (2000) One model, several results: the paradox of the Hosmer–Lemeshow goodness-of-fit test for the logistic regression model. J Epidemiol Biostat 5:251–253
Kramer AA, Zimmerman JE (2007) Assessing the calibration of mortality benchmarks in critical care: the Hosmer-Lemeshow test revisited. Crit Care Med 35:2052–2056
Cox D (1958) Two further applications of a model for a method of binary regression. Biometrika 45:562–565
Finazzi S, Poole D, Luciani D, Cogo PE, Bertolini G (2011) Calibration belt for quality-of-care assessment based on dichotomous outcomes. PLoS ONE 6:e16110
Moreno R, Apolone G, Miranda DR (1998) Evaluation of the uniformity of fit of general outcome prediction models. Intensive Care Med 24:40–47
Strand K, Soreide E, Aardal S, Flaatten H (2009) A comparison of SAPS II and SAPS 3 in a Norwegian intensive care unit population. Acta Anaesthesiol Scand 53:595–600
Capuzzo M, Scaramuzza A, Vaccarini B, Gilli G, Zannoli S, Farabegoli L, Felisatti G, Davanzo E, Alvisi R (2009) Validation of SAPS 3 Admission Score and comparison with SAPS II. Acta Anaesthesiol Scand 53:589–594
Soares M, Silva UV, Teles JM, Silva E, Caruso P, Lobo SM, Dal Pizzol F, Azevedo LP, de Carvalho FB, Salluh JI (2010) Validation of four prognostic scores in patients with cancer admitted to Brazilian intensive care units: results from a prospective multicenter study. Intensive Care Med 36:1188–1195
Ledoux D, Canivet JL, Preiser JC, Lefrancq J, Damas P (2008) SAPS 3 admission score: an external validation in a general intensive care population. Intensive Care Med 34:1873–1877
Murphy-Filkins R, Teres D, Lemeshow S, Hosmer DW (1996) Effect of changing patient mix on the performance of an intensive care unit severity-of-illness model: how to distinguish a general from a specialty intensive care unit. Crit Care Med 24:1968–1973
Miller ME, Hui SL, Tierney WM (1991) Validation techniques for logistic regression models. Stat Med 10:1213–1226
Metnitz PG, Lang T, Vesely H, Valentin A, Le Gall JR (2000) Ratios of observed to expected mortality are affected by differences in case mix and quality of care. Intensive Care Med 26:1466–1472
Nimgaonkar A, Karnad DR, Sudarshan S, Ohno-Machado L, Kohane I (2004) Prediction of mortality in an Indian intensive care unit. Comparison between APACHE II and artificial neural networks. Intensive Care Med 30:248–253
Zimmerman JE, Wagner DP, Draper EA, Wright L, Alzola C, Knaus WA (1998) Evaluation of acute physiology and chronic health evaluation III predictions of hospital mortality in an independent database. Crit Care Med 26:1317–1326
Babyak MA (2004) What you see may not be what you get: a brief, nontechnical introduction to overfitting in regression-type models. Psychosom Med 66:411–421
Acknowledgments
The study was wholly funded by GiViTI-Istituto di Ricerche Farmacologiche Mario Negri, Bergamo (Italy). GiViTI is the recipient of unconditioned grants from Bellco, Brahms, and Astellas, which did not, however, have any role in this study. The authors would like to thank Abramo Anghileri and Michele Giardino (Istituto di Ricerche Farmacologiche Mario Negri) for their contribution in developing and maintaining the data collection software.
Conflicts of interest
The authors declare they have no conflict of interest regarding the content of the manuscript.
Author information
Authors and Affiliations
Consortia
Corresponding author
Additional information
This article is discussed in the editorial available at:doi: 10.1007/s00134-012-2579-z.
The complete list of study participants appears in the Appendix.
Electronic supplementary material
Below is the link to the electronic supplementary material.
Appendix
Appendix
GiViTI steering committee members
Guido Bertolini (Ranica, BG); Daniela Boccalatte (Lucca); Arturo Chieregato (Cesena); Daniela Codazzi (Bari); Roberto Fumagalli (Monza); Giorgio Gambale (Forlì); Martin Langer (Milano); Sergio Livigni (Torino); Paolo Malacarne (Pisa); Giuseppe Nardi (Roma); Daniele Poole (Belluno); Danilo Radrizzani (Legnano, MI); Mario Tavola (Lecco).
List of study participants
Marco Abastanotti (Manerbio—BS), Claudia Acciari (Ascoli Piceno—AP), Annamaria Acquarolo (Brescia—BS), Arnaldo Alberti (Porto Viro—RO), Armando Alborghetti (Ponte San Pietro—BG), Paolo Antolini (Fermo—AP), Enrico Arditi (Genova—GE), Massimo Barattini (Firenze—FI), Arcangelo Bartoccini (Borgomanero—NO), Teresa Bartoli (Antella—FI), Francesco Bassi (Cinisello Balsamo—MI), Eduardo Beck (Desio—MI), Uberto Belloni (Ivrea—TO), Mario Bellorini (Tradate—VA), Cesare Benanti (Pescia—PT), Mara Olga Bernasconi (Rovigo—RO), Andrea Bianchin (Montebelluna—TV), Angelo Blasetti (Avezzano—AQ), Daniela Boccalatte (Lucca—LU), Andrea Bonfà (Prato—PO), Monica Bonfiglio (Lavagna—GE), Cesare Breschi (Pesaro—PU), Emanuela Brunori (Macerata—MC), Roberto Buonanno (Lacco Ameno—NA), Virginio Buzzetti (Orbetello Scalo—GR), Gian Virgilio Cabano (Busto Arsizio—VA), Giuseppe Calicchio (Salerno—SA), Livio Carnevale (Pavia—PV), Lucia Casagrande (Treviso—TV), Pierpaolo Casalini (Faenza—RA), Giacomo Castiglione (Catania—CT), Elena Chinelli (Bologna—BO), Maddalena Coaloa (Savigliano—CN), Riccardo Colombo (Milano—MI), Sergio Colombo (Milano—MI), Walter Corsini (Carrara—MS), Eleonora Costanzo (Asti—AT), Tito Cottignoli (Lugo di Romagna—RA), Luciano Crema (Cremona—CR), Dolores Da Re (Este—PD), Roberto Alberto De Blasi (Roma—RM), Elvio De Blasio (Benevento—BN), Marino Del Ferro (Oderzo—TV), Pierfrancesco Di Masi (Castellana Grotte—BA), Dino Di Pasquale (Pontedera—PI), Ivano Donato (Cantù—CO), Lea Paola Fabbri (Firenze—FI), Emilio Fabbri (Forlì—FC), Maria Cristina Fabi (Fano—PU), Luciana Faccio (Torino—TO), Paolo Febbrari (Brescia—BS), Francesco Ferla (Cefalù—PA), Gilberto Fiore (Moncalieri—TO), Claudio Fogliati (Orbassano—TO), Emiliano Gamberini (Cesena—FC), Alberto Garelli (Ravenna—RA), Massimo Gianni (Aosta—AO), Stefano Giannoni (Empoli—FI), Daniela Giudici (Milano—MI), Pier Mario Giugiaro (Ciriè—TO), Maurizio Greco (Ariano Irpino—AV), Michele Isetta (Genova—GE), Luigi Lacquaniti (Rossano—CS), Sergio Livigni (Torino—TO), Roberto Madonna (Grosseto—GR), Maria Federica Magatti (Como—CO), Paolo Malacarne (Pisa—PI), Alba Malinconi (Este—PD), Stefano Mancosu (Cagliari—CA), Valerio Mangani (Firenze—FI), Giorgio Mantovani (Ferrara—FE), Alessandro Mastroianni (Chieri—TO), Marianna Messina (Olbia—OT), Renzo Miglioranzi (Pieve di Coriano—MN), Salvatore Murru (Cagliari—CA), Giuseppe Nardi (Roma—RM), Giuseppe Natalini (Brescia—BS), Lorenzo Odetto (Orbassano—TO), Carlo Oliviero (Novara—NO), Daniela Maria Pacini (Genova—GE), Giorgio Paganini (Sanremo—IM), Maurizio Palmer (Aversa—CE), Luigi Panella (L’Aquila—AQ), Silvano Papiri (San Benedetto del Tronto—AP), Mauro Pastorelli (Pinerolo—TO), Maurizio Pegoraro (Castelfranco Veneto—TV), Angelo Pezzi (Milano—MI), Donatella Pinciroli (Legnano—MI), Marialuisa Pizzaballa (Zingonia—BG), Maurizio Postiglione (Napoli—NA), Massimo Raffaeli (Milano—MI), Paolo Riverso (Sanremo—IM), Mario Romitti (Piacenza—PC), Nicola Rossi (Monza—MI), Maurizio Rossi (Menaggio—CO), Federica Rottoli (Bergamo—BG), Rosa Salcuni (Ivrea—TO), Monica Savioli (Milano—MI), Silvia Scarrone (Alessandria—AL), Vincenzo Segala (Torino—TO), Pieraurelio Segalini (Piacenza—PC), Alberto Sicignano (Milano—MI), Maria Josè Sucre (Castellammare di Stabia—NA), Claudio Tagliaferri (Piacenza—PC), Romano Tetamo (Palermo—PA), Giuseppe Tibaldi (Brescia—BS), Silvia Tinacci (Fucecchio—FI), Livio Todesco (Cittadella—PD), Gabriele Tomasoni (Manerbio—BS), Mauro Torta (Torino—TO), Carlo Tripepi (Monselice—PD), Pietro Vecchiarelli (Viterbo—VT), Maria Grazia Visconti (Cernusco sul Naviglio—MI), Nicoletta Vivaldi (Alessandria—AL), Giuseppe Angelo Vulcano (Rossano—CS), Sergio Zappa (Brescia—BS), Francesco Zuccaro (Matera—MT).
Rights and permissions
About this article
Cite this article
Poole, D., Rossi, C., Latronico, N. et al. Comparison between SAPS II and SAPS 3 in predicting hospital mortality in a cohort of 103 Italian ICUs. Is new always better?. Intensive Care Med 38, 1280–1288 (2012). https://doi.org/10.1007/s00134-012-2578-0
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00134-012-2578-0