Advertisement

Intensive Care Medicine

, Volume 39, Issue 5, pp 942–950 | Cite as

Mortality prediction models for pediatric intensive care: comparison of overall and subgroup specific performance

  • Idse H. E. Visser
  • Jan A. Hazelzet
  • Marcel J. I. J. Albers
  • Carin W. M. Verlaat
  • Karin Hogenbirk
  • Job B. van Woensel
  • Marc van Heerde
  • Dick A. van Waardenburg
  • Nicolaas J. G. Jansen
  • Ewout W. Steyerberg
Pediatric Original

Abstract

Aim

To validate paediatric index of mortality (PIM) and pediatric risk of mortality (PRISM) models within the overall population as well as in specific subgroups in pediatric intensive care units (PICUs).

Methods

Variants of PIM and PRISM prediction models were compared with respect to calibration (agreement between predicted risks and observed mortality) and discrimination (area under the receiver operating characteristic curve, AUC). We considered performance in the overall study population and in subgroups, defined by diagnoses, age and urgency at admission, and length of stay (LoS) at the PICU. We analyzed data from consecutive patients younger than 16 years admitted to the eight PICUs in the Netherlands between February 2006 and October 2009. Patients referred to another ICU or deceased within 2 h after admission were excluded.

Results

A total of 12,040 admissions were included, with 412 deaths. Variants of PIM2 were best calibrated. All models discriminated well, also in patients <28 days of age (neonates), with overall higher AUC for PRISM variants (PIM = 0.83, PIM2 = 0.85, PIM2-ANZ06 = 0.86, PIM2-ANZ08 = 0.85, PRISM = 0.88, PRISM3-24 = 0.90). Best discrimination for PRISM3-24 was confirmed in 13 out of 14 subgroup categories. After recalibration PRISM3-24 predicted accurately in most (12 out of 14) categories. Discrimination was poorer for all models (AUC < 0.73) after LoS of >6 days at the PICU.

Conclusion

All models discriminated well, also in most subgroups including neonates, but had difficulties predicting mortality for patients >6 days at the PICU. In a western European setting both the PIM2(-ANZ06) or a recalibrated version of PRISM3-24 are suited for overall individualized risk prediction.

Keywords

Benchmarking Outcome/quality assessment Critical illness/mortality Intensive care units pediatric Risk adjustment Severity of illness index Validation studies Registries Diagnosis-related groups 

Notes

Acknowledgments

We like to thank Hester Lingsma and Yvonne Vergouwe from the Department of Medical Decision Making, Erasmus Medical Center, Rotterdam, The Netherlands, for R scripts to bootstraps and calibration plots, and Douwe R van der Heide and Leo Bakker, pediatric intensive care nurses for data quality control during local site visits.

Supplementary material

134_2013_2857_MOESM1_ESM.doc (4.7 mb)
Supplementary material 1 (DOC 4790 kb)

References

  1. 1.
    Yeh TS, Pollack MM, Ruttimann UE, Holbrook PR, Fields AI (1984) Validation of a physiologic stability index for use in critically ill infants and children. Pediatr Res 18(5):445–451PubMedCrossRefGoogle Scholar
  2. 2.
    Pollack MM, Ruttimann UE, Getson PR (1988) Pediatric risk of mortality (PRISM) score. Crit Care Med 16(11):1110–1116PubMedCrossRefGoogle Scholar
  3. 3.
    Examples of several countries with PICU registries, Australia and New-Zealand: ANZPIC Registry http://www.anzics.com.au/anzpicr. Accessed 8 Feb 2013; Netherlands: Dutch PICE registry http://www.pice.nl; Portugal: REUNIR-CIP http://newdbserver.med.up.pt/web.care/reunir/index.html; Italy: TIPNet http://tipnet.cineca.it; United Kingdom: PICANet http://www.picanet.org.uk; United States of America: the Virtual PICU Systems LLC (VPS) https://portal.myvps.org/
  4. 4.
    Visser DY, Jansen NJ, Ijland MM, de Koning TJ, van Hasselt PM (2011) Intracranial bleeding due to vitamin K deficiency: advantages of using a pediatric intensive care registry. Intensive Care Med 37(6):1014–1020. doi: 10.1007/s00134-011-2175-7 PubMedCrossRefGoogle Scholar
  5. 5.
    Peeters B, Jansen NJ, Bollen CW, van Vught AJ, van der Heide D, Albers MJ (2010) Off-hours admission and mortality in two pediatric intensive care units without 24-h in-house senior staff attendance. Intensive Care Med 36(11):1923–1927. doi: 10.1007/s00134-010-2020-4 PubMedCrossRefGoogle Scholar
  6. 6.
    Latour JM, van Goudoever JB, Duivenvoorden HJ, van Dam NA, Dullaart E, Albers MJ, Verlaat CW, van Vught EM, van Heerde M, Hazelzet JA (2009) Perceptions of parents on satisfaction with care in the pediatric intensive care unit: the EMPATHIC study. Intensive Care Med 35(6):1082–1089. doi: 10.1007/s00134-009-1491-7 PubMedCrossRefGoogle Scholar
  7. 7.
    Visser IHE, PICE-studygroup (2008) Dutch pediatric intensive care evaluation. PICE Report 2003–2005. ISBN 9789078022039. http://www.pice.nl/eng/gegevens-eng.htm. Accessed 10 Apr 2012
  8. 8.
    Gemke RJ, Bonsel GJ (1995) Comparative assessment of pediatric intensive care: a national multicenter study. Pediatric Intensive Care Assessment of Outcome (PICASSO) study group. Crit Care Med 23(2):238–245PubMedCrossRefGoogle Scholar
  9. 9.
    Pollack MM, Patel KM, Ruttimann UE (1996) PRISM III: an updated pediatric risk of mortality score. Crit Care Med 24(5):743–752PubMedCrossRefGoogle Scholar
  10. 10.
    Shann F, Pearson G, Slater A, Wilkinson K (1997) Paediatric index of mortality (PIM): a mortality prediction model for children in intensive care. Intensive Care Med 23(2):201–207PubMedCrossRefGoogle Scholar
  11. 11.
    Slater A, Shann F, Pearson G (2003) PIM2: a revised version of the paediatric index of mortality. Intensive Care Med 29(2):278–285PubMedGoogle Scholar
  12. 12.
    PICANet (2007) Paediatric intensive care audit network national report 2004–2006 (published June 2007): Universities of Leeds and Leicester. ISBN 9780853162643. http://www.picanet.org.uk/documentation.html. Accessed 6 Oct 2011
  13. 13.
    Alexander J, Tregea S, Slater A (2008) Report of the Australian and New Zealand paediatric intensive care registry 2006. Australian and New Zealand Intensive Care Society (ANZICS). ISBN 1876980303. http://www.anzics.com.au/anzpicr. Accessed 8 Feb 2013
  14. 14.
    Alexander J, Tregea S, Slater A (2010) Report of the Australian and New Zealand paediatric intensive care registry 2008. Australian and New Zealand Intensive Care Society (ANZICS). ISBN 1876980699. http://www.anzics.com.au/anzpicr. Accessed 8 Feb 2013
  15. 15.
    Wolfler A, Silvani P, Musicco M, Salvo I (2007) Pediatric index of mortality 2 score in Italy: a multicenter, prospective, observational study. Intensive Care Med 33(8):1407–1413. doi: 10.1007/s00134-007-0694-z PubMedCrossRefGoogle Scholar
  16. 16.
    Brady AR, Harrison D, Black S, Jones S, Rowan K, Pearson G, Ratcliffe J, Parry GJ (2006) Assessment and optimization of mortality prediction tools for admissions to pediatric intensive care in the United Kingdom. Pediatrics 117(4):e733–e742PubMedCrossRefGoogle Scholar
  17. 17.
    Slater A, Shann F (2004) The suitability of the pediatric index of mortality (PIM), PIM2, the pediatric risk of mortality (PRISM), and PRISM III for monitoring the quality of pediatric intensive care in Australia and New Zealand. Pediatr Crit Care Med 5(5):447–454PubMedCrossRefGoogle Scholar
  18. 18.
    Steyerberg EW (2009) Clinical prediction models. A practical approach to development, validation, and updating. Springer, New YorkGoogle Scholar
  19. 19.
    Collins GS, Moons KG (2012) Comparing risk prediction models. BMJ 344:e3186PubMedCrossRefGoogle Scholar
  20. 20.
    Liddell FD (1984) Simple exact analysis of the standardised mortality ratio. J Epidemiol Community Health 38(1):85–88PubMedCrossRefGoogle Scholar
  21. 21.
    Cox DR (1958) Two further applications of a model for binary regression. Biometrika 45(3/4):562–565CrossRefGoogle Scholar
  22. 22.
    Hanley JAMB (1982) The meaning and use of the area under receiver operating characteristics (ROC) curve. Radiology 143:29–36PubMedGoogle Scholar
  23. 23.
    Slater A, Shann F, McEniery J (2003) The ANZPIC registry diagnostic codes: a system for coding reasons for admitting children to intensive care. Intensive Care Med 29(2):271–277PubMedGoogle Scholar
  24. 24.
    Vergouwe Y, Steyerberg EW, Eijkemans MJ, Habbema JD (2005) Substantial effective sample sizes were required for external validation studies of predictive logistic regression models. J Clin Epidemiol 58(5):475–483. doi: 10.1016/j.jclinepi.2004.06.017 PubMedCrossRefGoogle Scholar
  25. 25.
    Thukral A, Lodha R, Irshad M, Arora NK (2006) Performance of pediatric risk of mortality (PRISM), pediatric index of mortality (PIM), and PIM2 in a pediatric intensive care unit in a developing country. Pediatr Crit Care Med 7(4):356–361. doi: 10.1097/01.PCC.0000227105.20897.89 PubMedCrossRefGoogle Scholar
  26. 26.
    Thorburn K, Baines P (2002) Predicting mortality or recognizing death in meningococcal disease? Pediatric risk of mortality (PRISM) and pediatric index of mortality (PIM). Crit Care Med 30(7):1675–1676PubMedCrossRefGoogle Scholar
  27. 27.
    Mildh L, Pettila V, Sairanen H, Rautiainen P (2007) Predictive value of paediatric risk of mortality score and risk adjustment for congenital heart surgery score after paediatric open-heart surgery. Interact Cardiovasc Thorac Surg 6(5):628–631. doi: 10.1510/icvts.2007.157917 PubMedCrossRefGoogle Scholar
  28. 28.
    Leteurtre S, Leclerc F, Martinot A, Cremer R, Fourier C, Sadik A, Grandbastien B (2001) Can generic scores (pediatric risk of mortality and pediatric index of mortality) replace specific scores in predicting the outcome of presumed meningococcal septic shock in children? Crit Care Med 29(6):1239–1246PubMedCrossRefGoogle Scholar
  29. 29.
    Czaja AS, Scanlon MC, Kuhn EM, Jeffries HE (2010) Performance of the pediatric index of mortality 2 for pediatric cardiac surgery patients. Pediatr Crit Care Med. doi: 10.1097/PCC.0b013e3181e89694 Google Scholar
  30. 30.
    Wang C, Xie G, Cheng B, Du L, Shi L, Tan L, Shu Q, Fang X (2009) Performance of pediatric risk of mortality, pediatric index of mortality and PIM2 in term Chinese neonates. J Trop Pediatr. doi: 10.1093/tropej/fmp124 Google Scholar
  31. 31.
    Steyerberg EW, Borsboom GJ, van Houwelingen HC, Eijkemans MJ, Habbema JD (2004) Validation and updating of predictive logistic regression models: a study on sample size and shrinkage. Stat Med 23(16):2567–2586. doi: 10.1002/sim.1844 PubMedCrossRefGoogle Scholar
  32. 32.
    Leteurtre S, Grandbastien B, Leclerc F, Parslow R (2012) International comparison of the performance of the paediatric index of mortality (PIM) 2 score in two national data sets. Intensive Care Med. doi: 10.1007/s00134-012-2580-6 PubMedGoogle Scholar
  33. 33.
    Gorman M, Sachdeva R, Rice T, Wetzel R (2009) Support for PICUs and the PRISM scoring algorithms. Pediatr Crit Care Med 10(1):143–144. doi: 10.1097/PCC.0b013e3181937bf4 PubMedCrossRefGoogle Scholar
  34. 34.
    Tilford JM, Roberson PK, Lensing S, Fiser DH (1998) Differences in pediatric ICU mortality risk over time. Crit Care Med 26(10):1737–1743PubMedCrossRefGoogle Scholar
  35. 35.
    Kramer AA, Zimmerman JE (2010) A predictive model for the early identification of patients at risk for a prolonged intensive care unit length of stay. BMC Med Inform Decis Mak 10:27. doi: 10.1186/1472-6947-10-27 PubMedCrossRefGoogle Scholar
  36. 36.
    Randolph AG (1997) Paediatric index of mortality (PIM): do we need another paediatric mortality prediction score? Intensive Care Med 23(2):141–142PubMedCrossRefGoogle Scholar
  37. 37.
    Steyerberg EW, Bleeker SE, Moll HA, Grobbee DE, Moons KG (2003) Internal and external validation of predictive models: a simulation study of bias and precision in small samples. J Clin Epidemiol 56(5):441–447 (pii: S0895435603000477)PubMedCrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg and ESICM 2013

Authors and Affiliations

  • Idse H. E. Visser
    • 1
  • Jan A. Hazelzet
    • 2
  • Marcel J. I. J. Albers
    • 3
  • Carin W. M. Verlaat
    • 4
  • Karin Hogenbirk
    • 5
  • Job B. van Woensel
    • 6
  • Marc van Heerde
    • 7
  • Dick A. van Waardenburg
    • 8
  • Nicolaas J. G. Jansen
    • 9
  • Ewout W. Steyerberg
    • 10
  1. 1.Department of Pediatrics, Erasmus MCSophia Children’s HospitalRotterdamThe Netherlands
  2. 2.Division of Pediatric Intensive Care, Department of Pediatrics, Erasmus MCSophia Children’s HospitalRotterdamThe Netherlands
  3. 3.Pediatric Intensive Care Unit, Beatrix Children’s HospitalUniversity Medical Center GroningenGroningenThe Netherlands
  4. 4.Department of Pediatric Intensive CareRadboud University Nijmegen Medical CenterNijmegenThe Netherlands
  5. 5.Pediatric Intensive Care UnitLeiden University Medical CenterLeidenThe Netherlands
  6. 6.Pediatric Intensive Care Unit, Academic Medical CenterEmma Children’s HospitalAmsterdamThe Netherlands
  7. 7.Department of Pediatric Intensive CareVU University Medical CenterAmsterdamThe Netherlands
  8. 8.Pediatric Intensive Care UnitMaastricht University Medical CenterMaastrichtThe Netherlands
  9. 9.Department of Pediatric Intensive Care, Wilhelmina Children’s HospitalUniversity Medical Center UtrechtUtrechtThe Netherlands
  10. 10.Department of Public Health, Center for Medical Decision Making, Erasmus MCUniversity Medical Center RotterdamRotterdamThe Netherlands

Personalised recommendations