Neurocritical Care

, Volume 30, Issue 2, pp 394–404 | Cite as

External Validation of Hematoma Expansion Scores in Spontaneous Intracerebral Hemorrhage in an Asian Patient Cohort

  • Jia Xu Lim
  • Julian Xinguang Han
  • Angela An Qi See
  • Voon Hao Lew
  • Wan Ting Chock
  • Vin Fei Ban
  • Sohil Pothiawala
  • Winston Eng Hoe Lim
  • Louis Elliot McAdory
  • Michael Lucas James
  • Nicolas Kon Kam KingEmail author
Original Article



Hematoma expansion (HE) occurs in approximately one-third of patients with intracerebral hemorrhage (ICH) and is known to be a strong predictor of neurological deterioration as well as poor functional outcome. This study aims to externally validate three risk prediction models of HE (PREDICT, 9-point, and BRAIN scores) in an Asian population.


A prospective cohort of 123 spontaneous ICH patients admitted to a tertiary hospital (certified stroke center) in Singapore was recruited. Logistic recalibrations were performed to obtain updated calibration slopes and intercepts for all models. The discrimination (c-statistic), calibration (Hosmer–Lemeshow test, le Cessie–van Houwelingen–Copas–Hosmer test, Akaike information criterion), overall performance (Brier score, R2), and clinical usefulness (decision curve analysis) of the risk prediction models were examined.


Overall, the recalibrated PREDICT performed best among the three models in our study cohort based on the novel matrix comprising of Akaike information criterion and c-statistic. The PREDICT model had the highest R2 (0.26) and lowest Brier score (0.14). Decision curve analyses showed that recalibrated PREDICT was more clinically useful than 9-point and BRAIN models over the greatest range of threshold probabilities. The two scores (PREDICT and 9-point) which incorporated computed tomography (CT) angiography spot sign outperformed the one without (BRAIN).


To our knowledge, this is the first study to validate HE scores, namely PREDICT, 9-Point and BRAIN, in a multi-ethnic Asian ICH patient population. The PREDICT score was the best performing model in our study cohort, based on the performance metrics employed in this study. Our findings also showed support for CT angiography spot sign as a predictor of outcome after ICH. Although the models assessed are sufficient for risk stratification, the discrimination and calibration are at best moderate and could be improved.


Intracerebral hemorrhage Hematoma expansion External validation Risk prediction 


Authors Contribution

JXL contributed to protocol development, data collection and management, and manuscript writing and editing. JH contributed to protocol development, data analysis, and manuscript writing and editing. AAQS contributed to protocol development, data analysis, and manuscript writing and editing. VHL, WTC, and VFB contributed to data collection and management and manuscript approval. SP, LEM, and WEHL contributed to protocol development, data collection, and manuscript approval. MLJ contributed to protocol development and manuscript writing and editing. NKKK contributed to protocol development, data analysis, and manuscript writing and editing. MLJ is the Basic Science Editor for Neurocritical Care.

Sources of support


Compliance with Ethical Standards

Conflict of interest

Michael James is the Basic Science Editor for Neurocritical Care. The other authors declare that they have no conflict of interest.

Human and Animal Rights

This is a prospective study, needs statement of ethics approval and consent.


  1. 1.
    Feigin VL, Krishnamurthi RV, Parmar P, Norrving B, Mensah GA, Bennett DA, GBD 2013 Writing Group, GBD 2013 Stroke Panel Experts Group. Update on the global burden of ischemic and hemorrhagic stroke in 1990–2013: the GBD 2013 study. Neuroepidemiology. 2015;2015(45):161–76.CrossRefGoogle Scholar
  2. 2.
    National Registry of Diseases Office, Ministry of Health Singapore. Trends in stroke in Singapore 2005-2013. Singapore, 2015. Available at: Accessed 5th May 2018
  3. 3.
    Kannel WB, Wolf PA, Verter J, McNamara PM. Epidemiologic assessment of the role of blood pressure in stroke: the Framingham study. JAMA. 1970;214:301–10.CrossRefGoogle Scholar
  4. 4.
    Broderick J, Connolly S, Feldmann E, Hanley D, Kase C, Krieger D, et al. Guidelines for the management of spontaneous intracerebral hemorrhage in adults: 2007 update: a guideline from the American Heart Association/American Stroke Association Stroke Council, High Blood Pressure Research Council, and the Quality of Care and Outcomes in Research Interdisciplinary Working Group. Circulation. 2007;116:e391–413.CrossRefGoogle Scholar
  5. 5.
    Hong KS, Bang OY, Kang DW, Yu KH, Bae HJ, Lee JS, et al. Stroke statistics in Korea: part I. Epidemiology and risk factors: a report from the korean stroke society and clinical research center for stroke. J Stroke. 2013;15:2–20.CrossRefGoogle Scholar
  6. 6.
    Toyoda K. Epidemiology and registry studies of stroke in Japan. J Stroke. 2013;15:21–6.CrossRefGoogle Scholar
  7. 7.
    Davis SM, Broderick J, Hennerici M, Brun NC, Diringer MN, Mayer SA, et al. Recombinant Activated Factor VII Intracerebral Hemorrhage Trial Investigators. Haematoma growth is a determinant of mortality and poor outcome after intracerebral hemorrhage. Neurology. 2006;66:1175–81.CrossRefGoogle Scholar
  8. 8.
    Dowlatshahi D, Demchuk AM, Flaherty ML, Ali M, Lyden PL, Smith EE, VISTA Collaboration. Defining hematoma expansion in intracerebral hemorrhage: relationship with patient outcomes. Neurology. 2011;76:1238–44.CrossRefGoogle Scholar
  9. 9.
    Huynh TJ, Aviv RI, Dowlatshahi D, Gladstone DJ, Laupacis A, Kiss A, et al. PREDICT/Sunnybrook CTA Investigators. Validation of the 9-point and 24-point hematoma expansion prediction scores and derivation of the PREDICT A/B scores. Stroke. 2015;46:3105–10.CrossRefGoogle Scholar
  10. 10.
    Brouwers HB, Chang Y, Falcone GJ, Cai X, Ayres AM, Battey TW, et al. Predicting hematoma expansion after primary intracerebral hemorrhage. JAMA Neurol. 2014;71:158–64.CrossRefGoogle Scholar
  11. 11.
    Wang X, Arima H, Al-Shahi SR, Woodward M, Heeley E, Stapf C, et al. INTERACT Investigators. Clinical prediction algorithm (BRAIN) to determine risk of hematoma growth in acute intracerebral hemorrhage. Stroke. 2015;46:376–81.CrossRefGoogle Scholar
  12. 12.
    Flaherty ML, Woo D, Broderick JP. Epidemiology. In Carhuapoma JR, Mayer SA, & Hanley DF. Intracerebral Hemorrhage. [Online] Cambridge: Cambridge University Press, 2009; 1–16.Google Scholar
  13. 13.
    Jiang B, Wang W, Chen H, Hong Z, Yang QD, Wu SP, et al. Incidence and trends of stroke and its subtypes in China: results from three large cities. Stroke. 2006;37:63–8.CrossRefGoogle Scholar
  14. 14.
    Flaherty ML, Woo D, Haverbusch M, Sekar P, Khoury J, Sauerbeck L, et al. Racial variations in location and risk of intracerebral hemorrhage. Stroke. 2005;36:934–7.CrossRefGoogle Scholar
  15. 15.
    Washington, DC: Central Intelligence Agency, 2018. The World Factbook 2018. Available at: Accessed 5th May 2018.
  16. 16.
    Collins GS, Reitsma JB, Altman DG, Moons KG. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD Statement. Br J Surg. 2015;102:148–58.CrossRefGoogle Scholar
  17. 17.
    Hemphill JC, Greenberg SM, Anderson CS, Becker K, Bendok BR, Cushman M, et al. American Heart Association Stroke Council; Council on Cardiovascular and Stroke Nursing; Council on Clinical Cardiology. Guidelines for the management of spontaneous intracerebral hemorrhage: a guideline for healthcare professionals from the American Heart Association/American Stroke Association. Stroke. 2015;46:2032–60.CrossRefGoogle Scholar
  18. 18.
    Ciura VA, Brouwers HB, Pizzolato R, Ortiz CJ, Rosand J, Goldstein JN, et al. Spot sign on 90-second delayed computed tomography angiography improves sensitivity for haematoma expansion and mortality: a prospective study. Stroke. 2014;45:3293–7.CrossRefGoogle Scholar
  19. 19.
    Yushkevich PA, Piven J, Hazlett HC, Smith RG, Ho S, Gee JC, et al. User-guided 3D active contour segmentation of anatomical structures: significantly improved efficiency and reliability. Neuroimage. 2006;31:1116–28.CrossRefGoogle Scholar
  20. 20.
    Thompson AL, Kosior JC, Gladstone DJ, Hopyan JJ, Symons SP, Romero F, et al. PREDICT/Sunnybrook ICH CTA Study Group. Defining the CT angiography ‘spot sign’ in primary intracerebral hemorrhage. Can J Neurol Sci. 2009;36:456–61.CrossRefGoogle Scholar
  21. 21.
    Steyerberg EW. Clinical prediction models: a practical approach to development, validation, and updating. New York: Springer; 2008.Google Scholar
  22. 22.
    Mushkudiani NA, Hukkelhoven CW, Hernández AV, Murray Gd, Choi SC, Maas AI, et al. A systematic review finds methodological improvements necessary for prognostic models in determining traumatic brain injury outcomes. J Clin Epidemiol. 2008;61:331–43.CrossRefGoogle Scholar
  23. 23.
    Hosmer DW, Lemeshow S. Applied Logistic Regression. 2nd ed. Wiley: New York; 2000.CrossRefGoogle Scholar
  24. 24.
    Hosmer DW, Hosmer T, Le Cessie S, Lemeshow S. A comparison of goodness-of-fit tests for the logistic regression model. Stat Med. 1997;16(9):965–80.CrossRefGoogle Scholar
  25. 25.
    Burnham KP, Anderson DR, Huyvaert KP. AIC model selection and multimodel inference in behavioral ecology: some background, observations, and comparisons. Behav Ecol Sociobiol. 2011;65:23–5.CrossRefGoogle Scholar
  26. 26.
    Vickers AJ, Elkin EB. Decision curve analysis: a novel method for evaluating prediction models. Med Decis Making. 2006;26:565–74.CrossRefGoogle Scholar
  27. 27.
    Goldstein JN, Fazen LE, Snider R, Schwab K, Greenberg SM, Smith EE, et al. Contrast extravasation on CT angiography predicts hematoma expansion in intracerebral hemorrhage. Neurology. 2007;68:889–94.CrossRefGoogle Scholar
  28. 28.
    Wada R, Aviv RI, Fox AJ, Sahlas DJ, Gladstone DJ, Tomlinson G, et al. CT angiography “spot sign” predicts hematoma expansion in acute intracerebral hemorrhage. Stroke. 2007;38:1257–62.CrossRefGoogle Scholar
  29. 29.
    Morotti A, Dowlatshahi D, Boulouis G, Al-Ajlan F, Demchuk AM, Aviv RI, et al. ATACH-II, NETT, and PREDICT Investigators. predicting intracerebral hemorrhage expansion with noncontrast computed tomography: the BAT score. Stroke. 2018;49:1163–9.CrossRefGoogle Scholar
  30. 30.
    Ye J, Rust G, Baltrus P, Daniels E. Cardiovascular risk factors among Asian Americans: results from a National Health Survey. Ann Epidemiol. 2009;19:718–23.CrossRefGoogle Scholar
  31. 31.
    Moons KG, Altman DG, Reitsma JB, Ioannidis JP, Macaskill P, Steyerberg EW, et al. Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): explanation and elaboration. Ann Intern Med. 2015;162:W1–73.CrossRefGoogle Scholar
  32. 32.
    Eikelboom J, Merli G. Bleeding with direct oral anticoagulants vs warfarin: clinical experience. Am J Med. 2016;129(11):S33–40.CrossRefGoogle Scholar
  33. 33.
    Demchuk AM, Dowlatshahi D, Rodriguez-Luna D, Molina CA, Blas YS, Dzialowski I, et al. PREDICT/Sunnybrook ICH CTA study group. Prediction of hematoma growth and outcome in patients with intracerebral hemorrhage using the CT-angiography spot sign (PREDICT): a prospective observational study. Lancet Neurol. 2012;11:307–14.CrossRefGoogle Scholar
  34. 34.
    Delcourt C, Huang Y, Wang J, Heeley E, Lindley R, Stapf C, et al. INTERACT2 Investigators. The second (main) phase of an open, randomised, multicentre study to investigate the effectiveness of an intensive blood pressure reduction in acute cerebral haemorrhage trial (INTERACT2). Int J Stroke. 2010;5:110–6.CrossRefGoogle Scholar
  35. 35.
    Delcourt C, Huang Y, Arima H, Chalmers J, Davis SM, Heeley EL, et al. INTERACT 1 Investigators. Hematoma growth and outcomes in intracerebral hemorrhage: the INTERACT1 study. Neurology. 2012;79:314–9.CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature and Neurocritical Care Society 2018

Authors and Affiliations

  • Jia Xu Lim
    • 1
    • 2
  • Julian Xinguang Han
    • 1
    • 2
  • Angela An Qi See
    • 1
    • 2
  • Voon Hao Lew
    • 1
  • Wan Ting Chock
    • 2
  • Vin Fei Ban
    • 2
  • Sohil Pothiawala
    • 3
  • Winston Eng Hoe Lim
    • 4
  • Louis Elliot McAdory
    • 4
  • Michael Lucas James
    • 5
    • 6
  • Nicolas Kon Kam King
    • 1
    • 2
    • 7
    Email author
  1. 1.Department of NeurosurgeryNational Neuroscience InstituteSingaporeSingapore
  2. 2.Department of NeurosurgerySingapore General HospitalSingaporeSingapore
  3. 3.Department of Emergency MedicineSingapore General HospitalSingaporeSingapore
  4. 4.Department of Diagnostic RadiologySingapore General HospitalSingaporeSingapore
  5. 5.Departments of AnesthesiologyBrain Injury Translational Research Center, Duke UniversityDurhamUSA
  6. 6.Departments of NeurologyBrain Injury Translational Research Center, Duke UniversityDurhamUSA
  7. 7.Duke-NUS Medical SchoolSingaporeSingapore

Personalised recommendations