Skip to main content
SpringerLink
Log in

External validity of an automated delirium prediction model (DEMO) and comparison to the manual VMS-questions: a retrospective cohort study

  • Research Article
  • Published:
International Journal of Clinical Pharmacy Aims and scope Submit manuscript

Abstract

Background

It is estimated that one-third of delirium cases in hospitals could be prevented with appropriate interventions. In Dutch hospitals a manual instrument (VMS-questions) is used to identify patients at-risk for delirium. Delirium Model (DEMO) is an automated model which could support delirium prevention more efficiently. However, it has not been validated beyond the hospital it was developed in.

Aim

To externally validate the DEMO and compare its performance to the VMS-questions.

Method

A retrospective cohort study between July and December 2018 was conducted. Delirium cases were identified through a chart review, and the VMS-questions were extracted from the electronic health records. The DEMO was validated in patients ≥ 60 years, and a comparison with the VMS-questions was made in patients ≥ 70 years.

Results

In total 1,345 admissions were included. The DEMO predicted 59 out of 75 delirium cases (sensitivity 0.79, 95% CI = 0.68–0.87; specificity 0.75, 95% CI = 0.72–0.77). Compared to the VMS-questions, the DEMO showed a lower specificity (0.64 vs. 0.72; p < 0.001) and a comparable sensitivity (0.83 vs. 0.80; p = 0.56). The VMS-questions were missing in 20% of admissions, in which the DEMO correctly predicted 10 of 12 delirium cases.

Conclusion

The DEMO showed acceptable performance for delirium prediction. Overall the DEMO predicted more delirium cases because the VMS-questions were missing in 20% of admissions. This study shows that automated instruments such as DEMO could play a key role in the efficient and timely deployment of measures to prevent delirium.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

References

  1. Davoudi A, Ebadi A, Rashidi P, et al. Delirium prediction using machine learning models on preoperative electronic health records data. Proc IEEE Int Symp Bioinformatics Bioeng. 2017;2017:568–73. https://doi.org/10.1109/BIBE.2017.00014.

    Article  PubMed  Google Scholar 

  2. Siddiqi N, House AO, Holmes JD. Occurrence and outcome of delirium in medical in-patients: a systematic literature review. Age Ageing. 2006;35(4):350–64. https://doi.org/10.1093/ageing/afl005.

    Article  PubMed  Google Scholar 

  3. Hapca S, Guthrie B, Cvoro V, et al. Mortality in people with dementia, delirium, and unspecified cognitive impairment in the general hospital: prospective cohort study of 6,724 patients with 2 years follow-up. Clin Epidemiol. 2018;10:1743–53. https://doi.org/10.2147/CLEP.S174807.

    Article  PubMed  PubMed Central  Google Scholar 

  4. Inouye SK, Westendorp RG, Saczynski JS. Delirium in elderly people. Lancet. 2014;383(9920):911–22. https://doi.org/10.1016/S0140-6736(13)60688-1.

    Article  PubMed  Google Scholar 

  5. Salluh JI, Wang H, Schneider EB, et al. Outcome of delirium in critically ill patients: systematic review and meta-analysis. BMJ. 2015;350:h2538. https://doi.org/10.1136/bmj.h2538.

    Article  PubMed  PubMed Central  Google Scholar 

  6. Marcantonio ER. Delirium in hospitalized older adults. N Engl J Med. 2017;377(15):1456–66. https://doi.org/10.1056/NEJMcp1605501.

    Article  PubMed  PubMed Central  Google Scholar 

  7. Carrasco MP, Villarroel L, Andrade M, et al. Development and validation of a delirium predictive score in older people. Age Ageing. 2014;43(3):346–51. https://doi.org/10.1093/ageing/aft141.

    Article  PubMed  Google Scholar 

  8. Strijbos MJ, Steunenberg B, van der Mast RC, et al. Design and methods of the Hospital Elder Life Program (HELP), a multicomponent targeted intervention to prevent delirium in hospitalized older patients: efficacy and cost-effectiveness in Dutch health care. BMC Geriatr. 2013;13:78. https://doi.org/10.1186/1471-2318-13-78.

    Article  PubMed  PubMed Central  Google Scholar 

  9. Setters B, Solberg LM. Delirium. Prim Care. 2017;44(3):541–59. https://doi.org/10.1016/j.pop.2017.04.010.

    Article  PubMed  Google Scholar 

  10. Lindroth H, Bratzke L, Purvis S, et al. Systematic review of prediction models for delirium in the older adult inpatient. BMJ Open. 2018;8(4):e019223. https://doi.org/10.1136/bmjopen-2017-019223.

    Article  PubMed  PubMed Central  Google Scholar 

  11. Chen Y, Du H, Wei BH, et al. Development and validation of risk-stratification delirium prediction model for critically ill patients: A prospective, observational, single-center study. Medicine (Baltimore). 2017;96(29):e7543. https://doi.org/10.1097/MD.0000000000007543.

    Article  PubMed  PubMed Central  Google Scholar 

  12. de Wit HA, Winkens B, Mestres Gonzalvo C, et al. The development of an automated ward independent delirium risk prediction model. Int J Clin Pharm. 2016;38(4):915–23. https://doi.org/10.1007/s11096-016-0312-7.

    Article  PubMed  Google Scholar 

  13. Douglas VC, Hessler CS, Dhaliwal G, et al. The AWOL tool: derivation and validation of a delirium prediction rule. J Hosp Med. 2013;8(9):493–9. https://doi.org/10.1002/jhm.2062.

    Article  PubMed  Google Scholar 

  14. Kukreja D, Gunther U, Popp J. Delirium in the elderly: current problems with increasing geriatric age. Indian J Med Res. 2015;142(6):655–62. https://doi.org/10.4103/0971-5916.174546.

    Article  PubMed  PubMed Central  Google Scholar 

  15. Oh ES, Fong TG, Hshieh TT, et al. Delirium in older persons: advances in diagnosis and treatment. JAMA. 2017;318(12):1161–74. https://doi.org/10.1001/jama.2017.12067.

    Article  PubMed  PubMed Central  Google Scholar 

  16. Yang Y, Zhao X, Dong T, et al. Risk factors for postoperative delirium following hip fracture repair in elderly patients: a systematic review and meta-analysis. Aging Clin Exp Res. 2017;29(2):115–26. https://doi.org/10.1007/s40520-016-0541-6.

    Article  CAS  PubMed  Google Scholar 

  17. Kinchin I, Mitchell E, Agar M, et al. The economic cost of delirium: a systematic review and quality assessment. Alzheimers Dement. 2021;17(6):1026–41. https://doi.org/10.1002/alz.12262.

    Article  PubMed  Google Scholar 

  18. Nederlandse Vereniging voor Klinische Geriatrie. Delier bij volwassenen en ouderen. 2020. https://richtlijnendatabase.nl/richtlijn/delier_bij_volwassenen_en_ouderen/startpagina_-_delier_bij_volwassenen_en_ouderen.html. Accessed 24 July 2023.

  19. Inouye SK, Bogardus ST, Charpentier PA, et al. A multicomponent intervention to prevent delirium in hospitalized older patients. N Engl J Med. 1999;340(9):669–76. https://doi.org/10.1056/nejm199903043400901.

    Article  CAS  PubMed  Google Scholar 

  20. Expertgroep Kwetsbare ouderen. VMS Veiligheidsprogramma: Kwetsbare ouderen. Den Haag. 2009. https://www.vmszorg.nl/wp-content/uploads/2017/11/web_2009.0104_praktijkgids_kwetsbare_ouderen.pdf. Accessed 24 July 2023.

  21. VMSzorg. Dutch patient safety programme. 2021. https://www.vmszorg.nl/vms-veiligheidsprogramma/dutch-patient-safety-programme-english/. Accessed 26 July 2023.

  22. Ettema R, Heim N, Hamaker M, et al. Validity of a screening method for delirium risk in older patients admitted to a general hospital in the Netherlands. Gen Hosp Psychiatry. 2018;55:44–50. https://doi.org/10.1016/j.genhosppsych.2018.09.004.

    Article  PubMed  Google Scholar 

  23. Mestres Gonzalvo C, de Wit H, van Oijen BPC, et al. Validation of an automated delirium prediction model (DElirium MOdel (DEMO)): an observational study. BMJ Open. 2017;7(11):e016654. https://doi.org/10.1136/bmjopen-2017-016654.

    Article  PubMed  PubMed Central  Google Scholar 

  24. Inouye SK, Leo-Summers L, Zhang Y, et al. A chart-based method for identification of delirium: validation compared with interviewer ratings using the confusion assessment method. J Am Geriatr Soc. 2005;53(2):312–8. https://doi.org/10.1111/j.1532-5415.2005.53120.x.

    Article  PubMed  Google Scholar 

  25. Wassenaar A, Schoonhoven L, Devlin JW, et al. Delirium prediction in the intensive care unit: comparison of two delirium prediction models. Crit Care. 2018;22(1):114. https://doi.org/10.1186/s13054-018-2037-6.

    Article  PubMed  PubMed Central  Google Scholar 

  26. Schulthess-Lisibach AE, Gallucci G, Benelli V, et al. Predicting delirium in older non-intensive care unit inpatients: development and validation of the DELIrium risK Tool (DELIKT). Int J Clin Pharm. 2023. https://doi.org/10.1007/s11096-023-01566-0.

    Article  PubMed  Google Scholar 

  27. Collins N, Blanchard MR, Tookman A, et al. Detection of delirium in the acute hospital. Age Ageing. 2010;39(1):131–5. https://doi.org/10.1093/ageing/afp201.

    Article  PubMed  Google Scholar 

  28. Inouye SK. Predisposing and precipitating factors for delirium in hospitalized older patients. Dement Geriatr Cogn Disord. 1999;10(5):393–400. https://doi.org/10.1159/000017177.

    Article  CAS  PubMed  Google Scholar 

  29. Young J, Leentjens AF, George J, et al. Systematic approaches to the prevention and management of patients with delirium. J Psychosom Res. 2008;65(3):267–72. https://doi.org/10.1016/j.jpsychores.2008.05.022.

    Article  PubMed  Google Scholar 

  30. Young J, Murthy L, Westby M, et al. Diagnosis, prevention, and management of delirium: summary of NICE guidance. BMJ. 2010;341:c3704. https://doi.org/10.1136/bmj.c3704.

    Article  PubMed  Google Scholar 

Download references

Funding

No specific funding was received.

Author information

Author notes

  1. Ma Ida Mohmaed Ali and Gerard A. Kalkman have equally contributed to this work.

Authors and Affiliations

  1. Department of Clinical Pharmacy, Canisius Wilhelmina Hospital, Nijmegen, The Netherlands

    Ma Ida Mohmaed Ali, Gerard A. Kalkman, Hanneke W. H. A. Fleuren, Cornelis Kramers & Hugo A. J. M. de Wit

  2. CWZ Academy, Canisius Wilhelmina Hospital, Nijmegen, The Netherlands

    Charlotte H. W. Wijers

  3. Department of Pharmacology-Toxicology, Radboud University Hospital, Nijmegen, The Netherlands

    Cornelis Kramers

Authors
  1. Ma Ida Mohmaed Ali
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Gerard A. Kalkman
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Charlotte H. W. Wijers
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Hanneke W. H. A. Fleuren
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Cornelis Kramers
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Hugo A. J. M. de Wit
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Gerard A. Kalkman.

Ethics declarations

Conflicts of interest

The authors have no conflicts of interest to declare.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ali, M.I.M., Kalkman, G.A., Wijers, C.H.W. et al. External validity of an automated delirium prediction model (DEMO) and comparison to the manual VMS-questions: a retrospective cohort study. Int J Clin Pharm 45, 1128–1135 (2023). https://doi.org/10.1007/s11096-023-01641-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11096-023-01641-6

Keywords

Search

Navigation