Skip to main content
SpringerLink
Log in

Design and validation of a questionnaire to evaluate the usability of computerized critical care information systems

  • Original Research
  • Published:
Journal of Clinical Monitoring and Computing Aims and scope Submit manuscript

Abstract

The implementation of computerized critical care information systems (CCIS) can improve the quality of clinical care and staff satisfaction, but also holds risks of disrupting the workflow with consecutive negative impacts. The usability of CCIS is one of the key factors determining their benefits and weaknesses. However, no tailored instrument exists to measure the usability of such systems. Therefore, the aim of this study was to design and validate a questionnaire that measures the usability of CCIS. Following a mixed-method design approach, we developed a questionnaire comprising two evaluation models to assess the usability of CCIS: (1) the task-specific model rates the usability individually for several tasks which CCIS could support and which we derived by analyzing work processes in the ICU; (2) the characteristic-specific model rates the different aspects of the usability, as defined by the international standard “ergonomics of human-system interaction”. We tested validity and reliability of the digital version of the questionnaire in a sample population. In the sample population of 535 participants both usability evaluation models showed a strong correlation with the overall rating of the system (multiple correlation coefficients ≥0.80) as well as a very high internal consistency (Cronbach’s alpha ≥0.93). The novel questionnaire is a valid and reliable instrument to measure the usability of CCIS and can be used to study the influence of the usability on their implementation benefits and weaknesses.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Apkon M, Singhaviranon P. Impact of an electronic information system on physician workflow and data collection in the intensive care unit. Intensive Care Med. 2001;27:122–30.

    Article  CAS  PubMed  Google Scholar 

  2. Backhaus C. Usability-engineering in der medizintechnik: grundlagen - methoden - beispiele. Berlin: Springer; 2010.

    Book  Google Scholar 

  3. Bosman RJ. Impact of computerized information systems on workload in operating room and intensive care unit. Best Pract Res Clin Anaesthesiol. 2009;23:15–26.

    Article  CAS  PubMed  Google Scholar 

  4. Bosman RJ, Rood E, Oudemans-van Straaten HM, et al. Intensive care information system reduces documentation time of the nurses after cardiothoracic surgery. Intensive Care Med. 2003;29:83–90. doi:10.1007/s00134-002-1542-9.

    Article  PubMed  Google Scholar 

  5. Bradshaw KE, Sittig DF, Gardner RM, et al. Computer-based data entry for nurses in the ICU. MD Comput Comput Med Pract. 1989;6:274–80.

    CAS  Google Scholar 

  6. Donati A, Gabbanelli V, Pantanetti S, et al. The impact of a clinical information system in an intensive care unit. J Clin Monit Comput. 2008;22:31–6. doi:10.1007/s10877-007-9104-x.

    Article  PubMed  Google Scholar 

  7. Fraenkel DJ, Cowie M, Daley P. Quality benefits of an intensive care clinical information system. Crit Care Med. 2003;31:120–5. doi:10.1097/01.CCM.0000045565.11622.C4.

    Article  PubMed  Google Scholar 

  8. Friesdorf W, Fuchs D, Podtschaske B. Medicoergonomics - industrial ergonomics adapted to clinical requirements. In: Duffy V, editor. Advanced in human factors ergonomics in healthcare. Boca Raton: CRC Press; 2010. p. 698–704.

    Google Scholar 

  9. Friesdorf W, Mendyk S, Sander H, Podtschaske B. Arbeitssystem Gesundheitswesen - eine Herausforderung für die Arbeitswissenschaft. In: Mensch Technik, Organisation - Vernetzung im Produktentstehungs- und -herstellungsprozess. Dortmund: GfA-Press; 2011. p. 417–420.

    Google Scholar 

  10. Fromm S. Faktoren- und Reliabilitätsanalyse. In: Datenanalyse mit SPSS für Fortgeschrittene 2: Multivariate Verfahren für Querschnittsdaten. Wiesbaden: Springer; 2012. p. 53–82.

    Chapter  Google Scholar 

  11. Hug BL, Witkowski DJ, Sox CM, et al. Adverse drug event rates in six community hospitals and the potential impact of computerized physician order entry for prevention. J Gen Intern Med. 2010;25:31–8. doi:10.1007/s11606-009-1141-3.

    Article  PubMed  Google Scholar 

  12. ISO 9241-11:1998 Ergonomic requirements for office work with visual display terminals (VDTs)—part 11: guidance on usability.

  13. ISO 9241-110:2006 Ergonomics of human-system interaction—part 110: dialogue principles.

  14. Kaushal R, Shojania KG, Bates DW. Effects of computerized physician order entry and clinical decision support systems on medication safety: a systematic review. Arch Intern Med. 2003;163:1409–16. doi:10.1001/archinte.163.12.1409.

    Article  PubMed  Google Scholar 

  15. Kilgore ML, Flint D, Pearce R. The varying impact of two clinical information systems in a cardiovascular intensive care unit. J Cardiovasc Manag Off J Am Coll Cardiovasc Adm. 1998;9:31–5.

    CAS  Google Scholar 

  16. Leung AA, Keohane C, Amato M, et al. Impact of vendor computerized physician order entry in community hospitals. J Gen Intern Med. 2012;27:801–7. doi:10.1007/s11606-012-1987-7.

    Article  PubMed  PubMed Central  Google Scholar 

  17. Mador RL, Shaw NT. The impact of a critical care information system (CCIS) on time spent charting and in direct patient care by staff in the ICU: a review of the literature. Int J Med Inf. 2009;78:435–45. doi:10.1016/j.ijmedinf.2009.01.002.

    Article  Google Scholar 

  18. Marasovic C, Kenney C, Elliott D, Sindhusake D. A comparison of nursing activities associated with manual and automated documentation in an Australian intensive care unit. Comput Nurs. 1997;15:205–11.

    CAS  PubMed  Google Scholar 

  19. Marsolek I. Clinical process optimization: methodische Grundlagen und praktische Umsetzung. Saarbrücken: VDM Publishing; 2008.

    Google Scholar 

  20. Martich GD, Waldmann CS, Imhoff M. Clinical informatics in critical care. J Intensive Care Med. 2004;19:154–63. doi:10.1177/0885066604264016.

    Article  PubMed  Google Scholar 

  21. Menke JA, Broner CW, Campbell DY, et al. Computerized clinical documentation system in the pediatric intensive care unit. BMC Med Inform Decis Mak. 2001;1:3.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  22. Murff HJ, Kannry J. Physician satisfaction with two order entry systems. J Am Med Inform Assoc JAMIA. 2001;8:499–509.

    Article  CAS  PubMed  Google Scholar 

  23. Pierpont GL, Thilgen D. Effect of computerized charting on nursing activity in intensive care. Crit Care Med. 1995;23:1067–73.

    Article  CAS  PubMed  Google Scholar 

  24. Podtschaske B, Fuchs D, Friesdorf W. Integrated therapy safety management system. Br J Clin Pharmacol. 2013;76(Suppl 1):5–13. doi:10.1111/bcp.12155.

    Article  PubMed  PubMed Central  Google Scholar 

  25. Van Rosse F, Maat B, Rademaker CMA, et al. The effect of computerized physician order entry on medication prescription errors and clinical outcome in pediatric and intensive care: a systematic review. Pediatrics. 2009;123:1184–90. doi:10.1542/peds.2008-1494.

    Article  PubMed  Google Scholar 

  26. Saarinen K, Aho M. Does the implementation of a clinical information system decrease the time intensive care nurses spend on documentation of care? Acta Anaesthesiol Scand. 2005;49:62–5. doi:10.1111/j.1399-6576.2005.00546.x.

    Article  CAS  PubMed  Google Scholar 

  27. Shaw NT, Mador RL, Ho S, et al. Understanding the impact on intensive care staff workflow due to the introduction of a critical care information system: a mixed methods research methodology. Stud Health Technol Inform. 2009;143:186–91.

    CAS  PubMed  Google Scholar 

  28. Shekelle PG, Morton SC, Keeler EB. Costs and benefits of health information technology. Evid Rep Technol Assess. 2006;132:1–71.

    Google Scholar 

  29. Stanton N, Salmon P, Rafferty L, et al. Human factors methods—a practical guide for engineering and design. Farnham: Ashgate; 2013.

    Google Scholar 

  30. Weiner M, Callahan CM, Tierney WM, et al. Using information technology to improve the health care of older adults. Ann Intern Med. 2003;139:430–6.

    Article  PubMed  Google Scholar 

  31. Wong DH, Gallegos Y, Weinger MB, et al. Changes in intensive care unit nurse task activity after installation of a third-generation intensive care unit information system. Crit Care Med. 2003;31:2488–94. doi:10.1097/01.CCM.0000089637.53301.EF.

    Article  PubMed  Google Scholar 

Download references

Acknowledgments

We would like to thank the expert group of experienced intensive care specialists that supported our participative analysis, especially the members of the Committee for CCIS (Forum Patientendatenmanagementsysteme) of the Scientific Workgroup for Intensive Care Medicine (Wissenschaftlicher Arbeitskreis Intensivmedizin) of the German Society of Anesthesiology and Intensive Care Medicine (Deutsche Gesellschaft für Anästhesiologie und Intensivmedizin).

Funding

The study was funded through the internal research funding of the Klinik für Anästhesiologie mit Schwerpunkt operative Intensivmedizin, Campus Charité Mitte und Campus Virchow-Klinikum, Charité – Universitätsmedizin Berlin.

Author information

Authors and Affiliations

  1. Klinik für Anästhesiologie mit Schwerpunkt operative Intensivmedizin, Charité - Universitätsmedizin Berlin, Campus Charité Mitte und Campus Virchow-Klinikum, Augustenburger Platz 1, 13353, Berlin, Germany

    Falk von Dincklage & Gregor Lichtner

  2. Klinik für Anästhesiologie und Intensivmedizin, Medizinische Hochschule Hannover, Hannover, Germany

    Klaudiusz Suchodolski

  3. Klinik und Poliklinik für Anästhesiologie und Intensivtherapie, Technische Universität Dresden, Universitätsklinikum Carl Gustav Carus, Dresden, Germany

    Maximilian Ragaller

  4. HCMB Institute for Health Care Systems Management Berlin eG, Berlin, Germany

    Wolfgang Friesdorf

  5. Clinical Excellence Research Center, Stanford University, Stanford, CA, USA

    Beatrice Podtschaske

Authors
  1. Falk von Dincklage
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Gregor Lichtner
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Klaudiusz Suchodolski
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Maximilian Ragaller
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Wolfgang Friesdorf
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Beatrice Podtschaske
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Falk von Dincklage.

Ethics declarations

Conflicts of interest

No competing interests, financial interests or conflicts of interest regarding the study exist.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (DOC 176 kb)

Supplementary material 2 (DOC 188 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

von Dincklage, F., Lichtner, G., Suchodolski, K. et al. Design and validation of a questionnaire to evaluate the usability of computerized critical care information systems. J Clin Monit Comput 31, 833–844 (2017). https://doi.org/10.1007/s10877-016-9892-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10877-016-9892-y

Keywords

Search

Navigation