Skip to main content


Log in

Development of a Novel Scoring System to Quantify the Severity of Incident Reports: An Exploratory Research Study

  • Original Paper
  • Published:
Journal of Medical Systems Aims and scope Submit manuscript


Incident reporting systems have been widely adopted to collect information about patient safety incidents. Much of the value of incident reports lies in the free-text section. Computer processing of semantic information may be helpful to analyze this. We developed a novel scoring system for decision making to assess the severity of incidents using the semantic characteristics of the text in incident reports, and compared its results with experts’ opinions. We retrospectively analyzed free-text data from incident reports from January 2012 to September 2021 at Nagoya University Hospital, Aichi, Japan. The sample was allocated to training and validation datasets using the hold-out method. Morphological analysis was used to segment terms in the training dataset. We calculated a severity term score, a severity report score and severity group score, by report volume size, and compared these with conventional severity classifications by patient safety experts and reporters. We allocated 96,082 incident reports into two groups. We calculated 1,802 severity term scores from the 48,041 reports in the training dataset. There was a significant difference in severity report score between reports categorized as severe and not severe by experts (95% confidence interval [CI] −0.83 to −0.80, p < 0.001, d = 0.81). Severity group scores were positively associated with severity ratings from experts and reporters (correlation coefficients 0.73 [95% CI 0.63–0.80, p < 0.001] and 0.79 [95% CI 0.71–0.85, p < 0.001]) for all departments. Our severity scoring system could therefore contribute to better organizational patient safety.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

Data availability

The data that support the findings of this study are available from the corresponding author upon reasonable request.


  1. Kohn KT, C.J., Donaldson MS, To Err Is Human: Building a Safer Health System. 1999: Washington, DC.

  2. Leape, L.L. and D.M. Berwick, Five years after To Err Is Human: what have we learned? Jama, 2005. 293(19): p. 2384-90.

    Article  CAS  Google Scholar 

  3. Dzau, V.J. and K.I. Shine, Two Decades Since To Err Is Human: Progress, but Still a "Chasm". Jama, 2020. 324(24): p. 2489-2490.

    Article  Google Scholar 

  4. Clancy, C.M., Where we are a decade after To err is human. J Patient Saf, 2009. 5(4): p. 199-200.

    Article  Google Scholar 

  5. Sexton, J.B., et al., The Safety Attitudes Questionnaire: psychometric properties, benchmarking data, and emerging research. BMC Health Serv Res, 2006. 6: p. 44.

    Article  Google Scholar 

  6. Sorra J, G.L., Streagle S, et al. AHRQ Hospital Survey on Patient Safety Culture: User’s Guide. 2018 [cited 2022; Available from:

  7. Vincent, C., S. Burnett, and J. Carthey, Safety measurement and monitoring in healthcare: a framework to guide clinical teams and healthcare organisations in maintaining safety. BMJ Qual Saf, 2014. 23(8): p. 670-7.

    Article  Google Scholar 

  8. Benn, J., et al., Feedback from incident reporting: information and action to improve patient safety. Qual Saf Health Care, 2009. 18(1): p. 11-21.

    Article  CAS  Google Scholar 

  9. Larizgoitia, I., M.C. Bouesseau, and E. Kelley, WHO Efforts to Promote Reporting of Adverse Events and Global Learning. J Public Health Res, 2013. 2(3): p. e29.

    Article  Google Scholar 

  10. Pham, J.C., T. Girard, and P.J. Pronovost, What to do with healthcare incident reporting systems. J Public Health Res, 2013. 2(3): p. e27.

    Article  Google Scholar 

  11. Howell, A.M., et al., Can Patient Safety Incident Reports Be Used to Compare Hospital Safety? Results from a Quantitative Analysis of the English National Reporting and Learning System Data. PLoS One, 2015. 10(12): p. e0144107.

    Article  Google Scholar 

  12. Wagner, C., et al., Unit-based incident reporting and root cause analysis: variation at three hospital unit types. BMJ Open, 2016. 6(6): p. e011277.

    Article  Google Scholar 

  13. Wang, Y., et al., Using multiclass classification to automate the identification of patient safety incident reports by type and severity. BMC Med Inform Decis Mak, 2017. 17(1): p. 84.

    Article  CAS  Google Scholar 

  14. Williams, S.D. and D.M. Ashcroft, Medication errors: how reliable are the severity ratings reported to the national reporting and learning system? Int J Qual Health Care, 2009. 21(5): p. 316-20.

    Article  Google Scholar 

  15. Cheng, L., et al., International comparative analyses of incidents reporting systems for healthcare risk management. J Evid Based Med, 2011. 4(1): p. 32-47.

    Article  Google Scholar 

  16. Choudhury, A. and O. Asan, Role of Artificial Intelligence in Patient Safety Outcomes: Systematic Literature Review. JMIR Med Inform, 2020. 8(7): p. e18599.

    Article  Google Scholar 

  17. Bates, D.W., et al., The potential of artificial intelligence to improve patient safety: a scoping review. NPJ Digit Med, 2021. 4(1): p. 54.

    Article  Google Scholar 

  18. Senders, J.T., et al., Natural Language Processing for Automated Quantification of Brain Metastases Reported in Free-Text Radiology Reports. JCO Clin Cancer Inform, 2019. 3: p. 1-9.

    Article  Google Scholar 

  19. Kreimeyer, K., et al., Natural language processing systems for capturing and standardizing unstructured clinical information: A systematic review. J Biomed Inform, 2017. 73: p. 14-29.

    Article  Google Scholar 

  20. Young, I.J.B., S. Luz, and N. Lone, A systematic review of natural language processing for classification tasks in the field of incident reporting and adverse event analysis. Int J Med Inform, 2019. 132: p. 103971.

    Article  Google Scholar 

  21. Wong, A., et al., Natural Language Processing and Its Implications for the Future of Medication Safety: A Narrative Review of Recent Advances and Challenges. Pharmacotherapy, 2018. 38(8): p. 822-841.

    Article  Google Scholar 

  22. Syed, K., et al., Automatic Incident Triage in Radiation Oncology Incident Learning System. Healthcare (Basel), 2020. 8(3).

  23. Wang, Y., E. Coiera, and F. Magrabi, Using convolutional neural networks to identify patient safety incident reports by type and severity. J Am Med Inform Assoc, 2019. 26(12): p. 1600-1608.

    Article  Google Scholar 

  24. Fukami, T., M. Uemura, and Y. Nagao, Significance of incident reports by medical doctors for organizational transparency and driving forces for patient safety. Patient Saf Surg, 2020. 14: p. 13.

    Article  Google Scholar 

  25. Guidelines for the public announcement of medical accidents at national university hospitals (revised edition). . 2012 [cited 2021; Available from:

  26. Hirose, M., et al., Lag time in an incident reporting system at a university hospital in Japan. Qual Saf Health Care, 2007. 16(2): p. 101-4.

    Article  Google Scholar 

  27. World Health Organization & WHO Patient Safety. Safety, Conceptual framework for the international classification for patient safety version 1.1: final technical report January 2009. 2010, World Health Organization: Geneva.

  28. Mitchell, I., et al., Patient safety incident reporting: a qualitative study of thoughts and perceptions of experts 15 years after 'To Err is Human'. BMJ Qual Saf, 2016. 25(2): p. 92-9.

    Article  Google Scholar 

  29. Stavropoulou, C., C. Doherty, and P. Tosey, How Effective Are Incident-Reporting Systems for Improving Patient Safety? A Systematic Literature Review. Milbank Q, 2015. 93(4): p. 826-66.

    Article  Google Scholar 

  30. Ong, M.S., F. Magrabi, and E. Coiera, Automated identification of extreme-risk events in clinical incident reports. J Am Med Inform Assoc, 2012. 19(e1): p. e110-8.

    Article  Google Scholar 

  31. Evans, H.P., et al., Automated classification of primary care patient safety incident report content and severity using supervised machine learning (ML) approaches. Health Informatics J, 2020. 26(4): p. 3123-3139.

    Article  Google Scholar 

  32. Liang, C. and Y. Gong, Automated Classification of Multi-Labeled Patient Safety Reports: A Shift from Quantity to Quality Measure. Stud Health Technol Inform, 2017. 245: p. 1070-1074.

    Google Scholar 

  33. Zhang, W., T. Yoshida, and X. Tang, A comparative study of TF*IDF, LSI and multi-words for text classification. Expert Systems with Applications, 2011. 38(3): p. 2758-2765.

    Article  Google Scholar 

Download references


We thank staff of the Department of Patient Safety at Nagoya University Hospital and members of ASUISHI and CQSO. We would also like to thank Atsushi Okawa, Nobuyuki Toyama, Yasuyuki Nasuhara, Toshihiro Kaneko, Masashi Uramatsu, Yumi Arai, and Kouichi Tanabe for their help in this research project. We thank Melissa Leffler, MBA, from Edanz ( for editing a draft of this manuscript.


This research was supported by Ministry of Health, Labour and Welfare Policy Research Grants, Research on Region Medical (20IA2001).

Author information

Authors and Affiliations



YN and MU conceived the idea for the study. HU led the writing of this paper. MU, MK, TU, MH, FK, TF and YN analyzed and interpreted the data. MU, MK, and YN contributed to the writing of the paper as well as participated in revising this manuscript. All authors contributed substantially to the writing of the paper, and all reviewed and approved the final draft.

Corresponding author

Correspondence to Haruhiro Uematsu.

Ethics declarations

Ethical approval

This study was approved by Nagoya University Hospital Research Ethics Committee [2020–0181].

Consent to participate

Written informed consent was not required for this study because patient data were anonymously used for the study and no interventions took place for the study.

Consent for publication

Not applicable.

Human and animal ethics

Not applicable.

Conflict of interests

The authors declared no potential conflicts of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Uematsu, H., Uemura, M., Kurihara, M. et al. Development of a Novel Scoring System to Quantify the Severity of Incident Reports: An Exploratory Research Study. J Med Syst 46, 106 (2022).

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: