Skip to main content
SpringerLink
Log in

Innovative Digital Tools and Surveillance Systems for the Timely Detection of Adverse Events at the Point of Care: A Proof-of-Concept Study

  • Original Research Article
  • Published:
Drug Safety Aims and scope Submit manuscript

Abstract

Introduction and Objective

Regulatory authorities often receive poorly structured safety reports requiring considerable effort to investigate potential adverse events post hoc. Automated question-and-answer systems may help to improve the overall quality of safety information transmitted to pharmacovigilance agencies. This paper explores the use of the VACC-Tool (ViVI Automated Case Classification Tool) 2.0, a mobile application enabling physicians to classify clinical cases according to 14 pre-defined case definitions for neuroinflammatory adverse events (NIAE) and in full compliance with data standards issued by the Clinical Data Interchange Standards Consortium.

Methods

The validation of the VACC-Tool 2.0 (beta-version) was conducted in the context of a unique quality management program for children with suspected NIAE in collaboration with the Robert Koch Institute in Berlin, Germany. The VACC-Tool was used for instant case classification and for longitudinal follow-up throughout the course of hospitalization. Results were compared to International Classification of Diseases , Tenth Revision (ICD-10) codes assigned in the emergency department (ED).

Results

From 07/2013 to 10/2014, a total of 34,368 patients were seen in the ED, and 5243 patients were hospitalized; 243 of these were admitted for suspected NIAE (mean age: 8.5 years), thus participating in the quality management program. Using the VACC-Tool in the ED, 209 cases were classified successfully, 69 % of which had been missed or miscoded in the ED reports. Longitudinal follow-up with the VACC-Tool identified additional NIAE.

Conclusion

Mobile applications are taking data standards to the point of care, enabling clinicians to ascertain potential adverse events in the ED setting and during inpatient follow-up. Compliance with Clinical Data Interchange Standards Consortium (CDISC) data standards facilitates data interoperability according to regulatory requirements.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  1. Varricchio F, Iskander J, Destefano F, Ball R, Pless R, Braun MM, et al. Understanding vaccine safety information from the vaccine adverse event reporting system. Pediatr Infect Dis J. 2004;23(4):287–94.

    Article  PubMed  Google Scholar 

  2. Raine J, Wise L, Blackburn S, Eichler HG, Breckenridge A. European perspective on risk management and drug safety. Clin Pharmacol Ther. 2011;89(5):650–4.

    Article  CAS  PubMed  Google Scholar 

  3. Miller ER, Haber P, Hibbs B, Broder K. Surveillance for adverse events following immunization using the Vaccine Adverse Event Reporting System (VAERS). 2014 1st April 2014. http://www.cdc.gov/vaccines/pubs/surv-manual/chpt21-surv-adverse-events.html#f25. Accessed 1 Jan 2016.

  4. Uppsala Monitoring Centre. To improve worldwide patient safety. 2015. http://www.who-umc.org/. Accessed 2 Jan 2016.

  5. Pal SN, Duncombe C, Falzon D, Olsson S. WHO strategy for collecting safety data in public health programmes: complementing spontaneous reporting systems. Drug Saf. 2013;36(2):75–81.

    Article  PubMed  PubMed Central  Google Scholar 

  6. Shimabukuro TT, Nguyen M, Martin D, DeStefano F. Safety monitoring in the vaccine adverse event reporting system (VAERS). Vaccine. 2015;33(36):4398–405.

    Article  PubMed  PubMed Central  Google Scholar 

  7. Linder JA, Haas JS, Iyer A, Labuzetta MA, Ibara M, Celeste M, et al. Secondary use of electronic health record data: spontaneous triggered adverse drug event reporting. Pharmacoepidemiol Drug Saf. 2010;19(12):1211–5.

    Article  PubMed  Google Scholar 

  8. Muehlhans S, Richard G, Ali M, Codarini G, Elemuwa C, Khamesipour A, et al. Safety reporting in developing country vaccine clinical trials-a systematic review. Vaccine. 2012;30(22):3255–65.

    Article  PubMed  Google Scholar 

  9. Schroll JB, Maund E, Gotzsche PC. Challenges in coding adverse events in clinical trials: a systematic review. PLoS One. 2012;7(7):e41174.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  10. Crepin S, Villeneuve C, Merle L. Quality of serious adverse events reporting to academic sponsors of clinical trials: far from optimal. Pharmacoepidemiol Drug Saf. 2016;25(6):719–24.

    Article  PubMed  Google Scholar 

  11. Baker MA, Kaelber DC, Bar-Shain DS, Moro PL, Zambarano B, Mazza M, et al. Advanced clinical decision support for vaccine adverse event detection and reporting. Clin Infect Dis. 2015;61(6):864–70.

    Article  PubMed  Google Scholar 

  12. Bailey C, Peddie D, Wickham ME, Badke K, Small SS, Doyle-Waters MM, et al. Adverse drug event reporting systems:a systematic review. Br J Clin Pharmacol. 2016 [Epub ahead of print].

  13. Clinical Data Interchange Standards Consortium. Analysis Data Model (ADaM): data structure for adverse event analysis. 2012. http://www.cdisc.org/system/files/all/standard_category/application/pdf/adam_ae_final_v1.pdf. Accessed 5 Jun 2016.

  14. Beresniak A, Schmidt A, Proeve J, Bolanos E, Patel N, Ammour N, et al. Cost-benefit assessment of using electronic health records data for clinical research versus current practices: contribution of the electronic health records for clinical research (EHR4CR) European project. Contemp Clin Trials. 2016;46:85–91.

    Article  PubMed  Google Scholar 

  15. De Moor G, Sundgren M, Kalra D, Schmidt A, Dugas M, Claerhout B, et al. Using electronic health records for clinical research: the case of the EHR4CR project. J Biomed Informat. 2015;53:162–73.

    Article  Google Scholar 

  16. Li Y, Ryan PB, Wei Y, Friedman C. A method to combine signals from spontaneous reporting systems and observational healthcare data to detect adverse drug reactions. Drug Saf. 2015;38(10):895–908.

    Article  PubMed  PubMed Central  Google Scholar 

  17. Lopez-Gonzalez E, Herdeiro MT, Figueiras A. Determinants of under-reporting of adverse drug reactions: a systematic review. Drug Saf. 2009;32(1):19–31.

    Article  CAS  PubMed  Google Scholar 

  18. Hazell L, Shakir SA. Under-reporting of adverse drug reactions: a systematic review. Drug Saf. 2006;29(5):385–96.

    Article  PubMed  Google Scholar 

  19. Rosenthal S, Chen R. The reporting sensitivities of two passive surveillance systems for vaccine adverse events. Am J Public Health. 1995;85(12):1706–9.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  20. Horwitz RI, Yu EC. Assessing the reliability of epidemiologic data obtained from medical records. J Chron Dis. 1984;37(11):825–31.

    Article  CAS  PubMed  Google Scholar 

  21. Rath B, Magnus M, Heininger U. Evaluating the Brighton Collaboration case definitions, aseptic meningitis, encephalitis, myelitis, and acute disseminated encephalomyelitis, by systematic analysis of 255 clinical cases. Vaccine. 2010;28(19):3488–95.

    Article  PubMed  Google Scholar 

  22. Buajordet I, Ebbesen J, Erikssen J, Brors O, Hilberg T. Fatal adverse drug events: the paradox of drug treatment. J Intern Med. 2001;250(4):327–41.

    Article  CAS  PubMed  Google Scholar 

  23. Evans SR. Clinical trial structures. J Exp Stroke Transl Med. 2010;3(1):8–18.

    Article  PubMed  PubMed Central  Google Scholar 

  24. Obermeier P, Muehlhans S, Hoppe C, Karsch K, Tief F, Seeber L, et al. Enabling precision medicine with digital case classification at the point-of-care. EBioMedicine. 2016;4:191–6.

    Article  PubMed  PubMed Central  Google Scholar 

  25. World Health Organization. Causality assessment of an adverse event following immunization (AEFI). 2013. http://www.who.int/vaccine_safety/publications/aefi_manual.pdf?ua=1. Accessed 28 Apr 2016.

  26. Sejvar JJ, Kohl KS, Gidudu J, Amato A, Bakshi N, Baxter R, et al. Guillain-Barre syndrome and Fisher syndrome: case definitions and guidelines for collection, analysis, and presentation of immunization safety data. Vaccine. 2011;29(3):599–612.

    Article  PubMed  Google Scholar 

  27. Sejvar JJ, Kohl KS, Bilynsky R, Blumberg D, Cvetkovich T, Galama J, et al. Encephalitis, myelitis, and acute disseminated encephalomyelitis (ADEM): case definitions and guidelines for collection, analysis, and presentation of immunization safety data. Vaccine. 2007;25(31):5771–92.

    Article  CAS  PubMed  Google Scholar 

  28. Tapiainen T, Prevots R, Izurieta HS, Abramson J, Bilynsky R, Bonhoeffer J, et al. Aseptic meningitis: case definition and guidelines for collection, analysis and presentation of immunization safety data. Vaccine. 2007;25(31):5793–802.

    Article  CAS  PubMed  Google Scholar 

  29. Haber P, Slade B, Iskander J. Letter to the Editor. Guillain-Barre Syndrome (GBS) after vaccination reported to the United States Vaccine Adverse Event Reporting System (VAERS) in 2004. Vaccine. 2007;25(48):8101.

    Article  PubMed  Google Scholar 

  30. Wender M. Acute disseminated encephalomyelitis (ADEM). J Neuroimmunol. 2011;231(1–2):92–9.

    Article  CAS  PubMed  Google Scholar 

  31. Joshi D, Alsentzer E, Edwards K, Norton A, Williams SE. An algorithm developed using the Brighton Collaboration case definitions is more efficient for determining diagnostic certainty. Vaccine. 2014;32(28):3469–72.

    Article  PubMed  Google Scholar 

  32. Clinical Data Interchange Standards Consortium. Clinical Data Acquisition Standards Harmonization (CDASH). 2016.; http://www.cdisc.org/cdash. Accessed 9 Mar 2016.

  33. Karsch K, Obermeier P, Seeber L, Chen X, Tief F, Muhlhans S, et al. Human parechovirus infections associated with seizures and rash in infants and toddlers. Pediatr Infect Dis J. 2015;34(10):1049–55.

    Article  PubMed  Google Scholar 

  34. Obermeier PE, Karsch K, Hoppe C, Seeber L, Schneider J, Muhlhans S, et al. Acute disseminated encephalomyelitis after human parechovirus infection. Pediatr Infect Dis J. 2016;35(1):35–8.

    PubMed  Google Scholar 

  35. Seeber L, Michl B, Rundblad G, Trusko B, Schnjakin M, Meinel C, et al. A design thinking approach to effective vaccine safety communication. Curr Drug Saf. 2015;10(1):31–40.

    Article  PubMed  Google Scholar 

  36. The Brighton Collaboration. AEFI case definition document. https://brightoncollaboration.org/public/what-we-do/setting-standards/case-definitions/process/main/02/link/Case_Definition_Format_Template.pdf. Accessed 3 Jan 2016.

  37. Clinical Data Interchange Standards Consortium. Study Data Tabulation Model (SDTM). http://www.cdisc.org/sdtm. Accessed 10 Jan 2016.

  38. Clinical Data Interchange Standards Consortium. Clinical Data Acquisition Standards Harmonization (CDASH). http://www.cdisc.org/cdash. Accessed 10 Jan 2016.

  39. Clinical Data Interchange Standards Consortium. Biomedical Research Integrated Domain Group (BRIDG). http://www.cdisc.org/bridg. Accessed 10 Jan 2016.

  40. U.S. Food and Drug Administration. Providing regulatory submissions in electronic format: standardized study data. December 2014. http://www.fda.gov/downloads/Drugs/GuidanceComplianceRegulatoryInformation/Guidances/UCM292334.pdf. Accessed 9 Mar 2016.

  41. U.S. Food and Drug Administration. Study data standards resources. October 2015. http://www.fda.gov/forindustry/datastandards/studydatastandards/default.htm. Accessed 9 Mar 2016.

  42. Nelson JC, Cook AJ, Yu O, Zhao S, Jackson LA, Psaty BM. Methods for observational post-licensure medical product safety surveillance. Stat Methods Med Res. 2015;24(2):177–93.

    Article  PubMed  Google Scholar 

  43. U.S. Food and Drug Administration. Statistical guidance on reporting results from studies evaluating diagnostic tests. 2007. http://www.fda.gov/RegulatoryInformation/Guidances/ucm071148.htm. Accessed 10 Jan 2016.

  44. Donner A, Rotundi MA. Sample size requirements for interval estimation of the kappa statistic for interobserver agreement studies with a binary outcome and multiple raters. Int J Biostat. 2010;6(1):Article 31.

  45. Hall MA. Correlation-based feature subset selection for machine learning. Hamilton: University of Waikato; 1998.

    Google Scholar 

  46. Kullback S. Letter to the eitor: the Kullback-Leibler distance. Am Stat. 1987;41(4):340–1.

    Google Scholar 

  47. Pillans PI. Clinical perspectives in drug safety and adverse drug reactions. Expert Rev Clin Pharmacol. 2008;1(5):695–705.

    Article  PubMed  Google Scholar 

  48. Miller E, Haber P, Hibbs B, Broder K. Surveillance for adverse events following immunization using the Vaccine Adverse Event Reporting System (VAERS). 2014. http://www.cdc.gov/vaccines/pubs/surv-manual/chpt21-surv-adverse-events.html#f5. Accessed 17 June 2016.

  49. Marks RG. Validating electronic source data in clinical trials. Control Clin Trials. 2004;25(5):437–46.

    Article  PubMed  Google Scholar 

  50. Rosa C, Campbell AN, Miele GM, Brunner M, Winstanley EL. Using e-technologies in clinical trials. Contemp Clin Trials. 2015;45(Pt A):41–54.

    Article  PubMed  Google Scholar 

  51. Baker MA, Nguyen M, Cole DV, Lee GM, Lieu TA. Post-licensure rapid immunization safety monitoring program (PRISM) data characterization. Vaccine. 2013;30(31 Suppl 10):K98–112.

    Article  Google Scholar 

  52. McNeil MM, Gee J, Weintraub ES, Belongia EA, Lee GM, Glanz JM, et al. The vaccine safety datalink: successes and challenges monitoring vaccine safety. Vaccine. 2014;32(42):5390–8.

    Article  PubMed  Google Scholar 

  53. Chen RT, Glasser JW, Rhodes PH, Davis RL, Barlow WE, Thompson RS, et al. Vaccine Safety Datalink project: a new tool for improving vaccine safety monitoring in the United States. The Vaccine Safety Datalink Team. Pediatrics. 1997;99(6):765–73.

    Article  CAS  PubMed  Google Scholar 

  54. Yih WK, Kulldorff M, Sandhu SK, Zichittella L, Maro JC, Cole DV, et al. Prospective influenza vaccine safety surveillance using fresh data in the sentinel system. Pharmacoepidemiol Drug Saf. 2016;25(5):481–92.

    Article  PubMed  Google Scholar 

  55. World Health Organization. International Statistical Classification of Diseases and Related Health Problems 10th Revision (ICD-10)-WHO Version for 2016. http://apps.who.int/classifications/icd10/browse/2016/en#/G51. Accessed 25 Feb 2016.

  56. St Germaine-Smith C, Metcalfe A, Pringsheim T, Roberts JI, Beck CA, Hemmelgarn BR, et al. Recommendations for optimal ICD codes to study neurologic conditions: a systematic review. Neurology. 2012;79(10):1049–55.

    Article  PubMed  PubMed Central  Google Scholar 

  57. Hughes PS, Jackson AC. Delays in initiation of acyclovir therapy in herpes simplex encephalitis. Can J Neurol Sci. 2012;39(5):644–8.

    Article  PubMed  Google Scholar 

  58. O’Riordan JI, Thompson AJ, Kingsley DP, MacManus DG, Kendall BE, Rudge P, et al. The prognostic value of brain MRI in clinically isolated syndromes of the CNS: a 10-year follow-up. Brain. 1998;121(Pt 3):495–503.

    Article  PubMed  Google Scholar 

  59. Alexander M, Murthy JM. Acute disseminated encephalomyelitis: treatment guidelines. Annal Indian Acad Neurol. 2011;14(Suppl 1):S60–4.

    Article  CAS  Google Scholar 

  60. Rath B, Gidudu JF, Anyoti H, Bollweg B, Caubel P, Chen YH, et al. Facial nerve palsy including Bell’s palsy: case definitions and guidelines for collection, analysis, and presentation of immunisation safety data. Vaccine. 2016 [Epub ahead of print].

  61. Marcy SM, Kohl KS, Dagan R, Nalin D, Blum M, Jones MC, et al. Fever as an adverse event following immunization: case definition and guidelines of data collection, analysis, and presentation. Vaccine. 2004;22(5–6):551–6.

    Article  PubMed  Google Scholar 

  62. Bonhoeffer J, Menkes J, Gold MS, de Souza-Brito G, Fisher MC, Halsey N, et al. Generalized convulsive seizure as an adverse event following immunization: case definition and guidelines for data collection, analysis, and presentation. Vaccine. 2004;22(5–6):557–62.

    Article  PubMed  Google Scholar 

  63. Britton PN, Dale RC, Elliott E, Festa M, Macartney K, Booy R, et al. Pilot surveillance for childhood encephalitis in Australia using the paediatric active enhanced disease surveillance (PAEDS) network. Epidemiol Infect. 2016;26:1–11.

    Article  Google Scholar 

  64. Twilt M. Precision medicine: the new era in medicine. 2016. http://dx.doi.org/10.1016/j.ebiom.2016.02.009. Accessed 17 June 2016.

  65. Vellozzi C, Iqbal S, Broder K. Guillain-Barre syndrome, influenza, and influenza vaccination: the epidemiologic evidence. Clin Infect Dis. 2014;58(8):1149–55.

    Article  PubMed  Google Scholar 

  66. Rellosa N, Bloch KC, Shane AL, Debiasi RL. Neurologic manifestations of pediatric novel h1n1 influenza infection. Pediatr Infect Dis J. 2011;30(2):165–7.

    Article  PubMed  Google Scholar 

  67. Evans SR. Fundamentals of clinical trial design. J Exp Stroke Transl Med. 2010;3(1):19–27.

    Article  PubMed  PubMed Central  Google Scholar 

  68. Roberts KB. Management and outcomes of care of fever in early infancy. J Pediatr. 2004;145(3):417.

    Article  PubMed  Google Scholar 

  69. Muehlhans S, von Kleist M, Gretchukha T, Terhardt M, Fegeler U, Maurer W, et al. Awareness and utilization of reporting pathways for adverse events following immunization: online survey among pediatricians in Russia and Germany. Paediatr Drugs. 2014;16(4):321–30.

    Article  PubMed  Google Scholar 

  70. Poli F, Overeem S, Lammers GJ, Plazzi G, Lecendreux M, Bassetti CL, et al. Narcolepsy as an adverse event following immunization: case definition and guidelines for data collection, analysis and presentation. Vaccine. 2013;31(6):994–1007.

    Article  PubMed  Google Scholar 

  71. Jones JF, Kohl KS, Ahmadipour N, Bleijenberg G, Buchwald D, Evengard B, et al. Fatigue: case definition and guidelines for collection, analysis, and presentation of immunization safety data. Vaccine. 2007;25(31):5685–96.

    Article  CAS  PubMed  Google Scholar 

  72. Mentzer D, Prestel J, Adams O, Gold R, Hartung HP, Hengel H, et al. Case definition for progressive multifocal leukoencephalopathy following treatment with monoclonal antibodies. J Neurol Neurosurg Psychiatry. 2012;83(9):927–33.

    Article  PubMed  Google Scholar 

  73. Bonhoeffer J, Vermeer P, Halperin S, Kempe A, Music S, Shindman J, et al. Persistent crying in infants and children as an adverse event following immunization: case definition and guidelines for data collection, analysis, and presentation. Vaccine. 2004;22(5–6):586–91.

    Article  PubMed  Google Scholar 

Download references

Author information

Author notes

    Authors and Affiliations

    1. Department of Pediatrics, Charité University Medical Center Berlin, Berlin, Germany

      Christian Hoppe, Patrick Obermeier, Susann Muehlhans, Maren Alchikh, Lea Seeber, Franziska Tief, Katharina Karsch, Xi Chen & Barbara Rath

    2. Vienna Vaccine Safety Initiative, Berlin, Germany

      Christian Hoppe, Patrick Obermeier, Susann Muehlhans, Maren Alchikh, Lea Seeber, Franziska Tief, Katharina Karsch, Xi Chen, Bron Kisler & Barbara Rath

    3. National Reference Centre for Poliomyelitis and Enteroviruses, Robert Koch Institute, Berlin, Germany

      Sindy Boettcher & Sabine Diedrich

    4. Department of Mathematics and Computer Sciences, Freie Universität Berlin, Berlin, Germany

      Tim Conrad

    5. Clinical Data Interchange Standards Consortium, Austin, TX, USA

      Bron Kisler

    Authors
    1. Christian Hoppe
      View author publications

      You can also search for this author in PubMed Google Scholar

    2. Patrick Obermeier
      View author publications

      You can also search for this author in PubMed Google Scholar

    3. Susann Muehlhans
      View author publications

      You can also search for this author in PubMed Google Scholar

    4. Maren Alchikh
      View author publications

      You can also search for this author in PubMed Google Scholar

    5. Lea Seeber
      View author publications

      You can also search for this author in PubMed Google Scholar

    6. Franziska Tief
      View author publications

      You can also search for this author in PubMed Google Scholar

    7. Katharina Karsch
      View author publications

      You can also search for this author in PubMed Google Scholar

    8. Xi Chen
      View author publications

      You can also search for this author in PubMed Google Scholar

    9. Sindy Boettcher
      View author publications

      You can also search for this author in PubMed Google Scholar

    10. Sabine Diedrich
      View author publications

      You can also search for this author in PubMed Google Scholar

    11. Tim Conrad
      View author publications

      You can also search for this author in PubMed Google Scholar

    12. Bron Kisler
      View author publications

      You can also search for this author in PubMed Google Scholar

    13. Barbara Rath
      View author publications

      You can also search for this author in PubMed Google Scholar

    Corresponding author

    Correspondence to Barbara Rath.

    Ethics declarations

    Funding

    No sources of funding were used to assist in the preparation of this study. Virus diagnostics for the quality management program were provided in-kind by the Robert Koch Institute.

    Conflicts of interest

    Christian Hoppe, Patrick Obermeier, Susann Muehlhans, Maren Alchikh, Lea Seeber, Franziska Tief, Katharina Karsch, Xi Chen, Sindy Boettcher, Sabine Diedrich, Tim Conrad, Bron Kisler, and Barbara Rath have no conflicts of interest that are directly relevant to the content of this study.

    Ethical approval

    The quality management program was approved by the Charité Institutional Review Board (EA2/161/11). Informed consent procedures were waived by the Institutional Review Board for the purpose of quality improvement and infection control.

    Additional information

    C. Hoppe, P. Obermeier and B. Rath contributed equally to the article.

    Rights and permissions

    Reprints and permissions

    About this article

    Check for updates. Verify currency and authenticity via CrossMark

    Cite this article

    Hoppe, C., Obermeier, P., Muehlhans, S. et al. Innovative Digital Tools and Surveillance Systems for the Timely Detection of Adverse Events at the Point of Care: A Proof-of-Concept Study. Drug Saf 39, 977–988 (2016). https://doi.org/10.1007/s40264-016-0437-6

    Download citation

    • Published:

    • Issue Date:

    • DOI: https://doi.org/10.1007/s40264-016-0437-6

    Keywords

    Search

    Navigation