Drug Safety

, Volume 39, Issue 10, pp 977–988 | Cite as

Innovative Digital Tools and Surveillance Systems for the Timely Detection of Adverse Events at the Point of Care: A Proof-of-Concept Study

  • Christian Hoppe
  • Patrick Obermeier
  • Susann Muehlhans
  • Maren Alchikh
  • Lea Seeber
  • Franziska Tief
  • Katharina Karsch
  • Xi Chen
  • Sindy Boettcher
  • Sabine Diedrich
  • Tim Conrad
  • Bron Kisler
  • Barbara Rath
Original Research Article


Introduction and Objective

Regulatory authorities often receive poorly structured safety reports requiring considerable effort to investigate potential adverse events post hoc. Automated question-and-answer systems may help to improve the overall quality of safety information transmitted to pharmacovigilance agencies. This paper explores the use of the VACC-Tool (ViVI Automated Case Classification Tool) 2.0, a mobile application enabling physicians to classify clinical cases according to 14 pre-defined case definitions for neuroinflammatory adverse events (NIAE) and in full compliance with data standards issued by the Clinical Data Interchange Standards Consortium.


The validation of the VACC-Tool 2.0 (beta-version) was conducted in the context of a unique quality management program for children with suspected NIAE in collaboration with the Robert Koch Institute in Berlin, Germany. The VACC-Tool was used for instant case classification and for longitudinal follow-up throughout the course of hospitalization. Results were compared to International Classification of Diseases , Tenth Revision (ICD-10) codes assigned in the emergency department (ED).


From 07/2013 to 10/2014, a total of 34,368 patients were seen in the ED, and 5243 patients were hospitalized; 243 of these were admitted for suspected NIAE (mean age: 8.5 years), thus participating in the quality management program. Using the VACC-Tool in the ED, 209 cases were classified successfully, 69 % of which had been missed or miscoded in the ED reports. Longitudinal follow-up with the VACC-Tool identified additional NIAE.


Mobile applications are taking data standards to the point of care, enabling clinicians to ascertain potential adverse events in the ED setting and during inpatient follow-up. Compliance with Clinical Data Interchange Standards Consortium (CDISC) data standards facilitates data interoperability according to regulatory requirements.


Aseptic Meningitis Progressive Multifocal Leukoencephalopathy Convulsive Seizure Vaccine Adverse Event Reporting System Case Classification 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Compliance with Ethical Standards


No sources of funding were used to assist in the preparation of this study. Virus diagnostics for the quality management program were provided in-kind by the Robert Koch Institute.

Conflicts of interest

Christian Hoppe, Patrick Obermeier, Susann Muehlhans, Maren Alchikh, Lea Seeber, Franziska Tief, Katharina Karsch, Xi Chen, Sindy Boettcher, Sabine Diedrich, Tim Conrad, Bron Kisler, and Barbara Rath have no conflicts of interest that are directly relevant to the content of this study.

Ethical approval

The quality management program was approved by the Charité Institutional Review Board (EA2/161/11). Informed consent procedures were waived by the Institutional Review Board for the purpose of quality improvement and infection control.


  1. 1.
    Varricchio F, Iskander J, Destefano F, Ball R, Pless R, Braun MM, et al. Understanding vaccine safety information from the vaccine adverse event reporting system. Pediatr Infect Dis J. 2004;23(4):287–94.CrossRefPubMedGoogle Scholar
  2. 2.
    Raine J, Wise L, Blackburn S, Eichler HG, Breckenridge A. European perspective on risk management and drug safety. Clin Pharmacol Ther. 2011;89(5):650–4.CrossRefPubMedGoogle Scholar
  3. 3.
    Miller ER, Haber P, Hibbs B, Broder K. Surveillance for adverse events following immunization using the Vaccine Adverse Event Reporting System (VAERS). 2014 1st April 2014. Accessed 1 Jan 2016.
  4. 4.
    Uppsala Monitoring Centre. To improve worldwide patient safety. 2015. Accessed 2 Jan 2016.
  5. 5.
    Pal SN, Duncombe C, Falzon D, Olsson S. WHO strategy for collecting safety data in public health programmes: complementing spontaneous reporting systems. Drug Saf. 2013;36(2):75–81.CrossRefPubMedPubMedCentralGoogle Scholar
  6. 6.
    Shimabukuro TT, Nguyen M, Martin D, DeStefano F. Safety monitoring in the vaccine adverse event reporting system (VAERS). Vaccine. 2015;33(36):4398–405.CrossRefPubMedPubMedCentralGoogle Scholar
  7. 7.
    Linder JA, Haas JS, Iyer A, Labuzetta MA, Ibara M, Celeste M, et al. Secondary use of electronic health record data: spontaneous triggered adverse drug event reporting. Pharmacoepidemiol Drug Saf. 2010;19(12):1211–5.CrossRefPubMedGoogle Scholar
  8. 8.
    Muehlhans S, Richard G, Ali M, Codarini G, Elemuwa C, Khamesipour A, et al. Safety reporting in developing country vaccine clinical trials-a systematic review. Vaccine. 2012;30(22):3255–65.CrossRefPubMedGoogle Scholar
  9. 9.
    Schroll JB, Maund E, Gotzsche PC. Challenges in coding adverse events in clinical trials: a systematic review. PLoS One. 2012;7(7):e41174.CrossRefPubMedPubMedCentralGoogle Scholar
  10. 10.
    Crepin S, Villeneuve C, Merle L. Quality of serious adverse events reporting to academic sponsors of clinical trials: far from optimal. Pharmacoepidemiol Drug Saf. 2016;25(6):719–24.CrossRefPubMedGoogle Scholar
  11. 11.
    Baker MA, Kaelber DC, Bar-Shain DS, Moro PL, Zambarano B, Mazza M, et al. Advanced clinical decision support for vaccine adverse event detection and reporting. Clin Infect Dis. 2015;61(6):864–70.CrossRefPubMedGoogle Scholar
  12. 12.
    Bailey C, Peddie D, Wickham ME, Badke K, Small SS, Doyle-Waters MM, et al. Adverse drug event reporting systems:a systematic review. Br J Clin Pharmacol. 2016 [Epub ahead of print].Google Scholar
  13. 13.
    Clinical Data Interchange Standards Consortium. Analysis Data Model (ADaM): data structure for adverse event analysis. 2012. Accessed 5 Jun 2016.
  14. 14.
    Beresniak A, Schmidt A, Proeve J, Bolanos E, Patel N, Ammour N, et al. Cost-benefit assessment of using electronic health records data for clinical research versus current practices: contribution of the electronic health records for clinical research (EHR4CR) European project. Contemp Clin Trials. 2016;46:85–91.CrossRefPubMedGoogle Scholar
  15. 15.
    De Moor G, Sundgren M, Kalra D, Schmidt A, Dugas M, Claerhout B, et al. Using electronic health records for clinical research: the case of the EHR4CR project. J Biomed Informat. 2015;53:162–73.CrossRefGoogle Scholar
  16. 16.
    Li Y, Ryan PB, Wei Y, Friedman C. A method to combine signals from spontaneous reporting systems and observational healthcare data to detect adverse drug reactions. Drug Saf. 2015;38(10):895–908.CrossRefPubMedPubMedCentralGoogle Scholar
  17. 17.
    Lopez-Gonzalez E, Herdeiro MT, Figueiras A. Determinants of under-reporting of adverse drug reactions: a systematic review. Drug Saf. 2009;32(1):19–31.CrossRefPubMedGoogle Scholar
  18. 18.
    Hazell L, Shakir SA. Under-reporting of adverse drug reactions: a systematic review. Drug Saf. 2006;29(5):385–96.CrossRefPubMedGoogle Scholar
  19. 19.
    Rosenthal S, Chen R. The reporting sensitivities of two passive surveillance systems for vaccine adverse events. Am J Public Health. 1995;85(12):1706–9.CrossRefPubMedPubMedCentralGoogle Scholar
  20. 20.
    Horwitz RI, Yu EC. Assessing the reliability of epidemiologic data obtained from medical records. J Chron Dis. 1984;37(11):825–31.CrossRefPubMedGoogle Scholar
  21. 21.
    Rath B, Magnus M, Heininger U. Evaluating the Brighton Collaboration case definitions, aseptic meningitis, encephalitis, myelitis, and acute disseminated encephalomyelitis, by systematic analysis of 255 clinical cases. Vaccine. 2010;28(19):3488–95.CrossRefPubMedGoogle Scholar
  22. 22.
    Buajordet I, Ebbesen J, Erikssen J, Brors O, Hilberg T. Fatal adverse drug events: the paradox of drug treatment. J Intern Med. 2001;250(4):327–41.CrossRefPubMedGoogle Scholar
  23. 23.
    Evans SR. Clinical trial structures. J Exp Stroke Transl Med. 2010;3(1):8–18.CrossRefPubMedPubMedCentralGoogle Scholar
  24. 24.
    Obermeier P, Muehlhans S, Hoppe C, Karsch K, Tief F, Seeber L, et al. Enabling precision medicine with digital case classification at the point-of-care. EBioMedicine. 2016;4:191–6.CrossRefPubMedPubMedCentralGoogle Scholar
  25. 25.
    World Health Organization. Causality assessment of an adverse event following immunization (AEFI). 2013. Accessed 28 Apr 2016.
  26. 26.
    Sejvar JJ, Kohl KS, Gidudu J, Amato A, Bakshi N, Baxter R, et al. Guillain-Barre syndrome and Fisher syndrome: case definitions and guidelines for collection, analysis, and presentation of immunization safety data. Vaccine. 2011;29(3):599–612.CrossRefPubMedGoogle Scholar
  27. 27.
    Sejvar JJ, Kohl KS, Bilynsky R, Blumberg D, Cvetkovich T, Galama J, et al. Encephalitis, myelitis, and acute disseminated encephalomyelitis (ADEM): case definitions and guidelines for collection, analysis, and presentation of immunization safety data. Vaccine. 2007;25(31):5771–92.CrossRefPubMedGoogle Scholar
  28. 28.
    Tapiainen T, Prevots R, Izurieta HS, Abramson J, Bilynsky R, Bonhoeffer J, et al. Aseptic meningitis: case definition and guidelines for collection, analysis and presentation of immunization safety data. Vaccine. 2007;25(31):5793–802.CrossRefPubMedGoogle Scholar
  29. 29.
    Haber P, Slade B, Iskander J. Letter to the Editor. Guillain-Barre Syndrome (GBS) after vaccination reported to the United States Vaccine Adverse Event Reporting System (VAERS) in 2004. Vaccine. 2007;25(48):8101.CrossRefPubMedGoogle Scholar
  30. 30.
    Wender M. Acute disseminated encephalomyelitis (ADEM). J Neuroimmunol. 2011;231(1–2):92–9.CrossRefPubMedGoogle Scholar
  31. 31.
    Joshi D, Alsentzer E, Edwards K, Norton A, Williams SE. An algorithm developed using the Brighton Collaboration case definitions is more efficient for determining diagnostic certainty. Vaccine. 2014;32(28):3469–72.CrossRefPubMedGoogle Scholar
  32. 32.
    Clinical Data Interchange Standards Consortium. Clinical Data Acquisition Standards Harmonization (CDASH). 2016.; Accessed 9 Mar 2016.
  33. 33.
    Karsch K, Obermeier P, Seeber L, Chen X, Tief F, Muhlhans S, et al. Human parechovirus infections associated with seizures and rash in infants and toddlers. Pediatr Infect Dis J. 2015;34(10):1049–55.CrossRefPubMedGoogle Scholar
  34. 34.
    Obermeier PE, Karsch K, Hoppe C, Seeber L, Schneider J, Muhlhans S, et al. Acute disseminated encephalomyelitis after human parechovirus infection. Pediatr Infect Dis J. 2016;35(1):35–8.PubMedGoogle Scholar
  35. 35.
    Seeber L, Michl B, Rundblad G, Trusko B, Schnjakin M, Meinel C, et al. A design thinking approach to effective vaccine safety communication. Curr Drug Saf. 2015;10(1):31–40.CrossRefPubMedGoogle Scholar
  36. 36.
  37. 37.
    Clinical Data Interchange Standards Consortium. Study Data Tabulation Model (SDTM). Accessed 10 Jan 2016.
  38. 38.
    Clinical Data Interchange Standards Consortium. Clinical Data Acquisition Standards Harmonization (CDASH). Accessed 10 Jan 2016.
  39. 39.
    Clinical Data Interchange Standards Consortium. Biomedical Research Integrated Domain Group (BRIDG). Accessed 10 Jan 2016.
  40. 40.
    U.S. Food and Drug Administration. Providing regulatory submissions in electronic format: standardized study data. December 2014. Accessed 9 Mar 2016.
  41. 41.
    U.S. Food and Drug Administration. Study data standards resources. October 2015. Accessed 9 Mar 2016.
  42. 42.
    Nelson JC, Cook AJ, Yu O, Zhao S, Jackson LA, Psaty BM. Methods for observational post-licensure medical product safety surveillance. Stat Methods Med Res. 2015;24(2):177–93.CrossRefPubMedGoogle Scholar
  43. 43.
    U.S. Food and Drug Administration. Statistical guidance on reporting results from studies evaluating diagnostic tests. 2007. Accessed 10 Jan 2016.
  44. 44.
    Donner A, Rotundi MA. Sample size requirements for interval estimation of the kappa statistic for interobserver agreement studies with a binary outcome and multiple raters. Int J Biostat. 2010;6(1):Article 31.Google Scholar
  45. 45.
    Hall MA. Correlation-based feature subset selection for machine learning. Hamilton: University of Waikato; 1998.Google Scholar
  46. 46.
    Kullback S. Letter to the eitor: the Kullback-Leibler distance. Am Stat. 1987;41(4):340–1.Google Scholar
  47. 47.
    Pillans PI. Clinical perspectives in drug safety and adverse drug reactions. Expert Rev Clin Pharmacol. 2008;1(5):695–705.CrossRefPubMedGoogle Scholar
  48. 48.
    Miller E, Haber P, Hibbs B, Broder K. Surveillance for adverse events following immunization using the Vaccine Adverse Event Reporting System (VAERS). 2014. Accessed 17 June 2016.
  49. 49.
    Marks RG. Validating electronic source data in clinical trials. Control Clin Trials. 2004;25(5):437–46.CrossRefPubMedGoogle Scholar
  50. 50.
    Rosa C, Campbell AN, Miele GM, Brunner M, Winstanley EL. Using e-technologies in clinical trials. Contemp Clin Trials. 2015;45(Pt A):41–54.CrossRefPubMedGoogle Scholar
  51. 51.
    Baker MA, Nguyen M, Cole DV, Lee GM, Lieu TA. Post-licensure rapid immunization safety monitoring program (PRISM) data characterization. Vaccine. 2013;30(31 Suppl 10):K98–112.CrossRefGoogle Scholar
  52. 52.
    McNeil MM, Gee J, Weintraub ES, Belongia EA, Lee GM, Glanz JM, et al. The vaccine safety datalink: successes and challenges monitoring vaccine safety. Vaccine. 2014;32(42):5390–8.CrossRefPubMedGoogle Scholar
  53. 53.
    Chen RT, Glasser JW, Rhodes PH, Davis RL, Barlow WE, Thompson RS, et al. Vaccine Safety Datalink project: a new tool for improving vaccine safety monitoring in the United States. The Vaccine Safety Datalink Team. Pediatrics. 1997;99(6):765–73.CrossRefPubMedGoogle Scholar
  54. 54.
    Yih WK, Kulldorff M, Sandhu SK, Zichittella L, Maro JC, Cole DV, et al. Prospective influenza vaccine safety surveillance using fresh data in the sentinel system. Pharmacoepidemiol Drug Saf. 2016;25(5):481–92.CrossRefPubMedGoogle Scholar
  55. 55.
    World Health Organization. International Statistical Classification of Diseases and Related Health Problems 10th Revision (ICD-10)-WHO Version for 2016. Accessed 25 Feb 2016.
  56. 56.
    St Germaine-Smith C, Metcalfe A, Pringsheim T, Roberts JI, Beck CA, Hemmelgarn BR, et al. Recommendations for optimal ICD codes to study neurologic conditions: a systematic review. Neurology. 2012;79(10):1049–55.CrossRefPubMedPubMedCentralGoogle Scholar
  57. 57.
    Hughes PS, Jackson AC. Delays in initiation of acyclovir therapy in herpes simplex encephalitis. Can J Neurol Sci. 2012;39(5):644–8.CrossRefPubMedGoogle Scholar
  58. 58.
    O’Riordan JI, Thompson AJ, Kingsley DP, MacManus DG, Kendall BE, Rudge P, et al. The prognostic value of brain MRI in clinically isolated syndromes of the CNS: a 10-year follow-up. Brain. 1998;121(Pt 3):495–503.CrossRefPubMedGoogle Scholar
  59. 59.
    Alexander M, Murthy JM. Acute disseminated encephalomyelitis: treatment guidelines. Annal Indian Acad Neurol. 2011;14(Suppl 1):S60–4.CrossRefGoogle Scholar
  60. 60.
    Rath B, Gidudu JF, Anyoti H, Bollweg B, Caubel P, Chen YH, et al. Facial nerve palsy including Bell’s palsy: case definitions and guidelines for collection, analysis, and presentation of immunisation safety data. Vaccine. 2016 [Epub ahead of print].Google Scholar
  61. 61.
    Marcy SM, Kohl KS, Dagan R, Nalin D, Blum M, Jones MC, et al. Fever as an adverse event following immunization: case definition and guidelines of data collection, analysis, and presentation. Vaccine. 2004;22(5–6):551–6.CrossRefPubMedGoogle Scholar
  62. 62.
    Bonhoeffer J, Menkes J, Gold MS, de Souza-Brito G, Fisher MC, Halsey N, et al. Generalized convulsive seizure as an adverse event following immunization: case definition and guidelines for data collection, analysis, and presentation. Vaccine. 2004;22(5–6):557–62.CrossRefPubMedGoogle Scholar
  63. 63.
    Britton PN, Dale RC, Elliott E, Festa M, Macartney K, Booy R, et al. Pilot surveillance for childhood encephalitis in Australia using the paediatric active enhanced disease surveillance (PAEDS) network. Epidemiol Infect. 2016;26:1–11.CrossRefGoogle Scholar
  64. 64.
    Twilt M. Precision medicine: the new era in medicine. 2016. Accessed 17 June 2016.
  65. 65.
    Vellozzi C, Iqbal S, Broder K. Guillain-Barre syndrome, influenza, and influenza vaccination: the epidemiologic evidence. Clin Infect Dis. 2014;58(8):1149–55.CrossRefPubMedGoogle Scholar
  66. 66.
    Rellosa N, Bloch KC, Shane AL, Debiasi RL. Neurologic manifestations of pediatric novel h1n1 influenza infection. Pediatr Infect Dis J. 2011;30(2):165–7.CrossRefPubMedGoogle Scholar
  67. 67.
    Evans SR. Fundamentals of clinical trial design. J Exp Stroke Transl Med. 2010;3(1):19–27.CrossRefPubMedPubMedCentralGoogle Scholar
  68. 68.
    Roberts KB. Management and outcomes of care of fever in early infancy. J Pediatr. 2004;145(3):417.CrossRefPubMedGoogle Scholar
  69. 69.
    Muehlhans S, von Kleist M, Gretchukha T, Terhardt M, Fegeler U, Maurer W, et al. Awareness and utilization of reporting pathways for adverse events following immunization: online survey among pediatricians in Russia and Germany. Paediatr Drugs. 2014;16(4):321–30.CrossRefPubMedGoogle Scholar
  70. 70.
    Poli F, Overeem S, Lammers GJ, Plazzi G, Lecendreux M, Bassetti CL, et al. Narcolepsy as an adverse event following immunization: case definition and guidelines for data collection, analysis and presentation. Vaccine. 2013;31(6):994–1007.CrossRefPubMedGoogle Scholar
  71. 71.
    Jones JF, Kohl KS, Ahmadipour N, Bleijenberg G, Buchwald D, Evengard B, et al. Fatigue: case definition and guidelines for collection, analysis, and presentation of immunization safety data. Vaccine. 2007;25(31):5685–96.CrossRefPubMedGoogle Scholar
  72. 72.
    Mentzer D, Prestel J, Adams O, Gold R, Hartung HP, Hengel H, et al. Case definition for progressive multifocal leukoencephalopathy following treatment with monoclonal antibodies. J Neurol Neurosurg Psychiatry. 2012;83(9):927–33.CrossRefPubMedGoogle Scholar
  73. 73.
    Bonhoeffer J, Vermeer P, Halperin S, Kempe A, Music S, Shindman J, et al. Persistent crying in infants and children as an adverse event following immunization: case definition and guidelines for data collection, analysis, and presentation. Vaccine. 2004;22(5–6):586–91.CrossRefPubMedGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  • Christian Hoppe
    • 1
    • 2
  • Patrick Obermeier
    • 1
    • 2
  • Susann Muehlhans
    • 1
    • 2
  • Maren Alchikh
    • 1
    • 2
  • Lea Seeber
    • 1
    • 2
  • Franziska Tief
    • 1
    • 2
  • Katharina Karsch
    • 1
    • 2
  • Xi Chen
    • 1
    • 2
  • Sindy Boettcher
    • 3
  • Sabine Diedrich
    • 3
  • Tim Conrad
    • 4
  • Bron Kisler
    • 2
    • 5
  • Barbara Rath
    • 1
    • 2
  1. 1.Department of PediatricsCharité University Medical Center BerlinBerlinGermany
  2. 2.Vienna Vaccine Safety InitiativeBerlinGermany
  3. 3.National Reference Centre for Poliomyelitis and EnterovirusesRobert Koch InstituteBerlinGermany
  4. 4.Department of Mathematics and Computer SciencesFreie Universität BerlinBerlinGermany
  5. 5.Clinical Data Interchange Standards ConsortiumAustinUSA

Personalised recommendations