System Assessment and Evaluation

  • Hsinchun Chen
  • Daniel Zeng
  • Ping Yan
Part of the Integrated Series in Information Systems book series (ISIS, volume 21)


Knowing how systems perform under various scenarios is important. We need to examine with which level of sensitivity and how quickly they can detect an outbreak or recognize a bioterrorism attack. Knowing the error rate of a system can help make decisions regarding how much effort is needed to investigate an alarm. The performance of the algorithms for outbreak characterization determines the amount of information they provide (e.g., sets of affected individuals, the outbreak size, and disease spreading rate), which provide important input for response planning.

Substantial costs can be incurred when developing or managing syndromic surveillance systems and investigating possible outbreaks based on the outputs of these systems (Reingold, 2003). For example, as reported in (Doroshenko et al., 2005), the annual cost of the NHS Direct Syndromic Surveillance System is about $280,000 and the usefulness of surveillance systems for early detection and response is yet to be established. Assessing the performance of surveillance systems is of significant importance for improving the efficacy of the investment in system development and management (Buehler et al., 2004).


False Alarm Rate Outbreak Detection Syndromic Surveillance System Outbreak Size Bioterrorism Attack 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. Aamodt, G., Samuelsen, S.O., and Skrondal, A. 2006. "A Simulation Study of Three Methods for Detecting Disease Clusters," International Journal of Health Geographics (5:15).PubMedCrossRefGoogle Scholar
  2. Buckeridge, D., Burkom, H., Campbell, M., Hogan, W., and Moore, A. 2005a. "Algorithms for Rapid Outbreak Detection: A Research Synthesis," Journal of Biomedical Informatics (38), pp. 99–113.PubMedCrossRefGoogle Scholar
  3. Buckeridge, D., Burkom, H., Moore, A., Pavlin, J., Cutchis, P., and Hogan, W. 2004. "Evaluation of Syndromic Surveillance Systems: Development of an Epidemic Simulation Model," MMWR (CDC) (53(Suppl.)), pp. 137–143.Google Scholar
  4. Buehler, J., Hopkins, R., Overhage, J., Sosin, D., and Tong, V. 2004. "Framework for Evaluating Public Health Surveillance Systems for Early Detection of Outbreaks: Recommendations from the Cdc Working Group," MMWR (CDC) (53(RR-5)), pp. 1–13.Google Scholar
  5. Carley, K., Fridsma, D., Casman, E., Altman, N., Chang, J., Kaminsky, B., Nave, D., and Yahja, A. 2003. "Biowar: Scalable Multi-Agent Social and Epidemiological Simulation of Bioterrorism Events."Google Scholar
  6. CDC. 2003. "HIPAA Privacy Rule and Public Health: Guidance from CDC and the US Department of Health and Human Services," MMWR (52(Suppl)), pp. 1–20.Google Scholar
  7. Chin, J.P., Diehl, V.A., and Norman, K.L. 1988. "Development of an Instrument Measuring User Satisfaction of the Human-Computer Interface," ACM CHI Washington, DC, pp. 213–218.Google Scholar
  8. Doroshenko, A., Cooper, D., Smith, G., Gerard, E., Chinemana, F., Verlander, N., and Nicoll, A. 2005. "Evaluation of Syndromic Surveillance Based on National Health Service Direct Derived Data - England and Wales," MMWR (54(Suppl)), pp. 117–122.PubMedGoogle Scholar
  9. Espino, J.U., and Wagner, M.M. 2001. "The Accuracy of ICD-9 Coded Chief Complaints for Detection of Acute Respiratory Illness," Proc AMIA Symp, pp. 164–168.Google Scholar
  10. German, R.R. 2000. "Sensitivity and Predictive Value Positive Measurements for Public Health Surveillance Systems," Epidemiology (11:6), pp. 720–727.PubMedCrossRefGoogle Scholar
  11. Goldenberg, A., Shmueli, G., Caruana, R.A., and Fienberg, S.E. 2002. "Early Statistical Detection of Anthrax Outbreaks by Tracking Over-The-Counter Medication Sales," Proceedings of the National Academy of Sciences USA (99:8), pp. 5237–5240.CrossRefGoogle Scholar
  12. Greenko, J., Mostashari, F., Fine, A., and Layton, M. 2003. "Clinical Evaluation of the Emergency Medica Services (EMS) Ambulance Dispatch-Based Syndromic Surveillance System, New York City," Journal of Urban Health: Bulletin of the New York Academy of Medicine (80:2), pp. i50–i56.Google Scholar
  13. Halkidi, M., Batistakis, Y., and Vazirgiannis, M. 2002. "Cluster Validity Methods: Part I " SIGMOD Rec. (31:2), pp 40–45.CrossRefGoogle Scholar
  14. Hogan, W.R., Tsui, F.-C., Ivanov, O., Gesteland, P.H., Grannis, S., Overhage, M., Robinson, J.M., and Wagner, M.M. 2003. "Detection of Pediatric Respiratory and Diarrheal Outbreaks from Sales of Over-The-Counter Electrolyte Products," Journal of American Medical Informatics Association (10:6), pp. 555–562.CrossRefGoogle Scholar
  15. Hu, P.J.-H., Zeng, D., Chen, H., Larson, C.A., Chang, W., and Tseng, C. 2005. "Evaluating an Infectious Disease Information Sharing and Analysis System," IEEE International Conference on Intelligence and Security Informatics (ISI-2005), Atlanta, Georgia: Springer Lecture Notes in Computer Science, pp. 412–417.Google Scholar
  16. Hutwagner, L., Browne, T., Seeman, G.M., and Fleischauer, A.T. 2005a. "Comparing Aberration Detection Methods with Simulated Data," Emerg Infect Dis [serial on the Internet] (11), pp. 314–316.Google Scholar
  17. Ivanov, O., Gesteland, P.H., Hogan, W., Mundorff, M.B., and Wagner, M.M. 2003. "Detection of Pediatric Respiratory and Gastrointestinal Outbreaks from Free-Text Chief Complaints," AMIA Annual Symposium Proceedings, pp. 318–322.Google Scholar
  18. Jackson, M.L., Baer, A., Painter, I., and Duchin, J. 2007. "A Simulation Study Comparing Aberration Detection Algorithms for Syndromic Surveillance," BMC Medical Informatics and Decision Making (7:6).PubMedCrossRefGoogle Scholar
  19. Kleinman, K., and Abrams, A. 2006. "Assessing Surveillance Using Sensitivity, Specificity and Timeliness," Statistical Methods in Medical Research (15:5), pp. 445–464.PubMedGoogle Scholar
  20. Kleinman, K., Abrams, A., Kulldorff, M., and Platt, R. 2005a. "A Model-Adjusted Spacetime Scan Statistic with an Application to Syndromic Surveillance," Epidemiology and Infection (119), pp. 409–419.CrossRefGoogle Scholar
  21. Lombardo, J., Burkom, H., Elbert, E., Magruder, S.F., Lewis, S.H., Loschen, W., Sari, J., Sniegoski, C., Wojcik, R., and Pavlin, J. 2003. "A Systems Overview of the Electronic Surveillance System for the Early Notification of Community-Based Epidemics (ESSENCE II)," Journal of Urban Health: Bulletin of the New York Academy of Medicine (80:2), pp. 32–42.Google Scholar
  22. Reingold, A. 2003. "If Syndromic Surveillance is the Answer, What is the Question?," Biosecurity and Bioterrorism: Biodefense Strategy, Practice, and Science (1), pp. 1–5.CrossRefGoogle Scholar
  23. Reis, B., and Mandl, K. 2003. "Time Series Modeling for Syndromic Surveillance," BMC Medical Informatics and Decision Making (3:2).PubMedCrossRefGoogle Scholar
  24. Reis, B., Pagano, M., and Mandl, K. 2003. "Using Temporal Context to Improve Biosurveillance," Proceedings of the National Academy of Sciences USA (100:4), pp. 1961–1965.CrossRefGoogle Scholar
  25. Romaguera, R.A., German, R.R., and Klaucke, D.N. 2000. "Evaluating Public Health Surveillance," Principles and Practice of Public Health Surveillance, 2nd Ed, S.M. Teutsch and R.E. Churchill (eds.). New York: Oxford University Press.Google Scholar
  26. Shneiderman, B. 1998. Designing the User Interface: Strategies for Effective Human-Computer Interaction, 3rd Ed. Reading, MA: Addison-Wesley.Google Scholar
  27. Siegrist, D., McClellan, G., Campbell, M., Foster, V., Burkom, H., Hogan, W., Cheng, K., Pavlin, J., and Kress, A. 2004. "Evaluation of Algorithms for Outbreak Detection Using Clinical Data from Five US Cities." Technical Report, Darpa Bio-Alirt Program.Google Scholar
  28. Siegrist, D., and Pavlin, J. 2004. "Bio-Alirt Biosurveillance Detection Algorithm Evaluation," MMWR (CDC) (53(Suppl)), pp. 152–158.Google Scholar
  29. Sonesson, C., and Bock, D. 2003. "A Review and Discussion of Prospective Statistical Surveillance in Public Health," Journal of the Royal Statistical Society Series A (166:1), pp. 5–21.Google Scholar
  30. Tsui, F.-C., Wagner, M.M., Dato, V.M., and Chang, C.C.H. 2001. "Value of ICD-9-Coded Chief Complaints for Detection of Epidemics," Symposium of Journal of American Medical Informatics Association.Google Scholar
  31. Wong, W.-K., Moore, A.W., Cooper, G.F., and Wagner, M.M. 2005. "What's Strange About Recent Events (WSARE): An Algorithm for the Early Detection of Disease Outbreaks," Journal of Machine Learning Research (6) pp. 1961–1998.Google Scholar
  32. Zhu, Y., Wang, W., Atrubin, D., and Wu, Y. 2005. "Initial Evaluation of the Early Aberration Reporting System - Florida," MMWR (CDC) (54(Suppl)), pp. 123–130.Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2010

Authors and Affiliations

  • Hsinchun Chen
    • 1
  • Daniel Zeng
    • 2
    • 3
  • Ping Yan
    • 2
  1. 1.Department of Management Information SystemsEller College of Management University of ArizonaTucsonUSA
  2. 2.Department of Management Information SystemsEller College of Management University of ArizonaTucsonUSA
  3. 3.Chinese Academy of SciencesBeijingChina

Personalised recommendations