Skip to main content
Book cover

Handbook of Dynamic Data Driven Applications Systems pp 147–160Cite as

Dynamic Space-Time Model for Syndromic Surveillance with Particle Filters and Dirichlet Process

  • 159 Accesses

Abstract

Massive spatio-temporal data are challenging for statistical analysis due to their low signal-to-noise ratios and high-dimensional spatio-temporal structure. To resolve these issues, we propose a novel Dirichlet process particle filter (DPPF) model. The Dirichlet process models a set of stochastic functions as probability distributions for dimension reduction, and the particle filter is used to solve the nonlinear filtering problem with sequential Monte Carlo steps where the data has a low signal-to-noise ratio. Our data set is derived from surveillance data on emergency visits for influenza-like and respiratory illness (from 2008 to 2010) from the Indiana Public Health Emergency Surveillance System. The DPPF develops a dynamic data-driven applications system (DDDAS) methodology for disease outbreak detection. Numerical results show that our model significantly improves the outbreak detection performance in real data analysis.

Keywords

  • Syndromic surveillance
  • Space-time model
  • Dirichlet process
  • Particle filters
  • Outbreak detection

This is a preview of subscription content, access via your institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-030-74568-4_7
  • Chapter length: 14 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   219.00
Price excludes VAT (USA)
  • ISBN: 978-3-030-74568-4
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Hardcover Book
USD   279.99
Price excludes VAT (USA)
Learn about institutional subscriptions
Fig. 7.1
Fig. 7.2
Fig. 7.3
Fig. 7.4
Fig. 7.5

References

  1. C.E. Antoniak, Mixtures of Dirichlet processes with applications to Bayesian nonparametric problems. Ann. Stat. 2, 1152–1174 (1974)

    MathSciNet  MATH  CrossRef  Google Scholar 

  2. C.H. Bishop, B.J. Etherton, S.J. Majumdar, Adaptive sampling with the ensemble transform Kalman filter. Part I: theoretical aspects. Mon. Weather Rev. 129(3), 420–436 (2001)

    Google Scholar 

  3. D. Blackwell, J.B. MacQueen, Ferguson distributions via pólya urn schemes. Ann. Stat. 1, 353–355 (1973)

    MATH  Google Scholar 

  4. R. Brookmeyer, D.F. Stroup, Monitoring the Health of Populations: Statistical Principles and Methods for Public Health Surveillance (Oxford University Press, New York, 2003)

    CrossRef  Google Scholar 

  5. G. Burgers, P. Jan van Leeuwen, G. Evensen, Analysis scheme in the ensemble Kalman filter. Mon. Weather Rev. 126(6), 1719–1724 (1998)

    CrossRef  Google Scholar 

  6. K. Burghardt et al., Testing modeling assumptions in the West Africa Ebola outbreak. Sci. Rep. 6, 34598 (2016). https://doi.org/10.1038/srep34598

    CrossRef  Google Scholar 

  7. B. Cai, A.B. Lawson, M. Hossain, J. Choi, R.S. Kirby, J. Liu et al., Bayesian semiparametric model with spatially–temporally varying coefficients selection. Stat. Med. 32(21), 3670–3685 (2013)

    MathSciNet  CrossRef  Google Scholar 

  8. CDC, Weekly u.s. influenza surveillance report, 2007–2008, 2008–2009, 2009–2010 (2016)

    Google Scholar 

  9. A.J. Chorin, M. Morzfeld, X. Tu, A survey of implicit particle filters for data assimilation, in State-Space Models, ed. by Y. Zeng, S. Wu (Springer, New York, 2013), pp. 63–88

    CrossRef  Google Scholar 

  10. Y. Chung, D.B. Dunson, The local Dirichlet process. Ann. Inst. Stat. Math. 63(1), 59–80 (2011)

    MathSciNet  MATH  CrossRef  Google Scholar 

  11. J.A. Duan, M. Guindani, A.E. Gelfand, Generalized spatial Dirichlet process models. Biometrika 94(4), 809–825 (2007)

    MathSciNet  MATH  CrossRef  Google Scholar 

  12. D.B. Dunson, J.-H. Park, Kernel stick-breaking processes. Biometrika 95(2), 307–323 (2008)

    MathSciNet  MATH  CrossRef  Google Scholar 

  13. M.D. Escobar, M. West, Bayesian density estimation and inference using mixtures. J. Am. Stat. Assoc. 90(430), 577–588 (1995)

    MathSciNet  MATH  CrossRef  Google Scholar 

  14. G. Evensen, Sequential data assimilation with a nonlinear quasi-geostrophic model using monte carlo methods to forecast error statistics. J. Geophys. Res. Oceans 99(C5), 10143–10162 (1994)

    CrossRef  Google Scholar 

  15. T.S. Ferguson, A Bayesian analysis of some nonparametric problems. Ann. Stat. 1, 209–230 (1973)

    MathSciNet  MATH  CrossRef  Google Scholar 

  16. R.D. Fricker, B.L. Hegler, D.A. Dunfee, Comparing syndromic surveillance detection methods: ears versus a cusum-based methology. Stat. Med. 27, 3407–3429 (2008)

    MathSciNet  CrossRef  Google Scholar 

  17. M. Fuentes, B. Reich, Multivariate spatial nonparametric modelling via kernel processes mixing. Stat. Sin. 23(1), 75–97 (2013)

    MathSciNet  MATH  Google Scholar 

  18. A.E. Gelfand, A. Kottas, S.N. MacEachern, Bayesian nonparametric spatial modeling with Dirichlet process mixing. J. Am. Stat. Assoc. 100(471), 1021–1035 (2005)

    MathSciNet  MATH  CrossRef  Google Scholar 

  19. P.J. Green, S. Richardson, Hidden Markov models and disease mapping. J. Am. Stat. Assoc. 97(460), 1055–1070 (2002)

    MathSciNet  MATH  CrossRef  Google Scholar 

  20. M.S. Grewal, A.P. Andrews, A.K. Filtering, Theory and practice using matlab, 3rd edn. (Wiley, Hoboken, 2001)

    Google Scholar 

  21. R.E. Kalman, A new approach to linear filtering and prediction problems. J. Basic Eng. 82(1), 35–45 (1960)

    MathSciNet  CrossRef  Google Scholar 

  22. K. Kleinman, Generalized linear models and generalized linear mixed models for small-area surveillance, in Spatial and Syndromic Surveillance for Public Health, ed. by A.B. Lawson, K. Kleinman (Wiley, West Sussex, 2005), pp. 77–94

    CrossRef  Google Scholar 

  23. L. Knorr-Held, S. Richardson, A hierarchical model for space–time surveillance data on meningococcal disease incidence. J. R. Stat. Soc. Ser. C Appl. Stat. 52(2), 169–183 (2003)

    MathSciNet  MATH  CrossRef  Google Scholar 

  24. A. Kottas, J.A. Duan, A.E. Gelfand, Modeling disease incidence data with spatial and spatio temporal Dirichlet process mixtures. Biom. J. 50(1), 29–42 (2008)

    MathSciNet  MATH  CrossRef  Google Scholar 

  25. A.B. Lawson, K. Kleinman et al., Spatial and Syndromic Surveillance for Public Health (Wiley, New York, 2005)

    CrossRef  Google Scholar 

  26. Y. Le Strat, F. Carrat, Monitoring epidemiologic surveillance data using hidden Markov models. Stat. Med. 18(24), 3463–3478 (1999)

    CrossRef  Google Scholar 

  27. J. Mandel, J.D. Beezley, An Ensemble Kalman-Particle Predictor-Corrector Filter for Non-Gaussian Data Assimilation (Springer, Berlin/Heidelberg, 2009), pp. 470–478

    Google Scholar 

  28. J. Mandel, J.D. Beezley, A.K. Kochanski, V.Y. Kondratenko, M. Kim, Assimilation of perimeter data and coupling with fuel moisture in a wildland fire–atmosphere DDDAS. Proc. Comput. Sci. 9, 1100–1109 (2012)

    CrossRef  Google Scholar 

  29. J. Mandel, L.S. Bennethum, M. Chen, J.L. Coen, C.C. Douglas, L.P. Franca, C.J. Johns, M. Kim, A.V. Knyazev, R. Kremens, V. Kulkarni, G. Qin, A. Vodacek, J. Wu, W. Zhao, A. Zornes, Towards a Dynamic Data Driven Application System for Wildfire Simulation (Springer, Berlin/Heidelberg, 2005), pp. 632–639

    Google Scholar 

  30. A. Patra, M. Bursik, J. Dehn, M. Jones, M. Pavolonis, E.B. Pitman, T. Singh, P. Singla, P. Webley, A DDDAS framework for volcanic ash propagation and hazard analysis. Proc. Comput. Sci. 9, 1090–1099 (2012)

    MATH  CrossRef  Google Scholar 

  31. A.K. Patra, M. Bursik, J. Dehn, M. Jones, R. Madankan, D. Morton, M. Pavolonis, E.B. Pitman, S. Pouget, T. Singh et al., Challenges in developing DDDAS based methodology for volcanic ash hazard analysis–effect of numerical weather prediction variability and parameter estimation. Proc. Comput. Sci. 18, 1871–1880 (2013)

    CrossRef  Google Scholar 

  32. A. Rodriguez, D.B. Dunson, A.E. Gelfand, The nested Dirichlet process. J. Am. Stat. Assoc. 103(483), 1131–1154 (2008)

    MathSciNet  MATH  CrossRef  Google Scholar 

  33. H. Seybold, S. Ravela, P. Tagade, Ensemble Learning in Non-Gaussian Data Assimilation (Springer, Cham, 2015), pp. 227–238

    Google Scholar 

  34. Y.W. Teh, M.I. Jordan, M.J. Beal, D.M. Blei, Hierarchical Dirichlet processes. J. Am. Stat. Assoc. 101, 1566–1581 (2006)

    MathSciNet  MATH  CrossRef  Google Scholar 

  35. A. Vodacek, J.P. Kerekes, M.J. Hoffman, Adaptive optical sensing in an object tracking DDDAS. Proc. Comput. Sci. 9, 1159–1166 (2012)

    CrossRef  Google Scholar 

  36. L.A. Waller, B.P. Carlin, H. Xia, A. Gelfand, Hierarchical spatio-temporal mapping of disease rates. J. Am. Stat. Assoc. 92, 607–617 (1997)

    MATH  CrossRef  Google Scholar 

  37. R.E. Watkins, S. Eagleson, B. Veenendaal, G. Wright, A.J. Plant, Disease surveillance using a hidden Markov model. BMC Med. Inform. Decis. Mak. 9(1), 1 (2009)

    Google Scholar 

  38. J. Zou, A.F. Karr, D. Banks, M.J. Heaton, G. Datta, J. Lynch, F. Vera, Bayesian methodology for the analysis of spatial–temporal surveillance data. Stat. Anal. Data Min. 5(3), 194–204 (2012)

    MathSciNet  MATH  CrossRef  Google Scholar 

  39. J. Zou, A.F. Karr, G. Datta, J. Lynch, S.J. Grannis, A Bayesian spatio-temporal approach for real-time detection of disease outbreaks: a case study. BMC Med. Inform. Decis. Mak. 14(108), 1–18 (2014)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Mathematical Sciences, Worcester Polytechnic Institute, Worcester, MA, USA

    Hong Yan, Zhongqiang Zhang & Jian Zou

Authors
  1. Hong Yan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zhongqiang Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jian Zou
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jian Zou .

Editor information

Editors and Affiliations

  1. Air Force Office of Scientific Research, Arlington, VA, USA

    Erik P. Blasch

  2. InfoSymbiotics Systems Society, Boston, MA, USA

    Dr. Frederica Darema

  3. Earth, Atmospheric and Planetary Sciences, Massachusetts Institute of Technology, Cambridge, MA, USA

    Dr. Sai Ravela

  4. Air Force Research Lab, Rome, NY, USA

    Dr. Alex J. Aved

Rights and permissions

Reprints and Permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this chapter

Verify currency and authenticity via CrossMark

Cite this chapter

Yan, H., Zhang, Z., Zou, J. (2022). Dynamic Space-Time Model for Syndromic Surveillance with Particle Filters and Dirichlet Process. In: Blasch, E.P., Darema, F., Ravela, S., Aved, A.J. (eds) Handbook of Dynamic Data Driven Applications Systems. Springer, Cham. https://doi.org/10.1007/978-3-030-74568-4_7

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-74568-4_7

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-74567-7

  • Online ISBN: 978-3-030-74568-4

  • eBook Packages: Computer ScienceComputer Science (R0)