Skip to main content
SpringerLink
Log in

Dynamic Space-Time Model for Syndromic Surveillance with Particle Filters and Dirichlet Process

  • Chapter
  • First Online:
Handbook of Dynamic Data Driven Applications Systems

  • 673 Accesses

Abstract

Massive spatio-temporal data are challenging for statistical analysis due to their low signal-to-noise ratios and high-dimensional spatio-temporal structure. To resolve these issues, we propose a novel Dirichlet process particle filter (DPPF) model. The Dirichlet process models a set of stochastic functions as probability distributions for dimension reduction, and the particle filter is used to solve the nonlinear filtering problem with sequential Monte Carlo steps where the data has a low signal-to-noise ratio. Our data set is derived from surveillance data on emergency visits for influenza-like and respiratory illness (from 2008 to 2010) from the Indiana Public Health Emergency Surveillance System. The DPPF develops a dynamic data-driven applications system (DDDAS) methodology for disease outbreak detection. Numerical results show that our model significantly improves the outbreak detection performance in real data analysis.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 149.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 199.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 249.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. C.E. Antoniak, Mixtures of Dirichlet processes with applications to Bayesian nonparametric problems. Ann. Stat. 2, 1152–1174 (1974)

    Article  MathSciNet  MATH  Google Scholar 

  2. C.H. Bishop, B.J. Etherton, S.J. Majumdar, Adaptive sampling with the ensemble transform Kalman filter. Part I: theoretical aspects. Mon. Weather Rev. 129(3), 420–436 (2001)

    Google Scholar 

  3. D. Blackwell, J.B. MacQueen, Ferguson distributions via pólya urn schemes. Ann. Stat. 1, 353–355 (1973)

    MATH  Google Scholar 

  4. R. Brookmeyer, D.F. Stroup, Monitoring the Health of Populations: Statistical Principles and Methods for Public Health Surveillance (Oxford University Press, New York, 2003)

    Book  Google Scholar 

  5. G. Burgers, P. Jan van Leeuwen, G. Evensen, Analysis scheme in the ensemble Kalman filter. Mon. Weather Rev. 126(6), 1719–1724 (1998)

    Article  Google Scholar 

  6. K. Burghardt et al., Testing modeling assumptions in the West Africa Ebola outbreak. Sci. Rep. 6, 34598 (2016). https://doi.org/10.1038/srep34598

    Article  Google Scholar 

  7. B. Cai, A.B. Lawson, M. Hossain, J. Choi, R.S. Kirby, J. Liu et al., Bayesian semiparametric model with spatially–temporally varying coefficients selection. Stat. Med. 32(21), 3670–3685 (2013)

    Article  MathSciNet  Google Scholar 

  8. CDC, Weekly u.s. influenza surveillance report, 2007–2008, 2008–2009, 2009–2010 (2016)

    Google Scholar 

  9. A.J. Chorin, M. Morzfeld, X. Tu, A survey of implicit particle filters for data assimilation, in State-Space Models, ed. by Y. Zeng, S. Wu (Springer, New York, 2013), pp. 63–88

    Chapter  Google Scholar 

  10. Y. Chung, D.B. Dunson, The local Dirichlet process. Ann. Inst. Stat. Math. 63(1), 59–80 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  11. J.A. Duan, M. Guindani, A.E. Gelfand, Generalized spatial Dirichlet process models. Biometrika 94(4), 809–825 (2007)

    Article  MathSciNet  MATH  Google Scholar 

  12. D.B. Dunson, J.-H. Park, Kernel stick-breaking processes. Biometrika 95(2), 307–323 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  13. M.D. Escobar, M. West, Bayesian density estimation and inference using mixtures. J. Am. Stat. Assoc. 90(430), 577–588 (1995)

    Article  MathSciNet  MATH  Google Scholar 

  14. G. Evensen, Sequential data assimilation with a nonlinear quasi-geostrophic model using monte carlo methods to forecast error statistics. J. Geophys. Res. Oceans 99(C5), 10143–10162 (1994)

    Article  Google Scholar 

  15. T.S. Ferguson, A Bayesian analysis of some nonparametric problems. Ann. Stat. 1, 209–230 (1973)

    Article  MathSciNet  MATH  Google Scholar 

  16. R.D. Fricker, B.L. Hegler, D.A. Dunfee, Comparing syndromic surveillance detection methods: ears versus a cusum-based methology. Stat. Med. 27, 3407–3429 (2008)

    Article  MathSciNet  Google Scholar 

  17. M. Fuentes, B. Reich, Multivariate spatial nonparametric modelling via kernel processes mixing. Stat. Sin. 23(1), 75–97 (2013)

    MathSciNet  MATH  Google Scholar 

  18. A.E. Gelfand, A. Kottas, S.N. MacEachern, Bayesian nonparametric spatial modeling with Dirichlet process mixing. J. Am. Stat. Assoc. 100(471), 1021–1035 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  19. P.J. Green, S. Richardson, Hidden Markov models and disease mapping. J. Am. Stat. Assoc. 97(460), 1055–1070 (2002)

    Article  MathSciNet  MATH  Google Scholar 

  20. M.S. Grewal, A.P. Andrews, A.K. Filtering, Theory and practice using matlab, 3rd edn. (Wiley, Hoboken, 2001)

    Google Scholar 

  21. R.E. Kalman, A new approach to linear filtering and prediction problems. J. Basic Eng. 82(1), 35–45 (1960)

    Article  MathSciNet  Google Scholar 

  22. K. Kleinman, Generalized linear models and generalized linear mixed models for small-area surveillance, in Spatial and Syndromic Surveillance for Public Health, ed. by A.B. Lawson, K. Kleinman (Wiley, West Sussex, 2005), pp. 77–94

    Chapter  Google Scholar 

  23. L. Knorr-Held, S. Richardson, A hierarchical model for space–time surveillance data on meningococcal disease incidence. J. R. Stat. Soc. Ser. C Appl. Stat. 52(2), 169–183 (2003)

    Article  MathSciNet  MATH  Google Scholar 

  24. A. Kottas, J.A. Duan, A.E. Gelfand, Modeling disease incidence data with spatial and spatio temporal Dirichlet process mixtures. Biom. J. 50(1), 29–42 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  25. A.B. Lawson, K. Kleinman et al., Spatial and Syndromic Surveillance for Public Health (Wiley, New York, 2005)

    Book  Google Scholar 

  26. Y. Le Strat, F. Carrat, Monitoring epidemiologic surveillance data using hidden Markov models. Stat. Med. 18(24), 3463–3478 (1999)

    Article  Google Scholar 

  27. J. Mandel, J.D. Beezley, An Ensemble Kalman-Particle Predictor-Corrector Filter for Non-Gaussian Data Assimilation (Springer, Berlin/Heidelberg, 2009), pp. 470–478

    Google Scholar 

  28. J. Mandel, J.D. Beezley, A.K. Kochanski, V.Y. Kondratenko, M. Kim, Assimilation of perimeter data and coupling with fuel moisture in a wildland fire–atmosphere DDDAS. Proc. Comput. Sci. 9, 1100–1109 (2012)

    Article  Google Scholar 

  29. J. Mandel, L.S. Bennethum, M. Chen, J.L. Coen, C.C. Douglas, L.P. Franca, C.J. Johns, M. Kim, A.V. Knyazev, R. Kremens, V. Kulkarni, G. Qin, A. Vodacek, J. Wu, W. Zhao, A. Zornes, Towards a Dynamic Data Driven Application System for Wildfire Simulation (Springer, Berlin/Heidelberg, 2005), pp. 632–639

    Google Scholar 

  30. A. Patra, M. Bursik, J. Dehn, M. Jones, M. Pavolonis, E.B. Pitman, T. Singh, P. Singla, P. Webley, A DDDAS framework for volcanic ash propagation and hazard analysis. Proc. Comput. Sci. 9, 1090–1099 (2012)

    Article  MATH  Google Scholar 

  31. A.K. Patra, M. Bursik, J. Dehn, M. Jones, R. Madankan, D. Morton, M. Pavolonis, E.B. Pitman, S. Pouget, T. Singh et al., Challenges in developing DDDAS based methodology for volcanic ash hazard analysis–effect of numerical weather prediction variability and parameter estimation. Proc. Comput. Sci. 18, 1871–1880 (2013)

    Article  Google Scholar 

  32. A. Rodriguez, D.B. Dunson, A.E. Gelfand, The nested Dirichlet process. J. Am. Stat. Assoc. 103(483), 1131–1154 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  33. H. Seybold, S. Ravela, P. Tagade, Ensemble Learning in Non-Gaussian Data Assimilation (Springer, Cham, 2015), pp. 227–238

    Google Scholar 

  34. Y.W. Teh, M.I. Jordan, M.J. Beal, D.M. Blei, Hierarchical Dirichlet processes. J. Am. Stat. Assoc. 101, 1566–1581 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  35. A. Vodacek, J.P. Kerekes, M.J. Hoffman, Adaptive optical sensing in an object tracking DDDAS. Proc. Comput. Sci. 9, 1159–1166 (2012)

    Article  Google Scholar 

  36. L.A. Waller, B.P. Carlin, H. Xia, A. Gelfand, Hierarchical spatio-temporal mapping of disease rates. J. Am. Stat. Assoc. 92, 607–617 (1997)

    Article  MATH  Google Scholar 

  37. R.E. Watkins, S. Eagleson, B. Veenendaal, G. Wright, A.J. Plant, Disease surveillance using a hidden Markov model. BMC Med. Inform. Decis. Mak. 9(1), 1 (2009)

    Google Scholar 

  38. J. Zou, A.F. Karr, D. Banks, M.J. Heaton, G. Datta, J. Lynch, F. Vera, Bayesian methodology for the analysis of spatial–temporal surveillance data. Stat. Anal. Data Min. 5(3), 194–204 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  39. J. Zou, A.F. Karr, G. Datta, J. Lynch, S.J. Grannis, A Bayesian spatio-temporal approach for real-time detection of disease outbreaks: a case study. BMC Med. Inform. Decis. Mak. 14(108), 1–18 (2014)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Mathematical Sciences, Worcester Polytechnic Institute, Worcester, MA, USA

    Hong Yan, Zhongqiang Zhang & Jian Zou

Authors
  1. Hong Yan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zhongqiang Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jian Zou
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jian Zou .

Editor information

Editors and Affiliations

  1. Air Force Office of Scientific Research, Arlington, VA, USA

    Erik P. Blasch

  2. InfoSymbiotics Systems Society, Boston, MA, USA

    Frederica Darema

  3. Earth, Atmospheric and Planetary Sciences, Massachusetts Institute of Technology, Cambridge, MA, USA

    Sai Ravela

  4. Air Force Research Lab, Rome, NY, USA

    Alex J. Aved

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Yan, H., Zhang, Z., Zou, J. (2022). Dynamic Space-Time Model for Syndromic Surveillance with Particle Filters and Dirichlet Process. In: Blasch, E.P., Darema, F., Ravela, S., Aved, A.J. (eds) Handbook of Dynamic Data Driven Applications Systems. Springer, Cham. https://doi.org/10.1007/978-3-030-74568-4_7

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-74568-4_7

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-74567-7

  • Online ISBN: 978-3-030-74568-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation