Advertisement

Earth Science Informatics

, Volume 12, Issue 1, pp 87–96 | Cite as

Efficient spatiotemporal interpolation with spark machine learning

  • Weitian TongEmail author
  • Lixin Li
  • Xiaolu Zhou
  • Jason Franklin
Research Article
First Online:
  • 38 Downloads

Abstract

To better assess the relationships between environmental exposures and health outcomes, an appropriate spatiotemporal interpolation is critical. Traditional spatiotemporal interpolation methods either consider the spatial and temporal dimensions separately or incorporate both dimensions simultaneously by simply treating time as another dimension in space. Such interpolation results suffer from relatively low accuracy as the true space-time domain is skewed inappropriately and the distance calculation in such domain is not accurate. We employ the efficient k-d tree structure to store spatiotemporal data and adopt several machine learning methods to learn optimal parameters. To overcome the computational difficulty with large data sets, we implement our method on an efficient cluster computing framework – Apache Spark. Real world PM2.5 data sets are utilized to test our implementation and the experimental results demonstrate the computational power of our method, which significantly outperforms the previous work in terms of both speed and accuracy.

Keywords

Spatiotemporal interpolation Spark Machine learning Inverse distance weighting (IDW) k-d tree Bootstrap aggregating 
This is a preview of subscription content, log in to check access.

Notes

Acknowledgements

We would like to thank Brandon Kimmons, Director of Computational Research Technical Support at Georgia Southern University, for helping us set up Spark. Franklin, Tong and Zhou were supported in part by funds from the Office of the Vice President for Research & Economic Development at Georgia Southern University. Beseny, Franklin, Li, Tong were supported in part by cooperative engineering and health sciences faculty seed grants from Allen E. Paulson College of Engineering & Information Technology, Georgia Southern University, and the College of Allied Health Sciences, Augusta University.

References

  1. Appice A, Ciampi A, Malerba D, Guccione P (2013) Using trend clusters for spatiotemporal interpolation of missing data in a sensor network. Journal of Spatial Information Science 2013:119–153Google Scholar
  2. Aslam JA, Popa RA, Rivest RL (2007) On estimating the size and confidence of a statistical audit. EVT 7:8–8Google Scholar
  3. Breiman L (1994) Bagging predictor. Tech. rep., Department of Statistics, University of CaliforniaGoogle Scholar
  4. Brunekreef B, Holgate ST (2002) Air pollution and health. The lancet 360:1233–1242CrossRefGoogle Scholar
  5. Bureau USC (2010) Geographic terms and concepts - block groups. https://www.census.gov/geo/reference/gtc/gtc_bg.html?cssp=SERP
  6. Clemons W, Grecol M, Losser T, Yorke C (2013) Monitoring pollution trend in the course of the year using inverse distance weighting spatio-temporal interpolation. Tech. rep., Department of Computer Sciences, Georgia Southern UniversityGoogle Scholar
  7. De Boor C (1978) A practical guide to splines, vol 27. Springer, New YorkCrossRefGoogle Scholar
  8. Devijver PA, Kittler J (1982) Pattern recognition: a statistical approach. Prentice Hall, Upper Saddle RiverGoogle Scholar
  9. Dhatt G, Lefranċois E, Touzot G (2012) Finite element method, Wiley, New YorkGoogle Scholar
  10. Duncan IG (2011) Healthcare risk adjustment and predictive modeling, Actex Publications, WinstedGoogle Scholar
  11. EPA (2016) Air quality system (aqs). available online: http://www3.epa.gov/pm
  12. Friedman JH, Bentley JL, Finkel RA (1977) An algorithm for finding best matches in logarithmic expected time. ACM Trans Math Softw (TOMS) 3:209–226CrossRefGoogle Scholar
  13. Geisser S (1993) Predictive inference, vol 55. CRC Press, Boca RatonCrossRefGoogle Scholar
  14. Ghim YS, Moon KC, Lee S, Kim YP (2005) Visibility trends in Korea during the past two decades. J Air Waste Manag Assoc 55:73–82CrossRefGoogle Scholar
  15. Gräler B, Rehr M, Gerharz L, Pebesma E (2013) Spatio-temporal analysis and interpolation of PM10, measurements in Europe for 2009Google Scholar
  16. Hong YC, Lee JT, Kim H, Ha EH, Schwartz J, Christiani DC (2002) Effects of air pollutants on acute stroke mortality. Environ Health Perspect 110:187CrossRefGoogle Scholar
  17. Iceland J, Steinmetz E (2003) The effects of using census block groups instead of census tracts when examining residential housing patterns. US Census Bureau, Washington DCGoogle Scholar
  18. Kohavi R (1995) A study of cross-validation and bootstrap for accuracy estimation and model selection. In: Ijcai, vol. 14, pp. 1137–1145Google Scholar
  19. Krall JR, Anderson GB, Dominici F, Bell ML, Peng RD (2013) Short-term exposure to particulate matter constituents and mortality in a national study of US urban communitiesGoogle Scholar
  20. Krieger N, Chen JT, Waterman PD, Soobader MJ, Subramanian S, Carson R (2002) Geocoding and monitoring of us socioeconomic inequalities in mortality and cancer incidence: does the choice of area-based measure and geographic level matter? the public health disparities geocoding project. Am J Epidemiol 156:471– 482CrossRefGoogle Scholar
  21. Krige DG (1951) A statistical approach to some mine valuations and allied problems at the witwatersrand. Master thesisGoogle Scholar
  22. Künzli N, Kaiser R, Medina S, Studnicka M, Chanel O, Filliger P, Herry M, Horak F, J, Puybonnieux-Texier V, Quénel P, Schneider J, Seethaler R, Vergnaud JC, Sommer H (2000) Public-health impact of outdoor and traffic-related air pollution: a european assessment. The Lancet 356:795–801CrossRefGoogle Scholar
  23. Laden F, Neas LM, Dockery DW, Schwartz J (2000) Association of fine particulate matter from different sources with daily mortality in six U.S. cities. Environ Health Perspect 108:941–947CrossRefGoogle Scholar
  24. Li L, Losser T, Yorke C, Piltner R (2014) Fast inverse distance weighting-based spatiotemporal interpolation: a web-based application of interpolating daily fine particulate matter PM2.5 in the contiguous US using parallel programming and k-d tree. International journal of environmental research and public health 11:9101–9141CrossRefGoogle Scholar
  25. Li L, Revesz P (2004) Interpolation methods for spatio-temporal geographic data. Comput Environ Urban Syst 28:201–227CrossRefGoogle Scholar
  26. Li L, Tian J, Zhang X, Holt JB, Piltner R (2012) Estimating population exposure to fine particulate matter in the conterminous us using shape function-based spatiotemporal interpolation method: a county level analysis. GSTF international journal on computing 1:24–30Google Scholar
  27. Liao D, Peuquet DJ, Duan Y, Whitsel EA, Dou J, Smith RL, Lin HM, Chen JC, Heiss G (2006) Gis approaches for the estimation of residential-level ambient pm concentrations. Environ Health Perspect 114(9):1374–1380CrossRefGoogle Scholar
  28. Losser T, Li L, Piltner R (2014) A spatiotemporal interpolation method using radial basis functions for geospatiotemporal big data. In: COM.Geo, pp 17–24Google Scholar
  29. Lu P, Abedi V, Mei Y, Hontecillas R, Hoops S, Carbo A, Bassaganya-Riera J (2015) Supervised learning methods in modeling of CD4+ t cell heterogeneity. BioData mining 8:1CrossRefGoogle Scholar
  30. McLachlan G, Do KA, Ambroise C (2005) Analyzing microarray gene expression data, vol 422. Wiley, New YorkGoogle Scholar
  31. Pagowski M, Grell GA, McKeen SA, Peckham SE, Devenyi D (2010) Three-dimensional variational data assimilation of ozone and fine particulate matter observations: some results using the weather research and forecasting—chemistry model and grid-point statistical interpolation. Q J R Meteorol Soc 136:2013–2024CrossRefGoogle Scholar
  32. Pebesma E (2012) spacetime: Spatio-temporal data in R. J Stat Softw 51:1–30CrossRefGoogle Scholar
  33. Pope CA, Burnett RT, Thurston GD, Thun MJ, Calle EE, Krewski D, Godleski JJ (2004) Cardiovascular mortality and long-term exposure to particulate air pollution epidemiological evidence of general pathophysiological pathways of disease. Circulation 109:71–77CrossRefGoogle Scholar
  34. Pope CA III, Dockery DW (2006) Health effects of fine particulate air pollution: lines that connect. J Air Waste Manag Assoc 56:709–742CrossRefGoogle Scholar
  35. Seaton A, Godden D, MacNee W, Donaldson K (1995) Particulate air pollution and acute health effects. The Lancet 345:176–178CrossRefGoogle Scholar
  36. Shepard D (1968) A two-dimensional interpolation function for irregularly-spaced data. In: Proceedings of the 23rd ACM national conference, pp 517–524Google Scholar
  37. Sloane CS, Watson J, Chow J, Pritchett L, Richards LW (1991) Size-segregated fine particle measurements by chemical species and their impact on visibility impairment in denver. Atmos Environ Part A General Topics 25:1013–1024CrossRefGoogle Scholar
  38. Spark A (2016) Apache spark is a fast and general engine for large-scale data processingGoogle Scholar
  39. Tobler WR (1970) A computer movie simulating urban growth in the detroit region. Econ Geogr 46:234–240CrossRefGoogle Scholar
  40. Valentini V, Schmoll HJ, van de Velde CJH (2012) Multidisciplinary management of rectal cancer: questions and answers. Springer, BerlinCrossRefGoogle Scholar
  41. Zaharia M, Chowdhury M, Franklin MJ, Shenker S, Stoica I (2010) Spark: cluster computing with working sets. HotCloud 10:10–10Google Scholar
  42. Zakir J, Seymour T, Berg K (2015) Big data analytics. Issues in Information Systems 16(II):81–90Google Scholar
  43. Zanobetti A, Schwartz J (2009) The effect of fine and coarse particulate air pollution on mortality: a national analysis. Environ Health Perspect 117:898–903CrossRefGoogle Scholar
  44. Zhu Z, Yuan D, Luo D, Lu X, Huang S (2015) Enrichment of minor alleles of common snps and improved risk prediction for parkinson’s disease. PloS one 10:e0133,421CrossRefGoogle Scholar
  45. Zurflueh EG (1967) Applications of two-dimensional linear wavelength filtering. Geophysics 32:1015–1035CrossRefGoogle Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2018

Authors and Affiliations

  • Weitian Tong
    • 1
    Email author
  • Lixin Li
    • 1
  • Xiaolu Zhou
    • 2
  • Jason Franklin
    • 1
  1. 1.Department of Computer ScienceGeorgia Southern UniversityStatesboroUSA
  2. 2.Department of Geology and GeographyGeorgia Southern UniversityStatesboroUSA

Personalised recommendations

Cite article

Buy options