Abstract
Earth observation (EO) and environmental geospatial datasets are growing at an unprecedented rate in size, variety and complexity, thus, creating new challenges and opportunities as far as their access, archiving, processing and analytics are concerned. Currently, huge imaging streams are reaching several petabytes in many satellite archives worldwide. In this chapter, we review the current state-of-the-art in big data frameworks able to access, handle, process, analyse and deliver geospatial data and value-added products. Operational services that feature efficient implementations and different architectures allowing in certain cases the online and near real-time processing and analytics are detailed. Based on the current status, state-of-the-art and emerging challenges, the present study highlights certain issues, insights and future directions towards the efficient exploitation of EO big data for important engineering, environmental and agricultural applications.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
- 2.
- 3.
- 4.
- 5.
- 6.
- 7.
- 8.
- 9.
- 10.
- 11.
- 12.
- 13.
References
Adamov A (2012) Distributed file system as a basis of data-intensive computing. In: 2012 6th International conference on application of information and communication technologies (AICT), pp 1–3. doi:10.1109/ICAICT.2012.6398484
Aiordachioaie A, Baumann P (2010) Petascope: An open-source implementation of the ogc wcs geo service standards suite. In: Gertz M, Ludascher B (eds) Scientific and statistical database management. Lecture Notes in Computer Science, vol 6187, Springer, Berlin/Heidelberg, pp 160–168
Aji A, Wang F, Vo H, Lee R, Liu Q, Zhang X, Saltz J (2013) Hadoop gis: A high performance spatial data warehousing system over mapreduce. Proc VLDB Endowment 6(11):1009–1020. doi:10.14778/2536222.2536227, http://dx.doi.org/10.14778/2536222.2536227
Asrar G, Kanemasu E, Yoshida M (1985) Estimates of leaf area index from spectral reflectance of wheat under different cultural practices and solar angles. Remote Sens Environ 17:1–11
Assuncao MD, Calheiros RN, Bianchi S, Netto MA, Buyya R (2014) Big data computing and clouds: trends and future directions. J Parallel Distrib Comput. doi:http://dx.doi.org/10.1016/j.jpdc.2014.08.003, http://www.sciencedirect.com/science/article/pii/S0743731514001452
Babaee M, Datcu M, Rigoll G (2013) Assessment of dimensionality reduction based on communication channel model; application to immersive information visualization. In: 2013 IEEE international conference on big data, pp 1–6. doi:10.1109/BigData.2013.6691726
Barroso L, Dean J, Holzle U (2003) Web search for a planet: the google cluster architecture. IEEE Micro 23(2):22–28. doi:10.1109/MM.2003.1196112
Baumann P (1994) Management of multidimensional discrete data. Int J Very Large Data Bases 4(3):401–444
Baumann P (1999) A database array algebra for spatio-temporal data and beyond. In: Next generation information technologies and systems, pp 76–93
Baumann P (2009) Array databases and raster data management. In: Ozsu T, Liu L (eds), Encyclopedia of database systems. Springer, New York
Baumann P (2010) The OGC web coverage processing service (WCPS) standard. GeoInformatica 14(4):447–479. doi:10.1007/s10707-009-0087-2
Baumann P (2012) OGC WCS 2.0 Interface Standard-Core: Corrigendum (OGC 09-110r4)
Baumann P (2014) rasdaman: array databases boost spatio-temporal analytics. In: 2014 fifth international conference on computing for geospatial research and application (COM.Geo), pp 54–54
Baumann P, Nativi S (2012) Adding big earth data analytics to geoss. Group on Earth Observations Ninth Plenary Session – GEO-IX. Brazil, 22–23 November
Baumann P, Dehmel A, Furtado P, Ritsch R, Widmann N (1998) The multidimensional database system rasdaman. In: Proceedings of the 1998 ACM SIGMOD international conference on management of data. ACM Press, New York, pp 575–577
Begoli E, Horey J (2012) Design principles for effective knowledge discovery from big data. In: 2012 joint working IEEE/IFIP conference on IEEE software architecture (WICSA) and European conference on software architecture (ECSA), pp 215–218
Buehler K, McKee L (2006) The openGIS guide (third edition). In: Technical Committee, version 1, Engineering Specification Best Practices, OGIS TC Doc. 96-001
Cammalleri C, Anderson M, Gao F, Hain C, Ku W (2014) Mapping daily evapotranspiration at field scales over rainfed and irrigated agricultural areas using remote sensing data fusion. Agr Forest Meteorol 186(0):1–11
Cappelaere P, Sanchez S, Bernabe S, Scuri A, Mandl D, Plaza A (2013) Cloud implementation of a full hyperspectral unmixing chain within the nasa web coverage processing service for EO-1. IEEE J Sel Top Appl Earth Obs Remote Sens 6(2):408–418. doi:10.1109/JSTARS.2013.2250256
CartoDB (Retrieved 2015) https://cartodb.com/platform
Chen J, Chen J, Liao A, Cao X, Chen L, Chen X, He C, Han G, Peng S, Lu M, Zhang W, Tong X, Mills J (2014) Global land cover mapping at 30m resolution: a POK-based operational approach. Int J Photogr Remote Sens. doi:http://dx.doi.org/10.1016/j.isprsjprs.2014.09.002
Choo J, Park H (2013) Customizing computational methods for visual analytics with big data. Computer Graphics and Applications, IEEE 33(4):22–28
Davis B (1996) GIS: A Visual Approach. OnWord Press
de la Beaujardiere J (2006) OpenGIS Web Map Server Implementation Specification (OGC 06-042)
Dean J, Ghemawat S (2008) Mapreduce: Simplified data processing on large clusters. Commun ACM 51(1):107–113. doi 10.1145/1327452.1327492, http://doi.acm.org/10.1145/1327452.1327492
Demchenko Y, Zhao Z, Grosso P, Wibisono A, De Laat C (2012) Addressing big data challenges for scientific data infrastructure. In: 2012 IEEE 4th international conference on cloud computing technology and science (CloudCom). IEEE, New York, pp 614–617
Espinoza-Molina D, Datcu M (2013) Earth-observation image retrieval based on content, semantics, and metadata. IEEE IEEE Trans Geosci Remote Sens 51(11):5145–5159. doi:10.1109/TGRS.2013.2262232
Evangelidis K, Ntouros K, Makridis S, Papatheodorou C (2014) Geospatial services in the cloud. Comput. Geosci. 63(0):116–122. doi:http://dx.doi.org/10.1016/j.cageo.2013.10.007, http://www.sciencedirect.com/science/article/pii/S0098300413002719
Foster I, Zhao Y, Raicu I, Lu S (2008) Cloud computing and grid computing 360-degree compared. In: Grid computing environments workshop, 2008 (GCE ’08), pp 1–10. doi:10.1109/GCE.2008.4738445
Furht B, Escalante A (2011) Handbook of cloud computing. Springer, New York
Garcia-Rojas A, Athanasiou S, Lehmann J, Hladky D (2013) Geoknow: leveraging geospatial data in the web of data. In: Open data on the web workshop, http://jens-lehmann.org/files/2013/odw_geoknow.pdf
gigaomcom (Retrieved 2015) Can you predict future traffic patterns? Nokia thinks it can. https://gigaom.com/2013/07/02/living-cities-lights-up-traffic-in-5-cities-with-interactive-data-visualization/
Glenn EP, Huete AR, Nagler PL, Nelson SG (2008) Relationship between remotely-sensed vegetation indices, canopy attributes and plant physiological processes: what vegetation indices can and cannot tell us about the landscape. Sensors 8(4):2136. doi:10.3390/s8042136, http://www.mdpi.com/1424-8220/8/4/2136
Gray J (2008) Distributed computing economics. Queue 6(3):63–68. doi:10.1145/1394127.1394131, http://doi.acm.org/10.1145/1394127.1394131
Habib S, Morozov V, Frontiere N, Finkel H, Pope A, Heitmann K (2013) Hacc: Extreme scaling and performance across diverse architectures. In: Proceedings of the international conference on high performance computing, networking, storage and analysis (SC ’13). ACM, New York, pp 6:1–6:10. doi:10.1145/2503210.2504566, http://doi.acm.org/10.1145/2503210.2504566
Han J, Haihong E, Le G, Du J (2011) Survey on nosql database. In: 2011 6th international conference on pervasive computing and applications (ICPCA), pp 363–366. doi:10.1109/ICPCA.2011.6106531
Han W, Yang Z, Di L, Yue P (2014) A geospatial web service approach for creating on-demand cropland data layer thematic maps. Transactions of the ASABE 57(1):239–247. doi:http://dx.doi.org/10.13031/trans.57.10020
Hansen MC, Potapov PV, Moore R, Hancher M, Turubanova SA, Tyukavina A, Thau D, Stehman SV, Goetz SJ, Loveland TR, Kommareddy A, Egorov A, Chini L, Justice CO, Townshend JRG (2013) High-resolution global maps of 21st-century forest cover change. Science 342(6160):850–853. doi:10.1126/science.1244693
Hatfield JL, Prueger JH (2010) Value of using different vegetative indices to quantify agricultural crop characteristics at different growth stages under varying management practices. Remote Sens 2(2):562. doi:10.3390/rs2020562, http://www.mdpi.com/2072-4292/2/2/562
Hunter PD, Tyler AN, Présing M, Kovács AW, Preston T (2008) Spectral discrimination of phytoplankton colour groups: the effect of suspended particulate matter and sensor spectral resolution. Remote Sens Environ 112(4):1527–1544. doi:http://dx.doi.org/10.1016/j.rse.2007.08.003, http://www.sciencedirect.com/science/article/pii/S0034425707004051, remote Sensing Data Assimilation Special Issue
Hwang K, Choi M (2013) Seasonal trends of satellite-based evapotranspiration algorithms over a complex ecosystem in East Asia. Remote Sens Environ 137(0):244–263
Idreos S, Kersten ML, Manegold S (2007) Database cracking. In: CIDR 2007, Third biennial conference on innovative data systems research, Asilomar, CA, January 7-10, 2007, Online Proceedings, pp 68–78, http://www.cidrdb.org/cidr2007/papers/cidr07p07.pdf
Idreos S, Groffen F, Nes N, Manegold S, Mullender S, Kersten M (2012) Monetdb: two decades of research in column-oriented database architectures. IEEE Data Eng Bull 35(1):40–45
Ivanova MG, Kersten ML, Nes NJ, Gonçalves RA (2010) An architecture for recycling intermediates in a column-store. ACM Trans Database Syst 35(4):24:1–24:43. doi:10.1145/1862919.1862921, http://doi.acm.org/10.1145/1862919.1862921
Ivanova M, Kersten M, Manegold S (2012) Data vaults: A symbiosis between database technology and scientific file repositories. In: Ailamaki A, Bowers S (eds) Scientific and statistical database management. Lecture notes in computer science, vol. 7338. Springer, Berlin/Heidelberg, pp 485–494. doi:10.1007/978-3-642-31235-9_32, http://dx.doi.org/10.1007/978-3-642-31235-9_32
Karantzalos K, Bliziotis D, Karmas A (2015) A scalable web geospatial service for near real-time, high-resolution land cover mapping. IEEE J Sel Top Appl Earth Obs Remote Sens Special Issue on ‘Big Data in Remote Sensing’ 8(10):4665–4674
Karantzalos K, Karmas A, Tzotsos A (2015) RemoteAgri: processing online big earth observation data for precision agriculture. In: European conference on precision agriculture
Karmas A, Karantzalos K (2015) Benchmarking server-side software modules for handling and processing remote sensing data through rasdaman. In: (WHISPERS) IEEE workshop on hyperspectral image and signal processing: evolution in remote sensing
Karmas A, Karantzalos K, Athanasiou S (2014) Online analysis of remote sensing data for agricultural applications. In: OSGeo’s European conference on free and open source software for geospatial
Karmas A, Tzotsos A, Karantzalos K (2015) Scalable geospatial web services through efficient, online and near real-time processing of earth observation data. In: (BigData Service 2015) IEEE international conference on big data computing service and applications
Kopsiaftis G, Karantzalos K (2015) Vehicle detection and traffic density monitoring from very high resolution satellite video data. In: IEEE international geoscience and remote sensing symposium (IGARSS 2015)
Koubarakis M, Kontoes C, Manegold S (2013) Real-time wildfire monitoring using scientific database and linked data technologies. In: 16th international conference on extending database technology
Kouzes R, Anderson G, Elbert S, Gorton I, Gracio D (2009) The changing paradigm of data-intensive computing. Computer 42(1):26–34. doi:10.1109/MC.2009.26
Laney D (Retrieved 6 February 2001) 3d data management: controlling data volume, velocity and variety. Gartner
Lee C, Gasster S, Plaza A, Chang CI, Huang B (2011) Recent developments in high performance computing for remote sensing: a review. IEEE J Selected Top Appl Earth Obsand Remote Sens 4(3):508–527. doi:10.1109/JSTARS.2011.2162643
Liu B, Blasch E, Chen Y, Shen D, Chen G (2013) Scalable sentiment classification for big data analysis using Naive Bayes Classifier. In: 2013 IEEE international conference on big data, pp 99–104. doi:10.1109/BigData.2013.6691740
Ma Y, Wang L, Liu P, Ranjan R (2014) Towards building a data-intensive index for big data computing - a case study of remote sensing data processing. Information Sciences. doi:http://dx.doi.org/10.1016/j.ins.2014.10.006
Ma Y, Wang L, Zomaya A, Chen D, Ranjan R (2014) Task-tree based large-scale mosaicking for massive remote sensed imageries with dynamic dag scheduling. IEEE Trans Parallel Distrib Syst 25(8):2126–2137. doi:10.1109/TPDS.2013.272
Ma Y, Wu H, Wang L, Huang B, Ranjan R, Zomaya A, Jie W (2014) Remote sensing big data computing: challenges and opportunities. Futur Gener Comput Syst. doi:http://dx.doi.org/10.1016/j.future.2014.10.029, http://www.sciencedirect.com/science/article/pii/S0167739X14002234
Menzies T, Zimmermann T (2013) Software analytics: so what? IEEE Softw 30(4):31–37
MonetDB (Retrieved 2015) https://www.monetdb.org/home/features
Nebert D, Whiteside A, Vretanos P (2007) OpenGIS Catalogue Services Specification (OGC 07-006r1)
NGA (2014) Digitalglobe application a boon to raster data storage, processing
NGA (Retrieved 2015) https://github.com/ngageoint/mrgeo/wiki
Nikolaou C, Kyzirakos K, Bereta K, Dogani K, Giannakopoulou S, Smeros P, Garbis G, Koubarakis M, Molina D, Dumitru O, Schwarz G, Datcu M (2014) Big, linked and open data: applications in the German aerospace center. In: The semantic web: ESWC 2014 satellite events. Lecture notes in computer science. Springer International Publishing, New York, pp 444–449. doi:10.1007/978-3-319-11955-7_64, http://dx.doi.org/10.1007/978-3-319-11955-7_64
OGC (Retrieved 20 June 2015) OGC abstract specifications. http://www.opengeospatial.org/standards/as
OGC (Retrieved 20 June 2015) OGC history. http://www.opengeospatial.org/ogc/historylong
Oosthoek J, Flahaut J, Rossi A, Baumann P, Misev D, Campalani P, Unnithan V (2013) Planetserver: innovative approaches for the online analysis of hyperspectral satellite data from Mars. Adv Space Res pp 219–244. doi:http://dx.doi.org/10.1016/j.asr.2013.07.002
Palmer SC, Hunter PD, Lankester T, Hubbard S, Spyrakos E, Tyler AN, Présing M, Horváth H, Lamb A, Balzter H, Tóth VR (2015) Validation of envisat {MERIS} algorithms for chlorophyll retrieval in a large, turbid and optically-complex shallow lake. Remote Sens Environ 157(0):158–169. doi:http://dx.doi.org/10.1016/j.rse.2014.07.024, http://www.sciencedirect.com/science/article/pii/S0034425714002739, [special Issue: Remote Sensing of Inland Waters]
Pelekis N, Theodoridis Y (2014) Mobility data management and exploration. Springer, New York
Pettorelli N, Vik J, Mysterud A, Gaillard J, Tucker C, Stenseth N (2005) Using the satellite-derived ndvi to assess ecological responses to environmental change. Trends Ecol Evol 20:503–510
Pijanowski BC, Tayyebi A, Doucette J, Pekin BK, Braun D, Plourde J (2014) A big data urban growth simulation at a national scale: configuring the GIS and neural network based land transformation model to run in a high performance computing (HPC) environment. Environ Model Software 51(0):250–268. doi:http://dx.doi.org/10.1016/j.envsoft.2013.09.015
Plaza AJ (2009) Special issue on architectures and techniques for real-time processing of remotely sensed images. J Real-Time Image Proc 4(3):191–193
Plaza AJ, Chang CI (2007) High performance computing in remote sensing. Chapman & Hall/CRC Press, New York
Repository CC (Retrieved 2015) https://github.com/cartodb/cartodb.js
Rouse JW Jr, Haas RH, Schell JA, Deering DW (1974) Monitoring vegetation systems in the great Plains with Erts, vol.351. NASA Special Publication, Washington p 309
Russom P (2011) Big data analytics. TDWI best practices report, The Data Warehousing Institute (TDWI) Research
Sakr S, Liu A, Batista D, Alomari M (2011) A survey of large scale data management approaches in cloud environments. IEEE Commun Surv Tutorials 13(3):311–336. doi:10.1109/SURV.2011.032211.00087
Sass G, Creed I, Bayley S, Devito K (2007) Understanding variation in trophic status of lakes on the boreal plain: a 20 year retrospective using landsat {TM} imagery. Remote Sens Environ 109(2):127–141
Schut P (2007) OpenGIS web processing service (OGC 05-007r7)
Vouk M (2008) Cloud computing 2014; issues, research and implementations. In: 30th international conference on information technology interfaces, 2008 (ITI 2008), pp 31–40. doi:10.1109/ITI.2008.4588381
Vretanos PPA (2010) OpenGIS Web Feature Service 2.0 Interface Standard (OGC 09-025r1 and ISO/DIS 19142)
Yu P (2013) On mining big data. In: Wang J, Xiong YH (ed) Web-age information management. Lecture notes in computer science. Springer, Berlin, Heidelberg
Yue P, Gong J, Di L, Yuan J, Sun L, Sun Z, Wang Q (2010) Geopw: laying blocks for the geospatial processing web. Trans GIS 14(6):755–772. doi:10.1111/j.1467-9671.2010.01232.x, http://dx.doi.org/10.1111/j.1467-9671.2010.01232.x
Yue P, Di L, Wei Y, Han W (2013) Intelligent services for discovery of complex geospatial features from remote sensing imagery. ISPRS J Photogramm Remote Sens 83(0):151–164. doi:http://dx.doi.org/10.1016/j.isprsjprs.2013.02.015, http://www.sciencedirect.com/science/article/pii/S0924271613000580
Zeiler M (1999) Modeling our world: the ESRI guide to geodatabase design. ESRI Press, Redlands
Zell E, Huff A, Carpenter A, Friedl L (2012) A user-driven approach to determining critical earth observation priorities for societal benefit. IEEE J Sel Top Appl Earth Obs Remote Sens 5(6):1594–1602. doi:10.1109/JSTARS.2012.2199467
Zhang X, Seelan S, Seielstad G (2010) Digital northern great plains: a web-based system delivering near real time remote sensing data for precision agriculture. Remote Sens 2(3):861. doi:10.3390/rs2030861, http://www.mdpi.com/2072-4292/2/3/861
Zhang Y, Kersten M, Ivanova M, Nes N (2011) Sciql: bridging the gap between science and relational dbms. In: Proceedings of the 15th symposium on international database engineering & Applications (IDEAS ’11). ACM, New York, NY, pp 124–133. doi:10.1145/2076623.2076639, http://doi.acm.org/10.1145/2076623.2076639
Zhang Y, Scheers B, Kersten MNN Mand Ivanova (2011) Astronomical data processing using sciql, an sql based query language for array data. In: Astronomical data analysis software and systems XXI, vol 461, p 729
Zhao P, Foerster T, Yue P (2012) The geoprocessing web. Comput Geosci 47(0): 3–12. doi:http://dx.doi.org/10.1016/j.cageo.2012.04.021, http://www.sciencedirect.com/science/article/pii/S0098300412001446, towards a Geoprocessing Web
Zikopoulos P, Eaton C (2012) Understanding big data: analytics for enterprise class hadoop and streaming data. McGraw-Hill Companies, Inc., New York
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2016 Springer International Publishing Switzerland
About this chapter
Cite this chapter
Karmas, A., Tzotsos, A., Karantzalos, K. (2016). Geospatial Big Data for Environmental and Agricultural Applications. In: Yu, S., Guo, S. (eds) Big Data Concepts, Theories, and Applications . Springer, Cham. https://doi.org/10.1007/978-3-319-27763-9_10
Download citation
DOI: https://doi.org/10.1007/978-3-319-27763-9_10
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-27761-5
Online ISBN: 978-3-319-27763-9
eBook Packages: Computer ScienceComputer Science (R0)