Earth Observation and Geospatial Implementation: Fueling Innovation in a Changing World
With an ever-increasing wealth of earth science data produced from various sources and platforms including earth observation, modeling and forecasts, the opportunities to exploit such vast amounts of data to produce valuable information products are challenging and exciting. These data are widely used for monitoring, simulation and analysis of measurements that are associated with physical, chemical and biological phenomena across the ocean, atmosphere and land. The volume, variety, and complexity of multidimensional earth science data have posed challenges in the past with how it is shared with a diverse community, visualized intuitively, or integrated for answering scientific questions. In this chapter, we will share broadly how geospatial technology, and Esri technology, is evolving to better support earth observation scientific data management, analysis and visualization. Methods for sharing data through information products as web services and tailored applications that expand discovery and access to a larger user community using a suite of ArcGIS web apps will be discussed. We will then briefly explore the suite of ArcGIS web applications and how they are revolutionizing the way we communicate and tell stories with dynamic data-driven apps such as Esri Story Map. Lastly, a brief overview of how Esri is moving towards improving BigData capabilities will be provided. These methods and applications, publishing and consuming services and performing analyses are key drivers to advancing innovation.
Earth Observation Scientific Data Access, Discovery, and Challenges
The intuitive nature of understanding information portrayed on a map makes them extremely useful to decision-makers. The tools used to interact with web-based maps increase the utility of information products. As more information about our world becomes open to an ever-growing number of people, we have more opportunities to help decision-makers incorporate that information into their processes and workflows. This fact is especially true when considering the vast amount of data—including scientific multidimensional data—that when made easily accessible, could be used more effectively through online maps and apps.
US government agencies such as NOAA, NASA, USGS, USDA and DOE produce large volumes of near real-time, forecast, and historical data that drives climatological and meteorological studies. These data underpin critical operations ranging from short term weather prediction to long term climate projections, covering temperature changes or anomalies, across land or sea and how it is driving sea ice loss (Shrestha et al. 2016).
Modern science is computationally intensive because of the availability of this enormous amount of scientific data, the adoption of data-driven analysis, and the need to share data and research results with the public. Advances in IT and mapping technology have made a tremendous impact on how we ingest, manage, analyze, visualize, and share the complex scientific data that can be consumed by scientists, policy makers and the public.
The volume, variety and complexity of multidimensional earth observation, model and forecast scientific data pose challenges with how it is shared with a diverse community, visualized intuitively, or fused for answering scientific questions. Implementing a spatial data infrastructure for Earth and Space Sciences that is built on top of frameworks that support satellite images, airborne data, climate and weather observations, simulations, and forecasts allows us to manage data, generate science information products, and develop focused tools that support the specific needs of end users (Shrestha et al. 2016).
How Are Maps and Earth Observation Data Transforming Our World?
The challenge of existing and upcoming earth observations scientific data is how to make complicated data available and interoperable for the growing community of potential users from scientists, data wranglers and facility managers to policy makers. Geospatial software is evolving in step with the technology industry to help meet these challenges. Two of the biggest changes in the geospatial community in recent years have been the increase in collaboration, and the growth of cloud computing. In the early days, most GIS analysts and image scientists were individuals working on their own project, on their own data, all on their computers. As technology evolved, people within organizations began to see benefit in sharing, and they developed centralized data storage and some centralized analytic services for use within their organization. Organizations like these, who had adopted a Services Oriented Architecture approach, have been early adopters moving to cloud based solutions, are now sharing their data and services with other organizations. Collaborative communities have emerged that reach beyond corporate or organizational boundaries. This system of systems is a collection of portals, or distributed data and distributed analytics, which can interoperate as a single system, allowing people to collaborate across space and discipline. One example of this is GEOSS, the Global Earth Observations System of Systems, which has 104 member countries and many affiliated research groups. Esri and many of its customers are active participants in GEOSS.
A large factor in this increase in collaboration has been the growth of cloud computing in geosciences. ArcGIS Online is Esri’s fully distributed collaborative environment hosted cloud solution providing data hosting, mapping, and spatial analysis. It is included with every Esri software license, has several million users, and serves over one billion maps per day. Many organizations still want or need to host their own data and applications on their portal. Although some still choose an on-premise solution where they purchase and manage their own hardware, we have seen a large increase in the use of commercial cloud services in the last 3 years. Commercial cloud providers such as Microsoft and Amazon have become more cost effective and easier to implement than building and configuring your own hardware. To simplify setup, Esri provides a collection of predefined machine images to choose from, making it as easy as visiting a webpage, picking the hardware and software configuration you want, and then you can be up and running with a 20 or 200 CPU cluster the same day.
Delivering Geospatial Data and Information Products with Web GIS
The capabilities to share geospatial data and information products to meet the need of varying levels of users is accomplished through Web GIS, which is an architectural approach, for implementing a modern GIS. It is powered by web services that deliver data and capabilities as well as connect components. It can be implemented in the cloud, on-premises, or more typically as a hybrid combination of the two, leveraging the best of both worlds.
Web GIS is a transformation of GIS that brings analytics to spatial data in a way that was not possible before. Previously, spatial data had to be processed, modified, and extracted to answer a set of questions. With Web GIS, the data is transformed into web maps or services that are mashed up with different layers, making the data to dynamically answer questions. It no longer needs to be processed for each individual parameter. Web GIS offers a much more flexible and agile workflow. It brings GIS into the hands of a much larger audience, reduces the need to create custom applications, provides a platform for integrating GIS with other business systems, and enables cross-organizational collaboration.
Web GIS (Esri 2017a) provides the capabilities to let organizations properly manage all of their geographic knowledge. At the heart of Web GIS is a map-centric content management system. It lets you share your data and information products as map services, and other types of services providing the flexibility to enable various access capabilities for user interaction. For example, if you want to make your imagery available through an open, recognized standard, you can enable the Web Coverage Service (WCS) capability on the image service. You can also enable similar capabilities on a map, feature data, analysis tasks, etc. When you publish a service with ArcGIS Enterprise, it exposes itself through the common web service technologies SOAP and REST. This lets you share your web maps using web service URLs, and gives you freedom over how you want your users to interact with your data and information products over the web. In nutshell, map services represent a map that you’ve made available to others on a server, and they are designed to work in many web and intranet scenarios. The same map service may be consumed by several users on different platforms at the same time without those users physically downloading the data. And as mentioned earlier, Esri web services are exposed through SOAP and REST interfaces, and it also supports applicable OGC standards, making them usable in a wide variety of Esri and non-Esri applications.
To help leverage this wealth of dynamic data-driven web services, Esri provides a large suite of configurable open source apps that help people interact with the maps and data on desktops, the web, smartphones, and tablets. This suite includes apps for the field (delivering focused workflows and tools for everyday tasks), apps for the office (which let you view, analyze, create and share maps and location information), and apps for the community (including easily configurable apps like Esri Story Maps as well as an open data portal to share important information with the community). Story Maps are a new approach on telling dynamic map-based stories and interactive narratives in a way that makes geographic information easy to understand. They use geography as a means of organizing and presenting information, and they are designed for technical as well as non-technical audiences. They combine interactive maps with other rich content—text, photos, video and audio—within user experiences that are basic and intuitive (Esri 2017b). By telling the story of a place, event, issue, trend or pattern in a geographic context, they are a great tool for public engagement. There are already hundreds of thousands of these stories created and shared on the web (http://storymaps.arcgis.com/en/gallery/). Most Story Maps are created using simple templates and do not require the user to write any code. But if they are interested in getting fancy and extending the templates, they are open source and available for customization.
Esri and Big Data
Big Data storage and processing is one of the Esri’s important development thrusts and it has become an integral component of the ArcGIS platform. GIS in the age of BigData is about organizing, visualizing, and analyzing massive and diverse data from various sources to make accurate, more informed decisions. There is high demand for using such capabilities to encapsulate BigData by large organizations. Large organizations can manage tremendous amounts of operational data using enterprise systems, creating a useful and overwhelming source of information. They can integrate their geospatial systems with their critical business systems and BigData solutions to analyze massive amounts of data across the enterprise. Esri has used its BigData tools, including ArcGIS Image Server, ArcGIS GeoEvent Server, and ArcGIS GeoAnalytics Server, to accomplish specific objectives in several domains including intelligence, telematics, telecommunications, insurance, advertising and more.
The value of such a user experience is the ability to instantaneously find and use data and currently receives 1–2 million requests per month. The user does not need to search, select and download, but can directly use in their desktop and web applications. The user can define processing to be applied and instantaneously get the results either visually or as data values without the need to download. The data and analysis capabilities are also available as image services for use in other applications.
Automated daily ingest, ortho-correction and serving of current image basemaps for web applications.
Segmentation and classification of very large high resolution image collections.
Interactive site selection suitability modeling for green infrastructure development for the entire United States at 30 m resolution.
Perform terrain analysis such as viewshed, elevation profiles, and watershed delineation anywhere in the world in seconds.
Using millions of emergency calls accumulated over decades, which areas had the highest rates of emergency calls?
What are the most popular locations for taxi pickups in Paris, and how is this trend changing weekly?
What is the flight path of all US commercial aircraft in 2016, and how many of those paths occurred within 100 km of a no-fly zone?
Ingest high velocity sensor network data and make is available as web services for visualization and analysis.
Set alerts for values exceeding predefined values when actions should occur.
Geofence areas of interest using existing feature data to detect the spatial proximity of events.
Notable Solutions and Capabilities
There have been several technological advances in recent years that are changing data usage across the earth science community, and driving new innovative research and development. Advances such as increased internet speeds, machine to machine communication, use of Application Program Interfaces (APIs), and the increased adoption of mobile devices have changed how organizations provide access to earth science data utilizing GIS.
With the rise of faster servers and increased internet speeds, the paradigm of downloading data to your local system is being replaced by accessing data through services over the web. These web services provide the capability to visualize the data, change projections, and perform processing and analysis on the fly. Any application and device now connected to the web can access and use these services, providing users direct access to just the data they need in the format they want.
As the proliferation of Earth Observation data meets emerging technology, it is becoming easier to share data with a rapidly growing and diverse audience. Data providers further enable this by providing not only data, but intelligent web services and information products that are tailored to user experiences. Driving this change is a shift toward open data portals and collaboration, increasingly hosted on commercial cloud infrastructure. Geospatial software is in many ways driving innovation by developing architecture and algorithms for leveraging this vast storage and computational power. The level of IT skill needed to spin up a compute cluster and share useful web services is dropping quickly. This is leading to democratization of geospatial BigData. We see growing excitement and activity in our work and that of our customers, merging image science and spatial science with data science techniques, leading to the emergence of geospatial data science as a new specialty. Combining the increasing Earth Observation data with new capabilities to manage, analyze and serve this data lead to improved decision making for our planet and many opportunities for new discovery.
- Baker P (2015) MRF cloud optimized image format and LERC compression. https://geonet.esri.com/docs/DOC-7473. Accessed 17 Mar 2017
- Esri (2017a) WebGIS. http://www.esri.com/esri-news/arcnews/fall13articles/implementing-web-gis. Accessed 17 Mar 2017
- Esri (2017b) ArcGIS book: tell your story using a map. http://learn.arcgis.com/en/arcgis-book/chapter3/#learn-more. Accessed 31 Mar 2017
- Esri (2017c) Raster functions and mosaic dataset. http://desktop.arcgis.com/en/arcmap/latest/manage-data/raster-and-images/what-are-the-functions-used-by-a-raster-or-mosaic-dataset.htm. Accessed 25 Mar 2017
- Esri (2017d) Unlock the Earth’s secret with ArcGIS and the Landsat earth observation satellites. http://www.esri.com/landing-pages/software/landsat/unlock-earths-secrets#. Accessed 25 Mar 2017
- Esri (2017e) GeoAnalytics server. https://server.arcgis.com/en/server/latest/get-started/windows/what-is-arcgis-geoanalytics-server-.htm. Accessed 03 Apr 2017
- Esri (2017f) GeoEvent server. https://server.arcgis.com/en/geoevent. Accessed 03 Apr 2017
- Shrestha, S. R. et al (2016). Dynamic analysis, transformation, dissemination and applications of scientific multidimensional data in ArcGIS Platform. In: Spatial data infrastructure for earth and space sciences: analyzing, visualizing, and sharing multidimensional earth science data II. American Geophysical Union (AGU) Fall Meeting, San Francisco, Dec 2016Google Scholar
<SimplePara><Emphasis Type="Bold">Open Access</Emphasis> This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.</SimplePara> <SimplePara>The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.</SimplePara>