Abstract
One of the primary challenges facing scientists is extracting understanding from the large amounts of data produced by simulations, experiments, and observational facilities. The use of data across the entire lifetime ranging from real-time to post-hoc analysis is complex and varied, typically requiring a collaborative effort across multiple teams of scientists. Over time, three sets of tools have emerged: one set for analysis, another for visualization, and a final set for orchestrating the tasks. This trifurcated tool set often results in the manual assembly of analysis and visualization workflows, which are one-off solutions that are often fragile and difficult to generalize. To address these challenges, we propose a serviced-based paradigm and a set of abstractions to guide its design. These abstractions allow for the creation of services that can access and interpret data, and enable interoperability for intelligent scheduling of workflow systems. This work results from a codesign process over analysis, visualization, and workflow tools to provide the flexibility required for production use. Finally, this paper describes a forward-looking research and development plan that centers on the concept of visualization and analysis technology as reusable services, and also describes several real-world use cases that implement these concepts.
D. Pugmire et al.—Contributed Equally.
This manuscript has been co-authored by UT-Battelle, LLC, under contract DE-AC05-00OR22725 with the US Department of Energy (DOE). The US government retains and the publisher, by accepting the article for publication, acknowledges that the US government retains a nonexclusive, paid-up, irrevocable, worldwide license to publish or reproduce the published form of this manuscript, or allow others to do so, for US government purposes. DOE will provide public access to these results of federally sponsored research in accordance with the DOE Public Access Plan (http://energy.gov/downloads/doe-public-access-plan).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Infrastructure as a service (2018). https://webobjects.cdw.com/webobjects/media/pdf/Solutions/cloud-computing/Cloud-IaaS.pdf
ADIS: Adaptive data interfaces and services. https://gitlab.kitware.com/vtk/adis. Accessed 10 June 2020
Ayachit, U., et al.: Paraview catalyst: Enabling in situ data analysis and visualization. In: Proceedings of the First Workshop on in Situ Infrastructures for Enabling Extreme-Scale Analysis and Visualization, pp. 25–29. ACM (2015)
Ayachit, U., et al.: Performance analysis, design considerations, and applications of extreme-scale in situ infrastructures. In: ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis (SC16). Salt Lake City, UT, USA (2016). https://doi.org/10.1109/SC.2016.78. LBNL-1007264
Ayachit, U., et al.: The sensei generic in situ interface. In: 2016 Second Workshop on In Situ Infrastructures for Enabling Extreme-Scale Analysis and Visualization (ISAV), pp. 40–44 (2016). https://doi.org/10.1109/ISAV.2016.013
Bauer, A., et al.: In situ methods, infrastructures, and applications on high performance computing platforms. In: Computer Graphics Forum, Vol. 35, pp. 577–597. Wiley Online Library (2016)
Binyahib, R., et al.: A lifeline-based approach for work requesting and parallel particle advection. In: 2019 IEEE 9th Symposium on Large Data Analysis and Visualization (LDAV), pp. 52–61 (2019)
Camp, D., et al.: Parallel stream surface computation for large data sets. In: IEEE Symposium on Large Data Analysis and Visualization (ldav), pp. 39–47. IEEE (2012)
Childs, H., et al.: Extreme scaling of production visualization software on diverse architectures. IEEE Comput. Graph. Appl. 30(3), 22–31(2010). https://doi.org/10.1109/MCG.2010.51.
Childs, H., et al.: Visualization at extreme scale concurrency. In: Bethel, E.W., Childs, H., Hansen, C. (eds.) High Performance Visualization: Enabling Extreme-Scale Scientific Insight. CRC Press, Boca Raton, FL (2012)
Choi, J.Y., et al.: ICEE: Wide-area in transit data processing framework for near real-time scientific applications. In: 4th SC Workshop on Petascale (Big) Data Analytics: Challenges and Opportunities in conjunction with SC13, vol. 11 (2013)
Choi, J.Y., et al.: Coupling exascale multiphysics applications: methods and lessons learned. In: 2018 IEEE 14th International Conference on e-Science (e-Science), pp. 442–452 (2018). https://doi.org/10.1109/eScience.2018.00133
Dominski, J., et al.: A tight-coupling scheme sharing minimum information across a spatial interface between Gyrokinetic turbulence codes. Phys. Plasmas 25(7), 072,308 (2018). https://doi.org/10.1063/1.5044707.
Dorier, M., et al.: Damaris/viz: A nonintrusive, adaptable and user-friendly in situ visualization framework. In: LDAV-IEEE Symposium on Large-Scale Data Analysis and Visualization (2013)
Dragoni, N., et al.: Microservices: yesterday, today, and tomorrow. CoRR abs/1606.04036 (2016). http://arxiv.org/abs/1606.04036
Duque, E.P., et al.: Epic-an extract plug-in components toolkit for in situ data extracts architecture. In: 22nd AIAA Computational Fluid Dynamics Conference, p. 3410 (2015)
Fabian, N., et al.: The paraview coprocessing library: a scalable, general purpose in situ visualization library. In: 2011 IEEE Symposium on Large Data Analysis and Visualization (LDAV), pp. 89–96. IEEE (2011)
Fogal, T., et al.: Freeprocessing: transparent in situ visualization via data interception. In: Eurographics Symposium on Parallel Graphics and Visualization: EG PGV:[proceedings]/Sponsored by Eurographics Association in Cooperation with ACM SIGGRAPH. Eurographics Symposium on Parallel Graphics and Visualization, vol. 2014, p. 49. NIH Public Access (2014)
Hang, D.: Software as a service. https://www.cs.colorado.edu/~kena/classes/5828/s12/presentation-materials/dibieogheneovohanghaojie.pdf
Joy, K.I., et al.: Streamline integration using MPI-hybrid parallelism on a large multicore architecture. IEEE Trans. Vis. Comput. Graph. 17(11), 1702–1713 (2011). https://doi.org/10.1109/TVCG.2010.259
Kim, M., et al.: In situ analysis and visualization of fusion simulations: lessons learned. In: Yokota, R., Weiland, M., Shalf, J., Alam, S. (eds.) ISC High Performance 2018. LNCS, vol. 11203, pp. 230–242. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-02465-9_16
Klasky, S., et al.: A view from ORNL: Scientific data research opportunities in the big data age. In: 2018 IEEE 38th International Conference on Distributed Computing Systems (ICDCS), pp. 1357–1368. IEEE (2018)
Kress, J., et al.: Visualization and analysis requirements for in situ processing for a large-scale fusion simulation code. In: 2016 Second Workshop on In Situ Infrastructures for Enabling Extreme-Scale Analysis and Visualization (ISAV), pp. 45–50. IEEE (2016)
Kress, J., et al.: Comparing the efficiency of in situ visualization paradigms at scale. In: Weiland, M., Juckeland, G., Trinitis, C., Sadayappan, P. (eds.) ISC High Performance 2019. LNCS, vol. 11501, pp. 99–117. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-20656-7_6
Kress, J., et al.: Opportunities for cost savings with in-transit visualization. In: ISC High Performance 2020. ISC (2020)
Labasan, S., et al.: Power and performance tradeoffs for visualization algorithms. In: Proceedings of IEEE International Parallel and Distributed Processing Symposium (IPDPS), pp. 325–334. Rio de Janeiro, Brazil (2019)
Larsen, M., et al.: Performance modeling of in situ rendering. In: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis SC 2016, pp. 276–287. IEEE (2016)
Larsen, M., et al.: The ALPINE In Situ Infrastructure: ascending from the Ashes of Strawman. In: Proceedings of the In Situ Infrastructures on Enabling Extreme-Scale Analysis and Visualization, pp. 42–46. ACM (2017)
Lawrence Livermore National Laboratory: Blueprint. https://llnl-conduit.readthedocs.io/en/latest/blueprint.html. Accessed 30 June 2020
Lian, M.: Introduction to service oriented architecture (2012). https://www.cs.colorado.edu/~kena/classes/5828/s12/presentation-materials/lianming.pdf
Liu, Q., et al.: Hello adios: the challenges and lessons of developing leadership class i/o frameworks. Concurr. Comput. Pract. Exp. 7, 1453–1473. https://doi.org/10.1002/cpe.3125
Lo, L., et al.: Piston: a portable cross-platform framework for data-parallel visualization operators. In: EGPGV, pp. 11–20 (2012)
Malakar, P., et al.: Optimal scheduling of in-situ analysis for large-scale scientific simulations. In: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, p. 52. ACM (2015)
Malakar, P., et al.: Optimal execution of co-analysis for large-scale molecular dynamics simulations. In: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, p. 60. IEEE Press (2016)
Meredith, J., et al.: A distributed data-parallel framework for analysis and visualization algorithm development. ACM Int. Conf. Proc. Ser. (2012). https://doi.org/10.1145/2159430.2159432
Meredith, J., et al.: EAVL: the extreme-scale analysis and visualization library. In: Eurographics Symposium on Parallel Graphics and Visualization, pp. 21–30. The Eurographics Association (2012)
Moreland, K., et al.: Dax toolkit: A proposed framework for data analysis and visualization at extreme scale. In: 2011 IEEE Symposium on Large Data Analysis and Visualization (LDAV), pp. 97–104 (2011)
Moreland, K., et al.: VTK-M: accelerating the visualization toolkit for massively threaded architectures. IEEE Comput. Graph. Appl. 36(3), 48–58 (2016)
Oak Ridge National Laboratory: ADIOS2: The ADaptable Input/Output System Version 2 (2018). https://adios2.readthedocs.io
Oldfield, R.A., et al.: Evaluation of methods to integrate analysis into a large-scale shock shock physics code. In: Proceedings of the 28th ACM international conference on Supercomputing, pp. 83–92. ACM (2014)
Pugmire, D., et al.: Scalable computation of streamlines on very large datasets. In: Proceedings of the ACM/IEEE Conference on High Performance Computing (SC 2009), Portland, OR (2009)
Pugmire, D., et al.: Parallel integral curves. In: High Performance Visualization-Enabling Extreme-Scale Scientific Insight. CRC Press/Francis-Taylor Group (2012). https://doi.org/10.1201/b12985-8
Pugmire, D., et al.: Towards scalable visualization plugins for data staging workflows. In: Big Data Analytics: Challenges and Opportunities (BDAC-2014) Workshop at Supercomputing Conference (2014)
Pugmire, D., et al.: Performance-Portable Particle Advection with VTK-m. In: Childs, H., Cucchietti, F., (eds.) Eurographics Symposium on Parallel Graphics and Visualization. The Eurographics Association (2018). https://doi.org/10.2312/pgv.20181094
Rivi, M., et al.: In-situ visualization: State-of-the-art and some use cases. Brussels, Belgium, PRACE White Paper; PRACE (2012)
Tchoua, R., et al.: Adios visualization schema: a first step towards improving interdisciplinary collaboration in high performance computing. In: 2013 IEEE 9th International Conference on eScience (eScience), pp. 27–34. IEEE (2013)
The HDF Group: Hdf5 users guide. https://www.hdfgroup.org/HDF5/doc/UG/. Accessed 20 June 2016
Whitlock, B., Favre, J.M., Meredith, J.S.: Parallel in situ coupling of simulation with a fully featured visualization system. In: Kuhlen, T., et al. (eds.) Eurographics Symposium on Parallel Graphics and Visualization. The Eurographics Association (2011). https://doi.org/10.2312/EGPGV/EGPGV11/101-109
Wong, P.C., et al.: The top 10 challenges in extreme-scale visual analytics. IEEE Comput. Graph. Appl. 32(4), 63 (2012)
Yenpure, A., et al.: Efficient point merge using data parallel techniques. In: Eurographics Symposium on Parallel Graphics and Visualization (EGPGV), pp. 79–88. Porto, Portugal (2019)
Yun, G., et al.: Development of Kstar ECE imaging system for measurement of temperature fluctuations and edge density fluctuations. Rev. Sci. Instrum. 81(10), 10D930 (2010)
Acknowledgment
This research was supported by the DOE SciDAC RAPIDS Institute and the Exascale Computing Project (17-SC-20-SC), a collaborative effort of DOE Office of Science and the National Nuclear Security Administration. This research used resources of the Argonne and Oak Ridge Leadership Computing Facilities, DOE Office of Science User Facilities supported under Contracts DE-AC02-06CH11357 and DE-AC05-00OR22725, respectively, as well as the National Energy Research Scientific Computing Center (NERSC), a DOE Office of Science User Facility operated under Contract No. DE-AC02-05CH11231.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 This is a U.S. government work and not under copyright protection in the U.S.; foreign copyright protection may apply
About this paper
Cite this paper
Pugmire, D. et al. (2020). Visualization as a Service for Scientific Data. In: Nichols, J., Verastegui, B., Maccabe, A.‘., Hernandez, O., Parete-Koon, S., Ahearn, T. (eds) Driving Scientific and Engineering Discoveries Through the Convergence of HPC, Big Data and AI. SMC 2020. Communications in Computer and Information Science, vol 1315. Springer, Cham. https://doi.org/10.1007/978-3-030-63393-6_11
Download citation
DOI: https://doi.org/10.1007/978-3-030-63393-6_11
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-63392-9
Online ISBN: 978-3-030-63393-6
eBook Packages: Computer ScienceComputer Science (R0)