Abstract
Today’s massively parallel simulation codes can produce output ranging up to many terabytes of data. Utilizing this data to support scientific inquiry requires analysis and visualization, yet the sheer size of the data makes it cumbersome or impossible to read without computational resources similar to the original simulation. We identify two broad classes of problems for reading data and present effective solutions for both. The first class of data reads depends on user requirements and available resources. Tasks such as visualization and user-guided analysis may be accomplished using only a subset of variables with a restricted spatial extent at a reduced resolution. The other class of reads requires full resolution multivariate data to be loaded, for example to restart a simulation. We show that utilizing the hierarchical multiresolution IDX data format enables scalable and efficient serial and parallel read access on a variety of hardware from supercomputers down to portable devices. We demonstrate interactive view-dependent visualization and analysis of massive scientific datasets using low-power commodity hardware, and we compare read performance with other parallel file formats for both full and partial resolution data.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
HDF5 home page, http://www.hdfgroup.org/HDF5/
OpenGL standard, http://www.opengl.org/
OpenGL view frustum culling, http://www.lighthouse3d.com/tutorials/view-frustum-culling/
VAPOR home page, http://www.vapor.ucar.edu/
Visit home page, https://wci.llnl.gov/codes/visit/
Ahrens, J.P., Woodring, J., DeMarle, D.E., Patchett, J., Maltrud, M.: Interactive remote large-scale data visualization via prioritized multi-resolution streaming. In: Proceedings of the 2009 Workshop on Ultrascale Visualization, UltraVis 2009, pp. 1–10. ACM, New York (2009)
Chen, J.H., Choudhary, A., de Supinski, B., DeVries, M., Hawkes, E.R., Klasky, S., Liao, W.K., Ma, K.L., Crummey, J.M., Podhorszki, N., Sankaran, R., Shende, S., Yoo, C.S.: Terascale direct numerical simulations of turbulent combustion using s3d. In: Computational Science and Discovery, vol. 2 (January 2009)
Chiueh, T.-C., Katz, R.H.: Multi-resolution video representation for parallel disk arrays. In: Proceedings of the First ACM International Conference on Multimedia, MULTIMEDIA 1993, pp. 401–409. ACM, New York (1993)
del Rosario, J.M., Bordawekar, R., Choudhary, A.: Improved parallel I/O via a two-phase run-time access strategy. SIGARCH Comput. Archit. News 21, 31–38 (1993)
Foley, J.D., van Dam, A., Feiner, S.K., Hughes, J.F.: Computer Graphics: Principles and Practice, 2nd edn. Addison-Wesley Longman Publishing Co., Inc., Boston (1990)
Guthe, S., Wand, M., Gonser, J., Strasser, W.: Interactive rendering of large volume data sets. In: Proceedings of the Conference on Visualization 2002, VIS 2002, pp. 53–60. IEEE Computer Society, Washington, DC (2002)
Hearn, D., Baker, M.P.: Computer graphics, C version, vol. 2. Prentice Hall, Upper Saddle River (1997)
Kumar, S., Pascucci, V., Vishwanath, V., Carns, P., Hereld, M., Latham, R., Peterka, T., Papka, M., Ross, R.: Towards parallel access of multi-dimensional, multi-resolution scientific data. In: 2010 5th Petascale Data Storage Workshop (PDSW), pp. 1–5 (2010)
Kumar, S., Vishwanath, V., Carns, P., Levine, J.A., Latham, R., Scorzelli, G., Kolla, H., Grout, R., Ross, R., Papka, M.E., Chen, J., Pascucci, V.: Efficient data restructuring and aggregation for I/O acceleration in pidx. In: Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis, SC 2012, pp. 50:1–50:11. IEEE Computer Society Press, Los Alamitos (2012)
Kumar, S., Vishwanath, V., Carns, P., Summa, B., Scorzelli, G., Pascucci, V., Ross, R., Chen, J., Kolla, H., Grout, R.: Pidx: Efficient parallel I/O for multi-resolution multi-dimensional scientific datasets. In: Proceedings of the 2011 IEEE International Conference on Cluster Computing, CLUSTER 2011, pp. 103–111. IEEE Computer Society, Washington, DC (2011)
Lawder, J.K., King, P.J.H.: Using space-filling curves for multi-dimensional indexing. In: Jeffery, K., Lings, B. (eds.) BNCOD 2000. LNCS, vol. 1832, p. 20. Springer, Heidelberg (2000)
Li, J., Liao, W.-K., Choudhary, A., Ross, R., Thakur, R., Gropp, W., Latham, R., Siegel, A., Gallagher, B., Zingale, M.: Parallel netCDF: A high-performance scientific I/O interface. In: Proceedings of SC 2003: High Performance Networking and Computing, Phoenix, AZ. IEEE Computer Society Press (November 2003)
Lofstead, J., Klasky, S., Schwan, K., Podhorszki, N., Jin, C.: Flexible IO and integration for scientific codes through the adaptable IO system (ADIOS). In: Proceedings of the 6th International Workshop on Challenges of Large Applications in Distributed Environments, CLADE 2008, pp. 15–24. ACM, New York (2008)
Meng, Q., Humphrey, A., Schmidt, J., Berzins, M.: Investigating applications portability with the uintah dag-based runtime system on petascale supercomputers. In: Proceedings of the 2013 ACM/IEEE Conference on Supercomputing (SC 2013). ACM (2013)
Pascucci, V., Frank, R.J.: Global static indexing for real-time exploration of very large regular grids. In: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (2001)
Pascucci, V., Scorzelli, G., Summa, B., Bremer, P.-T., Gyulassy, A., Christensen, C., Kumar, S.: Scalable visualization and interactive analysis using massive data streams. Advances in Parallel Computing: Cloud Computing and Big Data 23, 212–230 (2013)
Pascucci, V., Scorzelli, G., Summa, B., Bremer, P.-T., Gyulassy, A., Christensen, C., Philip, S., Kumar, S.: The visus visualization framework. In: Bethel, E.W., Childs, H., Hansen, C. (eds.) High Performance Visualization: Enabling Extreme-Scale Scientific Insight, ch. 19, pp. 401–414. Chapman & Hall and CRC Computational Science (2012)
Shirley, P., Marschner, S.: Fundamentals of Computer Graphics, 3rd edn. A. K. Peters, Ltd., Natick (2009)
Thakur, R., Gropp, W., Lusk, E.: On implementing MPI-IO portably and with high performance. In: Proceedings of the 6th Workshop on I/O in Parallel and Distributed Systems, pp. 23–32. ACM Press (1999)
Tian, Y., Klasky, S., Yu, W., Wang, B., Abbasi, H., Podhorszki, N., Grout, R.: Dynam: Dynamic multiresolution data representation for large-scale scientific analysis. In: 2013 IEEE Eighth International Conference on Networking, Architecture and Storage (NAS), pp. 115–124. IEEE (2013)
Wang, C., Gao, J., Li, L., Shen, H.-W.: A multiresolution volume rendering framework for large-scale time-varying data visualization. In: Fourth International Workshop on Volume Graphics, pp. 11–223 (June 2005)
Williams, L.: Pyramidal parametrics. SIGGRAPH Comput. Graph. 17(3), 1–11 (1983)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2014 Springer International Publishing Switzerland
About this paper
Cite this paper
Kumar, S. et al. (2014). Fast Multiresolution Reads of Massive Simulation Datasets. In: Kunkel, J.M., Ludwig, T., Meuer, H.W. (eds) Supercomputing. ISC 2014. Lecture Notes in Computer Science, vol 8488. Springer, Cham. https://doi.org/10.1007/978-3-319-07518-1_20
Download citation
DOI: https://doi.org/10.1007/978-3-319-07518-1_20
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-07517-4
Online ISBN: 978-3-319-07518-1
eBook Packages: Computer ScienceComputer Science (R0)