Advertisement

Fast Multiresolution Reads of Massive Simulation Datasets

  • Sidharth Kumar
  • Cameron Christensen
  • John A. Schmidt
  • Peer-Timo Bremer
  • Eric Brugger
  • Venkatram Vishwanath
  • Philip Carns
  • Hemanth Kolla
  • Ray Grout
  • Jacqueline Chen
  • Martin Berzins
  • Giorgio Scorzelli
  • Valerio Pascucci
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8488)

Abstract

Today’s massively parallel simulation codes can produce output ranging up to many terabytes of data. Utilizing this data to support scientific inquiry requires analysis and visualization, yet the sheer size of the data makes it cumbersome or impossible to read without computational resources similar to the original simulation. We identify two broad classes of problems for reading data and present effective solutions for both. The first class of data reads depends on user requirements and available resources. Tasks such as visualization and user-guided analysis may be accomplished using only a subset of variables with a restricted spatial extent at a reduced resolution. The other class of reads requires full resolution multivariate data to be loaded, for example to restart a simulation. We show that utilizing the hierarchical multiresolution IDX data format enables scalable and efficient serial and parallel read access on a variety of hardware from supercomputers down to portable devices. We demonstrate interactive view-dependent visualization and analysis of massive scientific datasets using low-power commodity hardware, and we compare read performance with other parallel file formats for both full and partial resolution data.

Keywords

parallel I/O multiresolution PIDX read performance interactive visualization S3D Uintah VisIt 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
  2. 2.
    OpenGL standard, http://www.opengl.org/
  3. 3.
  4. 4.
    VAPOR home page, http://www.vapor.ucar.edu/
  5. 5.
  6. 6.
    Ahrens, J.P., Woodring, J., DeMarle, D.E., Patchett, J., Maltrud, M.: Interactive remote large-scale data visualization via prioritized multi-resolution streaming. In: Proceedings of the 2009 Workshop on Ultrascale Visualization, UltraVis 2009, pp. 1–10. ACM, New York (2009)CrossRefGoogle Scholar
  7. 7.
    Chen, J.H., Choudhary, A., de Supinski, B., DeVries, M., Hawkes, E.R., Klasky, S., Liao, W.K., Ma, K.L., Crummey, J.M., Podhorszki, N., Sankaran, R., Shende, S., Yoo, C.S.: Terascale direct numerical simulations of turbulent combustion using s3d. In: Computational Science and Discovery, vol. 2 (January 2009)Google Scholar
  8. 8.
    Chiueh, T.-C., Katz, R.H.: Multi-resolution video representation for parallel disk arrays. In: Proceedings of the First ACM International Conference on Multimedia, MULTIMEDIA 1993, pp. 401–409. ACM, New York (1993)CrossRefGoogle Scholar
  9. 9.
    del Rosario, J.M., Bordawekar, R., Choudhary, A.: Improved parallel I/O via a two-phase run-time access strategy. SIGARCH Comput. Archit. News 21, 31–38 (1993)CrossRefGoogle Scholar
  10. 10.
    Foley, J.D., van Dam, A., Feiner, S.K., Hughes, J.F.: Computer Graphics: Principles and Practice, 2nd edn. Addison-Wesley Longman Publishing Co., Inc., Boston (1990)Google Scholar
  11. 11.
    Guthe, S., Wand, M., Gonser, J., Strasser, W.: Interactive rendering of large volume data sets. In: Proceedings of the Conference on Visualization 2002, VIS 2002, pp. 53–60. IEEE Computer Society, Washington, DC (2002)CrossRefGoogle Scholar
  12. 12.
    Hearn, D., Baker, M.P.: Computer graphics, C version, vol. 2. Prentice Hall, Upper Saddle River (1997)Google Scholar
  13. 13.
    Kumar, S., Pascucci, V., Vishwanath, V., Carns, P., Hereld, M., Latham, R., Peterka, T., Papka, M., Ross, R.: Towards parallel access of multi-dimensional, multi-resolution scientific data. In: 2010 5th Petascale Data Storage Workshop (PDSW), pp. 1–5 (2010)Google Scholar
  14. 14.
    Kumar, S., Vishwanath, V., Carns, P., Levine, J.A., Latham, R., Scorzelli, G., Kolla, H., Grout, R., Ross, R., Papka, M.E., Chen, J., Pascucci, V.: Efficient data restructuring and aggregation for I/O acceleration in pidx. In: Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis, SC 2012, pp. 50:1–50:11. IEEE Computer Society Press, Los Alamitos (2012)Google Scholar
  15. 15.
    Kumar, S., Vishwanath, V., Carns, P., Summa, B., Scorzelli, G., Pascucci, V., Ross, R., Chen, J., Kolla, H., Grout, R.: Pidx: Efficient parallel I/O for multi-resolution multi-dimensional scientific datasets. In: Proceedings of the 2011 IEEE International Conference on Cluster Computing, CLUSTER 2011, pp. 103–111. IEEE Computer Society, Washington, DC (2011)Google Scholar
  16. 16.
    Lawder, J.K., King, P.J.H.: Using space-filling curves for multi-dimensional indexing. In: Jeffery, K., Lings, B. (eds.) BNCOD 2000. LNCS, vol. 1832, p. 20. Springer, Heidelberg (2000)CrossRefGoogle Scholar
  17. 17.
    Li, J., Liao, W.-K., Choudhary, A., Ross, R., Thakur, R., Gropp, W., Latham, R., Siegel, A., Gallagher, B., Zingale, M.: Parallel netCDF: A high-performance scientific I/O interface. In: Proceedings of SC 2003: High Performance Networking and Computing, Phoenix, AZ. IEEE Computer Society Press (November 2003)Google Scholar
  18. 18.
    Lofstead, J., Klasky, S., Schwan, K., Podhorszki, N., Jin, C.: Flexible IO and integration for scientific codes through the adaptable IO system (ADIOS). In: Proceedings of the 6th International Workshop on Challenges of Large Applications in Distributed Environments, CLADE 2008, pp. 15–24. ACM, New York (2008)Google Scholar
  19. 19.
    Meng, Q., Humphrey, A., Schmidt, J., Berzins, M.: Investigating applications portability with the uintah dag-based runtime system on petascale supercomputers. In: Proceedings of the 2013 ACM/IEEE Conference on Supercomputing (SC 2013). ACM (2013)Google Scholar
  20. 20.
    Pascucci, V., Frank, R.J.: Global static indexing for real-time exploration of very large regular grids. In: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (2001)Google Scholar
  21. 21.
    Pascucci, V., Scorzelli, G., Summa, B., Bremer, P.-T., Gyulassy, A., Christensen, C., Kumar, S.: Scalable visualization and interactive analysis using massive data streams. Advances in Parallel Computing: Cloud Computing and Big Data 23, 212–230 (2013)Google Scholar
  22. 22.
    Pascucci, V., Scorzelli, G., Summa, B., Bremer, P.-T., Gyulassy, A., Christensen, C., Philip, S., Kumar, S.: The visus visualization framework. In: Bethel, E.W., Childs, H., Hansen, C. (eds.) High Performance Visualization: Enabling Extreme-Scale Scientific Insight, ch. 19, pp. 401–414. Chapman & Hall and CRC Computational Science (2012)Google Scholar
  23. 23.
    Shirley, P., Marschner, S.: Fundamentals of Computer Graphics, 3rd edn. A. K. Peters, Ltd., Natick (2009)Google Scholar
  24. 24.
    Thakur, R., Gropp, W., Lusk, E.: On implementing MPI-IO portably and with high performance. In: Proceedings of the 6th Workshop on I/O in Parallel and Distributed Systems, pp. 23–32. ACM Press (1999)Google Scholar
  25. 25.
    Tian, Y., Klasky, S., Yu, W., Wang, B., Abbasi, H., Podhorszki, N., Grout, R.: Dynam: Dynamic multiresolution data representation for large-scale scientific analysis. In: 2013 IEEE Eighth International Conference on Networking, Architecture and Storage (NAS), pp. 115–124. IEEE (2013)Google Scholar
  26. 26.
    Wang, C., Gao, J., Li, L., Shen, H.-W.: A multiresolution volume rendering framework for large-scale time-varying data visualization. In: Fourth International Workshop on Volume Graphics, pp. 11–223 (June 2005)Google Scholar
  27. 27.
    Williams, L.: Pyramidal parametrics. SIGGRAPH Comput. Graph. 17(3), 1–11 (1983)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Sidharth Kumar
    • 1
  • Cameron Christensen
    • 1
  • John A. Schmidt
    • 1
  • Peer-Timo Bremer
    • 1
    • 4
  • Eric Brugger
    • 4
  • Venkatram Vishwanath
    • 2
  • Philip Carns
    • 2
  • Hemanth Kolla
    • 3
  • Ray Grout
    • 5
  • Jacqueline Chen
    • 3
  • Martin Berzins
    • 1
  • Giorgio Scorzelli
    • 1
  • Valerio Pascucci
    • 1
  1. 1.SCI InstituteUniversity of UtahSalt Lake CityUSA
  2. 2.Argonne National LaboratoryArgonneUSA
  3. 3.Sandia National LaboratoryLivermoreUSA
  4. 4.Lawrence-Livermore National LaboratoryLivermoreUSA
  5. 5.National Renewable Energy LaboratoryGoldenUSA

Personalised recommendations