Computing and Visualization in Science

, Volume 19, Issue 5–6, pp 65–76 | Cite as

Multilevel techniques for compression and reduction of scientific data—the univariate case

  • Mark Ainsworth
  • Ozan Tugluk
  • Ben Whitney
  • Scott Klasky
Original Article


We present a multilevel technique for the compression and reduction of univariate data and give an optimal complexity algorithm for its implementation. A hierarchical scheme offers the flexibility to produce multiple levels of partial decompression of the data so that each user can work with a reduced representation that requires minimal storage whilst achieving the required level of tolerance. The algorithm is applied to the case of turbulence modelling in which the datasets are traditionally not only extremely large but inherently non-smooth and, as such, rather resistant to compression. We decompress the data for a range of relative errors, carry out the usual analysis procedures for turbulent data, and compare the results of the analysis on the reduced datasets to the results that would be obtained on the full dataset. The results obtained demonstrate the promise of multilevel compression techniques for the reduction of data arising from large scale simulations of complex phenomena such as turbulence modelling.


Data compression Data reduction Lossy compression Multilevel compression Error-controlled compression 


  1. 1.
    Ainsworth, M., Klasky, S., Whitney, B.: Compression using lossless decimation: analysis and application. SIAM J. Sci. Comput. 39(4), B732–B757 (2017)MathSciNetCrossRefzbMATHGoogle Scholar
  2. 2.
    Austin, W., Ballard, G., Kolda, T. G.: Parallel tensor compression for large-scale scientific data. In: 2016 IEEE international parallel and distributed processing symposium (IPDPS), pp. 912–922, May 2016Google Scholar
  3. 3.
    Bank, R.E., Dupont, T.F., Yserentant, H.: The hierarchical basis multigrid method. Numer. Math. 52(4), 427–458 (1988)MathSciNetCrossRefzbMATHGoogle Scholar
  4. 4.
    Bautista, G., Leonardo, A., Cappello, F.: Improving floating point compression through binary masks. In: 2013 IEEE international conference on big data, pp. 326–331, October 2013Google Scholar
  5. 5.
    Bornemann, F., Yserentant, H.: A basic norm equivalence for the theory of multilevel methods. Numer. Math. 64(1), 455–476 (1993)MathSciNetCrossRefzbMATHGoogle Scholar
  6. 6.
    Burtscher, M., Hari, M., Annie, Y., Farbod, H.: Real-time synthesis of compression algorithms for scientific data. In: SC ‘16: proceedings of the international conference for high performance computing, networking, storage and analysis, IEEE, pp. 264–275, November 2016Google Scholar
  7. 7.
    Cover, T.M., Thomas, J.A.: Elements of Information Theory. Wiley Series in Telecommunications, 1st edn. Wiley, New York (1991)CrossRefzbMATHGoogle Scholar
  8. 8.
    Dahmen, W., Kunoth, A.: Multilevel preconditioning. Numer. Math. 63(3), 315–344 (1992)MathSciNetCrossRefzbMATHGoogle Scholar
  9. 9.
    Daubechies, I.: The wavelet transform, time-frequency localization and signal analysis. IEEE Trans. Inf. Theory 36(5), 961–1005 (1990)MathSciNetCrossRefzbMATHGoogle Scholar
  10. 10.
    Di, S., Cappello, F.: Fast error-bounded lossy HPC data compression with SZ. In: 2016 IEEE 30th international parallel and distributed processing symposium, IEEE, Chicago, IL, USA, pp. 730–739, May 2016Google Scholar
  11. 11.
    Donoho, D.L., Vetterli, M., DeVore, R.A., Daubechies, I.: Data compression and harmonic analysis. IEEE Trans. Inf. Theory 44(6), 2435–2476 (1998)MathSciNetCrossRefzbMATHGoogle Scholar
  12. 12.
    Edmunds, D.E., Triebel, H.: Function Spaces, Entropy Numbers, Differential Operators, 1st edn. Cambridge University Press, Cambridge (1996)CrossRefzbMATHGoogle Scholar
  13. 13.
    Golub, G.H., Van Loan, C.F.: Matrix Computations, 3rd edn. The Johns Hopkins University Press, Baltimore (1996)zbMATHGoogle Scholar
  14. 14.
    Grgic, S., Kers, K., Grgic, M.: Image compression using wavelets. In: Proceedings of the IEEE international symposium on industrial electronics, 1999. ISIE ‘99, vol. 1, pp. 99–104 (1999)Google Scholar
  15. 15.
    Griebel, M., Oswald, P.: Stable splittings of Hilbert spaces of functions of infinitely many variables. J. Complex. 41, 126–151 (2017)MathSciNetCrossRefzbMATHGoogle Scholar
  16. 16.
    Ibarria, L., Lindstrom, P., Rossignac, J., Szymczak, A.: Out-of-core compression and decompression of large n-dimensional scalar fields. Comput. Graph. Forum 22(3), 343–348 (2003)CrossRefGoogle Scholar
  17. 17.
    Johns Hopkins Turbulence Databases. Forced isotropic turbulence dataset description, October 2017. Last update: 10/19/2017 5:55:14 PM. Accessed 01 Feb 2018Google Scholar
  18. 18.
    Kolmogorov, A.: The local structure of turbulence in incompressible viscous fluid for very large Reynolds’ numbers. Akademiia Nauk SSSR Doklady 30, 301–305 (1941)MathSciNetGoogle Scholar
  19. 19.
    Lakshminarasimhan, S., Shah, N., Ethier, S., Klasky, S., Latham, R., Ross, R., Samatova, N. F.: Compressing the incompressible with ISABELA: in-situ reduction of spatio-temporal data. In: Emmanuel J., Raymond N., Jean R. (eds) Euro-Par 2011: Parallel Processing Workshops, Lecture Notes in Computer Science, Bordeaux, France, Springer, Berlin, Heidelberg, vol. 6852, pp. 366–379, August 2011Google Scholar
  20. 20.
    Li, Y., Perlman, E., Wan, M., Yang, Y., Meneveau, C., Burns, R., Chen, S., Szalay, A., Eyink, G.: A public turbulence database cluster and applications to study Lagrangian evolution of velocity increments in turbulence. J. Turbul. 9, N31 (2008)CrossRefzbMATHGoogle Scholar
  21. 21.
    Lindstrom, P.: Fixed-rate compressed floating-point arrays. IEEE Trans. Vis. Comput. Graph. 20(12), 2674–2683 (2014)CrossRefGoogle Scholar
  22. 22.
    Lindstrom, P., Isenburg, M.: Fast and efficient compression of floating-point data. IEEE Trans. Vis. Comput. Graph. 12(5), 1245–1250 (2006)CrossRefGoogle Scholar
  23. 23.
    Marcellin, M. W., Gormish, M. J., Bilgin, A., Boliek, M. P.: An overview of JPEG-2000. In: Proceedings DCC 2000. Data compression conference, pp. 523–541 (2000)Google Scholar
  24. 24.
    Oswald, P.: Multilevel Finite Element Approximation. Theory and Applications. Teubner Skripten zur Numerik. B. G. Teubner, Stuttgart (1994)CrossRefzbMATHGoogle Scholar
  25. 25.
    Perlman, E., Burns, R., Li, Y., Meneveau, C.: Data exploration of turbulence simulations using a database cluster. In: Proceedings of the 2007 ACM/IEEE conference on supercomputing, ACM, Reno, NV, USA, vol. 23, November 2007Google Scholar
  26. 26.
    Salomon, D.: Data Compression: The Complete Reference, 4th edn. Springer, London (2007)zbMATHGoogle Scholar
  27. 27.
    Schendel, E. R., Jin, Y., Shah, N., Chen, J., Chang, C. S., Ku, S.-H., Ethier, S., Klasky, S., Latham, R., Ross, R., Samatova, N. F.: ISOBAR preconditioner for effective and high-throughput lossless data compression. In: 2012 IEEE 28th international conference on data engineering, pp. 138–149, April 2012Google Scholar
  28. 28.
    Schneider, K., Farge, M., Pellegrino, G., Rogers, M.M.: Coherent vertex simulation of three-dimensional turbulent mixing layers using orthogonal wavelets. J. Fluid Mech. 534, 39–66 (2005)MathSciNetCrossRefzbMATHGoogle Scholar
  29. 29.
    Shah, N., Schendel, E. R., Lakshminarasimhan, S., Pendse, S. V., Rogers, T., Samatova, N. F.: Improving I/O throughput with PRIMACY: preconditioning ID-mapper for compressing incompressibility. In: 2012 IEEE international conference on cluster computing, pp. 209–219, September 2012Google Scholar
  30. 30.
    Strengert, M., Magallón, M., Weiskopf, D., Guthe, S., Ertl, T.: Hierarchical visualization and compression of large volume datasets using GPU clusters. EGPGV, pp. 41–48 (2004)Google Scholar
  31. 31.
    Wallace, G. K.: The JPEG still picture compression standard. IEEE Trans. Consum. Electron. 38(1), xviii–xxxiv (1992)Google Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2018

Authors and Affiliations

  • Mark Ainsworth
    • 1
    • 2
  • Ozan Tugluk
    • 1
    • 3
  • Ben Whitney
    • 1
  • Scott Klasky
    • 2
  1. 1.Division of Applied MathematicsBrown UniversityProvidenceUSA
  2. 2.Computer Science and Mathematics DivisionOak Ridge National LaboratoryOak RidgeUSA
  3. 3.Department of Astronautical EngineeringUniversity of Turkish Aeronautical AssociationAnkaraTurkey

Personalised recommendations