Evaluating Lossy Compression on Climate Data

  • Nathanael Hübbe
  • Al Wegener
  • Julian Martin Kunkel
  • Yi Ling
  • Thomas Ludwig
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7905)


While the amount of data used by today’s high-performance computing (HPC) codes is huge, HPC users have not broadly adopted data compression techniques, apparently because of a fear that compression will either unacceptably degrade data quality or that compression will be too slow to be worth the effort. In this paper, we examine the effects of three lossy compression methods (GRIB2 encoding, GRIB2 using JPEG 2000 and LZMA, and the commercial Samplify APAX algorithm) on decompressed data quality, compression ratio, and processing time. A careful evaluation of selected lossy and lossless compression methods is conducted, assessing their influence on data quality, storage requirements and performance. The differences between input and decoded datasets are described and compared for the GRIB2 and APAX compression methods. Performance is measured using the compressed file sizes and the time spent on compression and decompression. Test data consists both of 9 synthetic data exposing compression behavior and 123 climate variables output from a climate model. The benefits of lossy compression for HPC systems are described and are related to our findings on data quality.


Data Compression GRIB2 JPEG 2000 APAX 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Christopoulos, C., Skodras, A., Ebrahimi, T.: The JPEG2000 still image coding system: an overview. IEEE Transactions on Consumer Electronics 46(4), 1103–1127 (2000)CrossRefGoogle Scholar
  2. 2.
    Dey, C., et al.: Guide to the WMO Table Driven Code Form Used for the Representation and Exchange of Regularly Spaced Dat. In: Binary Form: FM 92 GRIB Edition 2. Tech. rep., World Meteorological Organization (2007),
  3. 3.
    ECMA: Streaming lossless data compression algorithm - (sldc), ECMA Standart 321 (2001)Google Scholar
  4. 4.
    Hübbe, N., Kunkel, J.: Reducing the HPC-Datastorage Footprint with MAFISC - Multidimensional Adaptive Filtering Improved Scientific data Compression. In: Computer Science - Research and Development. Executive Committee. Springer, Heidelberg (2012), doi: Google Scholar
  5. 5.
    Iverson, J., Kamath, C., Karypis, G.: Fast and effective lossy compression algorithms for scientific datasets. In: Kaklamanis, C., Papatheodorou, T., Spirakis, P.G. (eds.) Euro-Par 2012. LNCS, vol. 7484, pp. 843–856. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  6. 6.
    Lakshminarasimhan, S., Shah, N., Ethier, S., Klasky, S., Latham, R., Ross, R., Samatova, N.F.: Compressing the incompressible with ISABELA: In-situ reduction of spatio-temporal data. In: Jeannot, E., Namyst, R., Roman, J. (eds.) Euro-Par 2011, Part I. LNCS, vol. 6852, pp. 366–379. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  7. 7.
    Lakshminarasimhan, S., Shah, N., Ethier, S., Ku, S.H., Chang, C.S., Klasky, S., Latham, R., Ross, R., Samatova, N.F.: Isabela for effective in situ compression of scientific data. Concurrency and Computation: Practice and Experience (2012)Google Scholar
  8. 8.
    Lindstrom, P., Isenburg, M.: Fast and efficient compression of floating-point data. IEEE Transactions on Visualization and Computer Graphics 12(5), 1245–1250 (2006)CrossRefGoogle Scholar
  9. 9.
    Sullivan, S.: Wavelet compression for floating point data–sengcom. Tech. rep., University Corporation for Atmospheric Research (2012),
  10. 10.
    Wegener, A.: Adaptive compression and decompression of bandlimited signals. US patent 7,009,533 (2006)Google Scholar
  11. 11.
    Woodring, J., Mniszewski, S., Brislawn, C., DeMarle, D., Ahrens, J.: Revisiting wavelet compression for large-scale climate data using JPEG2000 and ensuring data precision. In: 2011 IEEE Symposium on Large Data Analysis and Visualization (LDAV), pp. 31–38 (2011), doi:10.1109/LDAV.2011.6092314Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Nathanael Hübbe
    • 1
  • Al Wegener
  • Julian Martin Kunkel
    • 1
  • Yi Ling
  • Thomas Ludwig
    • 2
  1. 1.University of HamburgGermany
  2. 2.German Climate Computing CentreGermany

Personalised recommendations