Reducing the HPC-datastorage footprint with MAFISC—Multidimensional Adaptive Filtering Improved Scientific data Compression
- 328 Downloads
Large HPC installations today also include large data storage installations. Data compression can significantly reduce the amount of data, and it was one of our goals to find out, how much compression can do for climate data. The price of compression is, of course, the need for additional computational resources, so our second goal was to relate the savings of compression to the costs it necessitates.
In this paper we present the results of our analysis of typical climate data. A lossless algorithm based on these insights is developed and its compression ratio is compared to that of standard compression tools. As it turns out, this algorithm is general enough to be useful for a large class of scientific data, which is the reason we speak of MAFISC as a method for scientific data compression. A numeric problem for lossless compression of scientific data is identified and a possible solution is given. Finally, we discuss the economics of data compression in HPC environments using the example of the German Climate Computing Center.
KeywordsData compression NetCDF HDF5
We want to thank Wolfgang Stahl for his measurements of the tape drive compression ratio at the DKRZ and the fruitful discussion about compression impact on HPC installations.
- 1.Alfsen K, Skodvin T (2009) The intergovernmental panel on climate change (ipcc) and scientific consensus Google Scholar
- 4.Dutta S, Bhattacherjee S, Narang A (2011) Towards “intelligent compression” in streams: a biased reservoir sampling based bloom filter approach. arXiv e-prints Google Scholar
- 5.ECMA: Streaming lossless data compression algorithm—(sldc) (2001). ECMA Standart 321 Google Scholar
- 11.Latham R (2010) The parallel-netCDF I/O library Google Scholar
- 15.Taylor KE, Stouffer RJ, Meehl GA (2007) A summary of the CMIP5 experiment design. World 4 (January 2011), 1–33. http://cmip-pcmdi.llnl.gov/cmip5/docs/Taylor_CMIP5_design.pdf
- 16.Woodring J, Mniszewski S, Brislawn C, DeMarle D, Ahrens J (2011) Revisiting wavelet compression for large-scale climate data using JPEG2000 and ensuring data precision. In: IEEE symposium on large data analysis and visualization (LDAV), pp 31–38. doi: 10.1109/LDAV.2011.6092314 CrossRefGoogle Scholar