Chapter

Euro-Par 2011 Parallel Processing

Volume 6852 of the series Lecture Notes in Computer Science pp 366-379

Compressing the Incompressible with ISABELA: In-situ Reduction of Spatio-temporal Data

  • Sriram LakshminarasimhanAffiliated withNorth Carolina State UniversityOak Ridge National Laboratory
  • , Neil ShahAffiliated withNorth Carolina State University
  • , Stephane EthierAffiliated withPrinceton Plasma Physics Laboratory
  • , Scott KlaskyAffiliated withOak Ridge National Laboratory
  • , Rob LathamAffiliated withArgonne National Laboratory
  • , Rob RossAffiliated withArgonne National Laboratory
  • , Nagiza F. SamatovaAffiliated withNorth Carolina State UniversityOak Ridge National Laboratory

* Final gross prices may vary according to local VAT.

Get Access

Abstract

Modern large-scale scientific simulations running on HPC systems generate data in the order of terabytes during a single run. To lessen the I/O load during a simulation run, scientists are forced to capture data infrequently, thereby making data collection an inherently lossy process. Yet, lossless compression techniques are hardly suitable for scientific data due to its inherently random nature; for the applications used here, they offer less than 10% compression rate. They also impose significant overhead during decompression, making them unsuitable for data analysis and visualization that require repeated data access.

To address this problem, we propose an effective method for In-situ Sort-And-B-spline Error-bounded Lossy Abatement (ISABELA) of scientific data that is widely regarded as effectively incompressible. With ISABELA, we apply a preconditioner to seemingly random and noisy data along spatial resolution to achieve an accurate fitting model that guarantees a ≥ 0.99 correlation with the original data. We further take advantage of temporal patterns in scientific data to compress data by ≈ 85%, while introducing only a negligible overhead on simulations in terms of runtime. ISABELA  significantly outperforms existing lossy compression methods, such as Wavelet compression. Moreover, besides being a communication-free and scalable compression technique, ISABELA  is an inherently local decompression method, namely it does not decode the entire data, making it attractive for random access.

Keywords

Lossy Compression B-spline In-situ Processing Data-intensive Application High Performance Computing