Analytics-Driven Lossless Data Compression for Rapid In-situ Indexing, Storing, and Querying

  • John Jenkins
  • Isha Arkatkar
  • Sriram Lakshminarasimhan
  • Neil Shah
  • Eric R. Schendel
  • Stephane Ethier
  • Choong-Seock Chang
  • Jacqueline H. Chen
  • Hemanth Kolla
  • Scott Klasky
  • Robert Ross
  • Nagiza F. Samatova
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7447)

Abstract

The analysis of scientific simulations is highly data-intensive and is becoming an increasingly important challenge. Peta-scale data sets require the use of light-weight query-driven analysis methods, as opposed to heavy-weight schemes that optimize for speed at the expense of size. This paper is an attempt in the direction of query processing over losslessly compressed scientific data. We propose a co-designed double-precision compression and indexing methodology for range queries by performing unique-value-based binning on the most significant bytes of double precision data (sign, exponent, and most significant mantissa bits), and inverting the resulting metadata to produce an inverted index over a reduced data representation. Without the inverted index, our method matches or improves compression ratios over both general-purpose and floating-point compression utilities. The inverted index is light-weight, and the overall storage requirement for both reduced column and index is less than 135%, whereas existing DBMS technologies can require 200-400%. As a proof-of-concept, we evaluate univariate range queries that additionally return column values, a critical component of data analytics, against state-of-the-art bitmap indexing technology, showing multi-fold query performance improvements.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    IEEE standard for floating-point arithmetic. IEEE Standard 754-2008 (2008)Google Scholar
  2. 2.
    Abadi, D., Madden, S., Ferreira, M.: Integrating compression and execution in column-oriented database systems. In: Proceedings of the 2006 ACM SIGMOD International Conference on Management of Data, SIGMOD 2006, pp. 671–682. ACM, New York (2006)CrossRefGoogle Scholar
  3. 3.
    Fryxell, B., Olson, K., Ricker, P., Timmes, F.X., Zingale, M., Lamb, D.Q., MacNeice, P., Rosner, R., Truran, J.W., Tufo, H.: FLASH: An adaptive mesh hydrodynamics code for modeling astrophysical thermonuclear flashes. The Astrophysical Journal Supplement Series 131, 273–334 (2000)CrossRefGoogle Scholar
  4. 4.
    Burtscher, M., Ratanaworabhan, P.: High throughput compression of double-precision floating-point data. In: IEEE Data Compression Conference, pp. 293–302 (2007)Google Scholar
  5. 5.
    Burtscher, M., Ratanaworabhan, P.: FPC: A high-speed compressor for double-precision floating-point data. IEEE Transactions on Computers 58, 18–31 (2009)MathSciNetCrossRefGoogle Scholar
  6. 6.
    Chen, J.H., Choudhary, A., Supinski, B., DeVries, M., Hawkes, S.K.E.R., Liao, W., Ma, K., Mellor-Crummey, J., Podhorszki, N., Sankaran, S.S.R., Yoo, C.: Terascale direct numerical simulations of turbulent combustion using S3D. Comp. Sci. and Discovery 2(1)Google Scholar
  7. 7.
    Comer, D.: The ubiquitous B-Tree. ACM Comput. Surv. 11, 121–137 (1979)MATHCrossRefGoogle Scholar
  8. 8.
    Goeman, B., Vandierendonck, H., Bosschere, K.D.: Differential FCM: Increasing value prediction accuracy by improving table usage efficiency. In: Seventh International Symposium on High Performance Computer Architecture, pp. 207–216 (2001)Google Scholar
  9. 9.
    Graefe, G., Shapiro, L.: Data compression and database performance. In: Proceedings of the 1991 Symposium on Applied Computing, pp. 22–27 (April 1991)Google Scholar
  10. 10.
    Ibarria, L., Lindstrom, P., Rossignac, J., Szymczak, A.: Out-of-core compression and decompression of large n-dimensional scalar fields. Computer Graphics Forum 22, 343–348 (2003)CrossRefGoogle Scholar
  11. 11.
    Isenburg, M., Lindstrom, P., Snoeyink, J.: Lossless compression of predicted floating-point geometry. Computer-Aided Design 37(8), 869–877 (2005); CAD 2004 Special Issue: Modelling and Geometry Representations for CADMATHCrossRefGoogle Scholar
  12. 12.
    Iyer, B.R., Wilhite, D.: Data compression support in databases. In: Proceedings of the 20th International Conference on Very Large Data Bases, VLDB 1994, pp. 695–704. Morgan Kaufmann Publishers Inc., San Francisco (1994)Google Scholar
  13. 13.
    Wu, K., Ahern, S., Bethel, E.W., Chen, J., Childs, H., Cormier-Michel, E., Geddes, C., Gu, J., Hagen, H., Hamann, B., Koegler, W., Lauret, J., Meredith, J., Messmer, P., Otoo, E., Perevoztchikov, V., Poskanzer, A., Prabhat, Rubel, O., Shoshani, A., Sim, A., Stockinger, K., Weber, G., Zhang, W.-M.: FastBit: interactively searching massive data. Journal of Physics: Conference Series 180(1), 012053 (2009)Google Scholar
  14. 14.
    Ku, S., Chang, C., Diamond, P.: Full-f gyrokinetic particle simulation of centrally heated global ITG turbulence from magnetic axis to edge pedestal top in a realistic Tokamak geometry. Nuclear Fusion 49(11), 115021 (2009)CrossRefGoogle Scholar
  15. 15.
    Lindstrom, P., Isenburg, M.: Fast and efficient compression of floating-point data. IEEE Transactions on Visualization and Computer Graphics 12, 1245–1250 (2006)CrossRefGoogle Scholar
  16. 16.
    Sinha, R.R., Winslett, M.: Multi-resolution bitmap indexes for scientific data. ACM Trans. Database Syst. 32 (August 2007)Google Scholar
  17. 17.
    Wang, W.X., Lin, Z., Tang, W.M., Lee, W.W., Ethier, S., Lewandowski, J.L.V., Rewoldt, G., Hahm, T.S., Manickam, J.: Gyro-kinetic simulation of global turbulent transport properties in Tokamak experiments. Physics of Plasmas 13(9), 092505 (2006)Google Scholar
  18. 18.
    Westmann, T., Kossmann, D., Helmer, S., Moerkotte, G.: The implementation and performance of compressed databases. SIGMOD Rec. 29(3), 55–67 (2000)CrossRefGoogle Scholar
  19. 19.
    Witten, I.H., Moffat, A., Bell, T.C.: Managing Gigabytes: Compressing and Indexing Documents and Images, 2nd edn. Morgan Kaufmann (1999)Google Scholar
  20. 20.
    Wu, K.: Fastbit: an efficient indexing technology for accelerating data-intensive science. Journal of Physics: Conference Series 16, 556 (2005)CrossRefGoogle Scholar
  21. 21.
    Yiannakis, S., Smith, J.E.: The predictability of data values. In: Proceedings of the 30th Annual ACM/IEEE International Symposium on Microarchitecture, MICRO 30, pp. 248–258. IEEE Computer Society, Washington, DC (1997)Google Scholar
  22. 22.
    Zobel, J., Moffat, A.: Inverted files for text search engines. ACM Computing Surveys 38(2) (July 2006)Google Scholar
  23. 23.
    Zukowski, M., Heman, S., Nes, N., Boncz, P.: Super-scalar ram-cpu cache compression. In: Proceedings of the 22nd International Conference on Data Engineering, ICDE 2006. IEEE Computer Society, Washington, DC (2006)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • John Jenkins
    • 1
    • 2
  • Isha Arkatkar
    • 1
    • 2
  • Sriram Lakshminarasimhan
    • 1
    • 2
  • Neil Shah
    • 1
    • 2
  • Eric R. Schendel
    • 1
    • 2
  • Stephane Ethier
    • 3
  • Choong-Seock Chang
    • 3
  • Jacqueline H. Chen
    • 4
  • Hemanth Kolla
    • 4
  • Scott Klasky
    • 2
  • Robert Ross
    • 5
  • Nagiza F. Samatova
    • 1
    • 2
  1. 1.North Carolina State UniversityUSA
  2. 2.Oak Ridge National LaboratoryUSA
  3. 3.Princeton Plasma Physics LaboratoryPrincetonUSA
  4. 4.Sandia National LaboratoryLivermoreUSA
  5. 5.Argonne National LaboratoryArgonneUSA

Personalised recommendations