Advertisement

PLZMA: A Parallel Data Compression Method for Cloud Computing

  • Xin Wang
  • Lin Gan
  • Jingheng Xu
  • Jinzhe Yang
  • Maocai Xia
  • Haohuan Fu
  • Xiaomeng Huang
  • Guangwen Yang
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11336)

Abstract

Recent decades have seen the rapid development of cloud computing, resulting in a huge breakthrough for people to handle the data produced every second and everywhere. Meanwhile, data compression is becoming increasingly important, due to its great potential in benefiting both the network transportation and the storage. Based on the urgent demand in high-efficient compression method with balanced performance in both merits of compression time and ratio, this paper presents PLZMA, a parallel design of LZMA. Process-level and thread-level parallelisms are implemented according to the algorithm of LZMA, which have gained great improvement in compression time, while ensuring a fair compression ratio. Experimental results on real-world application showed that PLZMA is able to achieve more balanced performance over other famous methods. The parallel design is able to achieve a performance speedup of 8\(\times \) over the serial baseline, using 12 threads.

Keywords

Data compression Parallel computing LZMA 

Notes

Acknowledgement

L. Gan, and J. Xu are supported by the National Natural Science Foundation of China (grant no. 61702297); and the China Postdoctoral Science Foundation (grant no. 2016M601031).

H. Fu, and X. Wang are supported by the National Key Research & Development Plan of China (grant no. 2017YFA0604500), the National Natural Science Foundation of China (grant no. 91530323, 41661134014, 41504040 and 61361120098); and the Tsinghua University Initiative Scientific Research Program (grant no. 20131089356).

G. Yang, and J. Yang are supported by the National Key Research & Development Plan of China (grant no. 2016YFA0602200).

X. Huang is supported by a grant from the State’s Key Project of Research and Development Plan (2016YFB0201100) and the National Natural Science Foundation of China (41375102).

References

  1. 1.
    Dean, J., Ghemawat, S.: Mapreduce: simplified data processing on large clusters. Commun. ACM 51(1), 107–113 (2008)CrossRefGoogle Scholar
  2. 2.
    Motley, C.F.: Telecommunication data compression apparatus and method, April 13 2004. US Patent 6,721,282Google Scholar
  3. 3.
    Yan, C., Zhang, Y., Dai, F., Li, L.: Highly parallel framework for HEVC motion estimation on many-core platform. In: Data Compression Conference (DCC), pp. 63–72. IEEE (2013)Google Scholar
  4. 4.
    Gan, L., Haohuan, F., Luk, W., Yang, C., Xue, W., Yang, G.: Solving mesoscale atmospheric dynamics using a reconfigurable dataflow architecture. IEEE Micro 37(4), 40–50 (2017)CrossRefGoogle Scholar
  5. 5.
    Gan, L., Fu, H., Mencer, O., Luk, W., Yang, G.: Data flow computing in geoscience applications. Adv. Comput. 104, 125–158 (2017)Google Scholar
  6. 6.
    Burrows, M., Wheeler, D.J.: A block-sorting lossless data compression algorithm (1994)Google Scholar
  7. 7.
    Deutsch, P.L.: Deflate compressed data format specification version 1, 3 (1996)Google Scholar
  8. 8.
    Ziv, J., Lempel, A.: A universal algorithm for sequential data compression. IEEE Trans. Inf. Theory 23(3), 337–343 (1977)MathSciNetCrossRefGoogle Scholar
  9. 9.
    Gristwood, T., Fineran, P.C., Everson, L., Salmond, G.P.C.: PigZ, a TetR/AcrR family repressor, modulates secondary metabolism via the expression of a putative four-component resistance-nodulation-cell-division efflux pump, zrpadbc, in serratia sp. atcc 39006. Mol. Microbiol. 69(2), 418–435 (2008)CrossRefGoogle Scholar
  10. 10.
    Adiego, J., Fuente, P.D.L.: Merging prediction by partial matching with structural contexts model, p. 522 (2004)Google Scholar
  11. 11.
    Berners-Lee, T., Fielding, R., Frystyk, H.: Hypertext transfer protocol-http/1.0. Technical report (1996)Google Scholar
  12. 12.
    Woods, J.: PPP deflate protocol (1996)Google Scholar
  13. 13.
    Boutell, T.: PNG (portable network graphics) specification version 1.0. (1997)Google Scholar
  14. 14.
    Deutsch, P., Gailly, J.-L.: Zlib compressed data format specification version 3.3. Technical report (1996)Google Scholar
  15. 15.
    Zhu, W., Xu, J., Ding, W., Shi, Y.: Adaptive LZMA-based coding for screen content. In: Picture Coding Symposium, pp. 373–376 (2013)Google Scholar
  16. 16.
    Kärkkäinen, J.: Fast BWT in small space by blockwise suffix sorting. Elsevier Science Publishers Ltd. (2007)Google Scholar
  17. 17.
    Culler, M., Dunfield, N.M., Weeks, J.R.: Snappy, a computer program for studying the geometry and topology of 3-manifolds (2017)Google Scholar
  18. 18.
    Pavlov, I.: Lzma sdk (software development kit) (2007)Google Scholar
  19. 19.
    Reinhold, L.M.: Quicklz websiteGoogle Scholar
  20. 20.
    Oberhumer, M.F.X.J.: Lzo-a real-time data compression library (2008). http://www.oberhumer.com/opensource/lzo/
  21. 21.
    Varsaki, A., Afendra, A.S., Vartholomatos, G., Tegos, G., Drainas, C.: Production of ice nuclei from two recombinant zymomonas mobilis strains employing the inaZ gene of pseudomonas syringae. Biotechnol. Lett. 20(7), 647–651 (1998)CrossRefGoogle Scholar
  22. 22.
    Lembayung, W.: Comparative analysis on the izarc compression process and 7-zip (2011)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Xin Wang
    • 1
    • 3
  • Lin Gan
    • 1
    • 3
    • 5
  • Jingheng Xu
    • 1
    • 3
  • Jinzhe Yang
    • 3
    • 4
  • Maocai Xia
    • 1
    • 3
  • Haohuan Fu
    • 2
    • 3
    • 5
  • Xiaomeng Huang
    • 2
    • 3
    • 5
  • Guangwen Yang
    • 1
    • 2
    • 3
    • 5
  1. 1.Department of Computer Science and TechnologyTsinghua UniversityBeijingChina
  2. 2.Ministry of Education Key Lab. for Earth System Modeling, and Department of Earth System ScienceTsinghua UniversityBeijingChina
  3. 3.National Supercomputing CenterWuxiChina
  4. 4.Department of ComputingImperial College LondonLondonUK
  5. 5.Lab. for Regional Oceanography and Numerical ModelingQingdao National Lab. for Marine Science and TechnologyQingdaoChina

Personalised recommendations