Skip to main content
SpringerLink
Log in

EC-Bench: Benchmarking Onload and Offload Erasure Coders on Modern Hardware Architectures

  • Conference paper
  • First Online:
Book cover Benchmarking, Measuring, and Optimizing (Bench 2018)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 11459))

Included in the following conference series:

  • 1213 Accesses

Abstract

Various Erasure Coding (EC) schemes based on hardware accelerations have been proposed in the community to leverage the advanced compute capabilities on modern data centers, such as Intel ISA-L Onload EC coders and Mellanox InfiniBand Offload EC coders. These EC coders can play a vital role in designing next-generation distributed storage systems. Unfortunately, there does not exist a unified and easy way for distributed storage systems researchers and designers to benchmark, measure, and characterize the performance of these different EC coders. In this context, we propose a unified benchmark suite, called EC-Bench, to help the users to benchmark both onload and offload EC coders on modern hardware architectures. EC-Bench provides both encoding and decoding benchmarks with tunable parameter support. A rich set of metrics, including latency, actual and normalized throughput, CPU utilization, and cache pressure, can be reported through EC-Bench. Evaluations with EC-Bench demonstrate that hardware-optimized offload coders (e.g. Mellanox-EC) have lower demands on CPU and cache compared to onload coders, and highly optimized onload coders (e.g., Intel ISA-L) outperform offload coders for most configurations.

This research is supported in part by National Science Foundation grants CCF#1822987, CNS#1513120, IIS#1636846, and OAC#1664137.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    Github: https://github.com/jaredjennings/libgibraltar,

    commit: c93f9d8c3be70ded173822cdca2e51900a3f5ed1.

  2. 2.

    Github: https://github.com/Mellanox/EC,

    commit: 00bf091aa14322baf4425f8a6d5d134e91fe2a5c.

References

  1. Facebook’s Erasure Coded Hadoop Distributed File System (HDFS-RAID) (2010). https://github.com/facebookarchive/hadoop-20

  2. ec\(\_\)highlevel\(\_\)func.c (2018). https://github.com/intel/isa-l/blob/master/erasure_code/ec_highlevel_func.c#L98

  3. Apache Hadoop 3.0.0-alpha2 (2017). http://hadoop.apache.org/docs/r3.0.0-alpha2/

  4. Backblaze Online Backup (2015). https://www.backblaze.com/blog/reed-solomon/

  5. Ceph Erasure Coding (2016). http://docs.ceph.com/docs/master/rados/operations/erasure-code/

  6. Colossus: Successor to the Google File System (GFS) (2012). https://www.systutorials.com/3202/colossus-successor-to-google-file-system-gfs/

  7. Corbett, P., et al.: Row-diagonal parity for double disk failure correction. In: Proceedings of the 3rd USENIX Conference on File and Storage Technologies, pp. 1–14. USENIX Association, Berkeley (2004)

    Google Scholar 

  8. Curry, M., Skjellum, A., Lee Ward, H., Brightwell, R.: Gibraltar: a Reed-Solomon coding library for storage applications on programmable graphics processors. Concurr. Comput.: Pract. Exp. 23(18), 2477–2495 (2011)

    Article  Google Scholar 

  9. Dimakis, A.G., Godfrey, P.B., Wu, Y., Wainwright, M.J., Ramchandran, K.: Network coding for distributed storage systems. IEEE Trans. Inf. Theory 56(9), 4539–4551 (2010)

    Article  Google Scholar 

  10. Fujii, Y., Azumi, T., Nishio, N., Kato, S., Edahiro, M.: Data transfer matters for GPU computing. In: 2013 International Conference on Parallel and Distributed Systems (ICPADS), pp. 275–282. IEEE (2013)

    Google Scholar 

  11. Ghemawat, S., Gobioff, H., Leung, S.T.: The Google file system. ACM SIGOPS Oper. Syst. Rev. 37, 29–43 (2003)

    Article  Google Scholar 

  12. Greenan, K.M., Li, X., Wylie, J.J.: Flat XOR-based erasure codes in storage systems: constructions, efficient recovery, and tradeoffs. In: 2010 IEEE 26th Symposium on Mass Storage Systems and Technologies (MSST), pp. 1–14. IEEE (2010)

    Google Scholar 

  13. Hafner, J.L.: WEAVER codes: highly fault tolerant erasure codes for storage systems. In: Proceedings of the 4th Conference on USENIX Conference on File and Storage Technologies - FAST 2005, vol. 4, p. 16. USENIX Association, Berkeley (2005)

    Google Scholar 

  14. Huang, C., Xu, L.: STAR: an efficient coding scheme for correcting triple storage node failures. IEEE Trans. Comput. 57(7), 889–901 (2008)

    Article  MathSciNet  Google Scholar 

  15. Huang, C., et al.: Erasure coding in windows azure storage. In: USENIX Annual Technical Conference, Boston, pp. 15–26 (2012)

    Google Scholar 

  16. Intel Intelligent Storage Acceleration Library (Intel ISA-L) (2016). https://software.intel.com/en-us/storage/ISA-L

  17. Introduction to Intel\(^{\textregistered }\) Advanced Vector Extensions. https://software.intel.com/en-us/articles/introduction-to-intel-advanced-vector-extensions

  18. Lai, C., et al.: Atlas: Baidu’s key-value storage system for cloud data. In: 2015 31st Symposium on Mass Storage Systems and Technologies (MSST), pp. 1–14. IEEE (2015)

    Google Scholar 

  19. Li, M., Lee, P.P.: STAIR codes: a general family of erasure codes for tolerating device and sector failures. Trans. Storage 10(4), 14:1–14:30 (2014)

    Article  Google Scholar 

  20. Li, R., Li, X., Lee, P.P., Huang, Q.: Repair pipelining for erasure-coded storage. In: Proceedings of the 2017 USENIX Annual Technical Conference (USENIX ATC 2017), pp. 567–579 (2017)

    Google Scholar 

  21. Luby, M.: Benchmark comparisons of erasure codes (2002)

    Google Scholar 

  22. Marov, A., Fedorov, A.: Optimization of RAID erasure coding algorithms for Intel Xeon Phi. In: 2016 IEEE International Conference on Networking, Architecture and Storage (NAS), pp. 1–4. IEEE (2016)

    Google Scholar 

  23. Mellanox. HDFS Erasure Coding Offload Plugin (2016). https://github.com/Mellanox/EC/tree/master/HDFS

  24. Mellanox. Understanding Erasure Coding Offload (2016). https://community.mellanox.com/docs/DOC-2414

  25. Mitra, S., Panta, R., Ra, M.R., Bagchi, S.: Partial-parallel-repair (PPR): a distributed technique for repairing erasure coded storage. In: Proceedings of the Eleventh European Conference on Computer Systems, p. 30. ACM (2016)

    Google Scholar 

  26. Muralidhar, S., et al.: f4: Facebook’s warm BLOB storage system. In: Proceedings of the 11th USENIX Conference on Operating Systems Design and Implementation, pp. 383–398. USENIX Association (2014)

    Google Scholar 

  27. Ovsiannikov, M., Rus, S., Reeves, D., Sutter, P., Rao, S., Kelly, J.: The quantcast file system. Proc. VLDB Endow. 11, 1092–1101 (2013)

    Article  Google Scholar 

  28. Plank, J.S., Greenan, K.M., Miller, E.L.: Screaming fast galois field arithmetic using Intel SIMD instructions. In: 11th USENIX Conference on File and Storage Technologies (FAST 2013), San Jose, pp. 298–306. USENIX Association (2013)

    Google Scholar 

  29. Plank, J.S., et al.: A performance evaluation and examination of open-source erasure coding libraries for storage. In: Proccedings of the 7th Conference on File and Storage Technologies, FAST 2009, pp. 253–265. USENIX Association, Berkeley (2009)

    Google Scholar 

  30. Plank, J.S., Simmerman, S., Schuman, C.D.: Jerasure: a library in C/C++ facilitating erasure coding for storage applications (2008)

    Google Scholar 

  31. Rashmi, K.V., Chowdhury, M., Kosaian, J., Stoica, I., Ramchandran, K.: EC-Cache: load-balanced, low-latency cluster caching with online erasure coding. In: 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 2016). USENIX Association (2016)

    Google Scholar 

  32. Rashmi, K.V., Nakkiran, P., Wang, J., Shah, N.B., Ramchandran, K.: Having your cake and eating it too: jointly optimal erasure codes for i/o, storage, and network-bandwidth. In: FAST, pp. 81–94 (2015)

    Google Scholar 

  33. Rashmi, K.V., Shah, N.B., Gu, D., Kuang, H., Borthakur, D., Ramchandran, K.: A solution to the network challenges of data recovery in erasure-coded distributed storage systems: a study on the Facebook warehouse cluster. In: HotStorage (2013)

    Google Scholar 

  34. Reed, I.S., Solomon, G.: Polynomial codes over certain finite fields. J. Soc. Ind. Appl. Math. 8(2), 300–304 (1960)

    Article  MathSciNet  Google Scholar 

  35. Rodrigues, R., Liskov, B.: High availability in DHTs: erasure coding vs. replication. In: Castro, M., van Renesse, R. (eds.) IPTPS 2005. LNCS, vol. 3640, pp. 226–239. Springer, Heidelberg (2005). https://doi.org/10.1007/11558989_21

    Chapter  Google Scholar 

  36. Sathiamoorthy, M., et al.: XORing elephants: novel erasure codes for big data. Proc. VLDB Endow. 6(5), 325–336 (2013)

    Article  Google Scholar 

  37. Shankar, D., Lu, X., Panda, D.K.: High-performance and resilient key-value store with online erasure coding for big data workloads. In: 2017 IEEE 37th International Conference on Distributed Computing Systems (ICDCS), pp. 527–537. IEEE (2017)

    Google Scholar 

  38. Shi, H., Lu, X., Shankar, D., Panda, D.K.: High-performance multi-rail erasure coding library over modern data center architectures: early experiences. In: Proceedings of the ACM Symposium on Cloud Computing, pp. 530–531. ACM (2018)

    Google Scholar 

  39. Shvachko, K., Kuang, H., Radia, S., Chansler, R.: The hadoop distributed file system. In: 2010 IEEE 26th Symposium on Mass Storage Systems and Technologies (MSST), pp. 1–10. IEEE (2010)

    Google Scholar 

  40. Terpstra, D., Jagode, H., You, H., Dongarra, J.: Collecting performance data with PAPI-C. In: Müller, M., Schulz, A., Nagel, W. (eds.) Tools for High Performance Computing 2009, pp. 157–173. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-11261-4_11

    Chapter  Google Scholar 

  41. Using Intel\(^{\textregistered }\) Streaming SIMD Extensions and Intel\(^{\textregistered }\) Integrated Performance Primitives to Accelerate Algorithms (2016). https://software.intel.com/en-us/articles/

  42. Weatherspoon, H., Kubiatowicz, J.D.: Erasure coding vs. replication: a quantitative comparison. In: Druschel, P., Kaashoek, F., Rowstron, A. (eds.) IPTPS 2002. LNCS, vol. 2429, pp. 328–337. Springer, Heidelberg (2002). https://doi.org/10.1007/3-540-45748-8_31

    Chapter  MATH  Google Scholar 

  43. Weil, S.A., Brandt, S.A., Miller, E.L., Long, D.D., Maltzahn, C.: Ceph: a scalable, high-performance distributed file system. In: Proceedings of the 7th Symposium on Operating Systems Design and Implementation, pp. 307–320. USENIX Association (2006)

    Google Scholar 

  44. Xu, L., Bruck, J.: X-code: MDS array codes with optimal encoding. IEEE Trans. Inf. Theory 45(1), 272–276 (1999)

    Article  MathSciNet  Google Scholar 

  45. Zhang, H., Dong, M., Chen, H.: Efficient and available in-memory KV-store with hybrid erasure coding and replication. In: 14th USENIX Conference on File and Storage Technologies (FAST 2016), Santa Clara, pp. 167–180. USENIX Association (2016)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, The Ohio State University, Columbus, USA

    Haiyang Shi, Xiaoyi Lu & Dhabaleswar K. Panda

Authors
  1. Haiyang Shi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xiaoyi Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Dhabaleswar K. Panda
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Haiyang Shi .

Editor information

Editors and Affiliations

  1. Chinese Academy of Sciences, Beijing, China

    Chen Zheng

  2. Chinese Academy of Sciences, Beijing, China

    Jianfeng Zhan

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Shi, H., Lu, X., Panda, D.K. (2019). EC-Bench: Benchmarking Onload and Offload Erasure Coders on Modern Hardware Architectures. In: Zheng, C., Zhan, J. (eds) Benchmarking, Measuring, and Optimizing. Bench 2018. Lecture Notes in Computer Science(), vol 11459. Springer, Cham. https://doi.org/10.1007/978-3-030-32813-9_18

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-32813-9_18

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-32812-2

  • Online ISBN: 978-3-030-32813-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation