Skip to main content
Log in

A new piggybacking design for systematic MDS storage codes

  • Published:
Designs, Codes and Cryptography Aims and scope Submit manuscript

Abstract

Distributed storage codes have important applications in the design of modern storage systems. In a distributed storage system, every storage node has a probability to fail and once an individual storage node fails, it must be reconstructed using the data stored in the surviving nodes. Computation load and network bandwidth are two important issues we need to concern when repairing a failed node. Generally speaking, the naive maximum distance separable (MDS) storage codes have low repair complexity but high repair bandwidth. On the contrary, minimum storage regenerating codes have low repair bandwidth but high repair complexity. Fortunately, the newly introduced piggybacked codes combine the advantages of both ones. The main result of this paper is a novel piggybacking design framework for \((k+r,k)\) systematic MDS storage codes, where kr denote the number of systematic nodes and the number of parity nodes, respectively. In the new code, the average repair bandwidth rate for the systematic nodes, i.e., the ratio of the average repair bandwidth of a single failed systematic node and the amount of the original data, can be as low as \(\sqrt{\frac{2}{r}}+\frac{1}{2r}+\frac{3}{k}+\frac{\sqrt{2r}}{k^2}\), which is roughly \(\sqrt{\frac{2}{r}}+\frac{1}{2r}\) when the code has high rate \(k\gg r\). For relatively large r (e.g., \(r\ge 6\)), this result significantly improves the previously known one which has average repair bandwidth rate roughly \(\frac{r-1}{2r-1}\). In the meanwhile, every failed systematic node of the new code can be reconstructed quickly using the decoding algorithm of a classical MDS code, only with some additional additions over the underlying finite field.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Similar content being viewed by others

References

  1. Bhagwan R., Tati K., Cheng Y., Savage S., Voelker G.M.: Total recall: system support for automated availability management. Nsdi 1, 337–350 (2004).

    Google Scholar 

  2. Colossus: Successor to the Google File System. https://www.systutorials.com/3202/colossus-successor-to-google-file-system-gfs/.

  3. Dimakis A.G., Godfrey P.B., Wu Y., Wainwright M.J.: Network coding for distributed storage systems. IEEE Trans. Inf. Theory 56(9), 4539–4551 (2010).

    Article  Google Scholar 

  4. Gopalan P., Huang C., Simitci H., Yekhanin S.: On the locality of codeword symbols. IEEE Trans. Inf. Theory 58(11), 6925–6934 (2012).

    Article  MathSciNet  Google Scholar 

  5. Goparaju S., Tamo I., Calderbank R.: An improved sub-packetization bound for minimum storage regenerating codes. IEEE Trans. Inf. Theory 60(5), 2770–2779 (2014).

    Article  MathSciNet  Google Scholar 

  6. Goparaju S., Fazeli A., Vardy A.: Minimum storage regenerating codes for all parameters. IEEE Trans. Inf. Theory 63(10), 6318–6328 (2017).

    Article  MathSciNet  Google Scholar 

  7. HDFS-Raid. http://wiki.apache.org/hadoop/hdfs-raid.

  8. Huang C., Simitci H., Xu Y., Ogus A., Calder B., Gopalan P., Li J., Yekhanin S.: Erasure coding in windows azure storage. In: Usenix Conference on Technical Conference, p. 2 (2012).

  9. Kubiatowicz J., Bindel D., Chen Y., Czerwinski S., Eaton P., Geels D., Gummadi R., Rhea S., Weatherspoon H., Wells C.: OceanStore: an architecture for global-scale persistent storage. ACM SIGPLAN Not. 35(11), 190–201 (2002).

    Article  Google Scholar 

  10. Kumar S., i Amat A.G., Andriyanova I., Brännström F.: A family of erasure correcting codes with low repair bandwidth and low repair complexity. In: IEEE Global Communication Conference, pp. 1–6 (2015).

  11. Rashmi K.V., Shah N.B., Gu D., Kuang H., Borthakur D., Ramchandran K.: A solution to the network challenges of data recovery in erasure-coded distributed storage systems: a study on the facebook warehouse cluster. Usenix Hotstorage (2013)

  12. Rashmi K.V., Shah N.B., Gu D., Kuang D., Borthakur D., Ramchandran K.: A Hitchhiker’s guide to fast and efficient data reconstruction in erasure-coded data centers. ACM SIFCOMM Comput. Commun. Rev. 44, 331–342 (2014).

    Article  Google Scholar 

  13. Rashmi K.V., Shah N.B., Ramchandran K.: A piggybacking design framework for read-and download-efficient distributed storage codes. IEEE Trans. Inf. Theory 63(9), 5802–5820 (2017).

    MathSciNet  MATH  Google Scholar 

  14. Rawat A.S., Koyluoglu O.O., Vishwanath S.: Progress on high-rate MSR codes: enabling arbitrary number of helper nodes. In: Information Theory and Applications Workshop (ITA), IEEE, pp. 1–6 (2016).

  15. Sasidharan B., Vajha M., Kumar P.V.: An explicit, coupled-layer construction of a high-rate MSR code with low sub-packetization level, small field size and all-node repair. arXiv preprint arXiv:1607.07335 (2016).

  16. Sasidharan B., Vajha M., Kumar P.V.: An explicit, coupled-layer construction of a high-rate MSR code with low sub-packetization level, small field size and $d<(n- 1)$. In: Proceedings of IEEE International Symposium on Information Theory, pp. 2048–2052 (2017)

  17. Singleton R.C.: Maximum distance $q$-nary codes. IEEE Trans. Inf. Theory IT–10, 116–118 (1964).

    Article  MathSciNet  Google Scholar 

  18. Tamo I., Barg A.: A family of optimal locally recoverable codes. IEEE Trans. Inf. Theory 60(8), 4661–4676 (2013).

    Article  MathSciNet  Google Scholar 

  19. Tamo I., Wang Z., Bruck J.: Zigzag codes: MDS array codes with optimal rebuilding. IEEE Trans. Inf. Theory 59(3), 1597–1616 (2015).

    Article  MathSciNet  Google Scholar 

  20. Wang Z., Tamo I., Bruck J.: Long MDS codes for optimal repair bandwidth. In: Proceedings of IEEE International Symposium on Information Theory, pp. 1182–1186 (2012)

  21. Yang B., Tang X., Li J.: A systematic piggybacking design for minimum storage regenerating codes. IEEE Trans. Inf. Theory 61(11), 5779–5786 (2015).

    Article  MathSciNet  Google Scholar 

  22. Ye M., Barg A.: Explicit constructions of high-rate MDS array codes with optimal repair bandwidth. IEEE Trans. Inf. Theory 63(4), 2001–2014 (2017).

    Article  MathSciNet  Google Scholar 

Download references

Acknowledgements

The authors express their great gratitude to the two anonymous reviewers for their detailed and constructive comments which are very helpful to the improvement of the presentation of this paper.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Gennian Ge.

Additional information

Communicated by D. Panario.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Research supported by the National Natural Science Foundation of China under Grant Nos. 11431003 and 61571310, Beijing Scholars Program, Beijing Hundreds of Leading Talents Training Project of Science and Technology, and Beijing Municipal Natural Science Foundation.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Shangguan, C., Ge, G. A new piggybacking design for systematic MDS storage codes. Des. Codes Cryptogr. 87, 2753–2770 (2019). https://doi.org/10.1007/s10623-019-00650-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10623-019-00650-9

Keywords

Mathematics Subject Classification

Navigation