Abstract
Distributed storage codes have important applications in the design of modern storage systems. In a distributed storage system, every storage node has a probability to fail and once an individual storage node fails, it must be reconstructed using the data stored in the surviving nodes. Computation load and network bandwidth are two important issues we need to concern when repairing a failed node. Generally speaking, the naive maximum distance separable (MDS) storage codes have low repair complexity but high repair bandwidth. On the contrary, minimum storage regenerating codes have low repair bandwidth but high repair complexity. Fortunately, the newly introduced piggybacked codes combine the advantages of both ones. The main result of this paper is a novel piggybacking design framework for \((k+r,k)\) systematic MDS storage codes, where k, r denote the number of systematic nodes and the number of parity nodes, respectively. In the new code, the average repair bandwidth rate for the systematic nodes, i.e., the ratio of the average repair bandwidth of a single failed systematic node and the amount of the original data, can be as low as \(\sqrt{\frac{2}{r}}+\frac{1}{2r}+\frac{3}{k}+\frac{\sqrt{2r}}{k^2}\), which is roughly \(\sqrt{\frac{2}{r}}+\frac{1}{2r}\) when the code has high rate \(k\gg r\). For relatively large r (e.g., \(r\ge 6\)), this result significantly improves the previously known one which has average repair bandwidth rate roughly \(\frac{r-1}{2r-1}\). In the meanwhile, every failed systematic node of the new code can be reconstructed quickly using the decoding algorithm of a classical MDS code, only with some additional additions over the underlying finite field.
Similar content being viewed by others
References
Bhagwan R., Tati K., Cheng Y., Savage S., Voelker G.M.: Total recall: system support for automated availability management. Nsdi 1, 337–350 (2004).
Colossus: Successor to the Google File System. https://www.systutorials.com/3202/colossus-successor-to-google-file-system-gfs/.
Dimakis A.G., Godfrey P.B., Wu Y., Wainwright M.J.: Network coding for distributed storage systems. IEEE Trans. Inf. Theory 56(9), 4539–4551 (2010).
Gopalan P., Huang C., Simitci H., Yekhanin S.: On the locality of codeword symbols. IEEE Trans. Inf. Theory 58(11), 6925–6934 (2012).
Goparaju S., Tamo I., Calderbank R.: An improved sub-packetization bound for minimum storage regenerating codes. IEEE Trans. Inf. Theory 60(5), 2770–2779 (2014).
Goparaju S., Fazeli A., Vardy A.: Minimum storage regenerating codes for all parameters. IEEE Trans. Inf. Theory 63(10), 6318–6328 (2017).
HDFS-Raid. http://wiki.apache.org/hadoop/hdfs-raid.
Huang C., Simitci H., Xu Y., Ogus A., Calder B., Gopalan P., Li J., Yekhanin S.: Erasure coding in windows azure storage. In: Usenix Conference on Technical Conference, p. 2 (2012).
Kubiatowicz J., Bindel D., Chen Y., Czerwinski S., Eaton P., Geels D., Gummadi R., Rhea S., Weatherspoon H., Wells C.: OceanStore: an architecture for global-scale persistent storage. ACM SIGPLAN Not. 35(11), 190–201 (2002).
Kumar S., i Amat A.G., Andriyanova I., Brännström F.: A family of erasure correcting codes with low repair bandwidth and low repair complexity. In: IEEE Global Communication Conference, pp. 1–6 (2015).
Rashmi K.V., Shah N.B., Gu D., Kuang H., Borthakur D., Ramchandran K.: A solution to the network challenges of data recovery in erasure-coded distributed storage systems: a study on the facebook warehouse cluster. Usenix Hotstorage (2013)
Rashmi K.V., Shah N.B., Gu D., Kuang D., Borthakur D., Ramchandran K.: A Hitchhiker’s guide to fast and efficient data reconstruction in erasure-coded data centers. ACM SIFCOMM Comput. Commun. Rev. 44, 331–342 (2014).
Rashmi K.V., Shah N.B., Ramchandran K.: A piggybacking design framework for read-and download-efficient distributed storage codes. IEEE Trans. Inf. Theory 63(9), 5802–5820 (2017).
Rawat A.S., Koyluoglu O.O., Vishwanath S.: Progress on high-rate MSR codes: enabling arbitrary number of helper nodes. In: Information Theory and Applications Workshop (ITA), IEEE, pp. 1–6 (2016).
Sasidharan B., Vajha M., Kumar P.V.: An explicit, coupled-layer construction of a high-rate MSR code with low sub-packetization level, small field size and all-node repair. arXiv preprint arXiv:1607.07335 (2016).
Sasidharan B., Vajha M., Kumar P.V.: An explicit, coupled-layer construction of a high-rate MSR code with low sub-packetization level, small field size and $d<(n- 1)$. In: Proceedings of IEEE International Symposium on Information Theory, pp. 2048–2052 (2017)
Singleton R.C.: Maximum distance $q$-nary codes. IEEE Trans. Inf. Theory IT–10, 116–118 (1964).
Tamo I., Barg A.: A family of optimal locally recoverable codes. IEEE Trans. Inf. Theory 60(8), 4661–4676 (2013).
Tamo I., Wang Z., Bruck J.: Zigzag codes: MDS array codes with optimal rebuilding. IEEE Trans. Inf. Theory 59(3), 1597–1616 (2015).
Wang Z., Tamo I., Bruck J.: Long MDS codes for optimal repair bandwidth. In: Proceedings of IEEE International Symposium on Information Theory, pp. 1182–1186 (2012)
Yang B., Tang X., Li J.: A systematic piggybacking design for minimum storage regenerating codes. IEEE Trans. Inf. Theory 61(11), 5779–5786 (2015).
Ye M., Barg A.: Explicit constructions of high-rate MDS array codes with optimal repair bandwidth. IEEE Trans. Inf. Theory 63(4), 2001–2014 (2017).
Acknowledgements
The authors express their great gratitude to the two anonymous reviewers for their detailed and constructive comments which are very helpful to the improvement of the presentation of this paper.
Author information
Authors and Affiliations
Corresponding author
Additional information
Communicated by D. Panario.
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Research supported by the National Natural Science Foundation of China under Grant Nos. 11431003 and 61571310, Beijing Scholars Program, Beijing Hundreds of Leading Talents Training Project of Science and Technology, and Beijing Municipal Natural Science Foundation.
Rights and permissions
About this article
Cite this article
Shangguan, C., Ge, G. A new piggybacking design for systematic MDS storage codes. Des. Codes Cryptogr. 87, 2753–2770 (2019). https://doi.org/10.1007/s10623-019-00650-9
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10623-019-00650-9