Advertisement

Springer Nature is making SARS-CoV-2 and COVID-19 research free. View research | View latest news | Sign up for updates

Erasure coding for distributed storage: an overview

Abstract

In a distributed storage system, code symbols are dispersed across space in nodes or storage units as opposed to time. In settings such as that of a large data center, an important consideration is the efficient repair of a failed node. Efficient repair calls for erasure codes that in the face of node failure, are efficient in terms of minimizing the amount of repair data transferred over the network, the amount of data accessed at a helper node as well as the number of helper nodes contacted. Coding theory has evolved to handle these challenges by introducing two new classes of erasure codes, namely regenerating codes and locally recoverable codes as well as by coming up with novel ways to repair the ubiquitous Reed-Solomon code. This survey provides an overview of the efforts in this direction that have taken place over the past decade.

This is a preview of subscription content, log in to check access.

References

  1. 1

    Rashmi V, Shah N B, Gu D, et al. A solution to the network challenges of data recovery in erasure-coded distributed storage systems: a study on the facebook warehouse cluster. In: Proceedings of the 5th USENIX Workshop on Hot Topics in Storage and File Systems, San Jose, 2013

  2. 2

    Sathiamoorthy M, Asteris M, Papailiopoulos D, et al. XORing elephants. Proc VLDB Endow, 2013, 6: 325–336

  3. 3

    Dimakis G, Ramchandran K, Wu Y N, et al. A survey on network codes for distributed storage. Proc IEEE, 2011, 99: 476–489

  4. 4

    Datta A, Oggier F. An overview of codes tailor-made for better repairability in networked distributed storage systems. SIGACT News, 2013, 44: 89

  5. 5

    Li J, Li B. Erasure coding for cloud storage systems: a survey. Tinshhua Sci Technol, 2013, 18: 259–272

  6. 6

    Liu Q, Oggier F. An overview of coding for distributed storage systems. In: Network Coding and Subspace Designs. Berlin: Springer, 2018. 363–383

  7. 7

    Dimakis G, Godfrey P B, Wu Y, et al. Network coding for distributed storage systems. IEEE Trans Inf Theory, 2010, 56: 4539–4551

  8. 8

    Wu Y. Existence and construction of capacity-achieving network codes for distributed storage. IEEE J Sel Areas Commun, 2010, 28: 277–288

  9. 9

    Shah B, Rashmi K V, Kumar P V, et al. Interference alignment in regenerating codes for distributed storage: necessity and code constructions. IEEE Trans Inf Theory, 2012, 58: 2134–2158

  10. 10

    Hu C, Chen H C H, Lee P P C, et al. NCCloud: applying network coding for the storage repair in a cloud-of-clouds. In: Proceedings of the 10th USENIX Conference on File and Storage Technologies, San Jose, 2012

  11. 11

    Krishnan N, Kumar P V. On MBR codes with replication. In: Proceedings of IEEE International Symposium on Information Theory, Barcelona, 2016. 71–75

  12. 12

    Shah B. On minimizing data-read and download for storage-node recovery. IEEE Commun Lett, 2013, 17: 964–967

  13. 13

    Rashmi V, Shah N B, Kumar P V, et al. Explicit construction of optimal exact regenerating codes for distributed storage. In: Proceedings of the 47th Annual Allerton Conference on Communication, Control, and Computing, Monticello, 2009. 1243–1249

  14. 14

    Rashmi V, Shah N B, Kumar P V. Optimal exact-regenerating codes for distributed storage at the MSR and MBR points via a product-matrix construction. IEEE Trans Inf Theory, 2011, 57: 5227–5239

  15. 15

    Lin J, Chung W H. Novel repair-by-transfer codes and systematic exact-mbr codes with lower complexities and smaller field sizes. IEEE Trans Parallel Distrib Syst, 2014, 25: 3232–3241

  16. 16

    Han S, Pai H T, Zheng R, et al. Update-efficient error-correcting product-matrix codes. IEEE Trans Commun, 2015, 63: 1925–1938

  17. 17

    Raviv N. Asymptotically optimal regenerating codes over any field. In: Proceedings of IEEE International Symposium on Information Theory, Aachen, 2017. 1416–1420

  18. 18

    Mahdaviani K, Khisti A, Mohajer S. Bandwidth adaptive & error resilient MBR exact repair regenerating codes. 2017. ArXiv:1711.02770

  19. 19

    Wu Y, Dimakis G. Reducing repair traffic for erasure coding-based storage via interference alignment. In: Proceedings of IEEE International Symposium on Information Theory, Seoul, 2009. 2276–2280

  20. 20

    Suh C, Ramchandran K. Exact-repair mds code construction using interference alignment. IEEE Trans Inf Theory, 2011, 57: 1425–1442

  21. 21

    Lin J, Chung W H, Han Y S, et al. A unified form of exact-MSR codes via product-matrix frameworks. IEEE Trans Inf Theory, 2015, 61: 873–886

  22. 22

    Papailiopoulos S, Dimakis A G, Cadambe V R. Repair optimal erasure codes through Hadamard designs. IEEE Trans Inf Theory, 2013, 59: 3021–3037

  23. 23

    Cadambe R, Jafar S A, Maleki H, et al. Asymptotic interference alignment for optimal repair of MDS codes in distributed storage. IEEE Trans Inf Theory, 2013, 59: 2974–2987

  24. 24

    Tamo I, Wang Z, Bruck J. Zigzag codes: MDS array codes with optimal rebuilding. IEEE Trans Inf Theory, 2013, 59: 1597–1616

  25. 25

    Wang Y, Tamo I, Bruck J. On codes for optimal rebuilding access. In: Proceedings of the 49th Annual Allerton Conference on Communication, Control, and Computing, Monticello, 2011. 1374–1381

  26. 26

    Cadambe R, Huang C, Li J, et al. Polynomial length MDS codes with optimal repair in distributed storage. In: Proceedings of the 45th Asilomar Conference on Signals, Systems and Computers, Pacific Grove, 2011. 1850–1854

  27. 27

    Wang Y, Tamo I, Bruck J. Long MDS codes for optimal repair bandwidth. In: Proceedings of IEEE International Symposium on Information Theory, Cambridge, 2012. 1182–1186

  28. 28

    Tamo I, Wang Y, Bruck J. Access versus bandwidth in codes for storage. IEEE Trans Inf Theory, 2014, 60: 2028–2037

  29. 29

    Goparaju S, Tamo I, Calderbank R. An improved sub-packetization bound for minimum storage regenerating codes. IEEE Trans Inf Theory, 2014, 60: 2770–2779

  30. 30

    Sasidharan B, Agarwal K, Kumar P V. A high-rate MSR code with polynomial sub-packetization level. In: Proceedings of IEEE International Symposium on Information Theory, Hong Kong, 2015. 2051–2055

  31. 31

    Rawat S, Koyluoglu O O, Vishwanath S. Progress on high-rate MSR codes: enabling arbitrary number of helper nodes. In: Proceedings of Information Theory and Applications Workshop, La Jolla, 2016

  32. 32

    Goparaju S, Fazeli A, Vardy A. Minimum storage regenerating codes for all parameters. IEEE Trans Inf Theory, 2017, 63: 6318–6328

  33. 33

    Agarwal K, Sasidharan B, Kumar P V. An alternate construction of an access-optimal regenerating code with optimal sub-packetization level. In: Proceedings of the 21st National Conference on Communications, Mumbai, 2015

  34. 34

    Alon N. Combinatorial nullstellensatz. Combinator Probab Comp, 1999, 8: 7–29

  35. 35

    Raviv N, Silberstein N, Etzion T. Constructions of high-rate minimum storage regenerating codes over small fields. IEEE Trans Inf Theory, 2017, 63: 2015–2038

  36. 36

    Ye M, Barg A. Explicit constructions of high-rate MDS array codes with optimal repair bandwidth. IEEE Trans Inf Theory, 2017, 63: 2001–2014

  37. 37

    Ye M, Barg A. Explicit constructions of optimal-access MDS codes with nearly optimal sub-packetization. IEEE Trans Inf Theory, 2017, 63: 6307–6317

  38. 38

    Sasidharan B, Vajha M, Kumar P V. An explicit, coupled-layer construction of a high-rate MSR code with low sub-packetization level, small field size and all-node repair. 2016. ArXiv:1607.07335

  39. 39

    Li J, Tang H, Tian C. A generic transformation for optimal repair bandwidth and rebuilding access in MDS codes. In: Proceedings of IEEE International Symposium on Information Theory, Aachen, 2017. 1623–1627

  40. 40

    Balaji S B, Kumar P V. A tight lower bound on the sub-packetization level of optimal-access MSR and MDS codes. 2018. ArXiv:1710.05876

  41. 41

    Vajha M, Balaji S B, Kumar P V. Explicit MSR codes with optimal access, optimal sub-packetization and small field size for d = k + 1, k + 2, k + 3. 2018. ArXiv:1804.00598

  42. 42

    Mahdaviani K, Mohajer S, Khisti A. Product matrix MSR codes with bandwidth adaptive exact repair. IEEE Trans Inf Theory, 2018, 64: 3121–3135

  43. 43

    Shah B, Rashmi K V, Kumar P V, et al. Distributed storage codes with repair-by-transfer and nonachievability of interior points on the storage-bandwidth tradeoff. IEEE Trans Inf Theory, 2012, 58: 1837–1852

  44. 44

    Tian C. Characterizing the rate region of the (4,3,3) exact-repair regenerating codes. IEEE J Sel Areas Commun, 2014, 32: 967–975

  45. 45

    Information theory inequality prover. 2016. https://doi.org/user-www.ie.cuhk.edu.hk/~ITIP/

  46. 46

    Yeung W. A framework for linear information inequalities. IEEE Trans Inf Theory, 1997, 43: 1924–1934

  47. 47

    Tian C, Sasidharan B, Aggarwal V, et al. Layered exact-repair regenerating codes via embedded error correction and block designs. IEEE Trans Inf Theory, 2015, 61: 1933–1947

  48. 48

    Senthoor K, Sasidharan B, Kumar V. Improved layered regenerating codes characterizing the exact-repair storagerepair bandwidth tradeoff for certain parameter sets. In: Proceedings of IEEE Information Theory Workshop, Jerusalem, 2015

  49. 49

    Sasidharan B, Senthoor K, Kumar V. An improved outer bound on the storage repair-bandwidth tradeoff of exactrepair regenerating codes. In: Proceedings of IEEE International Symposium on Information Theory, Honolulu, 2014. 2430–2434

  50. 50

    Duursma I M. Outer bounds for exact repair codes. 2014. ArXiv:1406.4852

  51. 51

    Duursma M. Shortened regenerating codes. IEEE Trans Inf Theory (Early Access), 2018. doi: 10.1109/TIT.2018. 2840995

  52. 52

    Mohajer S, Tandon R. New bounds on the (n, k, d) storage systems with exact repair. In: Proceedings of IEEE International Symposium on Information Theory, Hong Kong, 2015. 2056–2060

  53. 53

    Sasidharan B, Prakash N, KrishnanM N, et al. Outer bounds on the storage-repair bandwidth trade-off of exact-repair regenerating codes. Int J Inf Coding Theory, 2016, 3: 255–298

  54. 54

    Elyasi M, Mohajer S. Determinant coding: a novel framework for exact-repair regenerating codes. IEEE Trans Inf Theory, 2016, 62: 6683–6697

  55. 55

    Elyasi M, Mohajer S. Exact-repair trade-off for (n, k = d - 1, d) regenerating codes. In: Proceedings of the 55th Annual Allerton Conference on Communication, Control, and Computing, Monticello, 2017. 934–941

  56. 56

    Prakash N, Krishnan N. The storage-repair-bandwidth trade-off of exact repair linear regenerating codes for the case d = k = n - 1. In: Proceedings of IEEE International Symposium on Information Theory, Hong Kong, 2015. 859–863

  57. 57

    Elyasi M, Mohajer S, Tandon R. Linear exact repair rate region of (k + 1, k, k) distributed storage systems: a new approach. In: Proceedings of IEEE International Symposium on Information Theory (ISIT), Hong Kong, 2015. 2061–2065

  58. 58

    Hu Y, Xu Y, Wang X, et al. Cooperative recovery of distributed storage systems from multiple losses with network coding. IEEE J Sel Areas Commun, 2010, 28: 268–276

  59. 59

    Kermarrec M, Scouarnec N L, Straub G. Repairing multiple failures with coordinated and adaptive regenerating codes. In: Proceedings of International Symposium on Networking Coding, Beijing, 2011

  60. 60

    Shum W, Hu Y. Cooperative regenerating codes. IEEE Trans Inf Theory, 2013, 59: 7229–7258

  61. 61

    Wang Y, Zhang Z F. Exact cooperative regenerating codes with minimum-repair-bandwidth for distributed storage. In: Proceedings of IEEE INFOCOM, Turin, 2013. 400–404

  62. 62

    Shum W, Chen J. Cooperative repair of multiple node failures in distributed storage systems. Int J Inf Coding Theory, 2016, 3: 299–323

  63. 63

    Scouarnec L. Exact scalar minimum storage coordinated regenerating codes. In: Proceedings of IEEE International Symposium on Information Theory, Cambridge, 2012. 1197–1201

  64. 64

    Ye M, Barg A. Optimal MDS codes for cooperative repair. 2018. ArXiv:1801.09665

  65. 65

    Liu Q, Oggier F E. On storage codes allowing partially collaborative repairs. In: Proceedings of IEEE International Symposium on Information Theory Proceedings, Honolulu, 2014. 2440–2444

  66. 66

    Liu Q, Oggier F E. Two storage code constructions allowing partially collaborative repairs. In: Proceedings of International Symposium on Information Theory and its Applications, Melbourne, 2014. 378–382

  67. 67

    Koyluoglu O, Rawat A S, Vishwanath S. Secure cooperative regenerating codes for distributed storage systems. IEEE Trans Inf Theory, 2014, 60: 5228–5244

  68. 68

    Huang K, Parampalli U, Xian M. Security concerns in minimum storage cooperative regenerating codes. IEEE Trans Inf Theory, 2016, 62: 6218–6232

  69. 69

    Rashmi V, Shah N B, Ramchandran K. A piggybacking design framework for read-and download-efficient distributed storage codes. IEEE Trans Inf Theory, 2017, 63: 5802–5820

  70. 70

    Guruswami V, Rawat S. MDS code constructions with small sub-packetization and near-optimal repair bandwidth. In: Proceedings of the 28th Annual ACM-SIAM Symposium on Discrete Algorithms, Barcelona, 2017. 2109–2122

  71. 71

    Kralevska K, Gligoroski D, Øverby H. General sub-packetized access-optimal regenerating codes. IEEE Commun Lett, 2016, 20: 1281–1284

  72. 72

    Rawat S, Tamo I, Guruswami V, et al. o-MSR codes with small sub-packetization. In: Proceedings of IEEE International Symposium on Information Theory, Aachen, 2017. 2043–2047

  73. 73

    Rouayheb S Y E, Ramchandran K. Fractional repetition codes for repair in distributed storage systems. In: Proceedings of the 48th Annual Allerton Conference on Communication, Control, and Computing, Allerton, 2010

  74. 74

    Pawar S, Noorshams N, Rouayheb S Y E, et al. DRESS codes for the storage cloud: simple randomized constructions. In: Proceedings of IEEE International Symposium on Information Theory Proceedings, St. Petersburg, 2011. 2338–2342

  75. 75

    Silberstein N, Etzion T. Optimal fractional repetition codes based on graphs and designs. IEEE Trans Inf Theory, 2015, 61: 4164–4180

  76. 76

    Olmez O, Ramamoorthy A. Fractional repetition codes with flexible repair from combinatorial designs. IEEE Trans Inf Theory, 2016, 62: 1565–1591

  77. 77

    Koo C, Gill J T. Scalable constructions of fractional repetition codes in distributed storage systems. In: Proceedings of the 49th Annual Allerton Conference on Communication, Control, and Computing, Monticello, 2011. 1366–1373

  78. 78

    Ernvall T. The existence of fractional repetition codes. 2012. ArXiv:1201.3547

  79. 79

    Pawar S, Rouayheb S Y E, Ramchandran K. Securing dynamic distributed storage systems against eavesdropping and adversarial attacks. IEEE Trans Inf Theory, 2011, 57: 6734–6753

  80. 80

    Rashmi V, Shah N B, Ramchandran K, et al. Regenerating codes for errors and erasures in distributed storage. In: Proceeding of IEEE International Symposium on Information Theory Proceedings, Cambridge, 2012. 1202–1206

  81. 81

    Rashmi V, Shah N B, Ramchandran K, et al. Information-theoretically secure erasure codes for distributed storage. IEEE Trans Inf Theory, 2018, 64: 1621–1646

  82. 82

    Tandon R, Amuru D, Clancy T C, et al. Toward optimal secure distributed storage systems with exact repair. IEEE Trans Inf Theory, 2016, 62: 3477–3492

  83. 83

    Rawat S, Koyluoglu O O, Silberstein N, et al. Optimal locally repairable and secure codes for distributed storage systems. IEEE Trans Inf Theory, 2014, 60: 212–236

  84. 84

    Goparaju S, Rouayheb S Y E, Calderbank R, et al. Data secrecy in distributed storage systems under exact repair. In: Proceedings of International Symposium on Network Coding, Calgary, 2013

  85. 85

    Huang K, Parampalli U, Xian M. On secrecy capacity of minimum storage regenerating codes. IEEE Trans Inf Theory, 2017, 63: 1510–1524

  86. 86

    Rawat S. Secrecy capacity of minimum storage regenerating codes. In: Proceedings of IEEE International Symposium on Information Theory, Aachen, 2017. 1406–1410

  87. 87

    Kadhe S, Sprintson A. Security for minimum storage regenerating codes and locally repairable codes. In: Proceedings of IEEE International Symposium on Information Theory, Aachen, 2017. 1028–1032

  88. 88

    Ye F, Shum W, Yeung R W. The rate region for secure distributed storage systems. IEEE Trans Inf Theory, 2017, 63: 7038–7051

  89. 89

    Shao S, Liu T, Tian C, et al. On the tradeoff region of secure exact-repair regenerating codes. IEEE Trans Inf Theory, 2017, 63: 7253–7266

  90. 90

    Han J, Lastras-Montano A. Reliable memories with subline accesses. In: Proceedings of IEEE International Symposium on Information Theory, Nice, 2007. 2531–2535

  91. 91

    Huang C, Chen H, Li J. Pyramid codes: flexible schemes to trade space for access efficiency in reliable data storage systems. In: Proceedings of the 6th IEEE International Symposium on Network Computing and Applications, Cambridge, 2007. 79–86

  92. 92

    Oggier F, Datta A. Self-repairing homomorphic codes for distributed storage systems. In: Proceedings of IEEE INFOCOM, Shanghai, 2011. 1215–1223

  93. 93

    Gopalan P, Huang C, Simitci H, et al. On the locality of codeword symbols. IEEE Trans Inf Theory, 2012, 58: 6925–6934

  94. 94

    Papailiopoulos D, Dimakis A. Locally repairable codes. In: Proceedings of IEEE International Symposium on Information Theory, Cambridge, 2012. 2771–2775

  95. 95

    Forbes M, Yekhanin S. On the locality of codeword symbols in non-linear codes. Discrete Math, 2014, 324: 78–84

  96. 96

    Prakash N. Kamath GM, Lalitha V, et al. Optimal linear codes with a local-error-correction property. In: Proceedings of IEEE International Symposium on Information Theory Proceedings, Cambridge, 2012. 2776–2780

  97. 97

    Silberstein N, Rawat S, Koyluoglu O O, et al. Optimal locally repairable codes via rank-metric codes. In: Proceedings of IEEE International Symposium on Information Theory, Istanbul, 2013. 1819–1823

  98. 98

    Prakash N, Lalitha V, Kumar V. Codes with locality for two erasures. In: Proceedings of IEEE International Symposium on Information Theory, Honolulu, 2014. 1962–1966

  99. 99

    Wang A, Zhang Z. An integer programming-based bound for locally repairable codes. IEEE Trans Inf Theory, 2015, 61: 5280–5294

  100. 100

    Zhang J, Wang X, Ge G N. Some improvements on locally repairable codes. 2015. ArXiv:1506.04822

  101. 101

    Mehrabi M, Ardakani M. On minimum distance of locally repairable codes. In: Proceedings of the 15th Canadian Workshop on Information Theory, Quebec, 2017

  102. 102

    Tamo I, Barg A. A family of optimal locally recoverable codes. IEEE Trans Inf Theory, 2014, 60: 4661–4676

  103. 103

    Ernvall T, Westerback T, Hollanti C. Constructions of optimal and almost optimal locally repairable codes. In: Proceedings of the 4th International Conference on Wireless Communications, Vehicular Technology, Information Theory and Aerospace Electronic Systems, Aalborg, 2014

  104. 104

    Liu J, Mesnager S, Chen L. New constructions of optimal locally recoverable codes via good polynomials. IEEE Trans Inf Theory, 2018, 64: 889–899

  105. 105

    Kolosov O, Barg A, Tamo I, et al. Optimal LRC codes for all lenghts n 6 q. 2018. ArXiv:1802.00157

  106. 106

    Jin L F, Ma L M, Xing C P. Construction of optimal locally repairable codes via automorphism groups of rational function fields. 2017. ArXiv:1710.09638

  107. 107

    Balaji B, Kumar P V. On partial maximally-recoverable and maximally-recoverable codes. In: Proceedings of IEEE International Symposium on Information Theory, Hong Kong, 2015. 1881–1885

  108. 108

    Cadambe R, Mazumdar A. Bounds on the size of locally recoverable codes. IEEE Trans Inf Theory, 2015, 61: 5787–5794

  109. 109

    Huang P, Yaakobi E, Uchikawa H, et al. Binary linear locally repairable codes. IEEE Trans Inf Theory, 2016, 62: 6268–6283

  110. 110

    Balaji B, Kumar P V. Bounds on the rate and minimum distance of codes with availability. In: Proceedings of IEEE International Symposium on Information Theory, Aachen, 2017. 3155–3159

  111. 111

    Wei K. Generalized Hamming weights for linear codes. IEEE Trans Inf Theory, 1991, 37: 1412–1418

  112. 112

    Wang Y, Zhang Z F, Lin D D. Bounds and constructions for linear locally repairable codes over binary fields. In: Proceedings of IEEE International Symposium on Information Theory, Aachen, 2017. 2033–2037

  113. 113

    Ma J X, Ge G N. Optimal binary linear locally repairable codes with disjoint repair groups. 2017. ArXiv:1711.07138

  114. 114

    Agarwal A, Barg A, Hu S, et al. Combinatorial alphabet-dependent bounds for locally recoverable codes. IEEE Trans Inf Theory, 2018, 64: 3481–3492

  115. 115

    Tamo I, Barg A, Goparaju S, et al. Cyclic LRC codes, binary LRC codes, and upper bounds on the distance of cyclic codes. 2016. ArXiv:1603.08878

  116. 116

    Goparaju S, Calderbank R. Binary cyclic codes that are locally repairable. In: Proceedings of IEEE International Symposium on Information Theory, Honolulu, 2014. 676–680

  117. 117

    Zeh A, Yaakobi E. Optimal linear and cyclic locally repairable codes over small fields. In: Proceedings of IEEE Information Theory Workshop, Jerusalem, 2015

  118. 118

    Tamo I, Barg A, Frolov A. Bounds on the parameters of locally recoverable codes. IEEE Trans Inf Theory, 2016, 62: 3070–3083

  119. 119

    Barg A, Tamo I, Vladut S. Locally recoverable codes on algebraic curves. IEEE Trans Inf Theory, 2017, 63: 4928–4939

  120. 120

    Li X D, Ma L M, Xing C P. Construction of asymptotically good locally repairable codes via automorphism groups of function fields. 2017. ArXiv:1711.07703

  121. 121

    Nam Y, Song H Y. Binary locally repairable codes with minimum distance at least six based on partial t-spreads. IEEE Commun Lett, 2017, 21: 1683–1686

  122. 122

    Silberstein N, Zeh A. Optimal binary locally repairable codes via anticodes. In: Proceedings of IEEE International Symposium on Information Theory, Hong Kong, 2015. 1247–1251

  123. 123

    Hao J, Xia T, Chen B. Some results on optimal locally repairable codes. In: Proceedings of IEEE International Symposium on Information Theory, Barcelona, 2016. 440–444

  124. 124

    Shahabinejad M, Khabbazian M, Ardakani M. A class of binary locally repairable codes. IEEE Trans Commun, 2016, 64: 3182–3193

  125. 125

    Hao J, Xia T, Chen B. On optimal ternary locally repairable codes. In: Proceedings of IEEE International Symposium on Information Theory, Aachen, 2017. 171–175

  126. 126

    Hao J, Xia S. Bounds and constructions of locally repairable codes: parity-check matrix approach. 2016. ArXiv:1601.05595

  127. 127

    Li X D, Ma L M, Xing C P. Optimal locally repairable codes via elliptic curves. 2017. ArXiv:1712.03744

  128. 128

    Kim C, No S. New constructions of binary and ternary locally repairable codes using cyclic codes. IEEE Commun Lett, 2018, 22: 228–231

  129. 129

    Luo Y, Xing C P, Yuan C. Optimal locally repairable codes of distance 3 and 4 via cyclic codes. 2018. ArXiv:1801.03623

  130. 130

    Krishnan N, Puranik B, Kumar P V, et al. Exploiting locality for improved decoding of binary cyclic codes. IEEE Trans Commun, 2018, 66: 2346–2358

  131. 131

    Vardy ABe’ery Y. Maximum-likelihood soft decision decoding of BCH codes. IEEE Trans Inf Theory, 1994, 40: 546–554

  132. 132

    Huang P, Yaakobi E, Uchikawa H, et al. Cyclic linear binary locally repairable codes. In: Proceedings of IEEE Information Theory Workshop, Jerusalem, 2015

  133. 133

    Chen M H, Huang C, Li J. On the maximally recoverable property for multi-protection group codes. In: Proceedings of IEEE International Symposium on Information Theory, Nice, 2007. 486–490

  134. 134

    Blaum M, Hafner L, Hetzler S. Partial-MDS codes and their application to RAID type of architectures. IEEE Trans Inf Theory, 2013, 59: 4510–4519

  135. 135

    Calis G, Koyluoglu O. A general construction for PMDS codes. IEEE Commun Lett, 2017, 21: 452–455

  136. 136

    Gabrys R, Yaakobi E, Blaum M, et al. Constructions of partial MDS codes over small fields. In: Proceedings of IEEE International Symposium on Information Theory, Aachen, 2017

  137. 137

    Gopalan P, Huang C, Jenkins B, et al. Explicit maximally recoverable codes with locality. IEEE Trans Inf Theory, 2014, 60: 5245–5256

  138. 138

    Hu G, Yekhanin S. New constructions of SD and MR codes over small finite fields. In: Proceedings of IEEE International Symposium on Information Theory, Barcelona, 2016. 1591–1595

  139. 139

    Chen Y, Shum K W, Yu Q, et al. Sector-disk codes and partial MDS codes with up to three global parities. In: Proceedings of IEEE International Symposium on Information Theory, Hong Kong, 2015. 1876–1880

  140. 140

    Blaum M. Construction of PMDS and SD codes extending RAID 5. 2013. ArXiv:1305.0032

  141. 141

    Blaum M, Plank S, Schwartz M, et al. Construction of partial MDS and sector-disk codes with two global parity symbols. IEEE Trans Inf Theory, 2016, 62: 2673–2681

  142. 142

    Lalitha V, Lokam V. Weight enumerators and higher support weights of maximally recoverable codes. In: Proceedings of the 53rd Annual Allerton Conference on Communication, Control, and Computing, Monticello, 2015. 835–842

  143. 143

    Kadhe S, Calderbank R. Rate optimal binary linear locally repairable codes with small availability. 2017. ArXiv:1701.02456

  144. 144

    Wang Y, Zhang Z F, Liu M L. Achieving arbitrary locality and availability in binary codes. In: Proceedings of IEEE International Symposium on Information Theory, Hong Kong, 2015. 1866–1870

  145. 145

    Wang Y, Zhang Z F. Repair locality with multiple erasure tolerance. IEEE Trans Inf Theory, 2014, 60: 6979–6987

  146. 146

    Song W T, Yuen C. Locally repairable codes with functional repair and multiple erasure tolerance. 2015. ArXiv:1507.02796

  147. 147

    Balaji B, Kini G R, Kumar P V. A tight rate bound and a matching construction for locally recoverable codes with sequential recovery from any number of multiple erasures. In: Proceedings of IEEE International Symposium on Information Theory, Aachen, 2017. 1778–1782

  148. 148

    Balaji S B, Kini G R, Kumar P V. A bound on rate of codes with locality with sequential recovery from multiple erasures. 2016. ArXiv:1611.08561

  149. 149

    Song T, Cai K, Yuen C, et al. On sequential locally repairable codes. IEEE Trans Inf Theory, 2018, 64: 3513–3527

  150. 150

    Balaji B, Kini G R, Kumar P V. A rate-optimal construction of codes with sequential recovery with low block length. In: Proceedings of National Conference on Communications, Hyderabad, 2018

  151. 151

    Rawat A S, Mazumdar A, Vishwanath S. Cooperative local repair in distributed storage. 2014. ArXiv:1409.3900

  152. 152

    Exoo G, Jajcay R. Dynamic cage survey. The Electronic Journal Combinatorics, 2013. https://doi.org/pdfs.semanticscholar.org/43b8/2016a2ef8f394f2cb1954439248198d2c274.pdf

  153. 153

    Song W, Dau H, Yuen C, et al. Optimal locally repairable linear codes. IEEE J Sel Areas Commun, 2014, 32: 1019–1036

  154. 154

    Chen B, Xia T, Hao J, et al. Constructions of optimal cyclic (r, d) locally repairable codes. IEEE Trans Inf Theory, 2018, 64: 2499–2511

  155. 155

    Hao J, Xia T, Chen B. On the linear codes with (r, d)-locality for distributed storage. In: Proceedings of IEEE International Conference on Communications, Paris, 2017

  156. 156

    Sasidharan B, Agarwal K, Kumar P V. Codes with hierarchical locality. In: Proceedings of International Symposium on Information Theory (ISIT), Hong Kong, 2015. 1257–1261

  157. 157

    Ballentine S, Barg A. Codes on curves with hierarchical locality. In: Proceedings of IEEE International Symposium on Information Theory (accepted), Hong Kong, 2018

  158. 158

    Duminuco A, Biersack E. Hierarchical codes: how to make erasure codes attractive for peer-to-peer storage systems. In: Proceedings of the 8th International Conference on Peer-to-Peer Computing, Aachen, 2008. 89–98

  159. 159

    Kamath M, Prakash N, Lalitha V, et al. Codes with local regeneration and erasure correction. IEEE Trans Inf Theory, 2014, 60: 4637–4660

  160. 160

    Gligoroski D, Kralevska K, Jensen R E, et al. Repair duality with locally repairable and locally regenerating codes. 2017. ArXiv:1701.06664

  161. 161

    Hollmann H D L. On the minimum storage overhead of distributed storage codes with a given repair locality. In: Proceedings of IEEE International Symposium on Information Theory, Honolulu, 2014. 1041–1045

  162. 162

    Ahmad I, Wang C. When locally repairable codes meet regenerating codes — what if some helpers are unavailable. In: Proceedings of IEEE International Symposium on Information Theory, Hong Kong, 2015. 849–853

  163. 163

    Krishnan M N, Anantha N R, Kumar P V. Codes with combined locality and regeneration having optimal Rate, dmin and linear field size. 2018. ArXiv:1804.00564

  164. 164

    Shanmugam K, Papailiopoulos S, Dimakis A G, et al. A repair framework for scalar MDS codes. IEEE J Sel Areas Commun, 2014, 32: 998–1007

  165. 165

    Guruswami V, Wootters M. Repairing Reed-Solomon codes. IEEE Trans Inf Theory, 2017, 63: 5684–5698

  166. 166

    MacWilliams J, Sloane N J A. The theory of error-correcting codes. Elsevier, 1977, 68: 185–186

  167. 167

    Dau H, Milenkovic O. Optimal repair schemes for some families of full-length Reed-Solomon codes. In: Proceedings of IEEE International Symposium on Information Theory, Aachen, 2017. 346–350

  168. 168

    Ye M, Barg A. Explicit constructions of MDS array codes and RS codes with optimal repair bandwidth. In: Proceedings of IEEE International Symposium on Information Theory, Barcelona, 2016. 1202–1206

  169. 169

    Chowdhury A, Vardy A. Improved schemes for asymptotically optimal repair of MDS codes. In: Proceedings of the 55th Annual Allerton Conference on Communication, Control, and Computing, Monticello, 2017. 950–957

  170. 170

    Tamo I, Ye M, Barg A. Optimal repair of reed-solomon codes: achieving the cut-set bound. In: Proceedings of the 58th IEEE Annual Symposium on Foundations of Computer Science, Berkeley, 2017. 216–227

  171. 171

    Dau S H, Duursma I M, Kiah H M, et al. Repairing Reed-Solomon codes with multiple erasures. 2016. ArXiv:1612.01361

  172. 172

    Bartan B, Wootters M. Repairing multiple failures for scalar MDS codes. In: Proceedings of the 55th Annual Allerton Conference on Communication, Control, and Computing, Monticello, 2017. 1145–1152

  173. 173

    Ye M, Barg A. Repairing Reed-Solomon codes: universally achieving the cut-set bound for any number of erasures. 2017. ArXiv:1710.07216

  174. 174

    Luby M. Capacity bounds for distributed storage. 2016. ArXiv:1610.03541

  175. 175

    Luby M, Padovani R, Richardson T J, et al. Liquid cloud storage. 2017. ArXiv:1705.07983

  176. 176

    Huang C, Simitci H, Xu Y, et al. Erasure coding in windows azure storage. In: Proceedings of USENIX Annual Technical Conference, Boston, 2012. 15–26

  177. 177

    Gantenbein D. A better way to store data. Microsoft research blog. https://doi.org/www.microsoft.com/en-us/research/blog/better-way-store-data/

  178. 178

    CEPH. Locally repairable erasure code plugin. https://doi.org/docs.ceph.com/docs/master/rados/operations/erasure-codelrc/

  179. 179

    Rashmi V, Shah N B, Gu D, et al. A “hitchhiker’s” guide to fast and efficient data reconstruction in erasure-coded data centers. In: Proceedings of ACM SIGCOMM Conference, Chicago, 2014. 331–342

  180. 180

    Kralevska K, Gligoroski D, Jensen E, et al. Hashtag erasure codes: from theory to practice. IEEE Trans Big Data, 2017. doi: 10.1109/TBDATA.2017.2749255

  181. 181

    Krishnan N, Prakash N, Lalitha V, et al. Evaluation of codes with inherent double replication for Hadoop. In: Proceedings of the 6th USENIX Workshop on Hot Topics in Storage and File Systems, Philadelphia, 2014

  182. 182

    Rashmi V, Nakkiran P, Wang J, et al. Having your cake and eating it too: jointly optimal erasure codes for I/O, storage, and network-bandwidth. In: Proceedings of the 13th USENIX Conference on File and Storage Technologies, Santa Clara, 2015. 81–94

  183. 183

    Li J, Li B. Beehive: erasure codes for fixing multiple failures in distributed storage systems. IEEE Trans Parallel Distrib Syst, 2017, 28: 1257–1270

  184. 184

    Pamies-Juarez L, Blagojevic F, Mateescu R, et al. Opening the chrysalis: on the real repair performance of MSR codes. In: Proceedings of the 14th USENIX Conference on File and Storage Technologies, Santa Clara, 2016. 81–94

  185. 185

    Gad E, Mateescu R, Blagojevic F, et al. Repair-optimal MDS array codes over GF(2). In: Proceedings of IEEE International Symposium on Information Theory, Istanbul, 2013. 887–891

  186. 186

    Vajha M, Ramkumar V, Puranik B, et al. Clay codes: moulding MDS codes to yield an MSR code. In: Proceedings of the 16th USENIX Conference on File and Storage Technologies, Oakland, 2018. 139–154

Download references

Acknowledgements

This work was supported in part by National Science Foundation of USA (Grant No. 1421848) and in part by an India-Israel UGC-ISF Joint Research Program Grant.

Author information

Correspondence to P. Vijay Kumar.

Additional information

Invited article

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Balaji, S.B., Krishnan, M.N., Vajha, M. et al. Erasure coding for distributed storage: an overview. Sci. China Inf. Sci. 61, 100301 (2018). https://doi.org/10.1007/s11432-018-9482-6

Download citation

Keywords

  • distributed storage
  • regenerating codes
  • locally recoverable codes
  • codes with locality
  • erasure codes
  • node repair