Skip to main content
SpringerLink
Log in
Book cover

Network Coding and Subspace Designs pp 363–383Cite as

An Overview of Coding for Distributed Storage Systems

  • Chapter
  • First Online:

Part of the book series: Signals and Communication Technology ((SCT))

Abstract

This chapter provides a short survey of coding for distributed storage systems. It describes the code design criteria for such codes, emphasizing what makes them different from traditional codes for communication. It then focuses on two large families of codes, regenerating codes and locally repairable codes, including a discussion on how these codes are used in an adversarial setting.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD   109.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

Notes

  1. 1.

    We are not claiming that Reed–Solomon codes are not used in distributed storage systems, we added “as such” as a way to say that something else has to be taken into account in all cases, even if one wants to use Reed–Solomon codes.

  2. 2.

    The same example can be done on a linear combination of \(\mathbf {f}_i\).

  3. 3.

    Since \(\gamma =d\beta \), we may consider the trade-off in terms of either \((\alpha ,\gamma )\) or \((\alpha ,\beta )\).

  4. 4.

    In fact, we will not discuss non-linear codes, even though some works have been done on this topic.

References

  1. M. Blaum, S.R. Hetzler, Integrated interleaved codes as locally recoverable codes: properties and performance. Int. J. Inf. Coding Theory 3(4), 324 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  2. A. Datta, L. Pamies-Juarez, F. Oggier, A study of the performance of novel storage-centric repairable codes. Springer Comput. 98(3) (2015)

    Google Scholar 

  3. T.K. Dikaliotis, A.G. Dimakis, T. Ho, Security in distributed storage systems by communicating a logarithmic number of bits, in Proceedings of the 2010 IEEE International Symposium on Information Theory (ISIT) (2010), pp. 1948–1952

    Google Scholar 

  4. A.G. Dimakis, P.B. Godfrey, Y. Wu, M.J. Wainwright, K. Ramchandran, Network coding for distributed storage systems. IEEE Trans. Inf. Theory 56(9) (2010)

    Google Scholar 

  5. I.M. Duursma, Shortened regenerating codes. CoRR (2015), arXiv:1505.00178

  6. M. Forbes, S. Yekhanin, On the locality of codeword symbols in non-linear codes. Discret. Math. 324, 78–84 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  7. P. Gopalan, C. Huang, H. Simitci, S. Yekhanin, On the locality of codeword symbols. IEEE Trans. Inf. Theory 58(11), 6925–6934 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  8. Y.S. Han, R. Zheng, W.H. Mow, Exact regenerating codes for Byzantine fault tolerance in distributed storage, in Proceedings of IEEE INFOCOM (2012), pp. 2498–2506

    Google Scholar 

  9. J. Harshan, F. Oggier, A. Datta, Sparsity exploiting erasure coding for distributed storage of versioned data. Springer Comput. (2016)

    Google Scholar 

  10. C. Huang, M, Chen, J. Li, Pyramid codes: flexible schemes to trade space for access efficiency in reliable data storage systems, in Sixth IEEE International Symposium on Network Computing and Applications, July 2007, pp. 79–86

    Google Scholar 

  11. C. Huang, H. Simitci, Y. Xu, A. Ogus, B. Calder, P. Gopalan, J. Lin, S. Yekhanin, Erasure coding in windows Azure storage, in 2012 USENIX Annual Technical Conference, 12–15 June 2012

    Google Scholar 

  12. K.A.S. Immink, Codes for Mass Data Storage Systems, Second fully revised edition (Shannon Foundation Publishers, Eindhoven, 2004), http://www.turing-machines.com/pdf/codes_for_mass_data2.pdf. ISBN 90-74249-27-2

  13. S. Kadhe, A. Sprintson, Codes with unequal locality, in IEEE International Symposium on Information Theory (ISIT) (2016), pp. 435–439

    Google Scholar 

  14. A.-M. Kermarrec, N. Le Scouarnec, G. Straub, Repairing multiple failures with coordinated and adaptive regenerating codes, in The 2011 International Symposium on Network Coding (NetCod 2011)

    Google Scholar 

  15. O.O. Koyluoglu, A.S. Rawat, S. Vishwanath, Secure cooperative regenerating codes for distributed storage systems. IEEE Trans. Inf. Theory 60(9), 5228–5244 (2014), arXiv:1210.3664

  16. M. Kuijper, D. Napp, Erasure codes with simplex locality, in 21st International Symposium on Mathematical Theory of Networks and Systems, Groningen, The Netherlands (2014)

    Google Scholar 

  17. M. Li, P.P.C. Lee, STAIR codes: a general family of erasure codes for tolerating device and sector failures in practical storage systems, in Proceedings of the 12th USENIX Conference on File and Storage Technologies (FAST’14), Santa Clara, CA (2014)

    Google Scholar 

  18. W.K. Lin, D.M. Chiu, Y.B. Lee, Erasure code replication revisited, P2P 2004

    Google Scholar 

  19. S. Liu, F. Oggier, On storage codes allowing partially collaborative repairs, in IEEE International Symposium on Information Theory (ISIT) 2014, pp. 2440–2444

    Google Scholar 

  20. S. Liu, F. Oggier, Partially collaborative storage codes in the presence of an eavesdropper. Int. J. Inf. Coding Theory 3(3), 177–196 (2016)

    Google Scholar 

  21. S. Mohajer, R. Tandon, New bounds on the (n, k, d) storage systems with exact repair, in IEEE International Symposium on Information Theory, ISIT 2015, Hong Kong, China, 14–19 June 2015, pp. 2056–2060

    Google Scholar 

  22. F. Oggier, A. Datta, Byzantine fault tolerance of regenerating codes, in The 11th IEEE International Conference on Peer-to-Peer Computing (P2P 2011), Kyoto, arXiv:1106.2275

  23. F. Oggier, A. Datta, Coding techniques for repairability in networked distributed storage systems, Foundations and Trends in Communications and Information Theory (Now Publishers, Breda, 2013)

    Google Scholar 

  24. F. Oggier, A. Datta, Self-repairing homomorphic codes for distributed storage systems, INFOCOM 2011

    Google Scholar 

  25. L. Pamies-Juarez, A. Datta, F. Oggier, RapidRAID: pipelined erasure codes for fast data archival in distributed storage systems, arXiv:1207.6744

  26. L. Pamies-Juarez, H.D.L. Hollmann, F. Oggier, Locally repairable codes with multiple repair alternatives, in IEEE International Symposium on Information Theory (ISIT) (2013), pp. 892–896

    Google Scholar 

  27. D.S. Papailiopoulos, A.G. Dimakis, Locally repairable codes, in IEEE International Symposium on Information Theory (ISIT) (2012), arXiv:1206.3804

  28. S. Pawar, S. El Rouayheb, K. Ramchandran, Securing dynamic distributed storage systems against eavesdropping and adversarial attacks. IEEE Trans. Inf. Theory (Spec. Issue Facet. Coding Theory Algorithm Netw.) 57(9) (2011)

    Google Scholar 

  29. K.V. Rashmi, N.B. Shah, K. Ramchandran, P.V. Kumar, Regenerating codes for errors and erasures in distributed storage, in Proceedings of the 2012 IEEE International Symposium on Information Theory (ISIT) (2012), pp. 1202–1206

    Google Scholar 

  30. A.S. Rawat, S. Vishwanath, On locality in distributed storage systems, in IEEE Information Theory Workshop (ITW) (2012), pp. 497–501

    Google Scholar 

  31. A.S. Rawat, O.O. Koyluoglu, N. Silberstein, S. Vishwanath, Optimal locally repairable and secure codes for distributed storage systems. IEEE Trans. Inf. Theory 60(1), 212–236 (2014)

    Article  Google Scholar 

  32. A.S. Rawat, D.S. Papailiopoulos, A.G. Dimakis, S. Vishwanath, Locality and availability in distributed storage. IEEE Trans. Inf. Theory 62(8), 4481–4493 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  33. I. Reed, G. Solomon, Polynomial codes over certain finite fields. J. Soc. Ind. Appl. Math. 8, 300–304 (1960)

    Article  MathSciNet  MATH  Google Scholar 

  34. B. Sasidharan, N. Prakash, M.N. Krishnan, M. Vajha, K. Senthoor, P.V. Kumar, Outer bounds on the storage-repair bandwidth tradeoff of exact-repair regenerating codes. Int. J. Inf. Coding Theory 3(4), 255–298 (2016)

    Google Scholar 

  35. B. Sasidharan, K. Senthoor, P.V. Kumar, An improved outer bound on the storage repair-bandwidth tradeoff of exact-repair regenerating codes, in IEEE International Symposium on Information Theory, ISIT 2014, pp. 2430–2434

    Google Scholar 

  36. K. Senthoor, B. Sasidharan, P.V. Kumar, Improved layered regenerating codes characterizing the exact-repair storage-repair bandwidth tradeoff for certain parameter sets, in 2015 IEEE Information Theory Workshop, ITW 2015, Jerusalem, Israel, 26 April–1 May 2015, p. 15

    Google Scholar 

  37. N.B. Shah, K.V. Rashmi, P.V. Kumar, Information theoretically secure regenerating codes for distributed storage, in Proceedings of IEEE Globecom (2011)

    Google Scholar 

  38. K.W. Shum, Cooperative regenerating codes for distributed storage systems, in The International Conference on Communications (ICC 2011), Kyoto, arXiv:1101.5257

  39. K.W. Shum, Y. Hu, Exact minimum-repair-bandwidth cooperative regenerating codes for distributed storage systems, in The International Symposium on Information Theory (ISIT 2011), Saint-Petersburg

    Google Scholar 

  40. K.W. Shum, Y. Hu, Cooperative regenerating codes. IEEE Trans. Inf. Theory 59(11), 7229–7258 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  41. N. Silberstein, A.S. Rawat, O.O. Koyluoglu, S. Vishwanath, Optimal locally repairable codes via rank-metric codes, in IEEE International Symposium on Information Theory (ISIT) (2013)

    Google Scholar 

  42. N. Silberstein, A.S. Rawat, S. Vishwanath, Error-correcting regenerating and locally repairable codes via rank-metric codes. IEEE Trans. Inf. Theory 61(11), 5765–5778 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  43. A. Singh Rawat, S. Vishwanat, A. Bhowmick, E. Soljanin, Update efficient codes for distributed storage, in IEEE International Symposium on Information Theory (ISIT) (2011)

    Google Scholar 

  44. I. Tamo, A. Barg, S. Goparaju, R. Calderbank, Cyclic LRC codes, binary LRC codes, and upper bounds on the distance of cyclic clodes. Int. J. Inf. Coding Theory 3(4), 32 (2016)

    Google Scholar 

  45. I. Tamo, A. Barg, A family of optimal locally recoverable codes. IEEE Trans. Inf. Theory 60(8), 4661–4676 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  46. C. Tian, Characterizing the rate region of the (4,3,3) exact-repair regenerating codes. IEEE J. Sel. Areas Commun. 32(5), 967–975 (2014)

    Article  Google Scholar 

  47. A. Wang, Z. Zhang, Repair locality with multiple erasure tolerance. IEEE Trans. Inf. Theory 60(11), 6979–6987 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  48. H. Weatherspoon, J. Kubiatowicz, Erasure coding vs replication: a quantitative comparison, Peer-to-Peer Systems, LNCS (2002)

    Google Scholar 

  49. A. Zeh, E. Yaakobi, Bound and constructions of codes with multiple localities, in IEEE International Symposium on Information Theory (ISIT) (2016), pp. 641–644

    Google Scholar 

Download references

Acknowledgements

This work is supported by the MoE Tier-2 grant eCODE: Erasure Codes for Datacenter Environments.

Author information

Authors and Affiliations

  1. Institute of Network Coding-The Chinese University of Hong Kong, Hong Kong, China

    Shiqiu Liu

  2. Division of Mathematical Sciences, School of Physical and Mathematical Sciences, Nanyang Technological University, Singapore, Singapore

    Frédérique Oggier

Authors
  1. Shiqiu Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Frédérique Oggier
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shiqiu Liu .

Editor information

Editors and Affiliations

  1. Department of Mathematics and Systems Analysis, Aalto University, Espoo, Finland

    Marcus Greferath

  2. Faculty of Electrical Engineering and Computing, Department of Applied Mathematics, University of Zagreb, Zagreb, Croatia

    Mario Osvin Pavčević

  3. Yahoo Research, Haifa, Israel

    Natalia Silberstein

  4. Department of Telecommunications and Systems Engineering, Universitat Autònoma de Barcelona, Cerdanyola del Vallès, Barcelona, Spain

    María Ángeles Vázquez-Castro

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer International Publishing AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Liu, S., Oggier, F. (2018). An Overview of Coding for Distributed Storage Systems. In: Greferath, M., Pavčević, M., Silberstein, N., Vázquez-Castro, M. (eds) Network Coding and Subspace Designs. Signals and Communication Technology. Springer, Cham. https://doi.org/10.1007/978-3-319-70293-3_14

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-70293-3_14

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-70292-6

  • Online ISBN: 978-3-319-70293-3

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation