Efficient Random Network Coding for Distributed Storage Systems

  • Ádám Visegrádi
  • Péter Kacsuk
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8374)

Abstract

Making distributed storage systems reliable is an important challenge. Simple replication may cause severe storage overhead when individual components of the system are very unreliable. Using erasure codes is a promising solution for this problem, but it presents its own challenges; it makes the design of such a system very complex, and it presents the problem of reparation. Network coding has been suggested to be used in the communication in these networks to help reduce overhead.

However, using random network coding as—not besides—erasure coding would be an even more promising field to investigate; such a system would have a simple design, need little or no centralization, and reparation of the system could be much simpler than it is in other erasure coding schemes.

The first step on this path is to investigate whether network coding can achieve such a performance that it is a feasible alternative to other erasure codes. This paper presents our experiences about the realization of random network coding based on the discrete logarithm of the finite field. We discuss possible performance optimizations for such a system, and provide performance measurement results focusing on data storage scenarios.

Keywords

Nite 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Random Network Coding Library, https://github.com/avisegradi/rnc-lib (accessed: October 09, 2013 )
  2. 2.
    Acedanski, S., Deb, S., Médard, M., Koetter, R.: How good is random linear coding based distributed networked storage. In: Workshop on Network Coding, Theory and Applications (2005)Google Scholar
  3. 3.
    Ahlswede, R., Cai, N., Li, S., Yeung, R.: Network information flow. IEEE Transactions on Information Theory 46(4), 1204–1216 (2000)CrossRefMATHMathSciNetGoogle Scholar
  4. 4.
    Anderson, D.: BOINC: a system for public-resource computing and storage. In: Proceedings of the Fifth IEEE/ACM International Workshop on Grid Computing, pp. 4–10 (November 2004)Google Scholar
  5. 5.
    Borthakur, D.: The hadoop distributed file system: Architecture and design. Hadoop Project Website 11, 21 (2007)Google Scholar
  6. 6.
    Dean, J., Ghemawat, S.: MapReduce: simplified data processing on large clusters. Communications of the ACM 51(1), 107–113 (2008)CrossRefGoogle Scholar
  7. 7.
    Dimakis, A., Godfrey, P., Wu, Y., Wainwright, M., Ramchandran, K.: Network coding for distributed storage systems. IEEE Transactions on Information Theory 56(9), 4539–4551 (2010)CrossRefGoogle Scholar
  8. 8.
    Dimakis, A., Ramchandran, K., Wu, Y., Suh, C.: A survey on network codes for distributed storage. Proceedings of the IEEE 99(3), 476–489 (2011)CrossRefGoogle Scholar
  9. 9.
    Ho, T., Médard, M., Koetter, R., Karger, D., Effros, M., Shi, J., Leong, B.: A random linear network coding approach to multicast. IEEE Transactions on Information Theory 52(10), 4413–4430 (2006)CrossRefGoogle Scholar
  10. 10.
    MacKay, D.: Fountain codes. IEE Proceedings-Communications 152, 1062–1068 (2005)CrossRefGoogle Scholar
  11. 11.
    Reed, I., Solomon, G.: Polynomial codes over certain finite fields. Journal of the Society for Industrial & Applied Mathematics 8(2), 300–304 (1960)CrossRefMATHMathSciNetGoogle Scholar
  12. 12.
    Rodrigues, R., Liskov, B.: High availability in DHTs: erasure coding vs. replication. In: Peer-to-Peer Systems IV, pp. 226–239 (2005)Google Scholar
  13. 13.
    Wang, M., Li, B.: How practical is network coding? In: 14th IEEE International Workshop on Quality of Service, IWQoS 2006, pp. 274–278 (2006)Google Scholar
  14. 14.
    Weatherspoon, H., Kubiatowicz, J.D.: Erasure coding vs. Replication: A quantitative comparison. In: Druschel, P., Kaashoek, M.F., Rowstron, A. (eds.) IPTPS 2002. LNCS, vol. 2429, pp. 328–337. Springer, Heidelberg (2002)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2014

Authors and Affiliations

  • Ádám Visegrádi
    • 1
  • Péter Kacsuk
    • 1
  1. 1.Computer and Automation Research InstituteHungarian Academy of SciencesHungary

Personalised recommendations