Advertisement

Efficient Reliability in Volunteer Storage Systems with Random Linear Coding

  • Ádám Visegrádi
  • Péter Kacsuk
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8805)

Abstract

Volunteer systems pose difficult challenges for data storage. Because of the extremely low reliability of volunteer nodes, these systems require so high redundancy that replication is infeasible. Erasure coding has been proposed to cope with this problem as it needs much less redundancy to achieve the same reliability. Its downside is that the reparation of the system creates high overhead, as fully decoding the original data is required to generate new coded data.

Random linear coding has been proposed to be used as a data storage method, as it provides a better redundancy/reliability ratio, and less control overhead. We propose that it also helps in the reparation of the system, as decoding is not required; instead, coded data can be generated from already existing coded data. However, it may be possible that this iterative reparation leads to degradation of data over time; even more so, if sparse coding is used to increase compute efficiency.

This paper examines the effects of random linear coding and the iterative reparation of the system. It shows the reliability that can be achieved with random linear coding in a highly volatile distributed system. We conclude that random linear coding can achieve high reliability even in highly volatile systems.

Keywords

Network Code Sparse Code Erasure Code Target Redundancy Original Block 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Random Network Coding Library, https://github.com/avisegradi/rnc-lib (accessed: September 10, 2013)
  2. 2.
    Efficient Random Network Coding for Distributed Storage Systems, Ádám Visegrádi and Péter Kacsuk. Euro-Par 2013, MHPC Workshop (2013) (in press)Google Scholar
  3. 3.
    Acedanski, S., Deb, S., Médard, M., Koetter, R.: How good is random linear coding based distributed networked storage. In: Workshop on Network Coding, Theory and Applications (2005)Google Scholar
  4. 4.
    Ahlswede, R., Cai, N., Li, S.Y.R., Yeung, R.W.: Network information flow. IEEE Transactions on Information Theory 46(4), 1204–1216 (2000)CrossRefMathSciNetMATHGoogle Scholar
  5. 5.
    Anderson, D.: BOINC volunteer data archival, https://boinc.berkeley.edu/trac/wiki/VolunteerDataArchival (accessed January 2014)
  6. 6.
    Anderson, D.: BOINC volunteer storage, https://boinc.berkeley.edu/trac/wiki/VolunteerStorage (accessed January 2014)
  7. 7.
    Anderson, D.P.: BOINC: a system for public-resource computing and storage. In: Proceedings of the Fifth IEEE/ACM International Workshop on Grid Computing, pp. 4–10 (November 2004)Google Scholar
  8. 8.
    Anderson, D.P., Fedak, G.: The computational and storage potential of volunteer computing. In: Sixth IEEE International Symposium on Cluster Computing and the Grid, CCGRID 2006, vol. 1, pp. 73–80. IEEE (2006)Google Scholar
  9. 9.
    Borthakur, D.: The hadoop distributed file system: Architecture and design. Hadoop Project Website 11(21) (2007)Google Scholar
  10. 10.
    Dean, J., Ghemawat, S.: MapReduce: simplified data processing on large clusters. Communications of the ACM 51(1), 107–113 (2008)CrossRefGoogle Scholar
  11. 11.
    Dimakis, A.G., Godfrey, P.B., Wu, Y., Wainwright, M.J., Ramchandran, K.: Network coding for distributed storage systems. IEEE Transactions on Information Theory 56(9), 4539–4551 (2010)CrossRefGoogle Scholar
  12. 12.
    Dimakis, A.G., Ramchandran, K., Wu, Y., Suh, C.: A survey on network codes for distributed storage. Proceedings of the IEEE 99(3), 476–489 (2011)CrossRefGoogle Scholar
  13. 13.
    Fragouli, C., Le Boudec, J.-Y., Widmer, J.: Network coding: an instant primer. ACM SIGCOMM Computer Communication Review 36(1), 63–68 (2006)CrossRefGoogle Scholar
  14. 14.
    Ho, T., Médard, M., Koetter, R., Karger, D., Effros, M., Shi, J., Leong, B.: A random linear network coding approach to multicast. IEEE Transactions on Information Theory 52(10), 4413–4430 (2006)CrossRefGoogle Scholar
  15. 15.
    Li, S.-Y.R., Yeung, R.W., Cai, N.: Linear network coding. IEEE Transactions on Information Theory 49(2), 371–381 (2003)CrossRefMathSciNetMATHGoogle Scholar
  16. 16.
    Ma, G., Xu, Y., Lin, M., Xuan, Y.: A content distribution system based on sparse linear network coding. In: NetCod 2007 (2007)Google Scholar
  17. 17.
    MacKay, D.J.C.: Fountain codes. IEEE Proceedings Communications 152, 1062–1068 (2005)CrossRefGoogle Scholar
  18. 18.
    Reed, I., Solomon, G.: Polynomial codes over certain finite fields. Journal of the Society for Industrial & Applied Mathematics 8(2), 300–304 (1960)CrossRefMathSciNetMATHGoogle Scholar
  19. 19.
    Rodrigues, R., Zhou, T.H.: High availability in dHTs: Erasure coding vs. Replication. In: van Renesse, R. (ed.) IPTPS 2005. LNCS, vol. 3640, pp. 226–239. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  20. 20.
    Wang, M., Li, B.: How practical is network coding? In: 14th IEEE International Workshop on Quality of Service, IWQoS 2006, pp. 274–278 (2006)Google Scholar
  21. 21.
    Weatherspoon, H., Kubiatowicz, J.D.: Erasure coding vs. Replication: A quantitative comparison. In: Druschel, P., Kaashoek, M.F., Rowstron, A. (eds.) IPTPS 2002. LNCS, vol. 2429, p. 328. Springer, Heidelberg (2002)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Ádám Visegrádi
    • 1
  • Péter Kacsuk
    • 1
  1. 1.Computer and Automation Research InstituteHungarian Academy of SciencesHungary

Personalised recommendations