A hybrid disaster-tolerant model with DDF technology for MooseFS open-source distributed file system

Abstract

In a distributed file system (DFS), a disaster-tolerant model provides high data security in the data disaster event by geographically isolates the backup data, while the hot-backup model provides high availability when the main server is offline. Therefore, the combined model is widely concerned in multiple fields. However, previous investigations have neglected to the disaster tolerance for open-source distributed file systems. When a server run into a fault status and performs backup switch, the remote data center cannot receive any data replicas. We proposed a hybrid model in this work to provide disaster-tolerance especially for open-source DFS by build geographically isolate replicas. In addition, we optimize the model with direct data fetch (DDF) technology. DDF can bypass the main server, and retrieve data directly from the data node. The model was implemented on an open-source DFS named MooseFS. Compared with HDFS and IBM/PPRC, the hybrid model with DDF provides a strong performance boost to the hybrid model during the switch procedure, and this hybrid model performs best for storage of large amount of files in small size.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

References

  1. 1.

    Manchester J, Saha D, Tripathi SK (2004) Guest editorial-protection, restoration, and disaster recovery. IEEE Netw 18(2):3–4

    Article  Google Scholar 

  2. 2.

    Ceph File System Docs (2016) http://docs.ceph.com/docs/master/cephfs/#ceph-filesystem. Accessed 8 Apr 2016

  3. 3.

    Moon YH, Youn CH (2015) Multihybrid job scheduling for fault-tolerant distributed computing in policy-constrained resource networks[J]. Comput Netw 82:81–95

    Article  Google Scholar 

  4. 4.

    Yanlong W (2008) Research of a soft-synchronous replication algorithm for disaster tolerance. Chin High Technol Lett

  5. 5.

    Chang F, Ji M, Leung S et al (2002) Myriad cost-effective disaster tolerance. In: Proceeedings of Fast. USENIX Association, pp 103–116

  6. 6.

    Taobao File System (2011) http://code.taobao.org/p/tfs/wiki/intro/. Accessed 12 Aug 2015

  7. 7.

    Deng K, Wang K, Ma D (2016) A new solution of distributed disaster recovery based on raptor code. In: Proceedings of the 2015 International Conference on Applied Mechanics, Mechatronics and Intelligent Systems (AMMIS2015), pp 825–832

  8. 8.

    Ghemawat S, Gobioff H, Leung S (2004) The Google file system. Proc ACM Sigmetrics Int Conf Meas Model Comput Syst ACM 37:29-43

  9. 9.

    Li-jiang P, Yong-tang B (2008) The analysis and application of Oracle DataGuard for Diff-Area disaster recovery. Comput Knowl Technol 2008:28

  10. 10.

    Massiglia P (2000) VERITAS volume replication and Oracle databases. VERITAS Corporation, Mountain View

  11. 11.

    Liu X-J (2010) A brief analysis of the disaster recovery backup technology in oracle database dataguard. In: 2010 2nd International Conference on Industrial and Information Systems, IIS, vol 2, pp 234–236

  12. 12.

    Brown TM, Lipets ML (2014) Metadata management. US20140344526, IEEE

  13. 13.

    Borthakur D (2007) The hadoop distributed file system: architecture and design. Hadoop Project Website 2007(11):21

    Google Scholar 

  14. 14.

    Depardon B, Mahec GL, Séguin C (2013) Analysis of six distributed file systems. HAL-INRIA 23(4):525–542

  15. 15.

    Yunxiao A, Yuesheng T, Jingyu W (2013) Chunkserver load balancing selection algorithm on MooseFS. In: Microcomputer & Its Applications

  16. 16.

    Rajasekar A, Moore R, Hou Cy, Lee CA, Marciano R, de Torcy A, Wan M, Schroeder W, Chen SY, Gilbert L, Tooby P, Zhu B (2010) iRODS primer: integrated rule-oriented data system. Morgan and Claypool Publishers, San Rafael

    Google Scholar 

  17. 17.

    Wang T, Gao H, Qiu J (2016) A combined fault-tolerant and predictive control for network-based industrial processes. IEEE Trans Industr Electron 63(4):2529–2536

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to Xigao Li.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Li, X., Qian, L. A hybrid disaster-tolerant model with DDF technology for MooseFS open-source distributed file system. J Supercomput 73, 2052–2068 (2017). https://doi.org/10.1007/s11227-016-1902-9

Download citation

Keywords

  • MooseFS
  • Disaster tolerant
  • Direct data fetch
  • Distributed file system
  • Hybrid model