NameNode and DataNode Coupling for a Power-Proportional Hadoop Distributed File System

  • Hieu Hanh Le
  • Satoshi Hikida
  • Haruo Yokota
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7826)

Abstract

Current works on power-proportional distributed file systems have not considered the cost of updating data sets that were modified (updated or appended) in a low-power mode, where a subset of nodes were powered off. Effectively reflecting the updated data is vital in making a distributed file system, such as the Hadoop Distributed File System (HDFS), power proportional. This paper presents a novel architecture, a NameNode and DataNode Coupling Hadoop Distributed File System (NDCouplingHDFS), which effectively reflects the updated blocks when the system goes into a high-power mode. This is achieved by coupling the metadata management and data management at each node to efficiently localize the range of blocks maintained by the metadata. Experiments using actual machines show that NDCouplingHDFS is able to significantly reduce the execution time required to move updated blocks by 46% relative to the normal HDFS. Moreover, NDCouplingHDFS is capable of increasing the throughput of the system that is supporting MapReduce by applying an index in metadata management.

Keywords

power-proportionality HDFS metadata management 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Amur, H., Cipar, J., Gupta, V., Ganger, G.R., Kozuch, M.A., Schwan, K.: Robust and Flexible Power-proportional Storage. In: Proc. the 1st ACM Symposium on Cloud Computing, SoCC 2010, pp. 217–228 (2010)Google Scholar
  2. 2.
    Kim, J., Rotem, D.: Energy Proportionality for Disk Storage using Replication. In: Proc. the 14th Int’l Conference on Extending Database Technology, pp. 81–92 (2011)Google Scholar
  3. 3.
    Apache Hadoop: HDFS Hadoop Wiki, http://wiki.apache.org/hadoop/HDFS
  4. 4.
    Yokota, H., Kanemasa, Y., Miyazaki, J.: Fat-Btree: An Update Conscious Parallel Directory Structure. In: Proc. the 15th Int’l Conference on Data Engineering, pp. 448–457. IEEE Computer Society (1999)Google Scholar
  5. 5.
    Narayanan, D., Donnelly, A., Rowstron, A.: Write Off-loading: Practical Power Management for Enterprise Storage. In: Proc. 6th USENIX Conference on File and Storage Technologies, pp. 253–267 (2008)Google Scholar
  6. 6.
    Le, H.H., Hikida, S., Yokota, H.: An Evaluation of Power-proportional Data Placement for Hadoop Distributed File Systems. In: Proc. Cloud and Green Computing, pp. 752–759. IEEE Computer Society (2011)Google Scholar
  7. 7.
    Yoshihara, T., Kobayashi, D., Yokota, H.: A Concurrency Control Protocol for Parallel B-tree Structures Without Latch-coupling for Explosively Growing Digital Content. In: Proc. the 11th Int’l Conference on Extending Database Technology: Advances in Database Technology, pp. 133–144. ACM (2008)Google Scholar
  8. 8.
    Rodeh, O., Teperman, A.: zFS-a Scalable Distributed File System using Object Disks. In: Proc. 20th IEEE/11th NASA Goddard Conference on Mass Storage Systems and Technologies (MSST 2003), pp. 207–218. IEEE (2003)Google Scholar
  9. 9.
    Braam, P.: The Lustre Storage ArchitectureGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Hieu Hanh Le
    • 1
  • Satoshi Hikida
    • 1
  • Haruo Yokota
    • 1
  1. 1.Department of Computer ScienceTokyo Institute of TechnologyJapan

Personalised recommendations