Abstract
As Hadoop becomes a popular distributed programming framework for processing large data on its distributed file system (HDFS), demands for secure computing and file storage grow quickly. However, the current Hadoop does not support encryption of storing HDFS blocks, which is a fundamental solution for secure Hadoop. Therefore, we propose a secure Hadoop architecture by adding encryption and decryption functions in HDFS. We have implemented secure HDFS by adding the AES encrypt/decrypt class to CompressionCodec in Hadoop. From experiments with a small Hadoop testbed, we have shown that the representative MapReduce job on encrypted HDFS generates affordable computation overhead less than 7%.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Hadoop, http://hadoop.apache.org/
Dean, J., Ghemawat, S.: MapReduce: Simplified Data Processing on Large Cluster. In: OSDI (2004)
Ghemawat, S., Gobioff, H., Leung, S.: The Google File System. In: ACM Symposium on Operating Systems Principles (October 2003)
O’Malley, O., Zhang, K., Radia, S., Marti, R., Harrell, C.: Hadoop Security Design, Technical Report (October 2009)
White, T.: Hadoop: The Definitive Guide, 1st edn. O’Reilly Media (2009)
Cordova, A.: MapReduce over Tahoe. Hadoop World (2009)
Majors, J.H.: Secdoop: a confidentiality service on Hadoop clusters. Auburn University Master Thesis (May 2011)
Lin, H., Seh, S., Tzeng, W., Lin, B.P.: Toward Data Confidentiality via Integrating Hybrid Encryption Schemes and Hadoop Distributed FileSystem. In: IEEE AINA (2012)
Yang, Y., Wu, Z., Yang, X., Zhang, L., Yu, X., Lao, Z., Wang, D., Long, M.: SAPSC: Security Architecture of Private Storage Cloud Based on HDFS. In: Proceedings of 26th IEEE Workshops of International Conference on Advanced Information Networking and Applications (2012)
NVIDIA CUDA Programming Guide, http://developer.download.nvidia.com/compute/DevZone/docs/html/C/doc/CUDA_C_ProgrammingGuide.pdf
Al-Kiswany, S., Gharaibeh, A., Santos-Neto, E., Yuan, G., Ripeanu, M.: StoreGPU: exploiting graphics processing units to accelerate distributed storage systems. In: ACM HPDC (2008)
Han, S., Jang, K., Park, K., Moon, S.: PacketShader: A GPU accelerated Software Router. In: Proceedings of the ACM SIGCOMM (2010)
Jang, K., Han, S., Han, S., Moon, S., Park, K.: SSLShader: Cheap SSL Acceleration with Commodity Processors. In: Proceedings of NSDI (2011)
Sun, W., Ricci, R., Curry, M.L.: GPUstore: Harnessing GPU Computing for Storage Systems in the OS Kernel. In: ACM SYSTOR (June 2012)
Bhatotia, P., Rodrigues, R., Verma, A.: Shredder: GPU-Accelerated Incremental Storage and Computation. In: USENIX FAST (February 2012)
Advanced Encryption Standard, http://en.wikipedia.org/wiki/Advanced_Encryption_Standard
Intel, http://software.intel.com/en-us/articles/intel-advanced-encryption-standard-instructions-aes-ni
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2013 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Park, S., Lee, Y. (2013). Secure Hadoop with Encrypted HDFS. In: Park, J.J.(.H., Arabnia, H.R., Kim, C., Shi, W., Gil, JM. (eds) Grid and Pervasive Computing. GPC 2013. Lecture Notes in Computer Science, vol 7861. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-38027-3_14
Download citation
DOI: https://doi.org/10.1007/978-3-642-38027-3_14
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-38026-6
Online ISBN: 978-3-642-38027-3
eBook Packages: Computer ScienceComputer Science (R0)