Advertisement

International Journal of Parallel Programming

, Volume 44, Issue 4, pp 832–866 | Cite as

MapReduce Parallel Programming Model: A State-of-the-Art Survey

  • Ren LiEmail author
  • Haibo Hu
  • Heng Li
  • Yunsong Wu
  • Jianxi Yang
Article

Abstract

With the development of information technologies, we have entered the era of Big Data. Google’s MapReduce programming model and its open-source implementation in Apache Hadoop have become the dominant model for data-intensive processing because of its simplicity, scalability, and fault tolerance. However, several inherent limitations, such as lack of efficient scheduling and iteration computing mechanisms, seriously affect the efficiency and flexibility of MapReduce. To date, various approaches have been proposed to extend MapReduce model and improve runtime efficiency for different scenarios. In this review, we assess MapReduce to help researchers better understand these novel optimizations that have been taken to address its limitations. We first present the basic idea underlying MapReduce paradigm and describe several widely used open-source runtime systems. And then we discuss the main shortcomings of original MapReduce. We also review these MapReduce optimization approaches that have recently been put forward, and categorize them according to the characteristics and capabilities. Finally, we conclude the paper and suggest several research works that should be carried out in the future.

Keywords

MapReduce Hadoop Cloud computing Big data Scalability 

Notes

Acknowledgments

This work was supported by the Scientific and Technological Research Program of Chongqing Municipal Education Commission under Grant KJ1500538, and the Fundamental Research Funds for the Central Universities of China under Grant 106112014CDJZR098801.

References

  1. 1.
    Shamsi, J., Khojaye, M.A., Qasmi, M.A.: Data-intensive cloud computing: requirement, expectations, challenges and solutions. J. Grid Comput. 11(2), 281–310 (2013)CrossRefGoogle Scholar
  2. 2.
    Meng, X., Ci, X.: Big data management: concepts, techniques and challenges. J. Comput. Res. Dev. 50(1), 146–169 (2013)Google Scholar
  3. 3.
    Wang, S., Wang, H., Qin, X., Zhou, X.: Architecting big data: challenges, studies and forecasts. Chin. J. Comput. 34(10), 1741–1752 (2013)CrossRefGoogle Scholar
  4. 4.
    Mell, P., Grance, T.: Definition of Cloud Computing. Technical report, National Institute of Standards and Technology (NIST) (2009)Google Scholar
  5. 5.
    Sakr, S., Liu, A., Batista, D.M., Alomari, M.: A survey of large scale data management approaches in cloud environments. IEEE Commun. Surv. Tutor. 13(3), 311–336 (2011)CrossRefGoogle Scholar
  6. 6.
    Dean, J., Ghemawat, S.: MapReduce: simplified data processing on large clusters. Commun. ACM 51(1), 107–133 (2008)CrossRefGoogle Scholar
  7. 7.
    Ghemawat, S., Gobioff, H., Leung, S.-T.: The Google file system. In: Proceedings of 19th ACM Symposium on Operating Systems Principles, pp. 29–43. ACM (2003)Google Scholar
  8. 8.
    Mika, P., Tummarello, G.: Web semantics in the clouds. IEEE Intell. Syst. 23(5), 82–87 (2008)CrossRefGoogle Scholar
  9. 9.
    Apache Hadoop. http://hadoop.apache.org/. Accessed Oct. 2014
  10. 10.
    Olston, C., Reed, B., Srivastava, U., Kumar, R., Tomkins, A.: PigLatin: a not-so-foreign language for data processing. In: Proceedings of ACM SIGMOD International Conference of Management of Data (2008)Google Scholar
  11. 11.
    Mahou. http://mahout.apache.org/. Accessed Oct 2014
  12. 12.
    Urbani, J., Kotoulas, S., Maassen, J., Harmelen, F.V., Bal, H.: WebPIE: a web-scale parallel inference engine using mapreduce. J. Web Semant. 10, 59–75 (2012)CrossRefGoogle Scholar
  13. 13.
    Berners-Lee, T., Hendler, J., Lassila, O.: The Semantic Web. Sci. Am. 284(5), 34–43 (2001)CrossRefGoogle Scholar
  14. 14.
    Ding, L., Xin, J., Wang, G., Huang, S.: Efficient skyline query processing of massive data based on MapReduce. Chin. J. Comput. 34(10), 1785–1796 (2011)CrossRefGoogle Scholar
  15. 15.
    Doulkeridis, C., Nørvåg, K.: A survey of large-scale analytical query processing in MapReduce. VLDB J. 23(3), 355–380 (2014)CrossRefGoogle Scholar
  16. 16.
    Li, F., Ooi, B.C., Özsu, M.T., Wu, S.: Distributed data management using MapReduce. ACM Comput. Surv. 46(3), 31 (2014)Google Scholar
  17. 17.
    Lee, K.H., Lee, Y.J., Choi, H., Chung, Y.D., Moon, B.: Parallel data processing with MapReduce: a survey. ACM SIGMOD Rec. 40(4), 11–20 (2011)CrossRefGoogle Scholar
  18. 18.
    Dean, J., Ghemawat, S.: MapReduce: a flexible data processing tool. Commun. ACM 53(1), 72–77 (2010)CrossRefGoogle Scholar
  19. 19.
    Van Biema, M.: Parallelism in Lisp. In: IJCAI87, pp. 56–61 (1987)Google Scholar
  20. 20.
    QT Concurrent. http://doc.qt.io/qt-5/qtconcurrent-index.html. Accessed Oct 2015
  21. 21.
    Ranger, C., Raghuraman, R., Penmetsa, A., Bradski, G., Kozyrakis, C.: Evaluating MapReduce for multi-core and multiprocessor systems. In: Proceedings of IEEE 13th International Symposium on High Performance Computer Architecture, pp. 13–24 (2007)Google Scholar
  22. 22.
    Talbot, J., Yoo, R.M., Kozyrakis, C.: Phoenix++: modular mapreduce for shared-memory systems. In: Proceedings of 2nd International Workshop on MapReduce and Its Applications, pp. 9–16 (2011)Google Scholar
  23. 23.
    Disco massive data—minimal code. http://discoproject.org/. Accessed Oct 2014
  24. 24.
    Geni, Skynet A Ruby MapReduce Framework. http://skynet.rubyforge.org/. Accessed Oct 2014
  25. 25.
    GridGain. http://www.gridgain.com/. Accessed Oct 2014
  26. 26.
    Ekanayake, J., Li, H., Zhang, B., Gunarathne, T., Bae, S.-H., Qiu, J., Fox, G.: Twister: a runtime for iterative MapReduce. In: Proceedings of First International Workshop on MapReduce and Its Applications of ACM, pp. 810–818. ACM (2010)Google Scholar
  27. 27.
    Dou, A.J., Kalogeraki, V., Gunopulos, D., Mielikainen, T., Tuulos, V.: Misco: a MapReduce Framework for mobile systems. In: Proceedings of 3rd International Conference on Pervasive Technologies Related to Assistive Environment (2010)Google Scholar
  28. 28.
    DeWitt, D., Stonebraker, M.: MapReduce: a major step backwards. The Database Column 1 (2008)Google Scholar
  29. 29.
    Sun, X., He, C., Lu, Y.: ESAMR: an enhanced self-adaptive MapReduce scheduling algorithm. In: Proceedings of the 2012 IEEE 18th International Conference on Parallel and Distributed Systems, pp. 148–155 (2012)Google Scholar
  30. 30.
    Husain, M.F., McGlothlin, J., Masud, M.M., Khan, L.R., Thuraisingham, B.: Heuristics-based query processing for large RDF graphs using cloud computing. IEEE Trans. Knowl. Data Eng. 23(9), 1312–1327 (2011)CrossRefGoogle Scholar
  31. 31.
    Sandholm, T., Lai, K.: MapReduce optimization using regulated dynamic prioritization. In: Proceedings of the 11th International Joint Conference on Measurement and Modeling of Computer Systems, pp. 299–310 (2009)Google Scholar
  32. 32.
    Seo, S., Jang, I., Woo, K., Kim, I., Kim, J.-S., Maeng, S.: HPMR: prefetching and pre-shuffling in shared MapReduce computation environment. In: Proceedings of 2009 IEEE International Conference on Cluster Computing and Workshops (2009)Google Scholar
  33. 33.
    Polo, J., Carrera, D., Becerra, Y., Torres, J., Ayguade, E., Steinder, M., Whalley, I.: Performance-driven task co-scheduling for MapReduce environments. In: Proceedings of 12th IEEE/IFIP Network Operations and Management Symposium, pp. 373–380 (2010)Google Scholar
  34. 34.
    Polo, J., Carrera, D., Becerra, Y., Beltran, V., Torres, J., Ayguade, E.: Performance management of accelerated MapReduce workloads in heterogeneous clusters. In: Proceedings of 39th International Conference on Parallel Processing, pp. 653–662 (2010)Google Scholar
  35. 35.
    Zaharia, M., Borthakur, D., Sarma, J.S., Elmeleegy, K., Shenker, S., Stoica, I.: Delay scheduling: a simple technique for achieving locality and fairness in cluster scheduling. In: EuroSyst, pp. 265–278 (2010)Google Scholar
  36. 36.
    Yao, Y., Tai, J., Sheng, B., Mi, N.: Scheduling heterogeneous MapReduce jobs for efficiency improvement in enterprise clusters. In: Proceedings of 2013 IFIP/IEEE International Symposium on Integrated Network Management, pp. 872–875 (2013)Google Scholar
  37. 37.
    Zaharia, M., Konwinski, A., Joseph, A. D., Katz, R., Stoica, I.: Improving MapReduce performance in heterogeneous environments. In: Proceedings of 8th USENIX Symposium on Operating System Design and Implementation, pp. 29–42 (2008)Google Scholar
  38. 38.
    Chen, Q., Zhang, D., Guo, M., Deng, Q., Guo, S.: SAMR: A self-adaptive MapReduce scheduling algorithm in heterogeneous environment. In: Proceedings of 10th IEEE International Conference on Computer and Information Technology, CIT-2010, 7th IEEE International Conference on Embedded Software and Systems, ICESS-2010, 10th IEEE International Conference on Scalable Computing and Communications, pp. 2736–2743 (2010)Google Scholar
  39. 39.
    Chen, Q., Guo, M.Y., Deng, Q.N., Zheng, L., Guo, S., Shen, Y.: HAT: history-based auto-tuning MapReduce in heterogeneous environments. J. Supercomput. 64(3), 1038–1054 (2013)CrossRefGoogle Scholar
  40. 40.
    Zhang, X., Feng, Y., Feng, S., Fan, J., Ming, Z.: An effective data locality aware task scheduling method for MapReduce framework in heterogeneous environments. In: Proceedings of 2011 International Conference on Cloud and Service Computing, pp. 235–242 (2011)Google Scholar
  41. 41.
    Ahmad, F., Chakradhar, S.T., Raghunathan, A., Vijaykumar, T.N.: Tarazu: optimizing MapReduce on heterogeneous clusters. In: Proceedings of 17th International Conference on Architectural Support for Programming Languages and Operating Systems, pp. 61–74 (2011)Google Scholar
  42. 42.
    Yang, Y., Shi, B., Jiang, B., Xiang, L.: Data-replicas scheduler for heterogeneous MapReduce cluster. J. Netw. 8(5), 1096–1103 (2013)Google Scholar
  43. 43.
    Sehrish, S., Mackey, G., Shang, P.J., Wang, J., Bent, J.: Supporting HPC analytics applications with access patterns using data restructuring and data-centric scheduling techniques in MapReduce. IEEE Trans. Parallel Distrib. Syst. 24(1), 158–169 (2013)CrossRefGoogle Scholar
  44. 44.
    Peng, D., Dabek, F., Inc, G.: Large-scale incremental processing using distributed transactions and notifications. In: Usenix Symposium on Operating Systems Design & Implementation, pp. 4–6 (2010)Google Scholar
  45. 45.
    Bu, X., Rao, J., Xu, C.: Interference and locality-aware task scheduling for MapReduce applications in virtual clusters. In: Proceedings of the 22nd ACM International Symposium on High-Performance Parallel and Distributed Computing, pp. 227–238 (2013)Google Scholar
  46. 46.
    Yang, H.-C., Dasdan, A., Hsiao, R.-L., Parker, D.S.: Map-Reduce-Merge: simplified relational data processing on large clusters. In: Proceedings of ACM SIGMOD International Conference on Management of Data, pp. 1029–1040 (2007)Google Scholar
  47. 47.
    Jiang, D., Tung, A.K.H., Chen, G.: Map–Join–Reduce: toward scalable and efficient data analysis on large clusters. IEEE Trans. Knowl. Data Eng. 23(9), 1299–1311 (2011)CrossRefGoogle Scholar
  48. 48.
    Ferrera, P., de Prado, I., Palacios, E., Fernandez-Marquez, J.L., Serugendo, G.D.: Tuple MapReduce: beyond classic MapReduce. In: Proceedings of 12th IEEE International Conference on Data Mining, pp. 260–269 (2012)Google Scholar
  49. 49.
    Vu, T.-T., Huet, F.: A lightweight continuous jobs mechanism for MapReduce frameworks. In: Proceedings of 13th IEEE/ACM International Symposium on Cluster, Cloud, and Grid Computing, pp. 269–276 (2013)Google Scholar
  50. 50.
    Premchaiswadi, W., Pomsaiyud, W.: Optimizing and tuning MapReduce jobs to improve the large-scale data analysis process. Int. J. Intell. Syst. 28(2), 185–200 (2013)CrossRefGoogle Scholar
  51. 51.
    Bu, Y., Howe, B., Balazinska, M., Ernst, M.D.: Haloop: efficient iterative data processing on large clusters. Proc. VLDB Endow. 3(1), 285–296 (2010)CrossRefGoogle Scholar
  52. 52.
    Kambatla, K., Rapolu, N., Jagannathan, S., Grama, A.: Asynchronous algorithms in MapReduce. In: Proceedings of 2010 IEEE International Conference on Cluster Computing, Cluster, pp. 245–254 (2010)Google Scholar
  53. 53.
    Elnikety, E., Elsayed, T., Ramadan, H.E.: IHadoop: asynchronous iterations for MapReduce. In: Proceedings of 2011 3rd IEEE International Conference on Cloud Computing Technology and Science, pp. 81–90 (2011)Google Scholar
  54. 54.
    Wang, L., Ni, Z., Zhang, Y., Wu, Z., Tang, L.: Pipelined-MapReduce: an improved MapReduce parallel programming model. In: Proceedings of 4th International Conference on Intelligent Computation Technology and Automation, pp. 871–874 (2011)Google Scholar
  55. 55.
    Xu, W., Gong, X.J., Li, X.Y.: MapCombine: a lightweight solution to improve the efficiency of iterative MapReduce. In: Proceedings of iCETS 2012, pp. 444–456Google Scholar
  56. 56.
    Zhang, Y.F., Gao, Q.X., Gao, L.X., Wang, C.R.: iMapReduce: a distributed computing framework for iterative computation. J. Grid Comput. 10(1), 47–68 (2012)CrossRefGoogle Scholar
  57. 57.
    Zaharia, M., Chowdhury, M., Das, T., Dave, A., Ma, J., Mccauley, M., Franklin, M., Shenker, S., Stoica, I.: Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing. In: USENIX Symposium on Networked Systems Design and Implementation, vol. 70, pp. 141–146 (2012)Google Scholar
  58. 58.
    Gunarathne, T., Zhang, B.J., Wu, T.L., Qiu, J.: Scalable parallel computing on clouds using Twister4Azure iterative MapReduce. Future Gener. Comput. Syst. 29(4), 1035–1048 (2013)CrossRefGoogle Scholar
  59. 59.
    Condie, T., Conway, N., Alvaro, P., Hellerstein, M., Elmeleegy, K., Sears, R.: MapReduce Online. EECS Department, University of California, Berkeley. Tech. Rep. http://www.eecs.berkeley.edu/Pubs/TechRpts/2009/EECS-2009-136.html
  60. 60.
    Böse, J. H., Andrzejak, A., Hogqvist, M.: Beyond online aggregation: Parallel and incremental data mining with online Map-Reduce. In: Proceedings of 2010 Workshop on Massive Data Analytics on the Cloud, MDAC 2010, in Association with the 19th Annual World Wide Web Conference (2010)Google Scholar
  61. 61.
    Phan, L.T.X., Zhang, Z., Loo, B.T., Lee, I.: Real-Time MapReduce Scheduling. Technical Report, University of Pennsylvania (2010)Google Scholar
  62. 62.
    Dou, A.J., Kalogeraki, V., Gunopulos, D., Mielikainen, T., Tuulos, V.: Scheduling for real-time mobile MapReduce systems. In: Proceedings of the 5th ACM International Conference on Distributed Event-Based Systems, pp. 247–258 (2011)Google Scholar
  63. 63.
    Peng, C.-Z., Jiang, Z.-J., Cai, X.-B., Zhang, Z.-K.: Real-time analytics processing with MapReduce. In: Proceedings of 2012 International Conference on Machine Learning and Cybernetics, vol. 4, pp. 1308–1311 (2012)Google Scholar
  64. 64.
    Qi, K., Zhao, Z., Fang, J., Ma, Q.: Real-time processing for high speed data stream over large scale data. Chin. J. Comput. 35(3), 477–490 (2012)CrossRefGoogle Scholar
  65. 65.
    Zaharia, M., Chowdhury, M., Franklin, M.J., Shenker, S., Stoica, I.: Spark: cluster computing with working sets. In: Proceedings of the 2nd USENIX Conference on Hot Topics in Cloud Computing (2010)Google Scholar
  66. 66.
    Hong, C., Chen, D., Chen, W., Zheng, W., Lin, H.: MapCG: Writing parallel program portable between CPU and GPU. In: Proceedings of 19th International Conference on Parallel Architecture and Compilation Techniques, pp. 217–226 (2010)Google Scholar
  67. 67.
    Jiang, W., Ravi, V.T., Agrawal, G.: A Map-reduce system with an Alternate API for multi-core environments. In: Proceedings of 10th IEEE/ACM International Symposium on Cluster, Cloud, and Grid Computing, pp. 84–93 (2010)Google Scholar
  68. 68.
    Stuart, J.A., Chen, C.-K., Ma, K.-L., Owens, J.D.: Multi-GPU volume rendering using MapReduce. In: Proceedings of 19th ACM International Symposium on High Performance Distributed Computing, pp. 841–848 (2010)Google Scholar
  69. 69.
    Fang, W., He, B., Luo, Q., Govindaraju, N.K.: Mars: accelerating MapReduce with graphics processors. IEEE Trans. Parallel Distrib. Syst. 22(4), 608–620 (2011)CrossRefGoogle Scholar
  70. 70.
    Elteir, M., Lin, H., Feng, W.-C., Scogland, T.: StreamMR: an optimized MapReduce framework for AMD GPUs. In: Proceedings of 17th IEEE International Conference on Parallel and Distributed Systems, pp. 364–371 (2011)Google Scholar
  71. 71.
    Chen, L., Agrawal, G.: Optimizing MapReduce for GPUs with effective shared memory usage. In: Proceedings of 21st ACM Symposium on High-Performance Parallel and Distributed Computing, pp. 199–210 (2012)Google Scholar
  72. 72.
    Basaran, C., Kang, K.D.: Grex: an efficient MapReduce framework for graphics processing units. J. Parallel Distrib. Comput. 73(4), 522–533 (2013)CrossRefGoogle Scholar
  73. 73.
    Chen, R., Chen, H.: Tiled-mapreduce: efficient and flexible mapreduce processing on multicore with tiling. Trans. Archit. Code Optim. 10, 1 (2013)Google Scholar
  74. 74.
    de Krujif, M., Sankaralingam, K.: MapReduce for the Cell B.E. architecture. IBM J. Res. Dev. 53(5), 10:1–10:12 (2009)Google Scholar
  75. 75.
    Rafique, M.M., Rose, B., Butt, A.R., Nikolopoulos, D.S.: Supporting MapReduce on large-scale asymmetric multi-core clusters. Oper. Syst. Rev. 43, 25–34 (2009)CrossRefGoogle Scholar
  76. 76.
    Papagiannis, A., Nikolopoulos, D.S.: Rearchitecting MapReduce for heterogeneous multicore processors with explicitly managed memories. In: Proceedings of 39th International Conference on Parallel Processing, pp. 121–130 (2010)Google Scholar
  77. 77.
    He, C., Weitzel, D., Swanson, D., Lu, Y.: HOG: Distributed Hadoop MapReduce on the grid. In: Proceedings of 2012 SC Companion: High Performance Computing, Networking Storage and Analysis, pp. 1276–1283 (2012)Google Scholar
  78. 78.
    Heintz, B., Wang, C., Chandra, A., Weissman, J.: Cross-phase optimization in mapreduce. In: Proceedings of 1st IEEE International Conference on Cloud Engineering, pp. 338–347 (2013)Google Scholar
  79. 79.
    Wang, L., Tao, J., Ranjan, R., Marten, H., Streit, A., Chen, J., Chen, D.: G-Hadoop: MapReduce across distributed data centers for data-intensive computing. Future Gener. Comput. Syst. 29(3), 739–750 (2013)CrossRefGoogle Scholar
  80. 80.
    Mantha, P.K., Luckow, A., Jha, S.: Pilot-MapReduce: an extensible and flexible MapReduce implementation for distributed data. In: Proceedings of 2012 3rd International Workshop on MapReduce and Its Applications, pp. 17–24 (2012)Google Scholar
  81. 81.
    Marozzo, F., Talia, D., Trunfio, P.: P2P-MapReduce: parallel data processing in dynamic cloud environments. J. Comput. Syst. Sci. 78(5), 1382–1402 (2012)CrossRefGoogle Scholar
  82. 82.
    Wang, G., Butt, A.R., Pandey, P., Gupta, K.: Using realistic simulation for performance analysis of MapReduce setups. In: Proceedings of 1st ACM Workshop on Large-Scale System and Application Performance, pp. 16–29 (2009)Google Scholar
  83. 83.
    Hammoud, S., Li, M., Liu, Y., Alham, N.K., Liu, Z.: MRSim: a discrete event based MapReduce simulator. In: Proceedings of 2010 Seventh International Conference on Fuzzy Systems and Knowledge Discovery, pp. 2993–2997 (2010)Google Scholar
  84. 84.
    Huang, D., Shi, X., Ibrahim, S., Lu, L., Liu, H., Wu, S., Jin, H.: MR-Scope: a real-time tracing tool for MapReduce. In: Proceedings of 19th ACM International Symposium on High Performance Distributed Computing, pp. 849–855 (2010)Google Scholar
  85. 85.
    Teng, F., Yu, L., Magoules, F.: SimMapReduce: a simulator for modeling MapReduce framework. In: Proceedings of the 2011 5th FTRA International Conference on Multimedia and Ubiquitous Engineering, pp. 277–282 (2011)Google Scholar
  86. 86.
    Wang, K., Lin, X., Tang, W.: Predator—an experience guided configuration optimizer for Hadoop MapReduce. In: Proceedings of 4th IEEE International Conference on Cloud Computing Technology and Science, pp. 419–426 (2012)Google Scholar
  87. 87.
    Yang, H.L., Luan, Z.Z., Li, W.J., Qian, D.P.: MapReduce workload modeling with statistical approach. J. Grid Comput. 10(2), 279–310 (2012)CrossRefGoogle Scholar
  88. 88.
    Kolberg, W., Marcos, P.D., Anjos, J.C.S., Miyazaki, A.K.S., Geyer, C.R., Arantes, L.B.: MRSG—a MapReduce simulator over SimGrid. Parallel Comput. 39(4–5), 233–244 (2013)CrossRefGoogle Scholar
  89. 89.
    Liu, Y., Li, M.Z., Alham, N.K., Hammoud, S.: HSim: a MapReduce simulator in enabling cloud computing. Future Gener. Comput. Syst. 29(1), 300–308 (2013)CrossRefGoogle Scholar
  90. 90.
    Vianna, E., Comarela, G., Pontes, T., Almeida, J., Almeida, V., Wilkinson, K., Kuno, H., Dayal, U.: Analytical performance models for MapReduce workloads. Int. J. Parallel Prog. 41(4), 495–525 (2013)CrossRefGoogle Scholar
  91. 91.
    Leverich, J., Kozyrakis, C.: On the energy (In) efficiency of Hadoop Clusters. In: Proceedings of HotPower (2009)Google Scholar
  92. 92.
    Lang, W., Patel, J.: Energy management for MapReduce clusters. In: VLDB (2010)Google Scholar
  93. 93.
    Kaushik, R.T., et al.: Evaluation and analysis if GreenHDFS: a selft-adaptive. Energy-conserving variant of the Hadoop distributed file system. In: Proceedings of IEEE CloudCom (2010)Google Scholar
  94. 94.
    Chen, Y., Alspaugh, S., Borthakur, D., Katz, R.: Energy efficiency for large-scale MapReduce workloads with significant interactive analysis. In: Proceedings of EuroSys (2012)Google Scholar
  95. 95.
    Yoon, E., Squicciarini, A.: Toward detecting compromised MapReduce workers through log analysis. In: Proceedings of 14th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (2014)Google Scholar
  96. 96.
    Roy, I., Setty, S.T.V., Kilzer, A., Shmatikov, V., Witchel, E.: Airavat: security and privacy for MapReduce. In: NSDI (2010)Google Scholar
  97. 97.
    Wei, W., Du, J., Yu, T., Gu, X.: SecureMR: a service integrity assurance framework for MapReduce. In: Proceedings of 2009 Annual Computer Security Applications Conference (2009)Google Scholar
  98. 98.
    Chen, X., Huang, Q.: The data protection of MapReduce using homomorphic encryption. In: Proceedings of 4th IEEE International Conference on Software Engineering and Service Science, pp. 419–421 (2013)Google Scholar
  99. 99.
    Guo, Z., Zhu, X., Guo, L., Kang, S.: Design of a security framework On MapReduce. In: Proceedings of 5th International Conference on Intelligent Networking and Collaborative Systems, pp. 139–145 (2013)Google Scholar
  100. 100.
    Han, H., Zheng, W.: A privacy data-oriented hierarchical MapReduce programming model. Telkomnika Indones. J. Electr. Eng. 11(8), 4587–4593 (2013)Google Scholar
  101. 101.
    Zhang, C., Chang, E.-C., Yap, R.H.C.: Tagged-MapReduce: a general framework for secure computing with mixed-sensitivity data on hybrid clouds. In: Proceedings of 14th IEEE/ACM International Symposium on Cluster, Cloud, and Grid Computing (2014)Google Scholar
  102. 102.
    Apache Pig. http://pig.apache.org/. Accessed May 2015
  103. 103.
    Cascading. http://www.cascading.org/. Accessed May 2015
  104. 104.
    Scalding. http://www.cascading.org/projects/scalding/. Accessed May 2015
  105. 105.
    Pike, R., Dorward, S., Griesemer, R., Quinlan, S.: Interpreting the data: parallel analysis with Sawzall. Sci. Program. 13(4), 277–298 (2005)Google Scholar
  106. 106.
    Summingbird. https://github.com/twitter/summingbird. Accessed May 2015
  107. 107.
    Akidau, T., Balikov, A., Chernyak, S., Haberman, J., Lax, R., Mcveety, S., Mills, D., Nordstrom, P., Whittle, S.: MillWheel: faulttolerant stream processing at Internet scale. In: Proceedings of the 39th International Conference on Very Large Data Bases, VLDB, vol. 6 (2013)Google Scholar
  108. 108.
    Lu, M., Zhang, L., Huynh, H.P., Ong, Z., Liang, Y., He, B., Goh, R.S.M., Huynh, R.: Optimizing the MapReduce framework on Intel Xeon Phi coprocessor. In: Proceedings of IEEE International Conference on Big Data (2013)Google Scholar
  109. 109.
    QT. http://www.qt.io/download/. Accessed Oct 2015

Copyright information

© Springer Science+Business Media New York 2015

Authors and Affiliations

  • Ren Li
    • 1
    Email author
  • Haibo Hu
    • 2
  • Heng Li
    • 2
  • Yunsong Wu
    • 3
  • Jianxi Yang
    • 1
  1. 1.College of Information Science and EngineeringChongqing Jiaotong UniversityChongqingChina
  2. 2.School of Software EngineeringChongqing UniversityChongqingChina
  3. 3.College of Computer ScienceChongqing UniversityChongqingChina

Personalised recommendations