Journal of Signal Processing Systems

, Volume 87, Issue 3, pp 357–369 | Cite as

An FPGA-based Integrated MapReduce Accelerator Platform

  • Christoforos KachrisEmail author
  • Dionysios Diamantopoulos
  • Georgios Ch. Sirakoulis
  • Dimitrios Soudris


MapReduce is a programming framework for distributed systems that is used to automatically parallelize and schedule the tasks to distributed resources. MapReduce is widely used in data centers to process enterprise databases and Big Data. This paper presents a novel MapReduce accelerator platform based on FPGAs that can be used to speedup the processing of the MapReduce data. The proposed platform consists of specialized hardware accelerators for the Map tasks and a shared configurable accelerator for the Reduce tasks. The hardware accelerators for the Map tasks are developed using a modified source-to-source High-level Synthesis (HLS) tool while the Reduce accelerator is based on a novel hashing scheme. The proposed scheme is implemented, mapped and evaluated to a Virtex 7 FGPA. The performance evaluation is based on a benchmark suite that represent typical MapReduce applications and it shows that the proposed scheme can achieve up to 2 orders of magnitude energy reduction compared to General Purpose Processors (GPPs).


MapReduce Accelerator Data center FPGAs Reconfigurable computing 


  1. 1.
    SMART 2020. Enabling the low carbon economy in the information age. A report by The Climate Group on behalf of the Global eSustainability Initiative (GeSI).Google Scholar
  2. 2.
    Where does power go? (2008). GreenDataProject, available online at:
  3. 3.
    Make IT Green (2010). Cloud computing and its contribution to climate change. Greenpeace.Google Scholar
  4. 4.
    Blott, M., Karras, K., Liu, L., Vissers, K., Bär, J., & István, Z. (2013). Achieving 10gbps line-rate key-value stores with fpgas. In Presented as part of the 5th USENIX workshop on hot topics in cloud computing.Google Scholar
  5. 5.
    Chen, L., Huo, X., & Agrawal, G. (2012). Accelerating mapreduce on a coupled cpu-gpu architecture. In Proceedings of the international conference on high performance computing, networking, storage and analysis, SC ’12 (pp. 25:1–25:11).Google Scholar
  6. 6.
    Dean, J., & Ghemawat, S. (2008). Mapreduce: simplified data processing on large clusters. Commununications of the ACM, 51(1), 107–113.CrossRefGoogle Scholar
  7. 7.
    Dionysis Diamantopoulos, C.K., & Soudris, D. (2015). High-level Synthesizable Dataflow MapReduce Accelerator for FPGA-coupled Data Centers. In Proceedings of the international conference on embedded computer systems: architectures, Modeling and Simulation (SAMOS) (pp. 13–24).Google Scholar
  8. 8.
    Hoelzle, U., & Barroso, L.A. (2009). The datacenter as a computer: An introduction to the design of warehouse-scale machines. Morgan and Claypool Publishers 1st edition.Google Scholar
  9. 9.
    István, Z., Alonso, G., Blott, M., & Vissers, K. (2015). A hash table for line-rate data processing. ACM Trans Reconfigurable Technology Systems, 8(2), 13:1–13:15.CrossRefGoogle Scholar
  10. 10.
    Kachris, C., Sirakoulis, G., & Soudris, D. (2014). A Reconfigurable MapReduce accelerator for multi-core all-programmable SoCs. In Proceedings of the international symposium on system-on-chip (soc) (pp. 1–6).Google Scholar
  11. 11.
    Pagh, R. & Rodler, F.F. (2001). Cuckoo Hashing. In Proceedings of ESA 2001, Lecture Notes in Computer Science, 2161.Google Scholar
  12. 12.
    Putnam, A., Caulfield, A., Chung, E., Chiou, D., Constantinides, K., Demme, J., Esmaeilzadeh, H., Fowers, J., Gopal, G.P., Gray, J., Haselman, M., Hauck, S., Heil, S., & Burger, D. (2014). A reconfigurable fabric for accelerating large-scale datacenter services. In 41St annual international symposium on computer architecture (ISCA).Google Scholar
  13. 13.
    Ranger, C., Raghuraman, R., Penmetsa, A., Bradski, G., & Kozyrakis, C. (2007). Evaluating MapReduce for Multi-core and Multiprocessor Systems. In Proceedings of the 2007 IEEE 13th international symposium on high performance computer architecture, HPCA ’07 (pp. 13–24).Google Scholar
  14. 14.
    Ranger, C., Raghuraman, R., Penmetsa, A., Bradski, G., & Kozyrakis, C. (2007). Evaluating mapreduce for multi-core and multiprocessor systems. In IEEE 13th international symposium on High performance computer architecture, 2007. HPCA 2007 (pp. 13–24).Google Scholar
  15. 15.
    Shan, Y., Wang, B., Yan, J., Wang, Y., Xu, N., & Yang, H. (2010). FPMR: Mapreduce Framework on FPGA: A Case Study of RankBoost Acceleration. In Proceedings of the international symposium on field programmable gate arrays (FPGA) (pp. 93–102).Google Scholar
  16. 16.
    Tsoi, K.H., & Axel, W.L. (2010). A heterogeneous cluster with fpgas and gpus. In Proceedings of the 18th annual ACM/SIGDA international symposium on field programmable gate arrays, FPGA ’10 (pp. 115–124).Google Scholar
  17. 17.
    Veen, A.H. (1986). Dataflow machine architecture. ACM Computing Surveys, 18(4), 365–396.CrossRefGoogle Scholar
  18. 18.
    Yin, D., Li, G., & Huang, K.-d. (2012) Scalable mapreduce framework on fpga accelerated commodity hardware. In S. Andreev, S. Balandin & Y. Koucheryavy (Eds.), Internet of Things, Smart Spaces, and Next Generation Networking, volume 7469 of Lecture Notes in Computer Science, (pp. 280–294). Berlin : Springer.Google Scholar

Copyright information

© Springer Science+Business Media New York 2016

Authors and Affiliations

  • Christoforos Kachris
    • 1
    Email author
  • Dionysios Diamantopoulos
    • 1
  • Georgios Ch. Sirakoulis
    • 2
  • Dimitrios Soudris
    • 1
  1. 1.School of Electrical and Computer EngineeringNational Technical University of AthensAthensGreece
  2. 2.Department of Electrical and Computer EngineeringDemocritus University of ThraceXanthiGreece

Personalised recommendations