Skip to main content

An FPGA-based Integrated MapReduce Accelerator Platform

Journal of Signal Processing Systems volume 87, pages 357–369 (2017)Cite this article

Abstract

MapReduce is a programming framework for distributed systems that is used to automatically parallelize and schedule the tasks to distributed resources. MapReduce is widely used in data centers to process enterprise databases and Big Data. This paper presents a novel MapReduce accelerator platform based on FPGAs that can be used to speedup the processing of the MapReduce data. The proposed platform consists of specialized hardware accelerators for the Map tasks and a shared configurable accelerator for the Reduce tasks. The hardware accelerators for the Map tasks are developed using a modified source-to-source High-level Synthesis (HLS) tool while the Reduce accelerator is based on a novel hashing scheme. The proposed scheme is implemented, mapped and evaluated to a Virtex 7 FGPA. The performance evaluation is based on a benchmark suite that represent typical MapReduce applications and it shows that the proposed scheme can achieve up to 2 orders of magnitude energy reduction compared to General Purpose Processors (GPPs).

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Figure 1
Figure 2
Figure 3
Figure 4
Figure 5
Figure 6
Figure 7

Notes

  1. https://01.org/powertop

  2. http://www.xilinx.com/products/design_tools/logic_design/verification/xpower.htm

References

  1. SMART 2020. Enabling the low carbon economy in the information age. A report by The Climate Group on behalf of the Global eSustainability Initiative (GeSI).

  2. Where does power go? (2008). GreenDataProject, available online at: http://www.greendataproject.org.

  3. Make IT Green (2010). Cloud computing and its contribution to climate change. Greenpeace.

  4. Blott, M., Karras, K., Liu, L., Vissers, K., Bär, J., & István, Z. (2013). Achieving 10gbps line-rate key-value stores with fpgas. In Presented as part of the 5th USENIX workshop on hot topics in cloud computing.

  5. Chen, L., Huo, X., & Agrawal, G. (2012). Accelerating mapreduce on a coupled cpu-gpu architecture. In Proceedings of the international conference on high performance computing, networking, storage and analysis, SC ’12 (pp. 25:1–25:11).

  6. Dean, J., & Ghemawat, S. (2008). Mapreduce: simplified data processing on large clusters. Commununications of the ACM, 51(1), 107–113.

    Article  Google Scholar 

  7. Dionysis Diamantopoulos, C.K., & Soudris, D. (2015). High-level Synthesizable Dataflow MapReduce Accelerator for FPGA-coupled Data Centers. In Proceedings of the international conference on embedded computer systems: architectures, Modeling and Simulation (SAMOS) (pp. 13–24).

  8. Hoelzle, U., & Barroso, L.A. (2009). The datacenter as a computer: An introduction to the design of warehouse-scale machines. Morgan and Claypool Publishers 1st edition.

  9. István, Z., Alonso, G., Blott, M., & Vissers, K. (2015). A hash table for line-rate data processing. ACM Trans Reconfigurable Technology Systems, 8(2), 13:1–13:15.

    Article  Google Scholar 

  10. Kachris, C., Sirakoulis, G., & Soudris, D. (2014). A Reconfigurable MapReduce accelerator for multi-core all-programmable SoCs. In Proceedings of the international symposium on system-on-chip (soc) (pp. 1–6).

  11. Pagh, R. & Rodler, F.F. (2001). Cuckoo Hashing. In Proceedings of ESA 2001, Lecture Notes in Computer Science, 2161.

  12. Putnam, A., Caulfield, A., Chung, E., Chiou, D., Constantinides, K., Demme, J., Esmaeilzadeh, H., Fowers, J., Gopal, G.P., Gray, J., Haselman, M., Hauck, S., Heil, S., & Burger, D. (2014). A reconfigurable fabric for accelerating large-scale datacenter services. In 41St annual international symposium on computer architecture (ISCA).

  13. Ranger, C., Raghuraman, R., Penmetsa, A., Bradski, G., & Kozyrakis, C. (2007). Evaluating MapReduce for Multi-core and Multiprocessor Systems. In Proceedings of the 2007 IEEE 13th international symposium on high performance computer architecture, HPCA ’07 (pp. 13–24).

  14. Ranger, C., Raghuraman, R., Penmetsa, A., Bradski, G., & Kozyrakis, C. (2007). Evaluating mapreduce for multi-core and multiprocessor systems. In IEEE 13th international symposium on High performance computer architecture, 2007. HPCA 2007 (pp. 13–24).

  15. Shan, Y., Wang, B., Yan, J., Wang, Y., Xu, N., & Yang, H. (2010). FPMR: Mapreduce Framework on FPGA: A Case Study of RankBoost Acceleration. In Proceedings of the international symposium on field programmable gate arrays (FPGA) (pp. 93–102).

  16. Tsoi, K.H., & Axel, W.L. (2010). A heterogeneous cluster with fpgas and gpus. In Proceedings of the 18th annual ACM/SIGDA international symposium on field programmable gate arrays, FPGA ’10 (pp. 115–124).

  17. Veen, A.H. (1986). Dataflow machine architecture. ACM Computing Surveys, 18(4), 365–396.

    Article  Google Scholar 

  18. Yin, D., Li, G., & Huang, K.-d. (2012) Scalable mapreduce framework on fpga accelerated commodity hardware. In S. Andreev, S. Balandin & Y. Koucheryavy (Eds.), Internet of Things, Smart Spaces, and Next Generation Networking, volume 7469 of Lecture Notes in Computer Science, (pp. 280–294). Berlin : Springer.

Download references

Author information

Authors and Affiliations

  1. School of Electrical and Computer Engineering, National Technical University of Athens, Athens, Greece

    Christoforos Kachris, Dionysios Diamantopoulos & Dimitrios Soudris

  2. Department of Electrical and Computer Engineering, Democritus University of Thrace, Xanthi, Greece

    Georgios Ch. Sirakoulis

Authors
  1. Christoforos Kachris
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Dionysios Diamantopoulos
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Georgios Ch. Sirakoulis
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Dimitrios Soudris
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Christoforos Kachris.

Additional information

This paper was partially supported by the GreenCenter project (http://green-center.weebly.com/) and the Horizon 2020 EU-funded project #644906, AEGLE, www.aegle-uhealth.eu/

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Kachris, C., Diamantopoulos, D., Sirakoulis, G.C. et al. An FPGA-based Integrated MapReduce Accelerator Platform. J Sign Process Syst 87, 357–369 (2017). https://doi.org/10.1007/s11265-016-1108-7

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11265-016-1108-7

Keywords

  • MapReduce
  • Accelerator
  • Data center
  • FPGAs
  • Reconfigurable computing