Supporting Columnar In-memory Formats on FPGA: The Hardware Design of Fletcher for Apache Arrow

  • Johan PeltenburgEmail author
  • Jeroen van Straten
  • Matthijs Brobbel
  • H. Peter Hofstee
  • Zaid Al-Ars
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11444)


As a columnar in-memory format, Apache Arrow has seen increased interest from the data analytics community. Fletcher is a framework that generates hardware interfaces based on this format, to be used in FPGA accelerators. This allows efficient integration of FPGA accelerators with various high-level software languages, while providing an easy-to-use hardware interface for the FPGA developer. The abstract descriptions of data sets stored in the Arrow format, that form the input of the interface generation step, can be complex. To generate efficient interfaces from it is challenging. In this paper, we introduce the hardware components of Fletcher that help solve this challenge. These components allow FPGA developers to express access to complex Arrow data records through row indices of tabular data sets, rather than through byte addresses. The data records are delivered as streams of the same abstract types as found in the data set, rather than as memory bus words. The generated interfaces allow for full system bandwidth to be utilized and have a low area profile. All components are open sourced and available for other researchers and developers to use in their projects.


FPGA Apache Arrow Fletcher 


  1. 1.
    Amazon Web Services: AWS EC2 FPGA Hardware and Software Development Kits (2018).
  2. 2.
    Gingold, T.: GHDL VHDL 2008/93/87 simulator (2018).
  3. 3.
    McKinney, W.: Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython. O’Reilly Media Inc., Newton (2012)Google Scholar
  4. 4.
    OpenPOWER foundation: CAPI SNAP Framework Hardware and Software (2018).
  5. 5.
    Owaida, M., Sidler, D., Kara, K., Alonso, G.: Centaur: a framework for hybrid CPU-FPGA databases. In: 2017 IEEE 25th Annual International Symposium on Field-Programmable Custom Computing Machines (FCCM), pp. 211–218, April 2017Google Scholar
  6. 6.
    Peltenburg, J., van Straten, J.: Fletcher: a framework to integrate Apache Arrow with FPGA accelerators (2018).
  7. 7.
    Sidler, D., István, Z., Owaida, M., Alonso, G.: Accelerating pattern matching queries in hybrid CPU-FPGA architectures. In: Proceedings of the 2017 ACM International Conference on Management of Data, SIGMOD 2017, pp. 403–415. ACM, New York (2017)Google Scholar
  8. 8.
    Stuecheli, J., Blaner, B., Johns, C., Siegel, M.: CAPI: a coherent accelerator processor interface. IBM J. Res. Dev. 59(1), 7:1–7:7 (2015)CrossRefGoogle Scholar
  9. 9.
    The Apache Software Foundation: Apache Arrow (2018).
  10. 10.
    The Apache Software Foundation: Apache Parquet (2018).
  11. 11.
    Winterstein, F., Bayliss, S., Constantinides, G.A.: High-level synthesis of dynamic data structures: a case study using Vivado HLS. In: 2013 International Conference on Field-Programmable Technology (FPT), pp. 362–365, December 2013Google Scholar
  12. 12.
    Zaharia, M., et al.: Apache spark: a unified engine for big data processing. Commun. ACM 59(11), 56–65 (2016)CrossRefGoogle Scholar
  13. 13.
    Zhang, H., Chen, G., Ooi, B.C., Tan, K.L., Zhang, M.: In-memory big data management and processing: a survey. IEEE Trans. Knowl. Data Eng. 27(7), 1920–1948 (2015)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Johan Peltenburg
    • 1
    Email author
  • Jeroen van Straten
    • 1
  • Matthijs Brobbel
    • 1
  • H. Peter Hofstee
    • 2
  • Zaid Al-Ars
    • 1
  1. 1.Delft University of TechnologyDelftThe Netherlands
  2. 2.IBM ResearchAustinUSA

Personalised recommendations