The OpenFAM API: A Programming Model for Disaggregated Persistent Memory

  • Kimberly KeetonEmail author
  • Sharad Singhal
  • Michael Raymond
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11283)


Recent technology advances in high-density, byte-addressable non-volatile memory (NVM) and low-latency interconnects have enabled building large-scale systems with a large disaggregated fabric-attached memory (FAM) pool shared across decentralized compute nodes. The OpenFAM API is an API for programming with persistent FAM that is inspired by the OpenSHMEM partitioned global address space (PGAS) model. Unlike OpenSHMEM, where each node contributes local memory toward a logically shared global address space, FAM isn’t associated with a particular node and can be addressed directly from any node without the cooperation or involvement of another node. The OpenFAM API enables programmers to manage FAM allocations, access FAM-resident data structures, and order FAM operations. Because state in FAM can survive program termination, the API also provides interfaces for naming and managing data beyond the lifetime of a single program invocation.


Disaggregated memory Fabric-attached memory Persistent memory Non-volatile memory Gen-Z 


  1. 1.
    Asanovic, K.: FireBox: a hardware building block for 2020 warehouse-scale computers. Keynote at USENIX Conference on File and Storage Technologies (FAST) 2014Google Scholar
  2. 2.
    Binnig, C., Crotty, A., Galakatos, A., Kraska, T., Zamanian, E.: The end of slow networks: it’s time for a redesign. Proc. VLDB Endow. 9(7), 528–539 (2016)CrossRefGoogle Scholar
  3. 3.
    Cavium, ThunderX2 ARM Processors. Accessed 22 July 2018
  4. 4.
    Cray: Cray XC Series GNI and DMAPP API User Guide (CLE 6.0.UP05) S-2446. Accessed 28 July 2018
  5. 5.
    Dulloor, S.R., et al.: Data tiering in heterogeneous memory systems. In: Proceedings of 11th ACM European Conference on Computer Systems (EuroSys) (2016)Google Scholar
  6. 6.
    Faraboschi, P., Keeton, K., Marsland, T., Milojicic, D.: Beyond processor-centric operating systems. In: Proceedings of Workshop on Hot Topics in Operating Systems (HotOS) (2015)Google Scholar
  7. 7.
    Funk, M.: Drilling down into the machine from HPE. The Next Platform, 4 January 2016. Accessed 22 July 2018
  8. 8.
    Funk, M.: Drilling into the CCIX coherence standard. The Next Platform, 13 July 2016. Accessed 22 July 2018
  9. 9.
    Gen-Z Consortium: Gen-Z Atomics, October 2017. Accessed 25 July 2018
  10. 10.
    Gen-Z Consortium: Gen-Z Coherency, October 2017. Accessed 25 July 2018
  11. 11.
    Gen-Z Consortium: Gen-Z Core Specification. Accessed 22 July 2018
  12. 12.
    Gen-Z Consortium: Gen-Z ZMMU and Memory Interleave, June 2018. Accessed 25 July 2018
  13. 13.
    Hewlett Packard Enterprise: HPE Persistent Memory: The Performance of Memory with the Persistence of Storage. Accessed 25 July 2018
  14. 14.
    Hewlett Packard Enterprise: New HPE Pointnext Capabilities Accelerate Transition to Memory-Driven Computing. Accessed 22 July 2018
  15. 15.
    Huawei: High Throughput Computing Data Center Architecture: Thinking of Data Center 3.0. Accessed 25 July 2018
  16. 16.
    Intel: Facebook Collaborate on Future Data Center Rack Technologies, Intel Newsroom, 16 January 2013. Accessed 25 July 2018
  17. 17.
    Intel: Intel Omni-Path Architecture (Intel OPA) Driving Exascale Computing and HPC. Accessed 25 July 2018
  18. 18.
    Intel Rack Scale Design: Data Center Agility at Scale. Accessed 25 July 2018
  19. 19.
    High Bandwidth Memory (HBM) DRAM, JEDEC Standard JESD235A, November 2015. Accessed 25 July 2018
  20. 20.
    Keeton, K.: Memory-driven computing. Keynote at 33rd International Conference on Massive Storage Systems and Technology (MSST), May 2017.
  21. 21.
    Kerrisk, M.: Linux/UNIX System Programming: POSIX Shared Memory, February 2015. Accessed 30 July 2018
  22. 22.
    Lim, K., et al.: System-level implications of disaggregated memory. In: Proceedings of International Symposium on High Performance Computer Architecture (HPCA) (2012)Google Scholar
  23. 23.
    Lockwood, G.: What’s so bad about POSIX I/O?. The Next Platform, 11 September 2017. Accessed 29 July 2018
  24. 24.
    Loesing, S., Pilman, M., Etter, T., Kossmann, D.: On the design and scalability of distributed shared-data databases. In: Proceedings of 15th ACM SIGMOD International Conference on Management of Data (SIGMOD), pp. 663–676 (2015)Google Scholar
  25. 25.
    Novakovic, S., Daglis, A., Bugnion, E., Falsafi, B., Grot, B.: Scale-out NUMA. In: Proceedings of Symposium on Architectural Support for Programming Languages and Operating Systems (ASPLOS) (2014)Google Scholar
  26. 26.
    OpenCAPI Overview, 14 October 2016. Accessed 25 July 2018
  27. 27.
    OpenSHMEM API Version 1.4. Accessed 25 July 2018
  28. 28.
    The OpenFAM API. Accessed 27 June 2018
  29. 29. Persistent Memory Programming. Accessed 29 July 2018
  30. 30.
    Qureshi, M.K., Srinivasan, V., Rivers, J.A.: Scalable high performance main memory system using phase-change memory technology. In: Proceedings of 36th International Symposium on Computer Architecture (ISCA), pp. 24–33 (2009)Google Scholar
  31. 31.
    RDMA Consortium: Accessed 25 July 2018
  32. 32.
    Seyedi, M.A., Fiorentino, M.: Silicon photonics; ring modulator transmitters. In: Guenther, R., Steel, D. (eds.) Encyclopedia of Modern Optics, 2nd edn, vol. 4, pp. 216–223 (2018)Google Scholar
  33. 33.
    Shah, H., Marti, F., Noureddine, W., Eiriksson, A., Sharp, R.: Remote Direct Memory Access (RDMA) Protocol Extensions, IETF RFC 7306, June 2014. Accessed 25 July 2018
  34. 34.
    Spelman, L.: Reimagining the Data Center Memory and Storage Hierarchy, Intel Newsroom, 30 May 2018. Accessed 25 July 2018
  35. 35.
    Storage Networking Industry Association (SNIA): NVM Programming Model (NPM), Version 1.2. Accessed 29 July 2018
  36. 36.
    Strukov, D.B., Snider, G.S., Steward, D.R., Williams, R.S.: The missing memristor found. Nature 453, 80–83 (2008)CrossRefGoogle Scholar
  37. 37.
    Xie, Y.: Modeling, architecture and applications for emerging memory technologies. IEEE Design and Test of Computers 28(1), 44–51 (2011)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Kimberly Keeton
    • 1
    Email author
  • Sharad Singhal
    • 1
  • Michael Raymond
    • 1
  1. 1.Hewlett Packard EnterprisePalo AltoUSA

Personalised recommendations