Skip to main content
SpringerLink
Log in

Scaling out NUMA-Aware Applications with RDMA-Based Distributed Shared Memory

  • Regular Paper
  • Published:
Journal of Computer Science and Technology Aims and scope Submit manuscript

Abstract

The multicore evolution has stimulated renewed interests in scaling up applications on shared-memory multiprocessors, significantly improving the scalability of many applications. But the scalability is limited within a single node; therefore programmers still have to redesign applications to scale out over multiple nodes. This paper revisits the design and implementation of distributed shared memory (DSM) as a way to scale out applications optimized for non-uniform memory access (NUMA) architecture over a well-connected cluster. This paper presents MAGI, an efficient DSM system that provides a transparent shared address space with scalable performance on a cluster with fast network interfaces. MAGI is unique in that it presents a NUMA abstraction to fully harness the multicore resources in each node through hierarchical synchronization and memory management. MAGI also exploits the memory access patterns of big-data applications and leverages a set of optimizations for remote direct memory access (RDMA) to reduce the number of page faults and the cost of the coherence protocol. MAGI has been implemented as a user-space library with pthread-compatible interfaces and can run existing multithreaded applications with minimized modifications. We deployed MAGI over an 8-node RDMAenabled cluster. Experimental evaluation shows that MAGI achieves up to 9.25x speedup compared with an unoptimized implementation, leading to a scalable performance for large-scale data-intensive applications.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Dice D, Marathe V J, Shavit N. Lock cohorting: A general technique for designing NUMA locks. In Proc. the 17th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, February 2012, pp.247-256.

  2. Calciu I, Dice D, Lev Y, Luchangco V, Marathe V J, Shavit N. Numa-aware reader-writer locks. In Proc. the 18th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, February 2013, pp.157-166.

  3. Boyd-Wickizer S, Kaashoek M F, Morris R, Zeldovich N. OpLog: A library for scaling update-heavy data structures. Technical Report, Massachusetts Institute of Technology, 2014. https://dspace.mit.edu/handle/1721.1/89653, September 2018.

  4. Majo Z, Gross T R. (Mis)understanding the NUMA memory system performance of multithreaded workloads. In Proc. the 2013 IEEE International Symposium on Workload Characterization, September 2013, pp.11-22.

  5. Majo Z, Gross T. R. A library for portable and composable data locality optimizations for NUMA systems. In Proc. the 20th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, February 2015, pp.227-238.

  6. Zhang K, Chen R, Chen H. NUMA-aware graph-structured analytics. In Proc. the 20th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, February 2015, pp.183-193.

  7. Calciu I, Sen S, Balakrishnan M, Aguilera M K. Blackbox concurrent data structures for NUMA architectures. In Proc. the 22nd International Conference on Architectural Support for Programming Languages and Operating Systems, April 2017, pp.207-221.

  8. Li K, Hudak P. Memory coherence in shared virtual memory systems. ACM Transactions on Computer Systems, 1989, 7(4): 321-359.

    Article  Google Scholar 

  9. Bennett J K, Carter J B, Zwaenepoel W. Munin: Distributed shared memory based on type-specific memory coherence. In Proc. the 2nd ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, March 1990, pp.168-176.

  10. Chapman M, Heiser G. vNUMA: A virtual shared-memory multiprocessor. In Proc. the 2009 USENIX Annual Technical Conference, June 2009, pp.349-362.

  11. Keleher P, Cox A L, Dwarkadas S, Zwaenepoel W. Tread-Marks: Distributed shared memory on standard workstations and operating systems. In Proc. the USENIX Winter 1994 Technical Conference, January 1994, pp.115-132.

  12. Fleisch B, Popek G. Mirage: A coherent distributed shared memory design. In Proc. the 12th ACM Symposium on Operating Systems Principles, December 1989, pp.211-223.

  13. Iftode L, Singh J P, Li K. Scope consistency: A bridge between release consistency and entry consistency. In Proc. the 8th Annual ACM Symposium on Parallel Algorithms and Architectures, June 1996, pp.277-287.

  14. Bershad B N, Zekauskas M J, Sawdon W A. The Midway distributed shared memory system. In Proc. Digest of Papers. Compcon Spring, February 1993, pp.528-537.

  15. Erlichson A, Nuckolls N, Chesson G, Hennessy J. Soft-FLASH: Analyzing the performance of clustered distributed virtual shared memory. In Proc. the 7th International Conference on Architectural Support for Programming Languages and Operating Systems, October 1996, pp.210-220.

  16. Stets R, Dwarkadas S, Hardavellas N, Hunt G, Kontothanassis L, Parthasarathy S, Scott M. Cashmere-2L: Software coherent shared memory on a clustered remotewrite network. In Proc. the 16th ACM Symposium on Operating Systems Principles, October 1997, pp.170-183.

  17. Charles P, Grothof C, Saraswat V, Donawa C, Kielstra A, Ebcioglu K, von Praun C, Sarkar V. X10: An objectoriented approach to non-uniform cluster computing. In Proc. the 20th Annual ACM SIGPLAN Conference on Object-Oriented Programming, Systems, Languages, and Applications, October 2005, pp.519-538.

  18. Dragojević A, Narayanan D, Hodson O, Castro M. FaRM: Fast remote memory. In Proc. the 11th USENIX Symposium on Networked Systems Design and Implementation, April 2014, pp.401-414.

  19. Wei X, Shi J, Chen Y, Chen R, Chen H. Fast in-memory transaction processing using RDMA and HTM. In Proc. the 25th Symposium on Operating Systems Principles, October 2015, pp.87-104.

  20. Nelson J, Holt B, Myers B, Briggs P, Ceze L, Kahan S, Oskin M. Latency-tolerant software distributed shared memory. In Proc. the 2015 USENIX Annual Technical Conference, July 2015, pp.291-305.

  21. Kalia A, Kaminsky M, Andersen D G. Design guidelines for high performance RDMA systems. In Proc. the 2016 USENIX Annual Technical Conference, June 2016, pp.437-450.

  22. Kumar M, Maass S, Kashyap S, Veselý J, Yan Z, Kim T, Bhattacharjee A, Krishna T. LATR: Lazy translation coherence. In Proc. the 23rd International Conference on Architectural Support for Programming Languages and Operating Systems, March 2018, pp.651-664.

  23. Dennard R H, Gaensslen F H, Rideout V L, Bassous E, LeBlanc A R. Design of ion-implanted MOSFET’s with very small physical dimensions. IEEE Journal of Solid-State Circuits, 1974, 9(5): 256-268.

    Article  Google Scholar 

  24. Esmaeilzadeh H, Blem E, Amant R S, Sankaralingam K, Burger D. Dark silicon and the end of multicore scaling. In Proc. the 38th International Symposium on Computer Architecture, June 2011, pp.365-376.

  25. Borkar S. Thousand core chips: A technology perspective. In Proc. the 44th Annual Design Automation Conference, June 2007, pp.746-749.

  26. Ranger C, Raghuraman R, Penmetsa A, Bradski G, Kozyrakis C. Evaluating MapReduce for multi-core and multiprocessor systems. In Proc. the 13th International Symposium on High-Performance Computer Architecture, February 2007, pp.13-24.

  27. Chen R, Chen H, Zang B. Tiled-MapReduce: Optimizing resource usages of data-parallel applications on multicore with tiling. In Proc. the 19th International Conference on Parallel Architectures and Compilation Techniques, September 2010, pp.523-534.

  28. Zhu X, Chen W, Zheng W, Ma X. Gemini: A computationcentric distributed graph processing system. In Proc. the 12th USENIX Symposium on Operating Systems Design and Implementation, November 2016, pp.301-316.

  29. Guo C, Wu H, Deng Z, Soni G, Ye J, Padhye J, Lipshteyn . RDMA over commodity Ethernet at scale. In Proc. the 2016 ACM SIGCOMM Conference, August 2016, pp.202-215.

  30. Scales D J, Gharachorloo K, Thekkath C A. Shasta: A low overhead, software-only approach for supporting fine-grain shared memory. In Proc. the 7th International Conference on Architectural Support for Programming Languages and Operating Systems, October 1996, pp.174-185.

  31. Karlsson M, Stenström P. Performance evaluation of a cluster-based multiprocessor built from ATM switches and bus-based multiprocessor servers. In Proc. the 2nd International Symposium on High-Performance Computer Architecture, February 1996, pp.4-13.

  32. Gaud F, Lepers B, Decouchant J, Funston J, Fedorova A, Quéma V. Large pages may be harmful on NUMA systems. In Proc. the 2014 USENIX Annual Technical Conference, June 2014, pp.231-242.

  33. Tsai S, Zhang Y. LITE kernel RDMA support for datacenter applications. In Proc. the 26th Symposium on Operating Systems Principles, October 2017, pp.306-324.

  34. Bienia C, Kumar S, Singh J P, Li K. The PARSEC benchmark suite: Characterization and architectural implications. In Proc. the 17th International Conference on Parallel Architectures and Compilation Techniques, October 2008, pp.72-81.

  35. Shun J, Blelloch G E. Ligra: A lightweight graph processing framework for shared memory. In Proc. the 18th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, February 2013, pp.135-146.

  36. Boldi P, Vigna S. The WebGraph framework I: Compression techniques. In Proc. the 13th International Conference on World Wide Web, May 2004, pp.595-602.

  37. Boldi P, Rosa M, Santini M, Vigna S. Layered label propagation: A multiresolution coordinate-free ordering for compressing social networks. In Proc. the 20th International Conference on World Wide Web, March 2011, pp.587-596.

  38. Raikin S, Liss L, Shachar A, Bloch N, Kagan M. Remote transactional memory. US Patent US20150269116, 2015. http://www.freepatentsonline.com/20150269116.pdf, September 2018.

  39. Daglis A, Ustiugov D, Novaković S, Bugnion E, Falsafi B, Grot B. SABRes: Atomic object reads for in-memory rackscale computing. In Proc. the 49th Annual ACM/IEEE International Symposium on Microarchitecture, October 2016, Article No. 6.

  40. Blumrich M A, Li K, Alpert R, Dubnicki C, Felten E W, Sandberg J. Virtual memory mapped network interface for the SHRIMP multicomputer. In Proc. the 21st Annual International Symposium on Computer Architecture, April 1994, pp.142-153.

  41. Kontothanassis L I, Scott M L. Using memory-mapped network interfaces to improve the performance of distributed shared memory. In Proc. the 2nd International Symposium on High-Performance Computer Architecture, February 1996, pp.166-177.

  42. Kalia A, Kaminsky M, Andersen D G. Using RDMA efficiently for key-value services. In Proc. the 2014 ACM Conference on SIGCOMM, August 2014, pp.295-306.

  43. Dragojević A, Narayanan D, Nightingale E B, Renzelmann M, Shamis A, Badam A, Castro M. No compromises: Distributed transactions with consistency, availability, and performance. In Proc. the 25th ACM Symposium on Operating Systems Principles, October 2015, pp.54-70.

  44. Kalia A, Kaminsky M, Andersen D G. FaSST: Fast, scalable and simple distributed transactions with two-sided (RDMA) datagram RPCs. In Proc. the 12th USENIX Symposium on Operating Systems Design and Implementation, November 2016, pp.185-201.

  45. Vasilevsky A, Lively D, Ofsthun S. Linux® virtualization on Virtual IronTM VFe. In Proc. the 2005 Ottawa Linux Symposium, July 2005, pp.235-250.

  46. Kaneda K, Oyama Y, Yonezawa A. A virtual machine monitor for providing a single system image. In Proc. the 17th IPSJ Computer System Symposium, November 2005, pp.3-12.

  47. Gillett R B. Memory channel network for PCI IEEE Micro, 1996, 16(1): 12-18.

    Article  Google Scholar 

  48. Blumrich M A, Alpert R D, Chen Y et al. Design choices in the SHRIMP system: An empirical study. In Proc. the 25th Annual International Symposium on Computer Architecture, June 1998, pp.330-341.

  49. Zhou Y, Iftode L, Li K. Performance evaluation of two home-based lazy release consistency protocols for shared virtual memory systems. In Proc. the 2nd USENIX Symposium on Operating Systems Design and Implementation, October 1996, pp.75-88.

  50. Yeung D, Kubiatowicz J, Agarwal A. MGS: A multigrain shared memory system. In Proc. the 23rd Annual International Symposium on Computer Architecture, May 1996, pp.44-55.

  51. Novakovic S, Daglis A, Bugnion E, Falsafi B, Grot B. Scaleout NUMA. In Proc. the 19th International Conference on Architectural Support for Programming Languages and Operating Systems, March 2014, pp.3-18.

Download references

Author information

Authors and Affiliations

  1. Shanghai Key Laboratory for Scalable Computing Systems, Shanghai Jiao Tong University, Shanghai, 200240, China

    Yang Hong, Yang Zheng, Fan Yang, Bin-Yu Zang, Hai-Bing Guan & Hai-Bo Chen

Authors
  1. Yang Hong
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yang Zheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Fan Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Bin-Yu Zang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Hai-Bing Guan
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Hai-Bo Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hai-Bo Chen.

Electronic supplementary material

ESM 1

(PDF 313 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hong, Y., Zheng, Y., Yang, F. et al. Scaling out NUMA-Aware Applications with RDMA-Based Distributed Shared Memory. J. Comput. Sci. Technol. 34, 94–112 (2019). https://doi.org/10.1007/s11390-019-1901-4

Download citation

  • Received:

  • Revised:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11390-019-1901-4

Keywords

Search

Navigation