Distributed Range-Based Meta-Data Management for an In-Memory Storage

  • Florian KleinEmail author
  • Kevin Beineke
  • Michael Schöttner
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9523)


Large-scale interactive applications and online graph processing require fast data access to billions of small data objects. DXRAM addresses this challenge by keeping all data always in RAM of potentially many nodes aggregated in a data center. Such storage clusters need a space-efficient and fast meta-data management. In this paper we propose a range-based meta-data management allowing fast node lookups while being space efficient by combining data object IDs in ranges. A super-peer overlay network is used to manage these ranges together with backup-node information allowing parallel and fast recovery of meta data and data of failed peers. Furthermore, the same concept can also be used for client-side caching. The measurement results show the benefits of the proposed concepts compared to other meta-data management strategies as well as its very good overall performance evaluated using the social network benchmark BG.


Hash Table Memory Consumption Meta Data Storage Node NoSQL Database 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Askitis, N.: Fast and compact hash tables for integer keys. In: Proceedings of the Thirty-Second Australasian Conference on Computer Science, ACSC 2009, Darlinghurst, Australia, vol. 91 (2009)Google Scholar
  2. 2.
    Atikoglu, B., et al.: Workload analysis of a large-scale key-value store. In: Proceedings of the 12th ACM SIGMETRICS/PERFORMANCE Joint International Conference on Measurement and Modeling of Computer Systems, SIGMETRICS 2012, New York, NY, USA (2012)Google Scholar
  3. 3.
    Barahmand, S., Ghandeharizadeh, S.: Bg: a benchmark to evaluate interactive social networking actions. In: CIDR. Citeseer (2013)Google Scholar
  4. 4.
    Dragojević, A., Narayanan, D., Castro, M., Hodson, O.: Farm: fast remote memory. In: 11th USENIX Symposium on Networked Systems Design and Implementation (NSDI 2014), pp. 401–414. USENIX Association, Seattle, April 2014Google Scholar
  5. 5.
    Klein, F., Beineke, K., Schöttner, M.: Memory management for billions of small objects in a distributed in-memory storage. In: IEEE Cluster 2014, September 2014Google Scholar
  6. 6.
    Klein, F., Schöttner, M.: Dxram: a persistent in-memory storage for billions of small objects. In: International Conference on Parallel and Distributed Computing, Applications and Technologies (PDCAT 2013), pp. 103–110, December 2013Google Scholar
  7. 7.
    Lakshman, A., Malik, P.: Cassandra: a decentralized structured storage system. SIGOPS Oper. Syst. Rev. 44(2), 35–40 (2010)CrossRefGoogle Scholar
  8. 8.
    Lu, H., Ng, Y.Y., Tian, Z.: T-tree or b-tree: main memory database index structure revisited. In: 11th Australasian Proceedings of Database Conference, ADC 2000, pp. 65–73 (2000)Google Scholar
  9. 9.
    Nicolae, B., Antoniu, G., Bougé, L.: Enabling high data throughput in desktop grids through decentralized data and metadata management: the blobseer approach. In: Sips, H., Epema, D., Lin, H.-X. (eds.) Euro-Par 2009. LNCS, vol. 5704, pp. 404–416. Springer, Heidelberg (2009) CrossRefGoogle Scholar
  10. 10.
    Nishtala, R., et al.: Scaling memcache at facebook. In: Proceedings of the 10th USENIX Symposium on Networked Systems Design and Implementation (NSDI 2013), Lombard, Illinois (2013)Google Scholar
  11. 11.
    Ongaro, D., et al.: Fast crash recovery in ramcloud. In: Proceedings of the Twenty-Third ACM Symposium on Operating Systems Principles, SOSP 2011, New York, NY, USA (2011)Google Scholar
  12. 12.
    Ousterhout, J., et al.: The case for ramclouds: scalable high-performance storage entirely in dram. SIGOPS Oper. Syst. Rev. 43(4), 92–105 (2010)CrossRefGoogle Scholar
  13. 13.
    Plugge, E., Hawkins, T., Membrey, P.: The Definitive Guide to MongoDB: The NoSQL Database for Cloud and Desktop Computing, 1st edn. Apress, Berkely (2010) CrossRefGoogle Scholar
  14. 14.
    Rumble, S.M.: Memory and Object Management in RAMCloud. Ph.D. thesis, Stanford University (2014)Google Scholar
  15. 15.
    Shao, B., Wang, H., Li, Y.: Trinity: a distributed graph engine on a memory cloud. In: Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data, SIGMOD 2013, New York, NY, USA (2013)Google Scholar
  16. 16.
    Stoica, I., Morris, R., Karger, D., Kaashoek, M.F., Balakrishnan, H.: Chord: a scalable peer-to-peer lookup service for internet applications. In: Proceedings of the 2001 Conference on Applications, Technologies, Architectures, and Protocols for Computer Communications, SIGCOMM 2001, pp. 149–160. ACM, New York (2001)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Florian Klein
    • 1
    Email author
  • Kevin Beineke
    • 1
  • Michael Schöttner
    • 1
  1. 1.Institut für InformatikHeinrich-Heine-Universität DüsseldorfDüsseldorfGermany

Personalised recommendations