A Large-Scale Object-Based Active Storage Platform for Data Analytics in the Internet of Things

  • Quanqing XuEmail author
  • Khin Mi Mi Aung
  • Yongqing Zhu
  • Khai Leong Yong
Conference paper
Part of the Lecture Notes in Electrical Engineering book series (LNEE, volume 354)


In this paper, we propose a large-scale object-based storage platform, named Gem, for data analytics in the Internet of Things (IoT). In Gem, a region covered by an IoT network is partitioned into sub-regions, each of which can be identified by a unique ID and known to all participants, which is automatic and economical. Gem can preserve object locality using type and location sensitive hashing, as well as dynamically distribute objects among a server cluster to keep load balancing. All data from the IoT can be stored as objects with attributes, methods and policies in Object Store Devices (OSDs). For some applications such as data analytics, application-specific operations are executed by OSDs, and only the results are returned to clients, rather than data files are read by the clients. Thus, the platform Gem is able to greatly reduce the overhead of data analytics applications in the IoT.


Object-based storage Data analytics Internets of things 



This work is supported by A*STAR under Grant No. DSI/14-300009.


  1. 1.
    Atzori, L., Iera, A., Morabito, G.: The Internet of things: a survey. Comput. Netw. 54, 2787–2805 (2010)CrossRefGoogle Scholar
  2. 2.
    Mesnier, M., Ganger, G.R., Riedel, E.: Object-based storage. IEEE Commun. Mag. 41(8), 84–90 (2003)CrossRefGoogle Scholar
  3. 3.
    Xu, Q., Shen, H.T., Chen, Z., Cui, B., Zhou, X., Dai, Y.: Hybrid retrieval mechanisms in vehicle-based P2P networks. In: Proceedings of the International Conference on Computational Science (ICCS’09). Lecture Notes in Computer Science, vol. 5544, pp. 303–314. Springer, Berlin (2009)Google Scholar
  4. 4.
    Shvachko, K., Huang, H., Radia, S., et al.: The hadoop distributed filesystem. In: MSST 2010 (2010)Google Scholar
  5. 5.
    Stoica, I., Morris, R., Karger, D.R., Kaashoek, M.F., Balakrishnan, H.: Chord: a scalable peer-to-peer lookup service for internet applications. In: SIGCOMM, pp. 149–160 (2001)Google Scholar
  6. 6.
    Xu, Q., Shen, H.T., Chen, Z., Cui, B., Zhou, X., Dai, Y.: Hybrid information retrieval policies based on cooperative cache in mobile P2P networks. Front. Comput. Sci. China 3(3), 381–395 (2009)CrossRefGoogle Scholar
  7. 7.
    Xu, Q., Arumugam, R.V., Yong, K.L., Mahadevan, S.: Efficient and scalable metadata management in EB-scale file systems. IEEE Trans. Parallel Distrib. Syst. 25(11), 2840–2850 (2014)CrossRefzbMATHGoogle Scholar
  8. 8.
    Chekuri, C., Khanna, S.: A polynomial time approximation scheme for the multiple knapsack problem. SIAM J. Comput. 35(3), 713–728 (2005)MathSciNetCrossRefGoogle Scholar
  9. 9.
    Xu, Q., Arumugam, R.V., Yong, K.L., Mahadevan, S.: DROP: facilitating distributed metadata management in EB-scale storage systems. In: MSST, pp. 1–10 (2013)Google Scholar
  10. 10.
    Xu, Q., Arumugam R.V., Yong K.L., Wen, Y., Ong, Y.S.: C2: adaptive load balancing for metadata server cluster in cloud-scale storage systems. In: IES, pp. 195–209 (2015)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2016

Authors and Affiliations

  • Quanqing Xu
    • 1
    Email author
  • Khin Mi Mi Aung
    • 1
  • Yongqing Zhu
    • 1
  • Khai Leong Yong
    • 1
  1. 1.Data Storage InstituteA*STARSingaporeSingapore

Personalised recommendations