A Large-Scale Object-Based Active Storage Platform for Data Analytics in the Internet of Things
In this paper, we propose a large-scale object-based storage platform, named Gem, for data analytics in the Internet of Things (IoT). In Gem, a region covered by an IoT network is partitioned into sub-regions, each of which can be identified by a unique ID and known to all participants, which is automatic and economical. Gem can preserve object locality using type and location sensitive hashing, as well as dynamically distribute objects among a server cluster to keep load balancing. All data from the IoT can be stored as objects with attributes, methods and policies in Object Store Devices (OSDs). For some applications such as data analytics, application-specific operations are executed by OSDs, and only the results are returned to clients, rather than data files are read by the clients. Thus, the platform Gem is able to greatly reduce the overhead of data analytics applications in the IoT.
KeywordsObject-based storage Data analytics Internets of things
This work is supported by A*STAR under Grant No. DSI/14-300009.
- 3.Xu, Q., Shen, H.T., Chen, Z., Cui, B., Zhou, X., Dai, Y.: Hybrid retrieval mechanisms in vehicle-based P2P networks. In: Proceedings of the International Conference on Computational Science (ICCS’09). Lecture Notes in Computer Science, vol. 5544, pp. 303–314. Springer, Berlin (2009)Google Scholar
- 4.Shvachko, K., Huang, H., Radia, S., et al.: The hadoop distributed filesystem. In: MSST 2010 (2010)Google Scholar
- 5.Stoica, I., Morris, R., Karger, D.R., Kaashoek, M.F., Balakrishnan, H.: Chord: a scalable peer-to-peer lookup service for internet applications. In: SIGCOMM, pp. 149–160 (2001)Google Scholar
- 9.Xu, Q., Arumugam, R.V., Yong, K.L., Mahadevan, S.: DROP: facilitating distributed metadata management in EB-scale storage systems. In: MSST, pp. 1–10 (2013)Google Scholar
- 10.Xu, Q., Arumugam R.V., Yong K.L., Wen, Y., Ong, Y.S.: C2: adaptive load balancing for metadata server cluster in cloud-scale storage systems. In: IES, pp. 195–209 (2015)Google Scholar