Online Indexing and Distributed Querying Model-View Sensor Data in the Cloud

Abstract

As various kinds of sensors penetrate our daily life (e.g., sensor networks for environmental monitoring), the efficient management of massive amount of sensor data becomes increasingly important at present. Traditional sensor data management systems based on relational database lack scalability to accommodate large-scale sensor data efficiently. Consequently, distributed key-value stores in the cloud is becoming the prime tool to manage sensor data. Meanwhile, model-view sensor data management stores the sensor data in the form of modelled segments. However, currently there is no index and query optimizations upon the modelled segments in the cloud, which results in full table scan in the worst case. In this paper, we propose an innovative model index for sensor data segments in key-value stores (KVM-index). KVM-index consists of two interval indices on the time and sensor value dimensions respectively, each of which has an in-memory search tree and a secondary list materialized in the key-value store. This composite structure enables to update new incoming sensor data segments with constant network I/O. Second, for time (or value)-range and point queries a MapReduce-based approach is designed to process the discrete predicate-related ranges of KVM-index, thereby eliminating computation and communication overheads incurred by accessing irrelevant parts of the index table in conventional MapReduce programs. Finally, we propose a cost based adaptive strategy for the KVM-index-MapReduce framework to process composite queries. As proved by extensive experiments, our approach outperforms in query response time both MapReduce-based processing of the raw sensor data and multiple alternative approaches of model-view sensor data.