Processing of Aggregate Continuous Queries in a Distributed Environment

Abstract

Data Stream Management Systems (DSMSs) performing online analytics rely on the efficient execution of large numbers of Aggregate Continuous Queries (ACQs). In this paper, we study the problem of generating high quality execution plans of ACQs in DSMSs deployed on multi-node (multi-core and multi-processor) distributed environments. Towards this goal, we classify optimizers based on how they partition the workload among computing nodes and on their usage of the concept of Weavability, which is utilized by the state-of-the-art WeaveShare optimizer to selectively combine ACQs and produce low cost execution plans for single-node environments. For each category, we propose an optimizer, which either adopts an existing strategy or develops a new one for assigning and grouping ACQs to computing nodes. We implement and experimentally compare all of our proposed optimizers in terms of (1) keeping the total cost of the ACQs execution plan low and (2) balancing the load among the computing nodes. Our extensive experimental evaluation shows that our newly developed Weave-Group to Nodes ($WG_{TN}$) and Weave-Group Inserted ($WG_{I}$) optimizers produce plans of significantly higher quality than the rest of the optimizers. $WG_{TN}$ minimizes the total cost, making it more suitable from a client perspective, and $WG_{I}$ achieves load balancing, making it more suitable from a system perspective.