The VLDB Journal

, Volume 25, Issue 2, pp 197–221 | Cite as

Avoiding class warfare: managing continuous queries with differentiated classes of service

  • Thao N. Pham
  • Panos K. Chrysanthis
  • Alexandros Labrinidis
Regular Paper


Data stream management systems (DSMSs) offer the most effective solution for processing data streams by efficiently executing continuous queries (CQs) over the incoming data. CQs inherently have different levels of criticality and hence different levels of expected quality of service (QoS) and quality of data (QoD). Adhering to such expected QoS/QoD metrics is even more important in cases of multi-tenant data stream management services. In this work, we propose DILoS, a framework that, through priority-based scheduling and load shedding, supports differentiated QoS and QoD for multiple classes of CQs. Unlike existing works that consider scheduling and load shedding separately, DILoS is a novel unified framework that exploits the synergy between scheduling and load shedding. We also propose ALoMa, a general, adaptive load manager that DILoS is built upon. By its design, ALoMa performs better than the state-of-the-art alternatives in three dimensions: (1) it automatically tunes the headroom factor, (2) it honors the delay target, (3) it is applicable to complex query networks with shared operators. We implemented DILoS and ALoMa in our real DSMS prototype system (AQSIOS) and evaluate their performance for a variety of real and synthetic workloads. Our experimental evaluation of ALoMa verified its clear superiority over the state-of-the-art approaches. Our experimental evaluation of the DILoS framework showed that it (a) allows the scheduler and load shedder to consistently honor CQs’ priorities, (b) significantly increases system capacity utilization by exploiting batch processing, and (c) enables operator sharing among query classes of different priorities while avoiding priority inversion, i.e., a lower-priority class never blocks a higher-priority one.


Data stream management system Continuous query Multi-tenant Load shedding Scheduling 



Our thanks to the anonymous reviewers for their insightful comments and Mark Silvis and Eric Gratta for their help with copyediting. This work was supported in part by NSF awards IIS-0534531, IIS-0746696, OIA-1028162, an Andrew Mellon Predoctoral Fellowship and EMC/ Greenplum.


  1. 1.
  2. 2.
  3. 3.
  4. 4.
    Pacific tsunami warning center.
  5. 5.
  6. 6.
    Tropical Atmosphere Ocean Project.
  7. 7.
    Abadi, D.J., Carney, D., Çetintemel, U., Cherniack, M., Convey, C., Lee, S., Stonebraker, M., Tatbul, N., Zdonik, S.: Aurora: a new model and architecture for data stream management. In: VLDBJ ’03Google Scholar
  8. 8.
    Al Moakar, L., Chrysanthis, P. K., Chung, C., Guirguis, S., Labrinidis, A., Neophytou, P., Pruhs, K.: Admission control mechanisms for continuous queries in the cloud. In: ICDE’10Google Scholar
  9. 9.
    Al Moakar, L., Labrinidis, A., Chrysanthis, P. K.: Adaptive class-based scheduling of continuous queries. In: SMDB ’12Google Scholar
  10. 10.
    Al Moakar, L., Pham, T. N., Neophytou, P., Chrysanthis, P. K., Labrinidis, A., Sharaf, M.: Class-based continuous query scheduling for data streams. In: DMSN ’09Google Scholar
  11. 11.
    Arasu, A., Cherniack, M., Galvez, E., Maier, D., Maskey, A. S., Ryvkina, E., Stonebraker, M., Tibbetts, R.: Linear road: a stream data management benchmark. In: VLDB’ 04Google Scholar
  12. 12.
    Babcock, B., Babu, S., Datar, M., Motwani, R., Thomas, D.: Operator scheduling in data stream systems. In: VLDBJ ’04Google Scholar
  13. 13.
    Babcock, B., Babu, S., Datar, M., Motwani, R., Widom, J.: Models and issues in data stream systems. In: PODS ’02Google Scholar
  14. 14.
    Babcock, B., Datar, M., Motwani, R.: Load shedding for aggregation queries over data streams. In: ICDE ’04Google Scholar
  15. 15.
    Carney, D., Çetintemel, U., Rasin, A., Zdonik, S., Cherniack, M., Stonebraker, M.: Operator scheduling in a data stream manager. In: VLDB’ 03Google Scholar
  16. 16.
    Castro Fernandez, R., Migliavacca, M., Kalyvianaki, E., Pietzuch, P.: Integrating scale out and fault tolerance in stream processing using operator state management. In: SIGMOD’13Google Scholar
  17. 17.
    Chakravarthy, S., Jiang, Q.: Stream Data Processing: A Quality of Service Perspective Modeling. Load Shedding, and Complex Event Processing. Springer, Scheduling (2009)Google Scholar
  18. 18.
    Chandrasekaran, S., Cooper, O., Deshpande, A., Franklin, M. J., Hellerstein, J. M., Hong, W., Krishnamurthy, S., Madden, S. R., Reiss, F., Shah, M. A.: TelegraphCQ: continuous dataflow processing. In: SIGMOD ’03Google Scholar
  19. 19.
    Chang, J.H., Kum, H.-C.M.: Frequency-based load shedding over a data stream of tuples. Inf. Sci. 179(21), 3733–3744 (2009)CrossRefGoogle Scholar
  20. 20.
    Chi, Y., Wang, H., Yu, P. S.: Loadstar: load shedding in data stream mining. In: VLDB ’05Google Scholar
  21. 21.
    Chrysanthis, P. K.: AQSIOS—Next Generation Data Stream Management System. CONET Newsletter, June 2010Google Scholar
  22. 22.
    Dash, R., Fegaras, L.: Synopsis based load shedding in XML streams. In: EDBT/ICDT ’09 WorkshopsGoogle Scholar
  23. 23.
    Feng, H., Liu, Z., Xia, C. H., Zhang, L.: Load shedding and distributed resource control of stream processing networks. In: Performance Evaluation (2007)Google Scholar
  24. 24.
    Gedik, B., Wu, K.-L., Yu, P., Liu, L.: GrubJoin: An Adaptive, Multi-Way. Windowed Stream Join with Time Correlation-Aware CPU Load Shedding, TKDE (2007)Google Scholar
  25. 25.
    Gedik, B., Wu, K.-L., Yu, P. S.: Efficient construction of compact shedding filters for data stream processing. In: ICDE ’08Google Scholar
  26. 26.
    Gedik, B., Wu, K.-L., Yu, P. S., Liu, L.: Mobiqual: Qos-aware load shedding in mobile CQ systems. In: ICDE ’08Google Scholar
  27. 27.
    Gedik, B., Wu, K.-L., Yu, P.S., Liu, L.: CPU load shedding for binary stream joins. Knowl. Inf. Syst. 13(3), 271–303 (2007)CrossRefGoogle Scholar
  28. 28.
    Gulisano, V., Jimenez-Peris, R., Patino-Martinez, M., Soriente, C., Valduriez, P.: Streamcloud: an elastic and scalable data streaming system. In: IEEE TPDS, 2012Google Scholar
  29. 29.
    Jagadish, H.V., Gehrke, J., Labrinidis, A., Papakonstantinou, Y., Patel, J.M., Ramakrishnan, R., Shahabi, C.: Big data and its technical challenges. CACM 57(7), 86–94 (2014)CrossRefGoogle Scholar
  30. 30.
    Kendai, B., Chakravarthy, S.: Load shedding in MavStream: analysis, implementation, and evaluation. In: BNCOD ’08Google Scholar
  31. 31.
    Kleiminger, W., Kalyvianaki, E., Pietzuch, P.: Balancing load in stream processing with the cloud. In: ICDEW’ 11Google Scholar
  32. 32.
    Kulkarni, D., Ravishankar, C. V., Cherniack, M.: Real-time load-adaptive processing of continuous queries over data streams. In: DEBS’ 08Google Scholar
  33. 33.
    Lei, C., Rundensteiner, E. A.: Robust distributed query processing for streaming data. In: ACM TODS, 2014Google Scholar
  34. 34.
    Mozafari, B., Zaniolo, C.: Optimal load shedding with aggregates and mining queries. In: ICDE ’10Google Scholar
  35. 35.
    Narayanan, S., Waas, F.: Dynamic prioritization of database queries. In: ICDE ’11Google Scholar
  36. 36.
    Nehme, R. V., Rundensteiner, E. A.: Clustersheddy: load shedding using moving clusters over spatio-temporal data streams. In: DASFAA’07Google Scholar
  37. 37.
    Pham, T. N., Al Moakar, L., Chrysanthis, P. K., Labrinidis, A.: DILoS: a dynamic integrated load manager and scheduler for continuous queries. In: SMDB ’11Google Scholar
  38. 38.
    Pham, T. N., Chrysanthis, P. K., Labrinidis, A.: Self-managing load shedding for data stream management systems. In: SMDB ’13Google Scholar
  39. 39.
    Reiss, F., Hellerstein, J. M.: Data triage: an adaptive architecture for load shedding in telegraphCQ. In: ICDE ’05Google Scholar
  40. 40.
    Sharaf, M. A., Chrysanthis, P. K., Labrinidis, A., Pruhs, K.: Algorithms and metrics for processing multiple heterogeneous continuous queries. In: ACM TODS, 2008Google Scholar
  41. 41.
    Tatbul, N.,  Çetintemel, U., Zdonik, S.: Staying FIT: efficient load shedding techniques for distributed stream processing. In: VLDB ’07Google Scholar
  42. 42.
    Tatbul, N., Çetintemel, U., Zdonik, S., Cherniack, M., Stonebraker, M.: Load shedding in a data stream manager. In: VLDB ’03Google Scholar
  43. 43.
    Tatbul, N., Zdonik, S.: Window-aware load shedding for aggregation queries over data streams. In: VLDB ’06Google Scholar
  44. 44.
    Tu, Y.-C., Liu, S., Prabhakar, S., Yao, B.: Load shedding in stream databases: a control-based approach. In: VLDB ’06Google Scholar
  45. 45.
    Wei, Y., Son, S. H., Stankovic, J. A.: RTSTREAM: real-time query processing for data streams. In: ISORC’ 06Google Scholar
  46. 46.
    Wolf, J., Bansal, N., Hildrum, K., Parekh, S., Rajan, D., Wagle, R., Wu, K.-L., Fleischer, L.: SODA: An optimizing scheduler for large-scale stream-based distributed computer systems. In: Middleware’ 08Google Scholar
  47. 47.
    Wu, S., Lv, Y., Yu, G., Gu, Y., Li, X.: A QoS-guaranteeing scheduling algorithm for continuous queries over streams. In: APWeb/WAIM’ 07Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2015

Authors and Affiliations

  • Thao N. Pham
    • 1
  • Panos K. Chrysanthis
    • 1
  • Alexandros Labrinidis
    • 1
  1. 1.Department of Computer ScienceUniversity of PittsburghPittsburghUSA

Personalised recommendations