Abstract
Real-time stream processing is essential for many real-life stream-based applications. Systems designed to run such applications must be prepared to operate under overloaded conditions. In this paper, the load shedding problem is studied for an important class of real-time data stream monitoring applications. In particular, we adopt the \((m,k)\) deadline model, instead of the commonly used random dropping policy, to capture the QoS requirements of such applications. Based on this model, we propose a Safe lOad Shedding Approach (SOSA) that aims to reduce the workload imposed on the system while at the same time preserve system timing constraints by exploiting data stream semantics. SOSA categorizes stream processing into two different modes and allows one to place provably lighter loads on streams that operate in one particular mode. To demonstrate the usefulness of SOSA, we introduce a concrete \((m,k)\) scheduling algorithm called SOSA-DBP by combining SOSA with DBP, a well-known \((m,k)\) scheduling algorithm. Probabilistic analysis and experimental results show that SOSA-DBP has significant performance gain over DBP.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Babcock B, Babu S, Datar M, Motwani R, Thomas D (2004a) Operator scheduling in data stream systems. VLDB J 13(4):333–353
Babcock B, Datar M, Motwani R (2004b) Load shedding for aggregation queries over data streams. In: ICDE, pp 350–361
Babcock B, Babu S, Datar M, Motwani R, Widom J (2002) Models and issues in data stream systems. In: PODS, pp 1–16
Bernat G, Burns A (1997) Combining (m, n)-hard deadlines and dual priority scheduling. In: RTSS, pp 46–57
Bernat G, Burns A, Llamosi A (2001) Weakly hard real-time systems. IEEE Trans. Comput. 50(4):308–321
Carney D, Cetintermel U, Rasin A, Zdonik S, Cherniack M, Stonebraker M (2003) Operator scheduling in a data stream manager. In: VLDB, pp 37–44, Aug 2003
Gedik B, Liu L, Wu K-L, Yu. Lira PS (2007a) Lightweight region-aware load shedding in mobile cq systems. In ICDE
Gedik B, Wu K-L, Yu PS, Liu L (2007b) A load shedding framework and optimizations for m-way windowed stream joins. In: ICDE
Hamdaoui M, Ramanathan P (1995) A dynamic priority assignement technique for streams with (m, k)-firm deadlines. IEEE Trans Comput 44(12):1443–1451
Hua S, Hu G (2004) Energy-efficient dual-voltage soft real-time system with (m, k)-firm deadline guarantee. In: International conference on compilers, architecture and synthesis for embedded systems
Koren G, Shasha D (1995) Skip-over: algorithms and complexity for overloaded systems that allow skips. In: RTSS, pp 261–272
Linsay W, Ramanathan P (1997) Dbp-m: a technique for meeting end-to-end (m, k)-firm guaranteerequirement in point-to-point networks. In: 22nd annual conference on local computer networks
Quan G, Huv (2000) Enhanced fixed-priority scheduling with (m, k)- firm guarantee. In: RTSS, pp 27–30, November 2000
Ramanathan P (1999) Overload management in real-time control applications using (m, k)-firm guarantee. IEEE Trans Parallel Distrib Syst 10(6):549–559
Schmidt S, Berthold H, Lehner W (2004) Qstream: deterministic querying of data streams. In: VLDB, pp 1365–1368
Schmidt S, Legler T, Schaller D, Lehner W (2005a) Real-time scheduling for data stream management systems. In: ECRTS, pp 167–176
Schmidt S, Legler T, Schär S, Lehner W (2005b) Robust real-time query processing with qstream. In: VLDB, pp 1299–1302
Stonebraker M, Çetintemel U, Zdonik SB (2005) The 8 requirements of real-time stream processing. SIGMOD Rec 34(4):42–47
Striegel A, Manimaran G (2000) Best-effort scheduling of (m, k)-firm real-time streams in multihop networks. Comput Commun 23(13):1292–1300
Tatbul N, Çetintemel U, Zdonik SB, Cherniack M, Stonebraker M (2003) Load shedding in a data stream manager. In: VLDB, pp 309–320
Tatbul N, Zdonik SB (2006a) Dealing with overload in distributed stream processing systems. In: ICDE workshops, p 24
Tatbul N, Zdonik SB (2006b) Window-aware load shedding for aggregation queries over data streams. In: VLDB, pp 799–810
Tu Y-C, Hefeeda M, Xia Y, Prabhakar S, Liu S (2005) Control-based quality adaptation in data stream management systems. In: DEXA, pp 746–755
Tu Y-C, Liu S, Prabhakar S, Yao B (2006) Load shedding in stream databases: a control-based approach. In: VLDB, pp 787–798
Wei Y, Prasad V, Son SH, Stankovic JA (2006) Prediction-based qos management for real-time data streams. In: RTSS, pp 344–358
Acknowledgments
The authors would like to thank the anonymous reviewers for their valuable comments and suggestions that improved the quality of this paper. The work reported in this paper is partially supported by NSFC under Grant Number 61370205, NSF of Shanghai under Grant Number 13ZR1400800 and the Fundamental Research Funds for the Central Universities.
Author information
Authors and Affiliations
Corresponding author
Additional information
Communicated by V. Loia.
Rights and permissions
About this article
Cite this article
Ning, G., Wang, H., Shu, L. et al. Towards load shedding and scheduling schemes for data streams that maintain quality and timing requirements of query results. Soft Comput 20, 1961–1976 (2016). https://doi.org/10.1007/s00500-015-1617-5
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00500-015-1617-5