Skip to main content
SpringerLink
Log in

Data Stream Management

  • Chapter
  • First Online:
Real-Time & Stream Data Management

Part of the book series: SpringerBriefs in Computer Science ((BRIEFSCOMPUTER))

  • 951 Accesses

Abstract

In some domains, data arrives so fast and in such great quantity that storing it in a database collection is simply infeasible. When the incoming data relates to ongoing (real-world) events that require immediate action, persistence may further not even be useful; for example, data in electronic trading, network monitoring, or real-time fraud detection is only valuable for a short amount of time and therefore has to be utilized immediately. To adapt to these circumstances, data stream management systems (DSMSs) introduce the data stream as an abstraction for an infinite sequence of database records that arrive over time. The raw data streams arriving at the systems are usually referred to as base streams, whereas those resulting from data transformations (e.g. queries) are called derived streams. Since a data stream is impossible to store entirely due to its unbounded nature, DSMSs drop the database requirement of eternal data persistence: They retain incoming records for limited time only and eventually discard them.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
eBook
USD 16.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 16.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    An attribute is monotonic, if all its values are either decreasing or increasing, such as arrival timestamps in a centralized DSMS. Similarly, an attribute is quasi-monotonic, if it is correlated to a monotonic attribute. For example, the time at which an event is registered according to a sensor’s local clock (event time) is quasi-monotonic, because it typically corresponds to the time at which it is received according to the server’s clock (arrival time) within a certain error margin (cf. Sect. 4.2).

  2. 2.

    All write operations in PipelineDB are coordinated synchronously via two-phase commit between all nodes [Pipb], so that highly distributed setups are likely to experience increased latency as well as reduced throughput and availability [Pan15, Sec. 3.1].

  3. 3.

    As an example, consider the graphical user interface of Aurora/Borealis which is based on arrows and boxes rather than SQL-style declarative statements [Çet+16].

  4. 4.

    Specifically, providing undo information requires buffering the original output [Aki+15, Sec. 2.3]. Likewise, reprocessing huge amounts of data to generate updated records can lead to CPU contention and can thus significantly impair overall system performance [Kre14c].

References

  1. Daniel J. Abadi et al. “Aurora: A New Model and Architecture for Data Stream Management”. In: The VLDB Journal 12.2 (Aug. 2003), pp. 120–139. issn: 1066–8888. url: https://doi.org/10.1007/s00778-003-0095-z. http://dx.doi.org/10.1007/s00778-003-0095-z.

  2. Daniel J Abadi et al. “The Design of the Borealis Stream Processing Engine”. In: Second Biennial Conference on Innovative Data Systems Research (CIDR 2005). Asilomar, CA, 2005.

    Google Scholar 

  3. Tyler Akidau, Alex Balikov, Kaya Bekiroglu, et al. “MillWheel: Fault-Tolerant Stream Processing at Internet Scale”. In: Very Large Data Bases. 2013, pp. 734–746.

    Google Scholar 

  4. Arvind Arasu, Shivnath Babu, and Jennifer Widom. “The CQL Continuous Query Language: Semantic Foundations and Query Execution”. In: The VLDB Journal 15.2 (June 2006), pp. 121–142. issn: 1066-8888. url: https://doi.org/10.1007/s00778-004-0147-z. http://dx.doi.org/10.1007/s00778-004-0147-z.

    Article  Google Scholar 

  5. Tyler Akidau et al. “The Dataflow Model: A Practical Approach to Balancing Correctness, Latency, and Cost in Massive-Scale, Unbounded, Out-of-Order Data Processing”. In: Proceedings of the VLDB Endowment8 (2015), pp. 1792–1803.

    Google Scholar 

  6. Tyler Akidau. “The world beyond batch: Streaming 101”. In: O’Reilly Media (Aug. 2015). Accessed: 2017-05-21. url: https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-101.

  7. M.H. Ali et al. “Microsoft CEP Server and Online Behavioral Targeting”. In: Proc. VLDB Endow 2.2 (Aug. 2009), pp. 1558–1561. issn: 2150-8097. url: https://doi.org/10.14778/1687553.1687590. https://doi.org/10.14778/1687553.1687590.

    Article  Google Scholar 

  8. Ahmed M. Ayad and Jeffrey F. Naughton. “Static Optimization of Conjunctive Queries with Sliding Windows over Infinite Streams”. In: Proceedings of the 2004 ACM SIGMOD International Conference on Management of Data SIGMOD’04. Paris, France: ACM, 2004, pp. 419–430. isbn: 1-58113-859-8. url: https://doi.org/10.1145/1007568.1007616. http://doi.acm.org/10.1145/1007568.1007616.

  9. Arvind Arasu et al. “Data Stream Management: Processing High-Speed Data Streams”. In: Data Stream Management: Processing High-Speed Data Streams. Ed. by Minos Garofalakis, Johannes Gehrke, and Rajeev Rastogi. Berlin, Heidelberg: Springer Berlin Heidelberg, 2016. Chap. STREAM: The Stanford Data Stream Management System, pp. 317–336. isbn:978-3-540-28608-0. url: https://doi.org/10.1007/978-3-540-28608-0_16. https://doi.org/10.1007/978-3-540-28608-0_16.

    Google Scholar 

  10. Mauricio Arango. “Mobile QoS Management Using Complex Event Processing”. In: Proceedings of the 7th ACM International Conference on Distributed Event-based Systems DEBS’ 13. Arlington, Texas, USA: ACM, 2013, pp. 115–122. isbn: 978-1-4503-1758-0. url: https://doi.org/10.1145/2488222.2488277. http://doi.acm.org/10.1145/2488222.2488277.

  11. Brian Babcock et al. “Models and Issues in Data Stream Systems”. In: Proceedings of the Twenty-first ACM SIGMOD-SIGACT-SIGART Symposium on Principles of Database Systems PODS’02. Madison, Wisconsin: ACM, 2002, pp. 1–16. isbn: 1-58113-507-6. url: https://doi.org/10.1145/543613.543615. http://doi.acm.org./10.1145/543613.543615.

  12. Ralf Bruns and Jürgen Dunkel. Complex Event Processing: Komplexe Analyse von massiven Datenströmen mit CEP. Springer Vieweg, 2015.

    Book  Google Scholar 

  13. Brian Babcock, Mayur Datar, and Rajeev Motwani. “Load Shedding in Data Stream Systems”. In: Data Streams – Models and Algorithms. Vol. 31. Advances in Database Systems. Springer, 2007, pp. 127–147.

    Google Scholar 

  14. Francesco Buccafurri and Gianluca Lax. “Approximating Sliding Windows by Cyclic Tree-like Histograms for Efficient Range Queries”. In: Data Knowl. Eng 69.9 (Sept. 2010), pp. 979–997. issn: 0169-023X. url: https://doi.org/10.1016/j.datak.2010.5002. http://dx.doi.org/10.1016/j.datak.2010.05.002.

  15. Shivnath Babu, Utkarsh Srivastava, and Jennifer Widom. “Exploiting K-constraints to Reduce Memory Overhead in Continuous Queries over Data Streams”. In: ACM Trans. Database Syst. 29.3 (Sept. 2004), pp. 545–580. issn: 0362-5915. url: https://doi.org/10.1145/1016028.1016032. http://doi.acm.org/10.1145/1016028.1016032.

    Article  Google Scholar 

  16. Streaming Accessed: 2017-11-26. Calcite. Nov 2017. url: https://calcite.apache.org/docs/stream.html.

  17. Don Carney et al. “Monitoring Streams: A New Class of Data Management Applications”. In: Proceedings of the 28th International Conference on Very Large Data Bases. VLDB’02. Hong Kong, China: VLDB Endowment, 2002, pp. 215–226. url: http://dl.acm.org/citation.cfm?id=1287369.1287389.

  18. Paris Carbone et al. “Large-Scale Data Stream Processing Systems”. In: Handbook of Big Data Technologies. Springer, 2017, pp. 219–260.

    Google Scholar 

  19. Ugur Cetintemel et al. “S-Store: A Streaming NewSQL System for Big Velocity Applications”. In: Proc. VLDB Endow 7.13 (Aug. 2014), pp. 1633–1636. issn: 2150-8097. url: https://doi.org/10.14778/2733004.2733048. http://dx.doi.org/10.14778/2733004.2733048.

    Article  Google Scholar 

  20. Sirish Chandrasekaran et al. “TelegraphCQ: Continuous Dataflow Processing”. In: Proceedings of the 2003 ACM SIGMOD International Conference on Management of Data SIGMOD ’03. San Diego, California: ACM, 2003, pp. 668–668. isbn: 1-58113-634-X. url: https://doi.org/10.1145/872757.872857. http://doi.acm.org/10.1145/872757.872857.

  21. Jianjun Chen et al. “NiagaraCQ: A Scalable Continuous Query System for Internet Databases”. In: Proceedings of the 2000 ACM SIGMOD International Conference on Management of Data SIGMOD ’00. Dallas, Texas, USA: ACM, 2000, pp. 379–390. isbn: 1-58113-217-4. url: https://doi.org/10.1145/342009.335432. http://doi.acm.org/10.1145/342009.335432.

  22. Graham Cormode and S. Muthukrishnan. “An Improved Data Stream Summary: The Count-min Sketch and Its Applications”. In: J. Algorithms 55.1 (Apr 2005), pp. 58–75. issn: 0196-6774. url: https://doi.org/10.1016/j.jalgor.2003.12.001. http://dxdoiorg/10.1016/j.jalgor.2003.12.001

    Article  MathSciNet  Google Scholar 

  23. Gianpaolo Cugola and Alessandro Margara. “Processing Flows of Information: From Data Stream to Complex Event Processing”. In: ACM Comput. Surv44.3 (June 2012), 15:1–15:62. issn: 0360-0300. url: https://doi.org/10.1145/2187671.2187677. http://doi.acm.org/10.1145/2187671.2187677.

    Article  Google Scholar 

  24. Antonio Carzaniga, David S. Rosenblum, and Alexander L. Wolf. “Design and Evaluation of a Wide-area Event Notification Service”. In: ACM Trans. Comput. Syst. 19.3 (Aug. 2001), pp. 332–383. issn: 0734-2071. url: https://doi.org/10.1145/380749.380767. http://doi.acm.org/10.1145/380749.380767.

    Article  Google Scholar 

  25. P. Carbone, K. Vandikas, and F Zaloshnja. “Towards Highly Available Complex Event Processing Deployments in the Cloud”. In: 2013 Seventh International Conference on Next Generation Mobile Apps, Services and Technologies. 2013, pp. 153–158. https://doi.org/10.1109/NGMAST.2013.35

  26. Mayur Datar et al. “Maintaining Stream Statistics over Sliding Windows”. In: SIAM Journal on Computing 31.6 (2002), pp. 1794–1813.

    Article  MathSciNet  Google Scholar 

  27. Erik D. Demaine, Alejandro López-Ortiz, and J. Ian Munro. “Frequency Estimation of Internet Packet Streams with Limited Space”. In: Proceedings of the 10th Annual European Symposium on Algorithms. ESA’02. London, UK, UK: Springer-Verlag, 2002, pp. 348–360. isbn: 3-540-44180-8. url: http://dlacmorg/citationcfm?id=647912.740658.

    Google Scholar 

  28. Opher Etzion, Peter Niblett, and David C Luckham. Event processing in action. Ed. by Sebastian Stirling. Manning Greenwich, 2011.

    Google Scholar 

  29. How does Esper scale? Accessed: 2016-09-19. EsperTech. 2016. url: http://www.espertech.com/esper/faq_esper.php#scaling.

  30. Leonidas Fegaras. “Incremental Query Processing on Big Data Streams”. In: IEEE Trans. on Knowl. and Data Eng. 28.11 (Nov 2016), pp. 2998–3012. issn: 1041-4347. url: https://doi.org/10.1109/TKDE.2016.2601103. https://doi.org/10.1109/TKDE.2016.2601103.

    Article  Google Scholar 

  31. Joan Feigenbaum et al. “An Approximate L1-Difference Algorithm for Massive Data Streams”. In: SIAM J Comput. 32.1 (Jan. 2003), pp. 131–151. issn: 0097-5397. url: https://doi.org/10.1137/S0097539799361701. https://doi.org/10.1137/S0097539799361701.

    Article  MathSciNet  Google Scholar 

  32. M. Ficco and L. Romano. “A Generic Intrusion Detection and Diagnoser System Based on Complex Event Processing”. In: 2011 First International Conference on Data Compression, Communications and Processing 2011, pp. 275–284. https://doi.org/10.1109/CCP.2011.43.

  33. Thanaa M. Ghanem, Walid G. Aref, and Ahmed K. Elmagarmid. “Ex- ploiting Predicate-window Semantics over Data Streams”. In: SIGMOD Rec. 35.1 (Mar 2006), pp. 3–8. issn: 0163-5808. url: https://doi.org/10.1145/1121995.1121996. http://doiacmorg/10.1145/1121995.1121996

    Article  Google Scholar 

  34. T M. Ghanem et al. “Incremental Evaluation of Sliding-Window Queries over Data Streams”. In: IEEE Transactions on Knowledge and Data Engineering 19.1 (2007), pp. 57–72. issn: 1041-4347. https://doi.org/10.1109/TKDE.2007.250585

    Article  Google Scholar 

  35. Piero Giacomelli. Hornetq messaging developer’s guide Ed. by Ankita Shashi. Packt Publishing Ltd., 2012.

    Google Scholar 

  36. Phillip B. Gibbons. “Distinct Sampling for Highly-Accurate Answers to Distinct Values Queries and Event Reports”. In: Proceedings of the 27th International Conference on Very Large Data Bases VLDB ’01. San Francisco, CA, USA: Morgan Kaufmann Publishers Inc., 2001, pp. 541–550. isbn: 1-55860-804-4. url: http://dl.acm.org/citation.cfm?id=645927.672351

  37. Michael Greenwald and Sanjeev Khanna. “Space-efficient Online Computation of Quantile Summaries”. In: Proceedings of the 2001 ACM SIGMOD International Conference on Management of Data. SIGMOD’01. Santa Barbara, California, USA: ACM, 2001, pp. 58–66. isbn: 1-58113-332-4. url: https://doi.org/10.1145/375663.375670. http://doi.acm.org/10.1145/375663.375670.

  38. Johannes Gehrke, Flip Korn, and Divesh Srivastava. “On Computing Correlated Aggregates over Continual Data Streams”. In: Proceedings of the 2001 ACM SIGMOD International Conference on Management of Data SIGMOD ’01. Santa Barbara, California, USA: ACM, 2001, pp. 13–24. isbn: 1-58113-332-4. url: https://doi.org/10.1145/375663.375665. http://doi.acm.org/10.1145/375663.375665

  39. Sudipto Guha, Nick Koudas, and Kyuseok Shim. “Data-streams and Histograms”. In: Proceedings of the Thirty-third Annual ACM Symposium on Theory of Computing STOC ’01. Hersonissos, Greece: ACM, 2001, pp. 471–475. isbn: 1-58113-349-9. url: https://doi.org/10.1145/380752.380841. http://doi.acm.org/10.1145/380752.380841.

  40. Phillip B. Gibbons and Yossi Matias. “New Sampling-based Summary Statistics for Improving Approximate Query Answers”. In: SIGMOD Rec.27.2 (June 1998), pp. 331–342. issn: 0163-5808. url: https://doi.org/10.1145/276305.276334. http://doi.acm.org/10.1145/276305.276334.

    Article  Google Scholar 

  41. Lukasz Golab. “Sliding Window Query Processing over Data Streams”. PhD thesis. University of Waterloo, Aug. 2006.

    Google Scholar 

  42. Lukasz Golab and M. Tamer Zsu. Data Stream Management. Morgan & Claypool Publishers, 2010. isbn: 1608452727, 9781608452729.

    Google Scholar 

  43. Namit Jain et al. “Towards a Streaming SQL Standard”. In: Proc. VLDB Endow 1.2 (Aug. 2008), pp. 1379–1390. issn: 2150-8097. url: https://doi.org/10.14778/14541591454179. http://dxdoiorg/10.14778/1454159.1454179

  44. Theodore Johnson et al. “A Heartbeat Mechanism and Its Application in Gigascope”. In: Proceedings of the 31st International Conference on Very Large Data Bases. VLDB ’05. Trondheim, Norway: VLDB Endowment, 2005, pp. 1079–1088. isbn: 1-59593-154-6. url: http://dl.acm.org/citation.cfm?id=1083592.1083716

  45. Robert Kallman et al. “H-store: A High-performance, Distributed Main Memory Transaction Processing System”. In: Proc. VLDB Endow 1.2 (Aug. 2008), pp. 1496–1499. issn: 2150-8097. url: https://doi.org/10.14778/1454159.1454211. http://dxdoiorg/10.14778/1454159.1454211.

    Article  Google Scholar 

  46. Konstantinos Karanasos, Asterios Katsifodimos, and Ioana Manolescu. “Delta: Scalable Data Dissemination Under Capacity Constraints”. In: Proc. VLDB Endow 7.4 (Dec. 2013), pp. 217–228. issn: 2150-8097. url: https://doi.org/10.14778/2732240.2732241. http://dxdoi.org/10.14778/2732240.2732241

    Article  Google Scholar 

  47. Jon Kleinberg. “Bursty and Hierarchical Structure in Streams”. In: Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining KDD ’02. Edmonton, Alberta, Canada: ACM, 2002, pp. 91–101. isbn: 1-58113-567-X. url: https://doi.org/10.1145/775047.775061. http://doi.acm.org/10.1145/775047.775061.

  48. Jay Kreps, Neha Narkhede, and Jun Rao. “Kafka: a Distributed Messaging System for Log Processing”. In: NetDB’11 2011.

    Google Scholar 

  49. Jay Kreps. “Why local state is a fundamental primitive in stream proc- essing”. In: O’Reilly Media (July 2014). Accessed: 2017-11-30. url: https://wwworeillycom/ideas/why-local-state-is-a-fundamental-primitive-in-stream-processing.

    Google Scholar 

  50. Kostas Kloudas and Chris Ward. “Complex Event Processing with Flink: An Update on the State of Flink CEP”. In: data Artisans Blog (Nov. 2017). Accessed: 2017-12-26. url: https://dataartisans.com/blog/complex-event-processing-flink-cep-update.

  51. Valerie Lampkin et al. Building smarter planet solutions with MQTT and IBM WebSphere MQ Telemetry. IBM Redbooks, 2012.

    Google Scholar 

  52. Jin Li et al. “Semantics and Evaluation Techniques for Window Aggregates in Data Streams”. In: Proceedings of the 2005 ACM SIGMOD International Conference on Management of Data SIGMOD ’05. Baltimore, Maryland: ACM, 2005, pp. 311–322. isbn: 1-59593-060-4. url: https://doi.org/10.1145/1066157.1066193. http://doi.acm.org/10.1145/1066157.1066193.

  53. X. Lin et al. “Continuously maintaining quantile summaries of the most recent N elements over a data stream”. In: Proceedings. 20th International Conference on Data Engineering. 2004, pp. 362–373. https://doi.org/10.1109/ICDE.2004.1320011.

  54. Xuemin Lin et al. “Stabbing the sky: efficient skyline computation over sliding windows”. In: 21st International Conference on Data Engineering (ICDE’05) 2005, pp. 502–513. https://doi.org/10.1109/ICDE.2005.137.

  55. Lisha Ma et al. “Stream Operators for Querying Data Streams”. In: Advances in Web-Age Information Management: 6th International Conference, WAIM 2005, Hangzhou, China, October 11–13, 2005. Proceedings. Ed. by Wenfei Fan, Zhaohui Wu, and Jun Yang. Berlin, Heidelberg: Springer Berlin Heidelberg, 2005, pp. 404–415. isbn: 978-3-540-32087-6. url: https://doi.org/10.1007/11563952_36. https://doi.org/10.1007/11563952_36.

    Chapter  Google Scholar 

  56. Ahmed Metwally, Divyakant Agrawal, and Amr El Abbadi. “Duplicate Detection in Click Streams”. In: Proceedings of the 14th Interna- tional Conference on World Wide Web. WWW ’05. Chiba, Japan: ACM, 2005, pp. 12–21. isbn: 1-59593-046-9. url: https://doi.org/10.1145/1060745.1060753. http://doi.acm.org/10.1145/1060745.1060753.

  57. Ahmed Metwally, Divyakant Agrawal, and Amr El Abbadi. “Efficient Computation of Frequent and Top-k Elements in Data Streams”. In: Proceedings of the 10th International Conference on Database Theory. ICDT’05. Edinburgh, UK: Springer-Verlag, 2005, pp. 398–412. isbn: 3-540-24288-0, 978-3-540-24288-8. url: https://doi.org/10.1007/9783540-30570-5_27. http://dxdoiorg/10.1007/978-3-540-30570-5_27.

  58. Kyriakos Mouratidis, Spiridon Bakiras, and Dimitris Papadias. “Continuous Monitoring of Top-k Queries over Sliding Windows”. In: Proceedings of the 2006 ACM SIGMOD International Conference on Management of Data SIGMOD ’06. Chicago, IL, USA: ACM, 2006, pp. 635–646. isbn: 1-59593-434-0. url: https://doi.org/10.1145/1142473.1142544. http://doi.acm.org/10.1145/1142473.1142544.

  59. Gurmeet Singh Manku and Rajeev Motwani. “Approximate Frequency Counts over Data Streams”. In: Proceedings of the 28th International Conference on Very Large Data Bases VLDB ’02. Hong Kong, China: VLDB Endowment, 2002, pp. 346–357. url: http://dl.acm.org/citation.cfm?id=1287369.1287400.

  60. Rajeev Motwani et al. “Query Processing, Approximation, and Resource Management in a Data Stream Management System”. In: CIDR. 2003.

    Google Scholar 

  61. Neha Narkhede. “Exactly-once Semantics are Possible: Heres How Kafka Does it”. In: Confluent Blog (June 2017). Accessed: 2017-11-18. url: https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/.

  62. Does NATS guarantee message delivery? Accessed: 2018-05-28. Cloud Native Computing Foundation. 2018. url: https://nats.io/documentation/faq/#gmd.

  63. Derek Nelson. “PipelineDB 0.9.7 – Delta Streams and Towards a Post- greSQL Extension”. In: PipelineDB Blog (Mar 2017). Accessed: 2017-11-25. url: https://www.pipelinedb.com/blog/pipelinedb-0-9-7-delta-streams-and-towards-a-postgresql-extension.

  64. Mark Palmer. “How To Analyze Sensor Data In Real-Time With CEP”. In: The StreamBase Event Processing Blog (Apr.2013). Accessed: 2018-02-14. url: http://streambase.typepad.com/streambase_stream_process/2013/04/time-windowing.html.

  65. Gene Pang. “Scalable Transactions for Scalable Distributed Database Systems”. PhD thesis. EECS Department, University of California, Berkeley, 2015. url: http://www2.eecs.berkeley.edu/Pubs/TechRpts/2015/EECS-2015-168.html.

  66. Streams. Accessed: 2017-11-25. PipelineDB. 2017. url: http://docs.pipelinedb.com/streams.html.

  67. Two Phase Commits Accessed: 2016-10-17. PipelineDB. 2015. url: http://enterprise.pipelinedb.com/docs/two-phase.html#two-phase.

  68. Kostas Patroumpas and Timos Sellis. “Window Specification over Data Streams”. In: Proceedings of the 2006 International Conference on Current Trends in Database Technology EDBT’06. Munich, Germany: Springer-Verlag, 2006, pp. 445–464. isbn: 3-540-46788-2, 978-3-540-46788-5. url: https://doi.org/10.1007/11896548_35. http://dx.doi.org/10.1007/11896548_35.

    Chapter  Google Scholar 

  69. A. Pavan and Srikanta Tirthapura. “Range-Efficient Computation of F” over Massive Data Streams”. In: Proceedings of the 21st International Conference on Data Engineering ICDE ’05. Washington, DC, USA: IEEE Computer Society 2005, pp. 32–43. isbn: 0-7695-2285-8. url: https://doi.org/10.1109/ICDE.2005.118. https://doi.org/10.1109/ICDE.2005.118

  70. David Rüdiger, Moritz Roidl, and Michael ten Hompel. “Towards Agile and Flexible Air Cargo Processes with Localization Based on RFID and Complex Event Processing”. In: Dynamics in Logistics: Third International Conference, LDIC 2012 Bremen, Germany, February/March 2012 Proceedings. Ed. by Hans-Jörg Kreowski, Bernd Scholz-Reiter, and Klaus-Dieter Thoben. Berlin, Heidelberg: Springer Berlin Heidelberg, 2013, pp. 235–246. isbn: 978-3-642-35966-8. url: https://doi.org/10.1007/978-3-642-35966-8_19. https://doi.org/10.1007/978-3-642-35966-8_19.

    Chapter  Google Scholar 

  71. E. Ryvkina et al. “Revision Processing in a Stream Processing Engine: A High-Level Design”. In: 22nd International Conference on Data En- gineering (ICDE’06) 2006, pp. 141–141. https://doi.org/10.1109/ICDE.2006.130.

  72. Salvatore Sanfilippo. Redis. Accessed: 2018-05-10. 2018. URL: https://redis.io/.

  73. Michael Stonebraker, Uǧur Cetintemel, and Stan Zdonik. “The 8 Requirements of Real-time Stream Processing”. In: SIGMOD Rec. 34.4 (Dec. 2005), pp. 42–47. ISSN: 0163–5808. URL: https://doi.org/10.1145/1107499.1107504. http://doi.acm.org/10.1145/1107499.1107504.

    Article  Google Scholar 

  74. S. Senthamilarasu and M. Hemalatha. “Load shedding techniques based on windows in data stream systems”. In: 2012 International Conference on Emerging Trends in Science, Engineering and Technology (INCOSET). 2012, pp. 68–73. https://doi.org/10.1109/INCOSET.2012.6513883.

  75. Utkarsh Srivastava and Jennifer Widom. “Flexible Time Management in Data Stream Systems”. In: Proceedings of the Twenty-third ACM SIGMOD-SIGACT-SIGART Symposium on Principles of Database Systems. PODS ’04. Paris, France: ACM, 2004, pp. 263–274. ISBN: 158113858X. url: https://doi.org/10.1145/1055558.1055596. http://doi.acm.org/10.1145/1055558.1055596.

  76. Nesime Tatbul et al. “Load Shedding in a Data Stream Manager”. In: Proceedings of the 29th International Conference on Very Large Data Bases - Volume 29. VLDB ’03. Berlin, Germany: VLDB Endowment, 2003, pp. 309–320. isbn: 0-12-722442-4. url: http://dl.acm.org/citation.cfm?id=1315451.1315479.

  77. Douglas Terry et al. “Continuous Queries over Append-only Databases”. In: SIGMOD Rec. 21.2 (June 1992), pp. 321–330. issn: 0163–5808. https://doi.org/10.1145/141484.130333. url: http://doi.acm.org/10.1145/141484.130333.

    Article  Google Scholar 

  78. Yufei Tao and Dimitris Papadias. “Maintaining sliding window skylines on data streams”. In: IEEE Transactions on Knowledge and Data Engineering 18.3 (2006), pp. 377–391. issn: 1041–4347. https://doi.org/10.1109/TKDE.2006.48.

    Article  Google Scholar 

  79. Tyler Treat. “You Cannot Have Exactly-Once Delivery”. In: Brave New Geek (Mar. 2015). Accessed: 2018-07-22. url: https://bravenewgeek.com/you-cannot-have-exactly-once-delivery/.

  80. Tyler Treat. “You Cannot Have Exactly-Once Delivery Redux”. In: Brave New Geek (June 2017). Accessed: 2018-07-22. url: https://bravenewgeek.com/you-cannot-have-exactly-once-delivery-redux/.

  81. P. A. Tucker et al. “Exploiting punctuation semantics in continuous data streams”. In: IEEE Transactions on Knowledge and Data Engineering 15.3 (2003), pp. 555–568. issn: 1041–4347. https://doi.org/10.1109/TKDE.2003.1198390.

    Article  Google Scholar 

  82. Paul Vincent. “CEP Tooling Market Survey 2016”. In: complexevents.com (May 2016). Accessed: 2017-12-14. url: http://www.complexevents.com/2016/05/12/cep-tooling-market-survey-2016/.

  83. Kreimir Vidakovi, Thomas Renner, and Sascha Rex. Marktübersicht Real-Time Monitoring Software: Event Processing Tools im Überblick. Tech. rep. Fraunhofer Verlag, Fraunhofer-Informationszentrum Raum und Bau IRB, 2010.

    Google Scholar 

  84. Jennifer Widom. “The Stanford Data Stream Management System”. In: Microsoft Research Lectures (July 2005). lecture video (relevant part: 25m45s to 26m48s); Accessed: 2018-07-30. url: https://www.microsoft.com/en-us/research/video/the-stanford-data-stream-management-system/.

  85. Apache Software Foundation. ActiveMQ. Accessed: 2018-05-10. 2018. url: https://activemq.apache.org/.

  86. Apache Software Foundation. Qpid. Accessed: 2018-05-10. 2018. url: https://qpid.apache.org/.

  87. IBM Corporation. Of Streams and Storms. Tech. rep. IBM Software Group, 2014.

    Google Scholar 

  88. Pivotal Software, Inc. RabbitMQ. Accessed: 2018-05-10. 2018. url: https://www.rabbitmq.com/.

  89. Uğur Çetintemel et al. “The Aurora and Borealis Stream Processing Engines”. In: Data Stream Management: Processing High-Speed Data Streams. Ed. by Minos Garofalakis, Johannes Gehrke, and Rajeev Rastogi. Berlin, Heidelberg: Springer Berlin Heidelberg, 2016, pp. 337–359. isbn: 978-3-540-28608-0.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Baqend GmbH, Hamburg, Hamburg, Germany

    Wolfram Wingerath & Felix Gessert

  2. University of Hamburg, Hamburg, Hamburg, Germany

    Norbert Ritter

Authors
  1. Wolfram Wingerath
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Norbert Ritter
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Felix Gessert
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Reprints and permissions

Copyright information

© 2019 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Wingerath, W., Ritter, N., Gessert, F. (2019). Data Stream Management. In: Real-Time & Stream Data Management. SpringerBriefs in Computer Science. Springer, Cham. https://doi.org/10.1007/978-3-030-10555-6_4

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-10555-6_4

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-10554-9

  • Online ISBN: 978-3-030-10555-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation