Skip to main content

Abstract

In the recent years, many emerging on-line data analysis applications require real-time delivery of the streaming data while dealing with unpredictable increase in the volume of data. In this paper we propose a novel approach for efficient stream processing of bursts in the Cloud. Our approach uses two queues to schedule requests pending execution. When bursts occur, incoming requests that exceed maximum processing capacity of the node, instead of being dropped, are diverted to a secondary queue. Requests in the secondary queue are concurrently scheduled with the primary queue, so that they can be immediately executed whenever the node has any processing power unused as the results of burst fluctuations. With this mechanism, processing power of nodes is fully utilized and the bursts are efficiently accommodated. Our experimental results illustrate the efficiency of our approach.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Similar content being viewed by others

References

  1. Chandrasekaran, S., Cooper, O., Deshpande, A., Franklin, M.J., Hellerstein, J.M., Hong, W., Krishnamurthy, S., Madden, S., Raman, V., Reiss, F., Shah, M.: TelegraphCQ: Continuous Dataflow Processing for an Uncertain World. In: CIDR, Asilomar, CA (January 2003)

    Google Scholar 

  2. Tatbul, N., Çetintemel, U., Zdonik, S.B., Cherniack, M., Stonebraker, M.: Load Shedding in a Data Stream Manager. In: VLDB 2003, Berlin, Germany, pp. 309–320 (2003)

    Google Scholar 

  3. Arasu, A., Babcock, B., Babu, S., Cieslewicz, J., Datar, M., Ito, K., Motwani, R., Srivastava, U., Widom, J.: STREAM: The Stanford Data Stream Management System (March 2005)

    Google Scholar 

  4. Madden, S., Gehrke, J.: Query Processing in Sensor Networks. IEEE Pervasive Computing, vol 3(1) (March 2004)

    Google Scholar 

  5. Lu, L., Varman, P., Doshi, K.: Graduated QoS by Decomposing Bursts: Don’t Let the Tail Wag Your Server. In: ICDCS 2009, Montreal, QC, Canada, pp. 12–21 (June 2009)

    Google Scholar 

  6. Repantis, T., Gu, X., Kalogeraki, V.: Synergy: Sharing-Aware Component Composition for Distributed Stream Processing Systems. In: van Steen, M., Henning, M. (eds.) Middleware 2006. LNCS, vol. 4290, pp. 322–341. Springer, Heidelberg (2006)

    Chapter  Google Scholar 

  7. Kalogeraki, V., Melliar-Smith, P.M., Moser, L.E.: Dynamic Scheduling of Distributed Method Invocations. In: IEEE Real-Time Systems Symposium (RTSS), Orlando, FL (December 2000)

    Google Scholar 

  8. FreePastry (2006), http://freepastry.org/FreePastry

  9. Drougas, Y., Kalogeraki, V.: RASC: Dynamic Rate Allocation for Distributed Stream Processing Applications. In: International Parallel and Distributed Processing Symposium (IPDPS), Long Beach, CA (March 2007)

    Google Scholar 

  10. Chen, F., Kalogeraki, V.: RUBEN: A Technique for Scheduling Multimedia Applications in Overlay Networks. In: Globecom 2004, Dalas, TX (November 2004)

    Google Scholar 

  11. Amini, L., Jain, N., Sehgal, A., Silber, J., Verscheure, O.: Adaptive Control of Extreme-scale Stream Processing Systems. In: ICDCS 2006, Lisboa, Portugal (2006)

    Google Scholar 

  12. Tatbul, N., Çetintemel, U., Zdonik, S.: Staying FIT: Efficient Load Shedding Techniques for Distributed Stream Processing. In: VLDB 2007, Vienna, Austria, pp. 159–170 (September 2007)

    Google Scholar 

  13. Chen, Y., Lu, C., Koutsoukos, X.: Optimal Discrete Rate Adaptation for Distributed Real-Time Systems. In: Real Time Systems Symposium (RTSS), Tucson, AZ (December 2007)

    Google Scholar 

  14. Drougas, Y., Kalogeraki, V.: Accommodating Bursts in Distributed Stream Processing Systems. In: 23rd International Parallel and Distributed Processing Symposium (IPDPS), Rome, Italy (May 2009)

    Google Scholar 

  15. Amazon Elastic Computer Cloud (Amazon EC2), http://aws.amazon.com/ec2/

  16. IBM Cloud Computing, http://www.ibm.com/ibm/cloud/

  17. Drougas, Y., Kalogeraki, V.: RASC: Dynamic Rate Allocation for Distributed Stream Processing Applications. In: International Parallel and Distributed Processing Symposium (IPDPS), Long Beach, CA (March 2007)

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2012 ICST Institute for Computer Science, Social Informatics and Telecommunications Engineering

About this paper

Cite this paper

Vu, D., Kalogeraki, V., Drougas, Y. (2012). Efficient Stream Processing in the Cloud. In: Zhang, X., Qiao, D. (eds) Quality, Reliability, Security and Robustness in Heterogeneous Networks. QShine 2010. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, vol 74. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-29222-4_19

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-29222-4_19

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-29221-7

  • Online ISBN: 978-3-642-29222-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics