Virtualizing Stream Processing

  • Michael Duller
  • Jan S. Rellermeyer
  • Gustavo Alonso
  • Nesime Tatbul
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7049)


Stream processing systems have evolved into established solutions as standalone engines but they still lack flexibility in terms of large-scale deployment, integration, extensibility, and interoperability. In the last years, a substantial ecosystem of new applications has emerged that can potentially benefit from stream processing but introduces different requirements on how stream processing solutions can be integrated, deployed, extended, and federated. To address these needs, we present an exoengine architecture and the associated ExoP platform. Together, they provide the means for encapsulating components of stream processing systems as well as automating the data exchange between components and their distributed deployment. The proposed solution can be used, e.g., to connect heterogeneous streaming engines, replace operators at runtime, and migrate operators across machines with a negligible overhead.


stream processing federation virtualization 


  1. 1.
    Abadi, D.J., Ahmad, Y., Balazinska, M., Çetintemel, U., Cherniack, M., Hwang, J.H., Lindner, W., Maskey, A.S., Rasin, A., Ryvkina, E., Tatbul, N., Xing, Y., Zdonik, S.: The Design of the Borealis Stream Processing Engine. In: CIDR (2005)Google Scholar
  2. 2.
    Abadi, D.J., Carney, D., Çetintemel, U., Cherniack, M., Convey, C., Lee, S., Stonebraker, M., Tatbul, N., Zdonik, S.: Aurora: A New Model and Architecture for Data Stream Management. The VLDB Journal 12(2), 120–139 (2003)CrossRefGoogle Scholar
  3. 3.
    Aberer, K., Hauswirth, M., Salehi, A.: A Middleware for Fast and Flexible Sensor Network Deployment. In: VLDB (2006)Google Scholar
  4. 4.
    Ali, M.H., Gerea, C., Raman, B.S., Sezgin, B., Tarnavski, T., Verona, T., Wang, P., Zabback, P., Ananthanarayan, A., Kirilov, A., Lu, M., Raizman, A., Krishnan, R., Schindlauer, R., Grabs, T., Bjeletich, S., Chandramouli, B., Goldstein, J., Bhat, S., Li, Y., Di Nicola, V., Wang, X., Maier, D., Grell, S., Nano, O., Santos, I.: Microsoft CEP server and online behavioral targeting. Proc. VLDB Endow. 2, 1558–1561 (2009)Google Scholar
  5. 5.
    Arasu, A., Cherniack, M., Galvez, E.F., Maier, D., Maskey, A., Ryvkina, E., Stonebraker, M., Tibbetts, R.: Linear Road: A Stream Data Management Benchmark.. In: VLDB (2004)Google Scholar
  6. 6.
    Botan, I., Cho, Y., Derakhshan, R., Dindar, N., Haas, L., Kim, K., Lee, C., Mundada, G., Shan, M., Tatbul, N., Yan, Y., Yun, B., Zhang, J.: Design and Implementation of the MaxStream Federated Stream Processing Architecture. Tech. Rep. TR-632, ETH Zurich Department of Computer Science (2009)Google Scholar
  7. 7.
    Botan, I., Kossmann, D., Fischer, P.M., Kraska, T., Florescu, D., Tamosevicius, R.: Extending XQuery with Window Functions. In: VLDB (2007)Google Scholar
  8. 8.
    Chamberlain, R.D., Franklin, M.A., Tyson, E.J., Buckley, J.H., Buhler, J., Galloway, G., Gayen, S., Hall, M., Shands, E.B., Singla, N.: Auto-Pipe: Streaming Applications on Architecturally Diverse Systems. IEEE Computer Magazine 43(3), 42–49 (2010)Google Scholar
  9. 9.
    Chandrasekaran, S., Cooper, O., Deshpande, A., Franklin, M.J., Hellerstein, J.M., Hong, W., Krishnamurthy, S., Madden, S., Raman, V., Reiss, F., Shah, M.: TelegraphCQ: Continuous Dataflow Processing for an Uncertain World. In: CIDR (2003)Google Scholar
  10. 10.
    Dean, J., Ghemawat, S.: MapReduce: Simplified Data Processing on Large Clusters. In: OSDI (2004)Google Scholar
  11. 11.
    Duller, M., Alonso, G.: A lightweight and extensible platform for processing personal information at global scale. Journal of Internet Services and Applications 1, 165–181 (2011)CrossRefGoogle Scholar
  12. 12.
    Engler, D.R., Kaashoek, M.F., O’Toole Jr., J.: Exokernel: An Operating System Architecture for Application-Level Resource Management. In: SOSP (1995)Google Scholar
  13. 13.
    Eugster, P.T., Felber, P.A., Guerraoui, R., Kermarrec, A.M.: The Many Faces of Publish/Subscribe. ACM Comput. Surv. 35(2), 114–131 (2003)CrossRefGoogle Scholar
  14. 14.
    Franklin, M.J., Jeffery, S.R., Krishnamurthy, S., Reiss, F., Rizvi, S., Wu, E., Cooper, O., Edakkunni, A., Hong, W.: Design Considerations for High Fan-In Systems: The HiFi Approach. In: CIDR (2005)Google Scholar
  15. 15.
    Gulisano, V., Jimenez-Peris, R., Patino-Martinez, M., Valduriez, P.: StreamCloud: A Large Scale Data Streaming System. In: ICDCS (2010)Google Scholar
  16. 16.
  17. 17.
    Jain, N., Amini, L., Andrade, H., King, R., Park, Y., Selo, P., Venkatramani, C.: Design, Implementation, and Evaluation of the Linear Road Benchmark on the Stream Processing Core. In: SIGMOD (2006)Google Scholar
  18. 18.
    Krämer, J., Seeger, B.: PIPES - A Public Infrastructure for Processing and Exploring Streams. In: SIGMOD (2004)Google Scholar
  19. 19.
    Leslie, I., McAuley, D., Black, R., Roscoe, T., Barham, P., Evers, D., Fairbairns, R., Hyden, E.: The Design and Implementation of an Operating System to Support Distributed Multimedia Applications. IEEE Journal on Selected Areas in Communications 14(7), 1280–1297 (1996)CrossRefGoogle Scholar
  20. 20.
    Motwani, R., Widom, J., Arasu, A., Babcock, B., Babu, S., Datar, M., Manku, G., Olston, C., Rosenstein, J., Varma, R.: Query Processing, Approximation, and Resource Management in a Data Stream Management System. In: CIDR (2003)Google Scholar
  21. 21.
    Neumeyer, L., Robbins, B., Nair, A., Kesari, A.: S4: Distributed Stream Computing Platform. In: ICDMW (2010)Google Scholar
  22. 22.
    OSGi Service Platform,
  23. 23.
    Papaemmanouil, O., Çetintemel, U., Jannotti, J.: Supporting Generic Cost Models for Wide-Area Stream Processing. In: ICDE (2009)Google Scholar
  24. 24.
    Pietzuch, P.R., Ledlie, J., Shneidman, J., Roussopoulos, M., Welsh, M., Seltzer, M.I.: Network-Aware Operator Placement for Stream-Processing Systems. In: ICDE (2006)Google Scholar
  25. 25.
    Rellermeyer, J.S., Alonso, G., Roscoe, T.: R-OSGi: Distributed Applications Through Software Modularization. In: Cerqueira, R., Pasquale, F. (eds.) Middleware 2007. LNCS, vol. 4834, pp. 1–20. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  26. 26.
    Schneider, S., Andrade, H., Gedik, B., Biem, A., Wu, K.L.: Elastic Scaling of Data Parallel Operators in Stream Processing. In: IPDPS (2009)Google Scholar
  27. 27.
    StreamBase Systems, Inc.,
  28. 28.
    Tatbul, N.: Streaming data integration: Challenges and opportunities. In: NTII (2010)Google Scholar
  29. 29.
    Truviso, Inc.,
  30. 30.
    Zeitler, E., Risch, T.: Scalable Splitting of Massive Data Streams. In: Kitagawa, H., Ishikawa, Y., Li, Q., Watanabe, C. (eds.) DASFAA 2010. LNCS, vol. 5982, pp. 184–198. Springer, Heidelberg (2010)CrossRefGoogle Scholar

Copyright information

© IFIP International Federation for Information Processing 2011

Authors and Affiliations

  • Michael Duller
    • 1
  • Jan S. Rellermeyer
    • 2
  • Gustavo Alonso
    • 1
  • Nesime Tatbul
    • 1
  1. 1.Systems Group, Department of Computer ScienceETH ZurichZurichSwitzerland
  2. 2.IBM Austin Research LaboratoryAustinU.S.A.

Personalised recommendations