A Sequence-Oriented Stream Warehouse Paradigm for Network Monitoring Applications

  • Lukasz Golab
  • Theodore Johnson
  • Subhabrata Sen
  • Jennifer Yates
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7192)


Network administrators are faced with the increasingly challenging task of monitoring their network’s health in real time, drawing upon diverse and voluminous measurement data feeds and extensively mining them. The role of database systems in network monitoring has traditionally been that of data repositories; even if an application uses a database, the application logic is implemented using external programs. While such programs are flexible, they tend to be ad-hoc, opaque, inefficient and hard to maintain over time. In this paper, we propose a new way of implementing network monitoring applications: directly within a database as continually updated tables defined using a declarative query language (SQL). We also address a crucial technical issue with realizing this approach: SQL was designed for set-oriented data transformations, but network monitoring involves sequence-oriented analysis. To solve this problem, we propose an extension to SQL that makes sequence-oriented analysis easier to express and faster to evaluate. Using a prototype implementation in a large-scale production data warehouse, we demonstrate how the declarative sequence-oriented query language simplifies application development and how the associated system optimizations improve application performance.


Application Logic External Program Base Table View Update Data Stream Management System 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Agrawal, J., et al.: Efficient pattern matching over event streams. In: SIGMOD 2008, pp. 147–160 (2008)Google Scholar
  2. 2.
    Ahuja, M., et al.: Peta-scale data warehousing at Yahoo! In: SIGMOD 2009, pp. 855–862 (2009)Google Scholar
  3. 3.
    Balazinska, M., et al.: Moirae: History-enhanced monitoring. In: CIDR 2007, pp. 375–386 (2007)Google Scholar
  4. 4.
    Cranor, C., et al.: A stream database for network applications. In: SIGMOD 2003, pp. 647–651 (2003)Google Scholar
  5. 5.
    Deri, L., Lorenzetti, V., Mortimer, S.: Collection and Exploration of Large Data Monitoring Sets Using Bitmap Databases. In: Ricciato, F., Mellia, M., Biersack, E. (eds.) TMA 2010. LNCS, vol. 6003, pp. 73–86. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  6. 6.
    Desnoyers, P., Shenoy, P.J.: Hyperion: High volume stream archival for retrospective querying. In: USENIX Annual Technical Conference, pp. 45–58 (2007)Google Scholar
  7. 7.
    Eriksson, B., et al.: Basisdetect: a model-based network event detection framework. In: IMC 2010, pp. 451–464 (2010)Google Scholar
  8. 8.
    Golab, L., et al.: Stream warehousing with DataDepot. In: SIGMOD 2009, pp. 847–854 (2009)Google Scholar
  9. 9.
    Golab, L., Johnson, T., Shkapenyuk, V.: Scheduling updates in a real-time stream warehouse. In: ICDE 2009, pp. 1207–1210 (2009)Google Scholar
  10. 10.
    Jain, N., et al.: Towards a streaming SQL standard. Proc. of the VLDB Endowment 1(2), 1379–1390 (2008)CrossRefGoogle Scholar
  11. 11.
    Kalmanek, C., et al.: Darkstar: Using exploratory data mining to raise the bar on network reliability and performance. In: DRCN 2009 (2009)Google Scholar
  12. 12.
    Li, X., et al.: Advanced indexing techniques for wide-area network monitoring. In: NetDB 2005 (2005)Google Scholar
  13. 13.
    Maier, G., et al.: Enriching network security analysis with time travel. SIGCOMM Comput. Commun. Rev. 38, 183–194 (2008)CrossRefGoogle Scholar
  14. 14.
    Markopoulou, A., et al.: Characterization of failures in an operational ip backbone network. IEEE/ACM Trans. Netw. 16(4), 749–762 (2008)CrossRefGoogle Scholar
  15. 15.
    Papadogiannakis, A., Polychronakis, M., Markatos, E.P.: RRDtrace: Long-term raw network traffic recording using fixed-size storage. In: MASCOTS 2010, pp. 101–110 (2010)Google Scholar
  16. 16.
    Qiu, T., et al.: What happened in my network: mining network events from router syslogs. In: IMC 2010, pp. 472–484 (2010)Google Scholar
  17. 17.
    Quass, D., Widom, J.: On-line warehouse view maintenance. In: SIGMOD 1997, pp. 393–404 (1997)Google Scholar
  18. 18.
    Reiss, F., et al.: Enabling real-time querying of live and historical stream data. In: SSDBM 2007, p. 28 (2007)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Lukasz Golab
    • 1
  • Theodore Johnson
    • 2
  • Subhabrata Sen
    • 2
  • Jennifer Yates
    • 2
  1. 1.University of WaterlooCanada
  2. 2.AT&T Labs - ResearchFlorham ParkUSA

Personalised recommendations