Persistent Temporal Streams

  • David Hilley
  • Umakishore Ramachandran
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5896)

Abstract

Distributed continuous live stream analysis applications are increasingly common. Video-based surveillance, emergency response, disaster recovery, and critical infrastructure protection are all examples of such applications. They are characterized by a variety of high- and low-bandwidth streams as well as a need for analyzing both live and archived streams. We present a system called Persistent Temporal Streams (PTS) that supports a higher-level, domain-targeted programming abstraction for such applications. PTS provides a simple but expressive stream abstraction encompassing transport, manipulation and storage of streaming data. In this paper, we present a system architecture for implementing PTS. We provide an experimental evaluation which shows the system-level primitives can be implemented in a lightweight and high-performance manner, and an application-based evaluation designed to show that a representative high-bandwidth stream analysis application can be implemented relatively simply and with good performance.

References

  1. 1.
    Gribble, S.D., Brewer, E.A., Hellerstein, J.M., Culler, D.: Scalable, Distributed Data Structures for Internet Service Construction. In: Proceedings of OSDI 2000, p. 22 (2000)Google Scholar
  2. 2.
    Olson, M.A., Bostic, K., Seltzer, M.: Berkeley DB. In: Proceedings of USENIX ATC 1999, p. 43 (1999)Google Scholar
  3. 3.
    MacCormick, J., et al.: Boxwood: Abstractions as the Foundation for Storage Infrastructure. In: Proceedings of OSDI 2004, p. 8 (2004)Google Scholar
  4. 4.
    Shin, J., et al.: ASAP: A Camera Sensor Network for Situation Awareness. In: Tovar, E., Tsigas, P., Fouchal, H. (eds.) OPODIS 2007. LNCS, vol. 4878, pp. 31–47. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  5. 5.
    Hilley, D., Ramachandran, U.: StampedeRT: Programming abstractions for live streaming applications. In: Proceedings of ICDCS 2007, June 2007, p. 65 (2007)Google Scholar
  6. 6.
    Schmuck, F., Haskin, R.: GPFS: A Shared-Disk File System for Large Computing Clusters. In: Proceedings of FAST 2002, Berkeley, CA, USA, p. 19 (2002)Google Scholar
  7. 7.
    Desnoyers, P., Shenoy, P.: Hyperion: High Volume Stream Archival for Retrospective Querying. In: Proceedings of USENIX ATC 2007, June 2007, pp. 45–58 (2007)Google Scholar
  8. 8.
    Mills, D.L., Thyagarajan, A.: Network Time Protocol Version 4 Proposed Changes. EE Deptartment Report 94-10-2, University of Delaware (October 1994)Google Scholar
  9. 9.
    Sweeney, A., et al.: Scalability in the XFS File System. In: Proceedings of USENIX ATC 1996, pp. 1–14 (1996)Google Scholar
  10. 10.
    Levon, J., Elie, P.: OProfile: A System Profiler for Linux, http://oprofile.sf.net
  11. 11.
    Cranor, C., et al.: Gigascope: A Stream Database for Network Applications. In: Proceedings of SIGMOD 2003, pp. 647–651. ACM Press, New York (2003)Google Scholar
  12. 12.
    Chandrasekaran, S., et al.: TelegraphCQ: Continuous Dataflow Processing for an Uncertain World. In: Proceedings of CIDR 2003 (January 2003)Google Scholar
  13. 13.
    Abadi, D.J., et al.: The Design of the Borealis Stream Processing Engine. In: Proceedings of CIDR 2005, Asilomar, CA (January 2005)Google Scholar
  14. 14.
    Balakrishnan, H., et al.: Retrospective on Aurora. The VLDB Journal 13(4), 370–383 (2004)CrossRefGoogle Scholar
  15. 15.
    Arasu, A., et al.: STREAM: The Stanford Data Stream Management System. In: Garofalakis, M., Gehrke, J., Rastogi, R. (eds.) Data Stream Management: Processing High-Speed Data Streams. Springer, Heidelberg (2008) (in press)Google Scholar
  16. 16.
    Amini, L., et al.: SPC: A Distributed, Scalable Platform for Data Mining. In: Proceedings of DMSSP 2006, pp. 27–37. ACM, New York (2006)Google Scholar
  17. 17.
    Arasu, A., Babu, S., Widom, J.: The CQL Continuous Query Language: Semantic Foundations & Query Execution. The VLDB Journal 15(2), 121–142 (2006)CrossRefGoogle Scholar
  18. 18.
    Girod, L., et al.: The Case for a Signal-Oriented Data Stream Management System. In: Proceedings of CIDR 2007, Monterey, CA (January 2007)Google Scholar
  19. 19.
    Girod, L., et al.: XStream: A Signal-Oriented Data Stream Management System. In: Proceedings of ICDE 2008, Canc’un, M’exico (April 2008)Google Scholar
  20. 20.
    Arasu, A., Cherniack, M., Galvez, E., Maier, D., Maskey, A.S., Ryvkina, E., Stonebraker, M., Tibbetts, R.: Linear Road: A Stream Data Management Benchmark. In: Proceedings of VLDB 2004, VLDB Endowment, pp. 480–491 (2004)Google Scholar
  21. 21.
    Thies, W., Karczmarek, M., Amarasinghe, S.P.: StreamIt: A Language for Streaming Applications. In: Horspool, R.N. (ed.) CC 2002. LNCS, vol. 2304, pp. 179–196. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  22. 22.
    Knobe, K., Offner, C.D.: TStreams: How to Write a Parallel Program. Technical Report HPL-2004-193, HP Laboratories Cambridge (October 2004)Google Scholar
  23. 23.
    Ritchie, D.M., Thompson, K.: The UNIX time-sharing system. Communications of the ACM 17(7), 365–375 (1974)CrossRefGoogle Scholar
  24. 24.
    Jones, S.P. (ed.): Haskell 98 Language and Libraries: The Revised Report. Cambridge University Press, Cambridge (2003)MATHGoogle Scholar
  25. 25.
    Dean, J., Ghemawat, S.: MapReduce: Simplified Data Processing on Large Clusters. In: Proceedings of OSDI 2004, p. 10 (2004)Google Scholar
  26. 26.
    Isard, M., et al.: Dryad: Distributed Data-Parallel Programs from Sequential Building Blocks. In: Proceedings of EuroSys 2007, pp. 59–72. ACM, New York (2007)Google Scholar
  27. 27.
    Pike, R., Dorward, S., Griesemer, R., Quinlan, S.: Interpreting the Data: Parallel Analysis with Sawzall. Scientific Programming 13(4), 277–298 (2005)CrossRefGoogle Scholar

Copyright information

© IFIP International Federation for Information Processing 2009

Authors and Affiliations

  • David Hilley
    • 1
  • Umakishore Ramachandran
    • 1
  1. 1.College of ComputingGeorgia Institute of TechnologyUSA

Personalised recommendations