Towards Efficient Feedback Control in Streaming Computer Vision Pipelines

  • Mohamed A. Helala
  • Ken Q. Pu
  • Faisal Z. Qureshi
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9009)


Stream processing is currently an active research direction in computer vision. This is due to the existence of many computer vision algorithms that can be expressed as a pipeline of operations, and the increasing demand for online systems that process image and video streams. Recently, a formal stream algebra has been proposed as an abstract framework that mathematically describes computer vision pipelines. The algebra defines a set of concurrent operators that can describe a pipeline of vision tasks, with image and video streams as operands. In this paper, we extend this algebra framework by developing a formal and abstract description of feedback control in computer vision pipelines. Feedback control allows vision pipelines to perform adaptive parameter selection, iterative optimization and performance tuning. We show how our extension can describe feedback control in the vision pipelines of two state-of-the-art techniques.


Feedback Control Tracking Algorithm Incoming Stream Iterative Optimization Output Stream 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Zhao, B., Fei-Fei, L., Xing, E.: Online detection of unusual events in videos via dynamic sparse coding. In: CVPR, Colorado Springs, pp. 3313–3320 (2011)Google Scholar
  2. 2.
    Helala, M., Pu, K., Qureshi, F.: Road boundary detection in challenging scenarios. In: AVSS, pp. 428–433 (2012)Google Scholar
  3. 3.
    Meghdadi, A., Irani, P.: Interactive exploration of surveillance video through action shot summarization and trajectory visualization. IEEE Trans. Vis. Comput. Graph. 19, 2119–2128 (2013)CrossRefGoogle Scholar
  4. 4.
    Yenikaya, S., Yenikaya, G., Düven, E.: Keeping the vehicle on the road: A survey on on-road lane detection systems. ACM Comput. Surv. 46, 1–2 (2013)CrossRefGoogle Scholar
  5. 5.
    Ozcanli, O., Dong, Y., Mundy, J., Webb, H., Hammoud, R., Victor, T.: Automatic geo-location correction of satellite imagery. In: IEEE CVPR Workshops (2014)Google Scholar
  6. 6.
    Wischounig-Strucl, D., Quartisch, M., Rinner, B.: Prioritized data transmission in airborne camera networks for wide area surveillance and image mosaicking. In: IEEE CVPR Workshops, pp. 17–24 (2011)Google Scholar
  7. 7.
    Yuping, L., Medioni, G.: Map-enhanced uav image sequence registration and synchronization of multiple image sequences. In: IEEE CVPR, Minneapolis, Minnesota, USA, pp. 1–7 (2007)Google Scholar
  8. 8.
    Ryoo, M.S.: Human activity prediction: Early recognition of ongoing activities from streaming videos. In: ICCV, Barcelona, Spain, pp. 1036–1043 (2011)Google Scholar
  9. 9.
    Lu, C., Shi, J., Jia, J.: Online robust dictionary learning. In: IEEE CVPR, pp. 415–422 (2013)Google Scholar
  10. 10.
    Xu, C., Xiong, C., Corso, J.J.: Streaming hierarchical video segmentation. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012, Part VI. LNCS, vol. 7577, pp. 626–639. Springer, Heidelberg (2012) CrossRefGoogle Scholar
  11. 11.
    Loy, C., Hospedales, T., Xiang, T., Gong, S.: Stream-based joint exploration-exploitation active learning. In: CVPR, pp. 1560–1567 (2012)Google Scholar
  12. 12.
    Al Harbi, N., Gotoh, Y.: Spatio-temporal human body segmentation from video stream. In: Wilson, R., Hancock, E., Bors, A., Smith, W. (eds.) CAIP 2013, Part I. LNCS, vol. 8047, pp. 78–85. Springer, Heidelberg (2013) CrossRefGoogle Scholar
  13. 13.
    Yang, J., Luo, J., Yu, J., Huang, T.: Photo stream alignment and summarization for collaborative photo collection and sharing. IEEE Trans. Multimedia 14, 1642–1651 (2012)CrossRefGoogle Scholar
  14. 14.
    Kim, G., Xing, E.: Jointly aligning and segmenting multiple web photo streams for the inference of collective photo storylines. In: CVPR, pp. 620–627 (2013)Google Scholar
  15. 15.
    Chkodrov, G., Ringseth, P., Tarnavski, T., Shen, A., Barga, R., Goldstein, J.: Implementation of stream algebra over class instances, Google patents (2013)Google Scholar
  16. 16.
    Broy, M., Stefanescu, G.: The algebra of stream processing functions. Theoret. Comput. Sci. 258, 99–129 (2001)CrossRefzbMATHMathSciNetGoogle Scholar
  17. 17.
    Carlson, J., Lisper, B.: An event detection algebra for reactive systems. In: Proceedings of the 4th ACM International Conference on Embedded Software, pp. 147–154 (2004)Google Scholar
  18. 18.
    Demers, A., Gehrke, J., Hong, M., Riedewald, M., White, W.: A general algebra and implementation for monitoring event streams. Cornell University, Technical report (2005)Google Scholar
  19. 19.
    Shen, C., Little, J., Fels, S.: Towards OpenVL: Improving real-time performance of computer vision applications. In: Kisačanin, B., Bhattacharyya, S.S., Chai, S. (eds.) Embedded Computer Vision. Advances in Pattern Recognition, pp. 195–216. Springer, London (2009)CrossRefGoogle Scholar
  20. 20.
    GStreamer. (2014). Accessed: 26 January 2014
  21. 21.
    Helala, M.A., Pu, K.Q., Qureshi, F.Z.: A stream algebra for computer vision pipelines. In: IEEE CVPR Workshops (2014)Google Scholar
  22. 22.
    Kisilev, P., Freedman, D.: Parameter tuning by pairwise preferences. In: BMVC (2010)Google Scholar
  23. 23.
    Sherrah, J.: Learning to adapt: a method for automatic tuning of algorithm parameters. In: Blanc-Talon, J., Bone, D., Philips, W., Popescu, D., Scheunders, P. (eds.) ACIVS 2010, Part I. LNCS, vol. 6474, pp. 414–425. Springer, Heidelberg (2010) CrossRefGoogle Scholar
  24. 24.
    Chau, D., Badie, J., Bremond, F., Thonnat, M.: Online tracking parameter adaptation based on evaluation. In: IEEE International Conference on AVSS, pp. 189–194 (2013)Google Scholar
  25. 25.
    III, J.S., Ramanan, D.: Self-paced learning for long-term tracking. In: CVPR, Washington, DC, USA, pp. 2379–2386 (2013)Google Scholar
  26. 26.
    Chau, D., Bremond, F., Thonnat, M.: A multi-feature tracking algorithm enabling adaptation to context variations. In: ICDP 2011, pp. 1–6 (2011)Google Scholar
  27. 27.
    Corvee, E., Bremond, F.: Body parts detection for people tracking using trees of histogram of oriented gradient descriptors. In: IEEE International Conference on AVSS, pp. 469–475 (2010)Google Scholar
  28. 28.
    Kim, G., Xing, E.: On multiple foreground cosegmentation. In: IEEE CVPR, pp. 837–844 (2012)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Mohamed A. Helala
    • 1
  • Ken Q. Pu
    • 1
  • Faisal Z. Qureshi
    • 1
  1. 1.Faculty of ScienceUniversity of Ontario Institute of TechnologyOshawaCanada

Personalised recommendations