Journal of Real-Time Image Processing

, Volume 7, Issue 1, pp 31–41 | Cite as

Real-time detection of colored objects in multiple camera streams with off-the-shelf hardware components

  • Christoph H. LampertEmail author
  • Jan Peters
Special Issue


We describe RTblob, a high speed vision system that detects objects in cluttered scenes based on their color and shape at a speed of over 800 frames/s. Because the system is available as open-source software and relies only on off-the-shelf PC hardware components, it can provide the basis for multiple application scenarios. As an illustrative example, we show how RTblob can be used in a robotic table tennis scenario to estimate ball trajectories through 3D space simultaneously from four cameras images at a speed of 200 Hz.


Image processing Object detection/tracking 3D trajectories GPU/CUDA Open source 


  1. 1.
    Handmann, U., Kalinke, T., Tzomakas, C., Werner, M., Seelen, W.: An image processing system for driver assistance. Image Vis. Comput. 18(5), 367–376 (2000)CrossRefGoogle Scholar
  2. 2.
    Miura, J., Kanda, T., Shirai, Y.: An active vision system for real-time traffic sign recognition. In: IEEE Intelligent Transportation Systems, pp. 52–57 (2000)Google Scholar
  3. 3.
    Tock D., Craw. I.: Tracking and measuring drivers’ eyes. Image Vis. Comput. 14(8), 541–547 (1996)CrossRefGoogle Scholar
  4. 4.
    Kirishima T., Sato K., Chihara, K.: Real-time gesture recognition by learning and selective control of visual interest points. IEEE Trans. Pattern Anal. Mach. Intell. 27(3), 351–364 (2005)CrossRefGoogle Scholar
  5. 5.
    Liang, R.H., Ouhyoung, M.: A real-time continuous gesture recognition system for sign language. In: International Conference on Automatic Face and Gesture Recognition, pp. 558–567 (1998)Google Scholar
  6. 6.
    Stauffer, C., Grimson, W.E.L.: Learning patterns of activity using real-time tracking. IEEE Trans. Pattern Anal. Mach. Intell. 22(8), 747–757 (2000)CrossRefGoogle Scholar
  7. 7.
    Williams, B., Klein, G., Reid, I.: Real-time SLAM relocalisation. In: International Conference on Computer Vision (2007)Google Scholar
  8. 8.
    Davison, A., Mayol, W., Murray, D.: Real-time localisation and mapping with wearable active vision. In: Proceedings of the IEEE International Symposium on Mixed and Augmented Reality, Tokyo (2003)Google Scholar
  9. 9.
    Comport, A., Marchand, E., Chaumette, F.: Statistically robust 2d visual servoing. IEEE Trans. Robot. 22(2), 415–421 (2006)CrossRefGoogle Scholar
  10. 10.
    Frese, U., Bäuml, B., Haidacher, S., Schreiber, G., Schaefer, I., Hähnle, M., Hirzinger, G.: Off-the-shelf vision for a robotic ball catcher. In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, 2001, vol. 3 (2001)Google Scholar
  11. 11.
    Jähne, B.: Digital Image Processing: Concepts, Algorithms, and Scientific Applications. Springer, Berlin (1995)zbMATHGoogle Scholar
  12. 12.
    Mizusawa, S., Namiki, A., Ishikawa, M.: Tweezers type tool manipulation by a multifingered hand using a high-speed visusal servoing. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, 2008. IROS 2008, pp. 2709–2714 (2008)Google Scholar
  13. 13.
    Viola, P., Jones, M.J.: Robust real-time face detection. Int. J. Comput. Vis. 57(2), 137–154 (2004)CrossRefGoogle Scholar
  14. 14.
    Avidan, S.: Support vector tracking. IEEE Trans. Pattern Anal. Mach. Intell. 26(8), 1064–1072 (2004)CrossRefGoogle Scholar
  15. 15.
    Shimizu, K., Hirai, S.: CMOS+FPGA vision system for visual feedback of mechanical systems. In: IEEE International Conference on Robotics and Automation, pp. 2060–2065 (2006)Google Scholar
  16. 16.
    Watanabe, Y., Komuro, T., Ishikawa, M.: 955-fps real-time shape measurement of a moving/deforming object using high-speed vision for numerous-point analysis. In: IEEE International Conference on Robotics and Automation, pp. 3192–3197 (2007)Google Scholar
  17. 17.
    Mühlmann, U., Ribo, M., Lang, P., Pinz, A.: A new high speed CMOS camera for real-time tracking applications. In: International Conference on Robotics and Automation, pp. 5195–5200 (2004)Google Scholar
  18. 18.
    Nakabo, Y., Ishikawa, M., Toyoda, H., Mizuno, S.: 1 ms column parallel vision system and its application of high speed target tracking. In: IEEE International Conference on Robotics and Automation, pp. 650–655 (2000)Google Scholar
  19. 19.
    Compute Unified Device Architecture Programming Guide, NVIDIA, Santa Clara, CA (2007)Google Scholar
  20. 20.
    Camera link: specifications of the camera link interface standard for digital cameras and frame grabbers (2000).
  21. 21.
    GigE vision: camera interface standard for machine vision (2005).
  22. 22.
    Tong, X., Lu, H., Liu, Q.: An effective and fast soccer ball detection and tracking method. In: International Conference on Pattern Recognition, vol. 4, pp. 795–798 (2004)Google Scholar
  23. 23.
    Pingali, G.S., Jean, Y., Carlbom, I.: Real time tracking for enhanced tennis broadcasts. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 260–265 (1998)Google Scholar
  24. 24.
    d’Orazio, T., Guaragnella, C., Leo, M., Distante, A.: A new algorithm for ball recognition using circle Hough transform and neural classifer. Pattern Recognit. 37, 393–408 (2003)CrossRefGoogle Scholar
  25. 25.
    Kimmel, R.: Demosaicing: image reconstruction from color CCD samples. IEEE Trans. Image Process. 8(9), 1221–1228 (1999)CrossRefGoogle Scholar
  26. 26.
    Lewis, J.: Fast template matching. Vis. Interface 10, 120–123 (1995)Google Scholar
  27. 27.
    Brunelli, R., Poggio, T.: Template matching: matched spatial filters and beyond. Pattern Recognit. 30(5), 751–768 (1997)CrossRefGoogle Scholar
  28. 28.
    Crow, F.: Summed-area tables for texture mapping. Comput. Graph. 18(3), 207–212 (1984)CrossRefGoogle Scholar
  29. 29.
    Blelloch, G.E.: Prefix Sums and Their Applications. Morgan Kaufmann, Menlo Park (1991)Google Scholar
  30. 30.
    Harris, M., Sengupta, S., Owens, J.: Parallel prefix sum (scan) with CUDA. GPU Gems 3(39), 851–876 (2007)Google Scholar
  31. 31.
    Tian, Q., Huhns, M.: Algorithms for subpixel registration. Comput. Vis. Graph. Image Process. 35(2), 220–233 (1986)CrossRefGoogle Scholar
  32. 32.
    Hartley, R., Zisserman, A.: Multiple View Geometry, 2nd edn. Cambridge University Press, London (2000)Google Scholar
  33. 33.
    Lindeberg, T.: Scale-Space Theory in Computer Vision. Kluwer, Norwell (1994)Google Scholar
  34. 34.
    Hinz, S.: Fast and subpixel precise blob detection and attribution. In: IEEE International Conference on Image Processing, vol. 3, pp. 457–460 (2005)Google Scholar
  35. 35.
    Turin, G.: An introduction to matched filters. IRE Trans. Inf. Theory 6(3), 311–329 (1960)MathSciNetCrossRefGoogle Scholar
  36. 36.
    Zeidler, J., McCool, J., Widrow, B.: Adaptive correlator, 1982. US Patent 4,355,368 (1982)Google Scholar
  37. 37.
    Flannery, D., Cartwright, S.: Optical adaptive correlator. In: Third Annual Aerospace Applications of Artificial Intelligence Conference, pp. 143–154 (1987)Google Scholar
  38. 38.
    Sung, K., Poggio, T.: Example-based learning for view-based human face detection. IEEE Trans. Pattern Anal. Mach. Intell. 20(1), 39–51 (1998)CrossRefGoogle Scholar
  39. 39.
    Lampert, C.H., Peters, J.: Active structured learning for high-speed object detection. In: Symposium of the German Pattern Recognition Society, pp. 221–231 (2009)Google Scholar
  40. 40.
    Triggs, B., McLauchlan, P., Hartley, R., Fitzgibbon, A.: Bundle adjustment—a modern synthesis. In: Vision Algorithms: Theory and Practice. LNCS, vol. 1883, pp. 298–372 (2000)Google Scholar
  41. 41.
    Freeman, W.T., Adelson, E.H.: The design and use of steerable filters. IEEE Trans. Pattern Anal. Mach. Intell. 13(9), 891–906 (1991)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag 2010

Authors and Affiliations

  1. 1.Institute of Science and Technology AustriaKlosterneuburgAustria
  2. 2.Max-Planck Institute for Biological CyberneticsTübingenGermany

Personalised recommendations